audio-embedding-vggish/README.md

# Pipeline: Audio Embedding using VGGish

Authors: Jael Gu

## Overview

Recommend upgrading Towhee to >=1.1.1 and using https://towhee.io/towhee/audio-embedding

This pipeline extracts features of a given audio file using a VGGish model implemented in Pytorch. This is a supervised model pre-trained with [AudioSet](https://research.google.com/audioset/), which contains over 2 million sound clips.

## Interface

**Input Arguments:**

- audio_path:
  - the input audio in `.wav`
  - supported types: `str` (path to the audio)
  - the audio should be as least 1 second

**Pipeline Output:**

The Operator returns a list of named tuple `[NamedTuple('AudioOutput', [('vec', 'ndarray')])]` containing following fields:

- each item in the output list represents for embedding(s) for an audio clip,
length & timestamps of which depend on `time-window` in [yaml](./audio_embedding_vggish.yaml) 
(You can modify `time_range_sec` & `time_step_sec` to change the way of audio split.)

- vec:
  - embeddings of input audio
  - data type: numpy.ndarray
  - shape: (num_clips, 128)

## How to use

1. Install [Towhee](https://github.com/towhee-io/towhee)

```bash
$ pip3 install towhee
```

> You can refer to [Getting Started with Towhee](https://towhee.io/) for more details. If you have any questions, you can [submit an issue to the towhee repository](https://github.com/towhee-io/towhee/issues).

2. Run it with Towhee

```python
>>> from towhee import pipeline

>>> embedding_pipeline = pipeline('towhee/audio-embedding-vggish')
>>> outs = embedding_pipeline('/path/to/your/audio')
>>> embeds = outs[0][0]
```

## How it works

This pipeline includes two main operator types:
[audio-decode](https://towhee.io/operators?limit=30&page=1&filter=1%3Aaudio-decode) & [audio-embedding](https://towhee.io/operators?limit=30&page=1&filter=3%3Aaudio-embedding).
By default, the pipeline uses [towhee/audio-decoder](https://towhee.io/towhee/audio-decoder) to load audio path as a list of audio frames in ndarray.
Then `time-window` will combine audio frames into a list of ndarray, each of which represents an audio clip in fixed length.
At the end, the [towhee/torch-vggish](https://hub.towhee.io/towhee/torch-vggish)) operator will generate a list of audio embeddings for each audio clip.




# More Resources

- [Exploring Multimodal Embeddings with FiftyOne and Milvus - Zilliz blog](https://zilliz.com/blog/exploring-multimodal-embeddings-with-fiftyone-and-milvus): This post explored how multimodal embeddings work with Voxel51 and Milvus.
- [About Towhee team | Zilliz](https://zilliz.com/authors/Towhee_team): Towhee is an open-source machine learning pipeline
that helps you encode your unstructured data into embeddings.
- [How to Get the Right Vector Embeddings - Zilliz blog](https://zilliz.com/blog/how-to-get-the-right-vector-embeddings): A comprehensive introduction to vector embeddings and how to generate them with popular open-source models.
- [The guide to clip-vit-base-patch32 | OpenAI](https://zilliz.com/ai-models/clip-vit-base-patch32): clip-vit-base-patch32: a CLIP multimodal model variant by OpenAI for image and text embedding.
- [Audio Retrieval Based on Milvus - Zilliz blog](https://zilliz.com/blog/audio-retrieval-based-on-milvus): Create an audio retrieval system using Milvus, an open-source vector database. Classify and analyze sound data in real time.
- [Vector Database Use Case: Audio Similarity Search - Zilliz](https://zilliz.com/vector-database-use-cases/audio-similarity-search): Building agile and reliable audio similarity search with Zilliz vector database (fully managed Milvus).
- [Making Machine Learning More Accessible for Developers  - Zilliz blog](https://zilliz.com/learn/making-machine-learning-more-accessible-for-application-developers): Learn how Towhee, an open-source embedding pipeline, supercharges the app development that requires embeddings and other ML tasks.
- [An Introduction to Vector Embeddings: What They Are and How to Use Them - Zilliz blog](https://zilliz.com/learn/everything-you-should-know-about-vector-embeddings): In this blog post, we will understand the concept of vector embeddings and explore its applications, best practices, and tools for working with embeddings.