audio-embedding-clmr/README.md

# Pipeline: Audio Embedding using CLMR

Authors: Jael Gu

## Overview

The pipeline uses a pre-trained CLMR model to extract embeddings of a given audio. It first transforms the input audio to a wave file with sample rate of 22050. Then the model splits the audio data into shorter clips with a fixed length. Finally it generates vectors of each clip, which composes the fingerprint of the input audio.

## Interface

**Input Arguments:**

- filepath:
  - the input audio in `.wav` (audio length > 3 seconds)
  - supported types: `str` (path to the audio)

**Pipeline Output:**

The Operator returns a tuple `Tuple[('embs', numpy.ndarray)]` containing following fields:

- embs:
  - embeddings of input audio
  - data type: numpy.ndarray
  - shape: (num_clips,512)

## How to use

1. Install [Towhee](https://github.com/towhee-io/towhee)

```bash
$ pip3 install towhee
```

> You can refer to [Getting Started with Towhee](https://towhee.io/) for more details. If you have any questions, you can [submit an issue to the towhee repository](https://github.com/towhee-io/towhee/issues).

2. Install `ffmpeg`

```bash
$ brew install ffmpeg # for Mac
```

OR

```bash
$ apt install ffmpeg # for Ubuntu
```

3. Run it with Towhee

```python
>>> from towhee import pipeline

>>> embedding_pipeline = pipeline('towhee/audio-embedding-clmr')
>>> embedding = embedding_pipeline('path/to/your/audio')
```

## How it works

This pipeline includes a main operator: [audio-embedding](https://towhee.io/operators?limit=30&page=1&filter=3%3Aaudio-embedding) (implemented as [towhee/clmr-magnatagatune](https://hub.towhee.io/towhee/clmr-magnatagatune)). The audio embedding operator encodes audio file and finally output a set of vectors of the given audio.


# More Resources

- [Exploring Multimodal Embeddings with FiftyOne and Milvus - Zilliz blog](https://zilliz.com/blog/exploring-multimodal-embeddings-with-fiftyone-and-milvus): This post explored how multimodal embeddings work with Voxel51 and Milvus.
- [How to Get the Right Vector Embeddings - Zilliz blog](https://zilliz.com/blog/how-to-get-the-right-vector-embeddings): A comprehensive introduction to vector embeddings and how to generate them with popular open-source models.
- [Audio Retrieval Based on Milvus - Zilliz blog](https://zilliz.com/blog/audio-retrieval-based-on-milvus): Create an audio retrieval system using Milvus, an open-source vector database. Classify and analyze sound data in real time.
- [Vector Database Use Case: Audio Similarity Search - Zilliz](https://zilliz.com/vector-database-use-cases/audio-similarity-search): Building agile and reliable audio similarity search with Zilliz vector database (fully managed Milvus).
- [Understanding Neural Network Embeddings - Zilliz blog](https://zilliz.com/learn/understanding-neural-network-embeddings): This article is dedicated to going a bit more in-depth into embeddings/embedding vectors, along with how they are used in modern ML algorithms and pipelines.