Pipeline: Audio Embedding using CLMR

Authors: Jael Gu

Overview

The pipeline uses a pre-trained CLMR model to extract embeddings of a given audio. It first transforms the input audio to a wave file with sample rate of 22050. Then the model splits the audio data into shorter clips with a fixed length. Finally it generates vectors of each clip, which composes the fingerprint of the input audio.

Interface

Input Arguments:

• filepath:
  • the input audio in .wav (audio length > 3 seconds)
  • supported types: str (path to the audio)

Pipeline Output:

The Operator returns a tuple Tuple[('embs', numpy.ndarray)] containing following fields:

• embs:
  • embeddings of input audio
  • data type: numpy.ndarray
  • shape: (num_clips,512)

How to use

1. Install Towhee

$ pip3 install towhee

You can refer to Getting Started with Towhee for more details. If you have any questions, you can submit an issue to the towhee repository.

2. Install ffmpeg

$ brew install ffmpeg # for Mac

OR

$ apt install ffmpeg # for Ubuntu

3. Run it with Towhee

>>> from towhee import pipeline

>>> embedding_pipeline = pipeline('towhee/audio-embedding-clmr')
>>> embedding = embedding_pipeline('path/to/your/audio')

How it works

This pipeline includes a main operator: audio-embedding (implemented as towhee/clmr-magnatagatune). The audio embedding operator encodes audio file and finally output a set of vectors of the given audio.

More Resources

• Exploring Multimodal Embeddings with FiftyOne and Milvus - Zilliz blog: This post explored how multimodal embeddings work with Voxel51 and Milvus.
• How to Get the Right Vector Embeddings - Zilliz blog: A comprehensive introduction to vector embeddings and how to generate them with popular open-source models.
• Audio Retrieval Based on Milvus - Zilliz blog: Create an audio retrieval system using Milvus, an open-source vector database. Classify and analyze sound data in real time.
• Vector Database Use Case: Audio Similarity Search - Zilliz: Building agile and reliable audio similarity search with Zilliz vector database (fully managed Milvus).
• Understanding Neural Network Embeddings - Zilliz blog: This article is dedicated to going a bit more in-depth into embeddings/embedding vectors, along with how they are used in modern ML algorithms and pipelines.