Audio Embedding with CLMR

Author: Jael Gu

Desription

The audio embedding operator converts an input audio into a dense vector which can be used to represent the audio clip's semantics. This operator is built on top of the original implementation of CLMR. The default model weight provided is pretrained on Magnatagatune Dataset with SampleCNN.

import numpy as np
from towhee import ops

audio_encoder = ops.audio_embedding.clmr()

# Path or url as input
audio_embedding = audio_encoder("/audio/path/or/url/")

# Audio data as input
audio_data = np.zeros((2, 441344))
sample_rate = 44100
audio_embedding = audio_encoder(audio_data, sample_rate)

Factory Constructor

Create the operator via the following factory method

ops.audio_embedding.clmr()

Interface

An audio embedding operator generates vectors in numpy.ndarray given an audio file path or audio data in numpy.ndarray.

Parameters:

​ None.

Returns: numpy.ndarray

​ Audio embeddings in shape (num_clips, 512).

Code Example

Generate embeddings for the audio "test.wav".

Write the pipeline in simplified style:

from towhee import dc

dc.glob('test.wav')
  .audio_embedding.clmr()
  .show()

Write a same pipeline with explicit inputs/outputs name specifications:

from towhee import dc

dc.glob['path']('test.wav')
  .audio_embedding.clmr['path', 'vecs']()
  .select('vecs')
  .show()