# Audio Embedding with CLMR

*Author: Jael Gu*


## Desription

The audio embedding operator converts an input audio into a dense vector which can be used to represent the audio clip's semantics.
Each vector represents for an audio clip with a fixed length of around 2s.
This operator is built on top of the original implementation of [CLMR](https://github.com/Spijkervet/CLMR).
The [default model weight](clmr_checkpoint_10000.pt) provided is pretrained on [Magnatagatune Dataset](https://paperswithcode.com/dataset/magnatagatune) with [SampleCNN](sample_cnn.py).

## Code Example

Generate embeddings for the audio "test.wav". 

 *Write the pipeline in simplified style*:

```python
from towhee import dc

dc.glob('test.wav')
  .audio_decode()
  .time_window(range=10)
  .audio_embedding.clmr()
  .show()
```
    |  |

*Write a same pipeline with explicit inputs/outputs name specifications:*

```python
from towhee import dc

dc.glob['path']('test.wav')
  .audio_decode['path', 'audio']()
  .time_window['audio', 'frames'](range=10)
  .audio_embedding.clmr['frames', 'vecs']()
  .select('vecs')
  .to_vec()
```

## Factory Constructor

Create the operator via the following factory method

***audio_embedding.clmr(framework="pytorch")***

**Parameters:**

​   *framework: str*

​   The framework of model implementation.
Default value is "pytorch" since the model is implemented in Pytorch.

## Interface

An audio embedding operator generates vectors in numpy.ndarray given an audio file path or a [towhee audio](link/to/AudioFrame/api/doc).

**Parameters:**

​   *Union[str, towhee.types.Audio]*

​   The audio path or link in string.
Or audio input data in towhee audio frames.
The input data should represent for an audio longer than 2s.

**Returns**:

​   *numpy.ndarray*

​   Audio embeddings in shape (num_clips, 512).
Each embedding stands for features of an audio clip with length of 2s.