# Audio Embedding with Vggish

*Author: [Jael Gu](https://github.com/jaelgu)*

<br />

## Desription

The audio embedding operator converts an input audio into a dense vector which can be used to represent the audio clip's semantics.
Each vector represents for an audio clip with a fixed length of around 0.9s.
This operator is built on top of [VGGish](https://github.com/tensorflow/models/tree/master/research/audioset/vggish) with Pytorch.
The model is a [VGG](https://arxiv.org/abs/1409.1556) variant pre-trained with a large scale of audio dataset [AudioSet](https://research.google.com/audioset).
As suggested, it is suitable to extract features at high level or warm up a larger model.

<br />

## Code Example

Generate embeddings for the audio "test.wav". 

*Write the pipeline in simplified style*:

```python
import towhee

(
    towhee.glob('test.wav')
          .audio_decode.ffmpeg()
          .runas_op(func=lambda x:[y[0] for y in x])
          .audio_embedding.vggish()
          .show()
)
```
    | [-0.4931737, -0.40068552, -0.032327592, ...] shape=(10, 128) |

*Write a same pipeline with explicit inputs/outputs name specifications:*

```python
import towhee

(
    towhee.glob['path']('test.wav')
          .audio_decode.ffmpeg['path', 'frames']()
          .runas_op['frames', 'frames'](func=lambda x:[y[0] for y in x])
          .audio_embedding.vggish['frames', 'vecs']()
          .show()
)
```
    [array([[-0.4931737 , -0.40068552, -0.03232759, ..., -0.33428153,
          0.1333081 , -0.25221825],
        [-0.49023268, -0.40161428, -0.03255743, ..., -0.33395663,
          0.13261834, -0.25324696],
        [-0.4992406 , -0.39848825, -0.03186834, ..., -0.33684137,
          0.13326398, -0.25385314],
        ...,
        [-0.49047503, -0.40119144, -0.03144619, ..., -0.33282205,
          0.13334712, -0.2520305 ],
        [-0.48861542, -0.40097567, -0.03173053, ..., -0.33255234,
          0.13278192, -0.25157905],
        [-0.4886143 , -0.40098593, -0.03175077, ..., -0.3325425 ,
          0.13271847, -0.25159872]], dtype=float32)]

<br />

## Factory Constructor

Create the operator via the following factory method

***audio_embedding.vggish(weights_path=None, framework="pytorch")***

**Parameters:**

*weights_path: str*

The path to model weights. If None, it will load default model weights.

*framework: str*

The framework of model implementation.
Default value is "pytorch" since the model is implemented in Pytorch.

<br />

## Interface

An audio embedding operator generates vectors in numpy.ndarray given an audio file path or towhee audio frames.

**Parameters:**

*data: List[towhee.types.audio_frame.AudioFrame]*

Input audio data is a list of towhee audio frames.
The input data should represent for an audio longer than 0.9s.


**Returns**:

*numpy.ndarray*

Audio embeddings in shape (num_clips, 128).
Each embedding stands for features of an audio clip with length of 0.9s.