# Audio Classification with PANNS

*Author: [Jael Gu](https://github.com/jaelgu)*

<br />

## Description

The audio classification operator classify the given audio data with 527 labels from the large-scale [AudioSet dataset](https://research.google.com/audioset/).
The pre-trained model used here is from the paper **PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition** ([paper link](https://arxiv.org/abs/1912.10211)).

<br />

## Code Example

Predict labels and generate embeddings given the audio path "test.wav". 

 *Write the pipeline in simplified style*:

```python
import towhee

(
    towhee.glob('test.wav')
          .audio_decode.ffmpeg()
          .runas_op(func=lambda x:[y[0] for y in x])
          .audio_classification.panns()
          .show() 
)
```

*Write a same pipeline with explicit inputs/outputs name specifications:*

```python
import towhee

(
    towhee.glob['path']('test.wav')
          .audio_decode.ffmpeg['path', 'frames']()
          .runas_op['frames', 'frames'](func=lambda x:[y[0] for y in x])
          .audio_classification.panns['frames', ('labels', 'scores', 'vec')]()
          .show() 
)
```

<br />

## Factory Constructor

Create the operator via the following factory method

***audio_classification.panns(weights_path=None, framework='pytorch',
sample_rate=32000, topk=5)***

**Parameters:**

*weights_path: str*

The path to model weights. If None, it will load default model weights.

*framework: str*

The framework of model implementation.
Default value is "pytorch" since the model is implemented in Pytorch.

*sample_rate: int*

The target sample rate of audio data after convention, defaults to 32000.

*topk: int*

The number of labels & corresponding scores to be returned, sorting by possibility from high to low.
Default value is 5.

<br/>

## Interface

An audio embedding operator generates vectors in numpy.ndarray given towhee audio frames.

**Parameters:**

*data: List[towhee.types.audio_frame.AudioFrame]*

Input audio data is a list of towhee audio frames.
The input data should represent for an audio longer than 2s.


**Returns**:

*labels, scores, vec: Tuple(List[str], List(float), numpy.ndarray)*

- labels: a list of topk predicted labels by model.
- scores: a list of scores corresponding to labels, representing for possibility.
- vec: a audio embedding generated by model, shape of which is (2048,)