Audio Classification with PANNS

Author: Jael Gu

Description

The audio classification operator classify the given audio data with 527 labels from the large-scale AudioSet dataset. The pre-trained model used here is from the paper PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition (paper link).

Code Example

Predict labels and generate embeddings given the audio path "test.wav".

Write a pipeline with explicit inputs/outputs name specifications:

from towhee.dc2 import pipe, ops, DataCollection

p = (
    pipe.input('path')
        .map('path', 'frame', ops.audio_decode.ffmpeg())
        .map('frame', ('labels', 'scores', 'vec'), ops.audio_classification.panns())
        .output('path', 'labels', 'scores', 'vec')
)

DataCollection(p('./test.wav')).show()

Factory Constructor

Create the operator via the following factory method

audio_classification.panns(weights_path=None, framework='pytorch', sample_rate=32000, topk=5)

Parameters:

weights_path: str

The path to model weights. If None, it will load default model weights.

framework: str

The framework of model implementation. Default value is "pytorch" since the model is implemented in Pytorch.

sample_rate: int

The target sample rate of audio data after convention, defaults to 32000.

topk: int

The number of labels & corresponding scores to be returned, sorting by possibility from high to low. Default value is 5.

Interface

An audio embedding operator generates vectors in numpy.ndarray given towhee audio frames.

Parameters:

data: List[towhee.types.audio_frame.AudioFrame]

Input audio data is a list of towhee audio frames. The input data should represent for an audio longer than 2s.

Returns:

labels, scores, vec: Tuple(List[str], List(float), numpy.ndarray)

• labels: a list of topk predicted labels by model.
• scores: a list of scores corresponding to labels, representing for possibility.
• vec: a audio embedding generated by model, shape of which is (2048,)