Audio Embedding with Vggish

Author: Jael Gu

Desription

The audio embedding operator converts an input audio into a dense vector which can be used to represent the audio clip's semantics. This operator is built on top of the VGGish model with Pytorch. It is originally implemented in Tensorflow. The model is pre-trained with a large scale of audio dataset AudioSet. As suggested, it is suitable to extract features at high level or warm up a larger model.

from towhee import ops

audio_encoder = ops.audio_embedding.vggish()
audio_embedding = audio_encoder("/path/to/audio")

Factory Constructor

Create the operator via the following factory method

ops.audio_embedding.vggish()

Interface

An audio embedding operator generates vectors in numpy.ndarray given an audio file path.

Parameters:

​ None.

Returns: numpy.ndarray

​ Audio embeddings.

Code Example

Generate embeddings for the audio "test.wav".

Write the pipeline in simplified style:

from towhee import dc

dc.glob('test.wav')
  .audio_embedding.vggish()
  .show()

Write a same pipeline with explicit inputs/outputs name specifications:

from towhee import dc

dc.glob['path']('test.wav')
  .audio_embedding.vggish['path', 'vecs']()
  .select('vecs')
  .show()