logo
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Readme
Files and versions

Updated 2 years ago

audio-embedding

Audio Embedding with Vggish

Author: Jael Gu


Desription

The audio embedding operator converts an input audio into a dense vector which can be used to represent the audio clip's semantics. Each vector represents for an audio clip with a fixed length of around 0.9s. This operator is built on top of VGGish with Pytorch. The model is a VGG variant pre-trained with a large scale of audio dataset AudioSet. As suggested, it is suitable to extract features at high level or warm up a larger model.


Code Example

Generate embeddings for the audio "test.wav".

Write the pipeline in simplified style:

import towhee

(
    towhee.glob('test.wav')
          .audio_decode.ffmpeg()
          .runas_op(func=lambda x:[y[0] for y in x])
          .audio_embedding.vggish()
          .show()
)
| [-0.4931737, -0.40068552, -0.032327592, ...] shape=(10, 128) |

Write a same pipeline with explicit inputs/outputs name specifications:

import towhee

(
    towhee.glob['path']('test.wav')
          .audio_decode.ffmpeg['path', 'frames']()
          .runas_op['frames', 'frames'](func=lambda x:[y[0] for y in x])
          .audio_embedding.vggish['frames', 'vecs']()
          .show()
)
[array([[-0.4931737 , -0.40068552, -0.03232759, ..., -0.33428153,
      0.1333081 , -0.25221825],
    [-0.49023268, -0.40161428, -0.03255743, ..., -0.33395663,
      0.13261834, -0.25324696],
    [-0.4992406 , -0.39848825, -0.03186834, ..., -0.33684137,
      0.13326398, -0.25385314],
    ...,
    [-0.49047503, -0.40119144, -0.03144619, ..., -0.33282205,
      0.13334712, -0.2520305 ],
    [-0.48861542, -0.40097567, -0.03173053, ..., -0.33255234,
      0.13278192, -0.25157905],
    [-0.4886143 , -0.40098593, -0.03175077, ..., -0.3325425 ,
      0.13271847, -0.25159872]], dtype=float32)]


Factory Constructor

Create the operator via the following factory method

audio_embedding.vggish(weights_path=None, framework="pytorch")

Parameters:

weights_path: str

The path to model weights. If None, it will load default model weights.

framework: str

The framework of model implementation. Default value is "pytorch" since the model is implemented in Pytorch.


Interface

An audio embedding operator generates vectors in numpy.ndarray given an audio file path or towhee audio frames.

Parameters:

data: List[towhee.types.audio_frame.AudioFrame]

Input audio data is a list of towhee audio frames. The input data should represent for an audio longer than 0.9s.

Returns:

numpy.ndarray

Audio embeddings in shape (num_clips, 128). Each embedding stands for features of an audio clip with length of 0.9s.

Jael Gu 822a4ae1c6 Debug 21 Commits
file-icon .gitattributes
48 B
download-icon
Refactor 3 years ago
file-icon README.md
2.9 KiB
download-icon
Debug 2 years ago
file-icon __init__.py
656 B
download-icon
Revert "update" 3 years ago
file-icon mel_features.py
9.6 KiB
download-icon
Refactor 3 years ago
file-icon requirements.txt
46 B
download-icon
Adapt audio-decode/ffmpeg 2 years ago
file-icon vggish.pth
275 MiB
download-icon
Refactor 3 years ago
file-icon vggish.py
2.3 KiB
download-icon
Debug 2 years ago
file-icon vggish_input.py
3.7 KiB
download-icon
Adapt audio-decode/ffmpeg 2 years ago
file-icon vggish_params.py
2.0 KiB
download-icon
Refactor 3 years ago