copied
Readme
Files and versions
Updated 2 years ago
audio-embedding
Audio Embedding with Vggish
Author: Jael Gu
Desription
The audio embedding operator converts an input audio into a dense vector which can be used to represent the audio clip's semantics. Each vector represents for an audio clip with a fixed length of around 0.9s. This operator is built on top of VGGish with Pytorch. The model is a VGG variant pre-trained with a large scale of audio dataset AudioSet. As suggested, it is suitable to extract features at high level or warm up a larger model.
Code Example
Generate embeddings for the audio "test.wav".
Write the pipeline in simplified style:
import towhee
(
towhee.glob('test.wav')
.audio_decode.ffmpeg()
.runas_op(func=lambda x:[y[0] for y in x])
.audio_embedding.vggish()
.show()
)
| [-0.4931737, -0.40068552, -0.032327592, ...] shape=(10, 128) |
Write a same pipeline with explicit inputs/outputs name specifications:
import towhee
(
towhee.glob['path']('test.wav')
.audio_decode.ffmpeg['path', 'frames']()
.runas_op['frames', 'frames'](func=lambda x:[y[0] for y in x])
.audio_embedding.vggish['frames', 'vecs']()
.show()
)
[array([[-0.4931737 , -0.40068552, -0.03232759, ..., -0.33428153,
0.1333081 , -0.25221825],
[-0.49023268, -0.40161428, -0.03255743, ..., -0.33395663,
0.13261834, -0.25324696],
[-0.4992406 , -0.39848825, -0.03186834, ..., -0.33684137,
0.13326398, -0.25385314],
...,
[-0.49047503, -0.40119144, -0.03144619, ..., -0.33282205,
0.13334712, -0.2520305 ],
[-0.48861542, -0.40097567, -0.03173053, ..., -0.33255234,
0.13278192, -0.25157905],
[-0.4886143 , -0.40098593, -0.03175077, ..., -0.3325425 ,
0.13271847, -0.25159872]], dtype=float32)]
Factory Constructor
Create the operator via the following factory method
audio_embedding.vggish(weights_path=None, framework="pytorch")
Parameters:
weights_path: str
The path to model weights. If None, it will load default model weights.
framework: str
The framework of model implementation. Default value is "pytorch" since the model is implemented in Pytorch.
Interface
An audio embedding operator generates vectors in numpy.ndarray given an audio file path or towhee audio frames.
Parameters:
data: List[towhee.types.audio_frame.AudioFrame]
Input audio data is a list of towhee audio frames. The input data should represent for an audio longer than 0.9s.
Returns:
numpy.ndarray
Audio embeddings in shape (num_clips, 128). Each embedding stands for features of an audio clip with length of 0.9s.
Jael Gu
822a4ae1c6
| 21 Commits | ||
---|---|---|---|
.gitattributes |
48 B
|
3 years ago | |
README.md |
2.9 KiB
|
2 years ago | |
__init__.py |
656 B
|
3 years ago | |
mel_features.py |
9.6 KiB
|
3 years ago | |
requirements.txt |
46 B
|
2 years ago | |
vggish.pth |
275 MiB
|
3 years ago | |
vggish.py |
2.3 KiB
|
2 years ago | |
vggish_input.py |
3.7 KiB
|
2 years ago | |
vggish_params.py |
2.0 KiB
|
3 years ago |