nnfp/performance.md

Inference Performance

Test Scripts

from towhee import ops
import time

decode = ops.audio_decode.ffmpeg()
audio = [x[0] for x in decode('path/to/test.wav')]

op = ops.audio_embedding.nnfp()
# op = ops.audio_embedding.nnfp(model_path='path/to/torchscript/model.pt')
# op = ops.audio_embedding.nnfp(model_path='path/to/model.onnx')


start = time.time()
for _ in range(100):
	embs = op(audio)
	assert(embs.shape == (10, 128))
end = time.time()

print((end-start) / 100)

Performance (Default model)

• Device: MacOS, 2.3 GHz Quad-Core Intel Core i7, 8 CPUs
• Input: 10s mono audio, shape (1, ), sr , loop for 100 times

inference method	mem usage	avg time
pytorch	0.3G	0.451s
torchscript	0.3G	0.470s
onnx	0.3G	0.378s

• Device: MacOS, 2.3 GHz Quad-Core Intel Core i7, 8 CPUs
• Input: 188s stereo audio, shape (2, 8328408), sr 44100, loop for 100 times

inference method	mem usage	avg time
pytorch	2.6G	8.162s
torchscript	2.8G	7.507s
onnx	1.7G	6.769s

• Device: MacOS, 2.3 GHz Quad-Core Intel Core i7, 8 CPUs
• Input: 600s stereo audio, shape (2, 28800000), sr 48000, loop for 20 times

inference method	mem usage	avg time
pytorch	5G	22.540s
torchscript	4.9G	22.514s
onnx	3.4G	17.874s

Performance (Distilled model)

• Device: MacOS, 2.3 GHz Quad-Core Intel Core i7, 8 CPUs
• Input: 188s stereo audio, shape (2, 8328408), sr 44100, loop for 20 times

inference method	mem usage	avg time
pytorch	2.6G	7.215s
torchscript	2.8G	7.220s
onnx	1G	6.410s

• Device: MacOS, 2.3 GHz Quad-Core Intel Core i7, 8 CPUs
• Input: 600s stereo audio, shape (2, 28800000), sr 48000, loop for 20 times

inference method	mem usage	avg time
pytorch	4.9G	22.482s
torchscript	5.1G	21.511s
onnx	3.4G	17.709s