transformers/benchmark

Evaluation

Model performance in sentence similarity

1. Download SentEval & test data

git clone https://github.com/facebookresearch/SentEval.git
cd SentEval/data/downstream
./get_transfer_data.bash

2. Run test script

python transformers_test.py MODEL_NAME

QPS Test

Please note that qps_test.py uses:

• localhost:8000: to connect triton client
• 'Hello, world.'': as test sentence

python qps_test --model paraphrase-albert-small-v2 --pipe --onnx --triton --num 100

Args:

• --model: mandatory, string, model name
• --pipe: optional, on/off flag to enable qps test for pipe
• --onnx: optional, on/off flag to enable qps test for onnx
• --triton: optional, on/off flag to enable qps for triton (please make sure that triton client is ready)
• --num: optional, integer, defaults to 100, batch size in each loop (10 loops in total)
• --device: optional, int, defaults to -1, cuda index or use cpu when -1