logo
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Readme
Files and versions

Updated 2 years ago

sentence-embedding

Evaluation

Model performance in sentence similarity

  1. Download SentEval & test data
git clone https://github.com/facebookresearch/SentEval.git
cd SentEval/data/downstream
./get_transfer_data.bash
  1. Run test script
python evaluate.py MODEL_NAME

QPS Test

Please note that qps_test.py uses:

  • localhost:8000: to connect triton client
  • 'Hello, world.'': as test sentence
python qps_test --model paraphrase-albert-small-v2 --pipe --onnx --triton --num 100

Args:

  • --model: mandatory, string, model name
  • --pipe: optional, on/off flag to enable qps test for pipe
  • --onnx: optional, on/off flag to enable qps test for onnx
  • --triton: optional, on/off flag to enable qps for triton (please make sure that triton client is ready)
  • --num: optional, integer, defaults to 100, batch size in each loop (10 loops in total)
  • --device: optional, int, defaults to -1, cuda index or use cpu when -1
Jael Gu 9f73f60382 Update benchmark 32 Commits
..
file-icon README.md
962 B
download-icon
Update benchmark 2 years ago
file-icon evaluate.py
2.0 KiB
download-icon
Update benchmark 2 years ago
file-icon qps_test.py
4.4 KiB
download-icon
Update qps test 2 years ago
file-icon test_client.py
387 B
download-icon
Update qps test 2 years ago