Browse Source
Update benchmark
Signed-off-by: Jael Gu <mengjia.gu@zilliz.com>
main
2 changed files with
3 additions and
4 deletions
-
benchmark/README.md
-
benchmark/evaluate.py
|
|
@ -11,7 +11,7 @@ cd SentEval/data/downstream |
|
|
|
|
|
|
|
|
2. Run test script |
|
|
2. Run test script |
|
|
```bash |
|
|
```bash |
|
|
python transformers_test.py MODEL_NAME |
|
|
|
|
|
|
|
|
python evaluate.py MODEL_NAME |
|
|
``` |
|
|
``` |
|
|
|
|
|
|
|
|
## QPS Test |
|
|
## QPS Test |
|
|
@ -30,4 +30,4 @@ python qps_test --model paraphrase-albert-small-v2 --pipe --onnx --triton --num |
|
|
- `--onnx`: optional, on/off flag to enable qps test for onnx |
|
|
- `--onnx`: optional, on/off flag to enable qps test for onnx |
|
|
- `--triton`: optional, on/off flag to enable qps for triton (please make sure that triton client is ready) |
|
|
- `--triton`: optional, on/off flag to enable qps for triton (please make sure that triton client is ready) |
|
|
- `--num`: optional, integer, defaults to 100, batch size in each loop (10 loops in total) |
|
|
- `--num`: optional, integer, defaults to 100, batch size in each loop (10 loops in total) |
|
|
- `--device`: optional, int, defaults to -1, cuda index or use cpu when -1 |
|
|
|
|
|
|
|
|
- `--device`: optional, int, defaults to -1, cuda index or use cpu when -1 |
|
|
|
|
|
@ -6,8 +6,7 @@ |
|
|
# |
|
|
# |
|
|
|
|
|
|
|
|
""" |
|
|
""" |
|
|
Clone GenSen repo here: https://github.com/Maluuba/gensen.git |
|
|
|
|
|
And follow instructions for loading the model used in batcher |
|
|
|
|
|
|
|
|
Clone repo here: https://github.com/facebookresearch/SentEval.git |
|
|
""" |
|
|
""" |
|
|
|
|
|
|
|
|
from __future__ import absolute_import, division, unicode_literals |
|
|
from __future__ import absolute_import, division, unicode_literals |
|
|
|
