You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
|
|
|
|
# Evaluation
|
|
|
|
|
|
|
|
|
|
## Model performance in sentence similarity
|
|
|
|
|
|
|
|
|
|
1. Download SentEval & test data
|
|
|
|
|
```bash
|
|
|
|
|
git clone https://github.com/facebookresearch/SentEval.git
|
|
|
|
|
cd SentEval/data/downstream
|
|
|
|
|
./get_transfer_data.bash
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
2. Run test script
|
|
|
|
|
```bash
|
|
|
|
|
python transformers_test.py MODEL_NAME
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## QPS Test
|
|
|
|
|
|
|
|
|
|
Please note that `qps_test.py` uses:
|
|
|
|
|
- `localhost:8000`: to connect triton client
|
|
|
|
|
- `'Hello, world.''`: as test sentence
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
python qps_test --model paraphrase-albert-small-v2 --pipe --onnx --triton --num 100
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Args:**
|
|
|
|
|
- `--model`: mandatory, string, model name
|
|
|
|
|
- `--pipe`: optional, on/off flag to enable qps test for pipe
|
|
|
|
|
- `--onnx`: optional, on/off flag to enable qps test for onnx
|
|
|
|
|
- `--triton`: optional, on/off flag to enable qps for triton (please make sure that triton client is ready)
|
|
|
|
|
- `--num`: optional, integer, defaults to 100, batch size in each loop (10 loops in total)
|
|
|
|
|
- `--device`: optional, int, defaults to -1, cuda index or use cpu when -1
|
