|
|
|
|
# Evaluation Method
|
|
|
|
|
|
|
|
|
|
- Model performance (WIP)
|
|
|
|
|
- Pipeline speed
|
|
|
|
|
|
|
|
|
|
## Model Performance
|
|
|
|
|
|
|
|
|
|
Build an image classification system based on similarity search across embeddings.
|
|
|
|
|
|
|
|
|
|
The core ideas in `performance.py`:
|
|
|
|
|
1. create a new Milvus collection each time
|
|
|
|
|
2. extract embeddings using a pretrained model with model name specified by `--model`
|
|
|
|
|
3. specify inference method with `--format` in value of `pytorch` or `onnx`
|
|
|
|
|
4. insert & search embeddings with Milvus collection without index
|
|
|
|
|
5. measure performance with accuracy at top 1, 5, 10
|
|
|
|
|
1. vote for the prediction from topk search results (most frequent one)
|
|
|
|
|
2. compare final prediction with ground truth
|
|
|
|
|
3. calculate percent of correct predictions over all queries
|
|
|
|
|
|
|
|
|
|
### Example Usage
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
# Option 1:
|
|
|
|
|
python performance.py --model MODEL_NAME --format pytorch
|
|
|
|
|
python performance.py --model MODEL_NAME --format onnx
|
|
|
|
|
|
|
|
|
|
# Option 2:
|
|
|
|
|
chmod +x performance.sh
|
|
|
|
|
./performance.sh
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Pipeline Speed
|
|
|
|
|
|
|
|
|
|
QPS test of the embedding pipeline including steps below:
|
|
|
|
|
1. load image from path (pipe.input)
|
|
|
|
|
2. decode image into arrays (ops.image_decode)
|
|
|
|
|
3. generate image embedding (preprocess, model inference, post-process)
|
|
|
|
|
|
|
|
|
|
There are 3 methods with different pipeline speeds:
|
|
|
|
|
- Towhee pipe (regular method)
|
|
|
|
|
- Onnxruntime (model inference using onnx at local)
|
|
|
|
|
- TritonServe with onnx enabled (request as client)
|
|
|
|
|
|
|
|
|
|
### Example usage
|
|
|
|
|
|
|
|
|
|
Please note that `qps_test.py` uses:
|
|
|
|
|
- `localhost:8000`: to connect triton client
|
|
|
|
|
- `../towhee/jpeg`: as test image path
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
python qps_test.py --model 'resnet50' --pipe --onnx --triton --num 100 --device cuda:0
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Args:**
|
|
|
|
|
- `--model`: mandatory, string, model name
|
|
|
|
|
- `--pipe`: optional, on/off flag to enable qps test for pipe
|
|
|
|
|
- `--onnx`: optional, on/off flag to enable qps test for onnx
|
|
|
|
|
- `--triton`: optional, on/off flag to enable qps for triton (please make sure that triton client is ready)
|
|
|
|
|
- `--num`: optional, integer, defaults to 100, batch size in each loop (10 loops in total)
|
|
|
|
|
- `--device`: optional, string, defaults to 'cpu'
|
