transformers/benchmark

Evaluate with Similarity Search

Introduction

Build a classification system based on similarity search across embeddings. The core ideas in run.py:

1. create a new Milvus collection each time
2. extract embeddings using a pretrained model with model name specified by --model
3. specify inference method with --format in value of pytorch or onnx
4. insert & search embeddings with Milvus collection without index
5. measure performance with accuracy at top 1, 5, 10
   1. vote for the prediction from topk search results (most frequent one)
   2. compare final prediction with ground truth
   3. calculate percent of correct predictions over all queries

Example Usage

python evaluate.py --model MODEL_NAME --format pytorch
python evaluate.py --model MODEL_NAME --format onnx