logo
rerank
repo-copy-icon

copied

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Readme
Files and versions

Updated 6 months ago

towhee

Rerank QA Content

Description

The Rerank operator is used to reorder the list of relevant documents for a query. It uses the MS MARCO Cross-Encoders model to get the relevant scores and then reorders the documents.


Code Example

  • Run with ops
from towhee import ops

op = ops.rerank(threshold=0)
res = op('What is Towhee?',
         ['Towhee is Towhee is a cutting-edge framework to deal with unstructure data.', 'I do not know about towhee', 'Towhee has many powerful operators.', 'The weather is good' ])
  • Run a pipeline
from towhee import ops, pipe, DataCollection

p = (pipe.input('query', 'doc')
         .map(('query', 'doc'), ('doc', 'score'), ops.rerank(threshold=0))
         .flat_map(('doc', 'score'), ('doc', 'score'), lambda x, y: [(i, j) for i, j in zip(x, y)])
         .output('query', 'doc', 'score')
     )

DataCollection(p('What is Towhee?',
                 ['Towhee is Towhee is a cutting-edge framework to deal with unstructure data.', 'I do not know about towhee', 'Towhee has many powerful operators.', 'The weather is good' ])
              ).show()


Factory Constructor

Create the operator via the following factory method

towhee.rerank(model_name: str = 'cross-encoder/ms-marco-MiniLM-L-12-v2')

Parameters:

model_name: str

​ The model name of CrossEncoder, you can set it according to the Model List.

threshold: float

​ The threshold for filtering with score device: str

Interface

This operator is used to sort the documents of the query content and return the score, and can also set a threshold to filter the results.

Parameters:

query: str

The query content.

docs: list

A list of sentences to check the correlation with the query content.


Return: List[str], List[float]

The list of documents after rerank and the list of corresponding scores.

More Resources

Jael Gu 0ffe4ec196 Add more resources 24 Commits
file-icon .gitattributes
1.1 KiB
download-icon
Initial commit 2 years ago
file-icon README.md
3.2 KiB
download-icon
Add more resources 6 months ago
file-icon __init__.py
91 B
download-icon
Add rerank op 2 years ago
file-icon requirements.txt
13 B
download-icon
Add transformers dep 1 year ago
file-icon rerank.py
5.6 KiB
download-icon
Fix token count 10 months ago
file-icon result.png
47 KiB
download-icon
Update README 2 years ago
file-icon test_onnx_and_qps.py
4.1 KiB
download-icon
test onnx and qps 2 years ago