|
@ -7,7 +7,7 @@ |
|
|
## Description |
|
|
## Description |
|
|
|
|
|
|
|
|
A text embedding operator takes a sentence, paragraph, or document in string as an input |
|
|
A text embedding operator takes a sentence, paragraph, or document in string as an input |
|
|
and output an embedding vector in ndarray which captures the input's core semantic elements. |
|
|
|
|
|
|
|
|
and outputs token embeddings which captures the input's core semantic elements. |
|
|
This operator is implemented with pre-trained models from [Huggingface Transformers](https://huggingface.co/docs/transformers). |
|
|
This operator is implemented with pre-trained models from [Huggingface Transformers](https://huggingface.co/docs/transformers). |
|
|
|
|
|
|
|
|
<br /> |
|
|
<br /> |
|
@ -329,18 +329,12 @@ If None, the operator will use default tokenizer by `model_name` from Huggingfac |
|
|
|
|
|
|
|
|
<br /> |
|
|
<br /> |
|
|
|
|
|
|
|
|
***return_sentence_emb***: *bool* |
|
|
|
|
|
|
|
|
|
|
|
The flag to output a sentence embedding for each text, defaults to True. |
|
|
|
|
|
If False, the operator returns token embeddings for each text. |
|
|
|
|
|
|
|
|
|
|
|
<br /> |
|
|
|
|
|
|
|
|
|
|
|
## Interface |
|
|
## Interface |
|
|
|
|
|
|
|
|
The operator takes a piece of text in string as input. |
|
|
The operator takes a piece of text in string as input. |
|
|
It loads tokenizer and pre-trained model using model name. |
|
|
It loads tokenizer and pre-trained model using model name. |
|
|
and then return text embedding in ndarray. |
|
|
|
|
|
|
|
|
and then return text embedding(s) in ndarray. |
|
|
|
|
|
|
|
|
***\_\_call\_\_(txt)*** |
|
|
***\_\_call\_\_(txt)*** |
|
|
|
|
|
|
|
@ -349,8 +343,8 @@ and then return text embedding in ndarray. |
|
|
***data***: *Union[str, list]* |
|
|
***data***: *Union[str, list]* |
|
|
|
|
|
|
|
|
The text in string or a list of texts. |
|
|
The text in string or a list of texts. |
|
|
If data is string, the operator returns embedding(s) in ndarray. |
|
|
|
|
|
If data is a list, the operator returns embedding(s) in a list. |
|
|
|
|
|
|
|
|
If data is string, the operator returns token embedding(s) in ndarray. |
|
|
|
|
|
If data is a list, the operator returns token embedding(s) in a list. |
|
|
|
|
|
|
|
|
**Returns**: |
|
|
**Returns**: |
|
|
|
|
|
|
|
|