# Text Embedding with Transformers

*author: [Jael Gu](https://github.com/jaelgu)*

<br />

## Description

A text embedding operator takes a sentence, paragraph, or document in string as an input
and outputs token embeddings which captures the input's core semantic elements.
This operator is implemented with pre-trained models from [Huggingface Transformers](https://huggingface.co/docs/transformers).

<br />

## Code Example

Use the pre-trained model 'distilbert-base-cased'
to generate a text embedding for the sentence "Hello, world.".

*Write a pipeline with explicit inputs/outputs name specifications:*

```python
from towhee.dc2 import pipe, ops, DataCollection

p = (
    pipe.input('text')
        .map('text', 'vec', ops.text_embedding.transformers(model_name='distilbert-base-cased'))
        .output('text', 'vec')
)

DataCollection(p('Hello, world.')).show()
```

<img src="./result.png" width="800px"/>

<br />

## Factory Constructor

Create the operator via the following factory method:

***text_embedding.transformers(model_name=None)***

**Parameters:**

***model_name***: *str*

The model name in string, defaults to None.
If None, the operator will be initialized without specified model.

Please note only supported models are tested by us:

<details><summary>Albert</summary>

  - albert-base-v1
  - albert-large-v1
  - albert-xlarge-v1
  - albert-xxlarge-v1
  - albert-base-v2
  - albert-large-v2
  - albert-xlarge-v2
  - albert-xxlarge-v2
</details>

<details><summary>Bart</summary>

  - facebook/bart-large
</details>

<details><summary>Bert</summary>

  - bert-base-cased
  - bert-base-uncased
  - bert-large-cased
  - bert-large-uncased
  - bert-base-multilingual-uncased
  - bert-base-multilingual-cased
  - bert-base-chinese
  - bert-base-german-cased
  - bert-large-uncased-whole-word-masking
  - bert-large-cased-whole-word-masking
  - bert-large-uncased-whole-word-masking-finetuned-squad
  - bert-large-cased-whole-word-masking-finetuned-squad
  - bert-base-cased-finetuned-mrpc
  - bert-base-german-dbmdz-cased
  - bert-base-german-dbmdz-uncased
  - cl-tohoku/bert-base-japanese-whole-word-masking
  - cl-tohoku/bert-base-japanese-char
  - cl-tohoku/bert-base-japanese-char-whole-word-masking
  - TurkuNLP/bert-base-finnish-cased-v1
  - TurkuNLP/bert-base-finnish-uncased-v1
  - wietsedv/bert-base-dutch-cased
</details>

<details><summary>BertGeneration</summary>

  - google/bert_for_seq_generation_L-24_bbc_encoder
</details>

<details><summary>BigBird</summary>

  - google/bigbird-roberta-base
  - google/bigbird-roberta-large
  - google/bigbird-base-trivia-itc
</details>

<details><summary>BigBirdPegasus</summary>

  - google/bigbird-pegasus-large-arxiv
  - google/bigbird-pegasus-large-pubmed
  - google/bigbird-pegasus-large-bigpatent
</details>

<details><summary>CamemBert</summary>

  - camembert-base
  - Musixmatch/umberto-commoncrawl-cased-v1
  - Musixmatch/umberto-wikipedia-uncased-v1
</details>

<details><summary>Canine</summary>

  - google/canine-s
  - google/canine-c
</details>

<details><summary>Convbert</summary>

  - YituTech/conv-bert-base
  - YituTech/conv-bert-medium-small
  - YituTech/conv-bert-small
</details>

<details><summary>CTRL</summary>
  - ctrl
</details>

<details><summary>DeBERTa</summary>

  - microsoft/deberta-base
  - microsoft/deberta-large
  - microsoft/deberta-xlarge
  - microsoft/deberta-base-mnli
  - microsoft/deberta-large-mnli
  - microsoft/deberta-xlarge-mnli
  - microsoft/deberta-v2-xlarge
  - microsoft/deberta-v2-xxlarge
  - microsoft/deberta-v2-xlarge-mnli
  - microsoft/deberta-v2-xxlarge-mnli
</details>

<details><summary>DistilBert</summary>

  - distilbert-base-uncased
  - distilbert-base-uncased-distilled-squad
  - distilbert-base-cased
  - distilbert-base-cased-distilled-squad
  - distilbert-base-german-cased
  - distilbert-base-multilingual-cased
  - distilbert-base-uncased-finetuned-sst-2-english
</details>

<details><summary>Electral</summary>

  - google/electra-small-generator
  - google/electra-base-generator
  - google/electra-large-generator
  - google/electra-small-discriminator
  - google/electra-base-discriminator
  - google/electra-large-discriminator
</details>

<details><summary>Flaubert</summary>

  - flaubert/flaubert_small_cased
  - flaubert/flaubert_base_uncased
  - flaubert/flaubert_base_cased
  - flaubert/flaubert_large_cased
</details>

<details><summary>FNet</summary>

  - google/fnet-base
  - google/fnet-large
</details>

<details><summary>FSMT</summary>

  - facebook/wmt19-ru-en
</details>

<details><summary>Funnel</summary>

  - funnel-transformer/small
  - funnel-transformer/small-base
  - funnel-transformer/medium
  - funnel-transformer/medium-base
  - funnel-transformer/intermediate
  - funnel-transformer/intermediate-base
  - funnel-transformer/large
  - funnel-transformer/large-base
  - funnel-transformer/xlarge-base
  - funnel-transformer/xlarge
</details>

<details><summary>GPT</summary>

  - openai-gpt
  - gpt2
  - gpt2-medium
  - gpt2-large
  - gpt2-xl
  - distilgpt2
  - EleutherAI/gpt-neo-1.3B
  - EleutherAI/gpt-j-6B
</details>

<details><summary>I-Bert</summary>

  - kssteven/ibert-roberta-base
</details>

<details><summary>LED</summary>

  - allenai/led-base-16384
</details>

<details><summary>MobileBert</summary>

  - google/mobilebert-uncased
</details>

<details><summary>MPNet</summary>

  - microsoft/mpnet-base
</details>

<details><summary>Nystromformer</summary>

  - uw-madison/nystromformer-512
</details>

<details><summary>Reformer</summary>

  - google/reformer-crime-and-punishment
</details>

<details><summary>Splinter</summary>

  - tau/splinter-base
  - tau/splinter-base-qass
  - tau/splinter-large
  - tau/splinter-large-qass
</details>

<details><summary>SqueezeBert</summary>

  - squeezebert/squeezebert-uncased
  - squeezebert/squeezebert-mnli
  - squeezebert/squeezebert-mnli-headless
</details>    

<details><summary>TransfoXL</summary>

  - transfo-xl-wt103
</details>

<details><summary>XLM</summary>

  - xlm-mlm-en-2048
  - xlm-mlm-ende-1024
  - xlm-mlm-enfr-1024
  - xlm-mlm-enro-1024
  - xlm-mlm-tlm-xnli15-1024
  - xlm-mlm-xnli15-1024
  - xlm-clm-enfr-1024
  - xlm-clm-ende-1024
  - xlm-mlm-17-1280
  - xlm-mlm-100-1280
</details>

<details><summary>XLMRoberta</summary>

  - xlm-roberta-base
  - xlm-roberta-large
  - xlm-roberta-large-finetuned-conll02-dutch
  - xlm-roberta-large-finetuned-conll02-spanish
  - xlm-roberta-large-finetuned-conll03-english
  - xlm-roberta-large-finetuned-conll03-german
</details>

<details><summary>XLNet</summary>

  - xlnet-base-cased
  - xlnet-large-cased
</details>

<details><summary>Yoso</summary>

  - uw-madison/yoso-4096
</details>
  
<br /> 

***checkpoint_path***: *str*

The path to local checkpoint, defaults to None.
If None, the operator will download and load pretrained model by `model_name` from Huggingface transformers.

<br />

***device***: *str*

The device in string, defaults to None. If None, it will enable "cuda" automatically when cuda is available.

<br /> 

***tokenizer***: *object*

The method to tokenize input text, defaults to None.
If None, the operator will use default tokenizer by `model_name` from Huggingface transformers.

<br />


## Interface

The operator takes a piece of text in string as input.
It loads tokenizer and pre-trained model using model name.
and then return text embedding(s) in ndarray.

***\_\_call\_\_(txt)***

**Parameters:**

***data***: *Union[str, list]*

​	The text in string or a list of texts.
If data is string, the operator returns token embedding(s) in ndarray.
If data is a list, the operator returns token embedding(s) in a list.

**Returns**:

*numpy.ndarray or list*

​	The text embedding (or token embeddings) extracted by model.

<br />

***save_model(format='pytorch', path='default')***

Save model to local with specified format.

**Parameters:**

***format***: *str*

​	The format of saved model, defaults to 'pytorch'.

***path***: *str*

​	The path where model is saved to. By default, it will save model to the operator directory.

```python
from towhee import ops

op = ops.text_embedding.transformers(model_name='distilbert-base-cased').get_op()
op.save_model('onnx', 'test.onnx')
```
PosixPath('/Home/.towhee/operators/text-embedding/transformers/main/test.onnx')

<br />

***supported_model_names(format=None)***

Get a list of all supported model names or supported model names for specified model format.

**Parameters:**

***format***: *str*

​	The model format such as 'pytorch', 'torchscript'.

```python
from towhee import ops


op = ops.text_embedding.transformers().get_op()
full_list = op.supported_model_names()
onnx_list = op.supported_model_names(format='onnx')
print(f'Onnx-support/Total Models: {len(onnx_list)}/{len(full_list)}')
```
    2022-12-13 16:25:15,916 - 140704500614336 - auto_transformers.py-auto_transformers:68 - WARNING: The operator is initialized without specified model.
    Onnx-support/Total Models: 111/126


<br />

## Fine-tune
### Requirement
If you want to train this operator, besides dependency in requirements.txt, you need install these dependencies.
```python
! python -m pip install datasets evaluate scikit-learn
```
### Get start
We have prepared some most typical use of [finetune examples](https://github.com/towhee-io/examples/tree/main/fine_tune/6_train_language_modeling_tasks).

Simply speaking, you only need to construct an op instance and pass in some configurations to train the specified task.
```python
import towhee

bert_op = towhee.ops.text_embedding.transformers(model_name='bert-base-uncased').get_op()
data_args = {
    'dataset_name': 'wikitext',
    'dataset_config_name': 'wikitext-2-raw-v1',
}
training_args = {
    'num_train_epochs': 3, # you can add epoch number to get a better metric.
    'per_device_train_batch_size': 8,
    'per_device_eval_batch_size': 8,
    'do_train': True,
    'do_eval': True,
    'output_dir': './tmp/test-mlm',
    'overwrite_output_dir': True
}
bert_op.train(task='mlm', data_args=data_args, training_args=training_args)

```
For more infos, refer to the [examples](https://github.com/towhee-io/examples/tree/main/fine_tune/6_train_language_modeling_tasks).

### Dive deep and customize your training
You can change the [training script](https://towhee.io/text-embedding/transformers/src/branch/main/train_clm_with_hf_trainer.py) in your customer way. 
Or your can refer to the original [hugging face transformers training examples](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling).