# Video-Text Retrieval Embedding with BridgeFormer

*author: Jinling Xu*

<br />

## Description

This operator extracts features for video or text with [BridgeFormer](https://arxiv.org/pdf/2201.04850.pdf) which can generate embeddings for text and video by jointly training a video encoder and text encoder to maximize the cosine similarity.


<br />


## Code Example

Load a video from path './demo_video.mp4' to generate a video embedding. 

Read the text 'kids feeding and playing with the horse' to generate a text embedding. 

- Encode video (default):
```python
from towhee import pipe, ops, DataCollection

p = (
    pipe.input('video_path') \
        .map('video_path', 'video_frames', ops.video_decode.ffmpeg()) \
        .map('video_frames', 'vec', ops.video_text_embedding.bridge_former(model_name='frozen_model', modality='video')) \
        .output('video_path', 'video_frames', 'vec')
)

DataCollection(p('./demo_video.mp4')).show()
```
<img src="./video_emb_result.png" width="800px"/>

- Encode text:
```python
from towhee import pipe, ops, DataCollection

p = (
    pipe.input('text') \
        .map('text', 'vec', ops.video_text_embedding.bridge_former(model_name='frozen_model', modality='text')) \
        .output('text', 'vec')
)

DataCollection(p('kids feeding and playing with the horse')).show()
```
<img src="./text_emb_result.png" width="800px"/>


<br />



## Factory Constructor

Create the operator via the following factory method

***bridge_former(model_name, modality, weight_path)***

**Parameters:**

​   ***model_name:*** *str*

​   The model name of frozen in time. Supported model names: 
- frozen_model
- clip_initialized_model


​   ***modality:*** *str*

​   Which modality(*video* or *text*) is used to generate the embedding. 

​   ***weight_path:*** *str*

​   pretrained model weights path.  

<br />



## Interface

An video-text embedding operator takes a list of [Towhee VideoFrame](link/to/towhee/image/api/doc) or string as input and generate an embedding in ndarray.


**Parameters:**

​	***data:*** *List[towhee.types.Image]*  or *str*

​  The data (list of  Towhee VideoFrame (which is uniform subsampled from a video) or text based on specified modality) to generate embedding.	



**Returns:** *numpy.ndarray*

​   The data embedding extracted by model.






# More Resources

- [Vector Database Use Cases: Video Similarity Search - Zilliz](https://zilliz.com/vector-database-use-cases/video-similarity-search): Experience a 10x performance boost and unparalleled precision when your video similarity search system is powered by Zilliz Cloud.
- [Sentence Transformers for Long-Form Text - Zilliz blog](https://zilliz.com/learn/Sentence-Transformers-for-Long-Form-Text): Deep diving into modern transformer-based embeddings for long-form text.
- [How to Get the Right Vector Embeddings - Zilliz blog](https://zilliz.com/blog/how-to-get-the-right-vector-embeddings): A comprehensive introduction to vector embeddings and how to generate them with popular open-source models.
- [What is BERT (Bidirectional Encoder Representations from Transformers)? - Zilliz blog](https://zilliz.com/learn/what-is-bert): Learn what Bidirectional Encoder Representations from Transformers (BERT) is and how it uses pre-training and fine-tuning to achieve its remarkable performance.
- [Supercharged Semantic Similarity Search in Production - Zilliz blog](https://zilliz.com/learn/supercharged-semantic-similarity-search-in-production): Building a Blazing Fast, Highly Scalable Text-to-Image Search with CLIP embeddings and Milvus, the most advanced open-source vector database.
- [Tutorial: Diving into Text Embedding Models | Zilliz Webinar](https://zilliz.com/event/tutorial-text-embedding-models): Register for a free webinar diving into text embedding models in a presentation and tutorial
- [Tutorial: Diving into Text Embedding Models | Zilliz Webinar](https://zilliz.com/event/tutorial-text-embedding-models/success): Register for a free webinar diving into text embedding models in a presentation and tutorial
- [Sparse and Dense Embeddings: A Guide for Effective Information Retrieval with Milvus | Zilliz Webinar](https://zilliz.com/event/sparse-and-dense-embeddings-webinar): Zilliz webinar covering what sparse and dense embeddings are and when you'd want to use one over the other.
- [Sparse and Dense Embeddings: A Guide for Effective Information Retrieval with Milvus | Zilliz Webinar](https://zilliz.com/event/sparse-and-dense-embeddings-webinar/success): Zilliz webinar covering what sparse and dense embeddings are and when you'd want to use one over the other.