drl/README.md

# Video-Text Retrieval Embedding with DRL

*author: Chen Zhang*


<br />



## Description

This operator extracts features for video or text with [DRL(Disentangled Representation Learning for Text-Video Retrieval)](https://arxiv.org/pdf/2203.07111v1.pdf), and then it can get the similarity by  Weighted Token-wise Interaction (WTI) module.


<br />

![](WTI.png)

## Code Example

Load an video from path './demo_video.mp4' to generate a video embedding. 

Read the text 'kids feeding and playing with the horse' to generate a text embedding. 

 *Write the pipeline in simplified style*:

```python
import towhee

towhee.dc(['./demo_video.mp4']) \
    .video_decode.ffmpeg(sample_type='uniform_temporal_subsample', args={'num_samples': 12}) \
    .runas_op(func=lambda x: [y for y in x]) \
    .drl(base_encoder='clip_vit_b32', modality='video', device='cpu') \
    .show()

towhee.dc(['kids feeding and playing with the horse']) \
    .drl(base_encoder='clip_vit_b32', modality='text', device='cpu') \
    .show()
```

![](vect_simplified_video.png)   
![](vect_simplified_text.png)   

*Write a same pipeline with explicit inputs/outputs name specifications:*

```python
import towhee

towhee.dc['path'](['./demo_video.mp4']) \
        .video_decode.ffmpeg['path', 'frames'](sample_type='uniform_temporal_subsample', args={'num_samples': 12}) \
        .runas_op['frames', 'frames'](func=lambda x: [y for y in x]) \
        .drl['frames', 'vec'](base_encoder='clip_vit_b32', modality='video', device='cpu') \
        .show(formatter={'path': 'video_path'})

towhee.dc['text'](['kids feeding and playing with the horse']) \
      .drl['text','vec'](base_encoder='clip_vit_b32', modality='text', device='cpu') \
      .select['text', 'vec']() \
      .show()
```

![](vect_explicit_video.png)    
![](vect_explicit_text.png)   

<br />



## Factory Constructor

Create the operator via the following factory method

***drl(base_encoder, modality)***

**Parameters:**

​   ***base_encoder:*** *str*

​   The base CLIP encode name in DRL model. Supported model names: 
- clip_vit_b32


​   ***modality:*** *str*

​   Which modality(*video* or *text*) is used to generate the embedding. 


<br />



## Interface

An video-text embedding operator takes a list of [towhee VideoFrame](link/to/towhee/image/api/doc) or string as input and generate an embedding in ndarray.


**Parameters:**

​	***data:*** *List[towhee.types.VideoFrame]*  or *str*

​  The data (list of VideoFrame(which is uniform subsampled from a video) or text based on specified modality) to generate embedding.	



**Returns:** *numpy.ndarray*

​   The data embedding extracted by model. When text, the shape is (text_token_num, model_dim), when video, the shape is (video_token_num, model_dim)