# Video Classification with Pytorchvideo

*Author: [Jael Gu](https://github.com/jaelgu)*

<br />

## Description

A video classification operator is able to predict labels (and corresponding scores)
and extracts features given the input video.
It preprocesses video frames with video transforms and then loads pre-trained models by model names.
This operator has implemented pre-trained models from [Pytorchvideo](https://github.com/facebookresearch/pytorchvideo)
and maps vectors with labels provided by the [Kinetics400 Dataset](https://deepmind.com/research/open-source/kinetics).

<br />

## Code Example

Use the pretrained SLOWFAST model ('slowfast_r50')
to classify and generate a vector for the given video path './archery.mp4' ([download](https://dl.fbaipublicfiles.com/pytorchvideo/projects/archery.mp4)).

 *Write the pipeline in simplified style*:

```python
import towhee

(
    towhee.glob('./archery.mp4')
          .video_decode.ffmpeg()
          .video_classification.pytorchvideo(model_name='slowfast_r50')
          .show()
)
```
<img src="./result1.png" height="45px"/>

*Write a same pipeline with explicit inputs/outputs name specifications:*

```python
import towhee

(
    towhee.glob['path']('./archery.mp4')
          .video_decode.ffmpeg['path', 'frames']()
          .video_classification.pytorchvideo['frames', ('labels', 'scores', 'features')](
                model_name='slowfast_r50')
          .select['labels', 'scores', 'features']()
          .show(formatter={'path':'video_path'})
)
```
<img src="./result2.png" height="80px"/>

<br />

## Factory Constructor

Create the operator via the following factory method

***video_classification.pytorchvideo(
model_name='x3d_xs', skip_preprocess=False, classmap=None, topk=5)***

**Parameters:**

​	***model_name***: *str*

​	The name of pre-trained model from pytorchvideo hub.

​    Supported model names:
- c2d_r50
- i3d_r50
- slow_r50
- slowfast_r50
- slowfast_r101
- x3d_xs
- x3d_s
- x3d_m
- mvit_base_16x4
- mvit_base_32x3

​	***skip_preprocess***: *bool*

​	Flag to control whether to skip UniformTemporalSubsample in video transforms, defaults to False.
If set to True, the step of UniformTemporalSubsample will be skipped.
In this case, the user should guarantee that all the input video frames are already reprocessed properly,
and thus can be fed to model directly.

​	***classmap***: *Dict[str: int]*:

​	Dictionary that maps class names to one hot vectors.
If not given, the operator will load the default class map dictionary.

​	***topk***: *int*

​	The topk labels & scores to present in result. The default value is 5.

## Interface

Given a video data, the video classification operator predicts a list of class labels
and generates a video embedding in numpy.ndarray.

**Parameters:**

​	***frames***: *List[VideoFrame]*

​	Video frames in towhee.types.video_frame.VideoFrame.


**Returns**:

​   ***labels, scores， features***: *Tuple(List[str], List[float], numpy.ndarray)*

- labels: predicted class names.
- scores: possibility scores ranking from high to low corresponding to predicted labels.
- features: a video embedding in shape of (num_features,) representing features extracted by model.