pytorchvideo/README.md

# Action Classification with Pytorchvideo

*Author: [Jael Gu](https://github.com/jaelgu)*

<br />

## Description

An action classification operator is able to predict labels of human activities (with corresponding scores)
and extracts features given the input video.
It preprocesses video frames with video transforms and then loads pre-trained models by model names.
This operator has implemented pre-trained models from [Pytorchvideo](https://github.com/facebookresearch/pytorchvideo)
and maps vectors with labels provided by the [Kinetics400 Dataset](https://deepmind.com/research/open-source/kinetics).

<br />

## Code Example

Use the pretrained SLOWFAST model ('slowfast_r50')
to classify and generate a vector for the given video path './archery.mp4' ([download](https://dl.fbaipublicfiles.com/pytorchvideo/projects/archery.mp4)).

*Write a pipeline with explicit inputs/outputs name specifications:*

```python
from towhee import pipe, ops, DataCollection

p = (
    pipe.input('path')
        .map('path', 'frames', ops.video_decode.ffmpeg())
        .map('frames', ('labels', 'scores', 'features'),
             ops.action_classification.pytorchvideo(model_name='slowfast_r50'))
        .output('path', 'labels', 'scores', 'features')
)

DataCollection(p('./archery.mp4')).show()
```
<img src="./result.png" width="800px"/>

<br />

## Factory Constructor

Create the operator via the following factory method

***action_classification.pytorchvideo(
model_name='x3d_xs', skip_preprocess=False, classmap=None, topk=5)***

**Parameters:**

​	***model_name***: *str*

​	The name of pre-trained model from pytorchvideo hub.

​    Supported model names:
- c2d_r50
- i3d_r50
- slow_r50
- slowfast_r50
- slowfast_r101
- x3d_xs
- x3d_s
- x3d_m
- mvit_base_16x4
- mvit_base_32x3

​	***skip_preprocess***: *bool*

​	Flag to control whether to skip UniformTemporalSubsample in video transforms, defaults to False.
If set to True, the step of UniformTemporalSubsample will be skipped.
In this case, the user should guarantee that all the input video frames are already reprocessed properly,
and thus can be fed to model directly.

​	***classmap***: *Dict[str: int]*:

​	Dictionary that maps class names to one hot vectors.
If not given, the operator will load the default class map dictionary.

​	***topk***: *int*

​	The topk labels & scores to present in result. The default value is 5.

## Interface

Given a video data, the video classification operator predicts a list of class labels
and generates a video embedding in numpy.ndarray.

**Parameters:**

​	***frames***: *List[VideoFrame]*

​	Video frames in towhee.types.video_frame.VideoFrame.


**Returns**:

​   ***labels, scores， features***: *Tuple(List[str], List[float], numpy.ndarray)*

- labels: predicted class names.
- scores: possibility scores ranking from high to low corresponding to predicted labels.
- features: a video embedding in shape of (num_features,) representing features extracted by model.


# More Resources

- [Understanding Class Activation Mapping (CAM) in Deep Learning - Zilliz blog](https://zilliz.com/learn/class-activation-mapping-CAM): Class Activation Mapping (CAM) is used to visualize and understand the decision-making of convolutional neural networks (CNNs) for computer vision tasks.
- [How to Get the Right Vector Embeddings - Zilliz blog](https://zilliz.com/blog/how-to-get-the-right-vector-embeddings): A comprehensive introduction to vector embeddings and how to generate them with popular open-source models.
- [What is a Convolutional Neural Network? An Engineer's Guide](https://zilliz.com/glossary/convolutional-neural-network): Convolutional Neural Network is a type of deep neural network that processes images, speeches, and videos. Let's find out more about CNN.
- [Understanding ImageNet: A Key Resource for Computer Vision and AI Research](https://zilliz.com/glossary/imagenet): The large-scale image database with over 14 million annotated images. Learn how this dataset supports advancements in computer vision.
- [Everything You Need to Know About Zero Shot Learning - Zilliz blog](https://zilliz.com/learn/what-is-zero-shot-learning): A comprehensive guide to Zero-Shot Learning, covering its methodologies, its relations with similarity search, and popular Zero-Shot Classification Models.
- [What is a Generative Adversarial Network? An Easy Guide](https://zilliz.com/glossary/generative-adversarial-networks): Just like we classify animal fossils into domains, kingdoms, and phyla, we classify AI networks, too. At the highest level, we classify AI networks as "discriminative" and "generative." A generative neural network is an AI that creates something new. This differs from a discriminative network, which classifies something that already exists into particular buckets. Kind of like we're doing right now, by bucketing generative adversarial networks (GANs) into appropriate classifications.
So, if you were in a situation where you wanted to use textual tags to create a new visual image, like with Midjourney, you'd use a generative network. However, if you had a giant pile of data that you needed to classify and tag, you'd use a discriminative model.
- [Enhancing Information Retrieval with Sparse Embeddings | Zilliz Learn - Zilliz blog](https://zilliz.com/learn/enhancing-information-retrieval-learned-sparse-embeddings): Explore the inner workings, advantages, and practical applications of learned sparse embeddings with the Milvus vector database
- [Zilliz partnership with PyTorch - View image search solution tutorial](https://zilliz.com/partners/pytorch): Zilliz partnership with PyTorch