You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Files and versions

Updated 1 year ago


Video Classification with Omnivore

Author: Xinyu Ge


A video classification operator generates labels (and corresponding scores) and extracts features for the input video. It transforms the video into frames and loads pre-trained models by model names. This operator has implemented pre-trained models from Omnivore and maps vectors with labels provided by datasets used for pre-training.

Code Example

Use the pretrained Omnivore model to classify and generate a vector for the given video path './archery.mp4' (download).

Write a pipeline with explicit inputs/outputs name specifications:

from towhee import pipe, ops, DataCollection

p = (
        .map('path', 'frames', ops.video_decode.ffmpeg())
        .map('frames', ('labels', 'scores', 'features'),
        .output('path', 'labels', 'scores', 'features')


Factory Constructor

Create the operator via the following factory method

action_classification.omnivore( model_name='omnivore_swinT', skip_preprocess=False, classmap=None, topk=5)


model_name: str

​ The name of pre-trained Omnivore model.

​ Supported model names:

  • omnivore_swinT
  • omnivore_swinS
  • omnivore_swinB
  • omnivore_swinB_in21k
  • omnivore_swinL_in21k

skip_preprocess: bool

​ Flag to control whether to skip video transforms, defaults to False. If set to True, the step to transform videos will be skipped. In this case, the user should guarantee that all the input video frames are already reprocessed properly, and thus can be fed to model directly.

classmap: Dict[str: int]:

​ Dictionary that maps class names to one hot vectors. If not given, the operator will load the default class map dictionary.

topk: int

​ The topk labels & scores to present in result. The default value is 5.


A video classification operator generates a list of class labels and a corresponding vector in numpy.ndarray given a video input data.


video: Union[str, numpy.ndarray]

​ Input video data using local path in string or video frames in ndarray.

Returns: (list, list, torch.Tensor)

​ A tuple of (labels, scores, features), which contains lists of predicted class names and corresponding scores.

Jael Gu 29afc321fc Remove dc2 22 Commits
file-icon .gitattributes
1.1 KiB
Initial commit 2 years ago
file-icon README.md
2.6 KiB
Remove dc2 1 year ago
file-icon __init__.py
680 B
modify readme 2 years ago
file-icon kinetics_400.json
10 KiB
add omnivore 2 years ago
file-icon omnivore.py
4.0 KiB
modify supported model name 2 years ago
file-icon requirements.txt
59 B
Update requirements 2 years ago
file-icon result.png
17 KiB
Update 1 year ago