Readme
Files and versions
2.8 KiB
Action Classification with VideoSwinTransformer
Author: Jinling xu
Description
An action classification operator generates labels of human activities (with corresponding scores) and extracts features for the input video. It transforms the video into frames and loads pre-trained models by model names. This operator has implemented pre-trained models from VideoSwinTransformer and maps vectors with labels.
Code Example
Use the pretrained VideoSwinTransformer model ('swin_t_k400_1k') to classify and generate a vector for the given video path './archery.mp4' (download).
Write a pipeline with explicit inputs/outputs name specifications:
from towhee import pipe, ops, DataCollection
p = (
pipe.input('path')
.map('path', 'frames', ops.video_decode.ffmpeg())
.map('frames', ('labels', 'scores', 'features'),
ops.action_classification.video_swin_transformer(model_name='swin_t_k400_1k'))
.output('path', 'labels', 'scores', 'features')
)
DataCollection(p('./archery.mp4')).show()
Factory Constructor
Create the operator via the following factory method
action_classification.video_swin_transformer( model_name='swin_tiny_patch244_window877_kinetics400_1k', skip_preprocess=False, classmap=None, topk=5)
Parameters:
model_name: str
The name of pre-trained model. Supported model names:
- swin_t_k400_1k
- swin_s_k400_1k
- swin_b_k400_1k
- swin_b_k400_22k
skip_preprocess: bool
Flag to control whether to skip UniformTemporalSubsample in video transforms, defaults to False. If set to True, the step of UniformTemporalSubsample will be skipped. In this case, the user should guarantee that all the input video frames are already reprocessed properly, and thus can be fed to model directly.
classmap: Dict[str: int]:
Dictionary that maps class names to one hot vectors. If not given, the operator will load the default class map dictionary.
topk: int
The topk labels & scores to present in result. The default value is 5.
Interface
A video classification operator generates a list of class labels and a corresponding vector in numpy.ndarray given a video input data.
Parameters:
video: List[towhee.types.VideoFrame]
Input video data should be a list of towhee.types.VideoFrame representing video frames in order.
Returns:
labels, scores, features: Tuple(List[str], List[float], numpy.ndarray)
- labels: predicted class names.
- scores: possibility scores ranking from high to low corresponding to predicted labels.
- features: a video embedding in shape of (768,) representing features extracted by model.
2.8 KiB
Action Classification with VideoSwinTransformer
Author: Jinling xu
Description
An action classification operator generates labels of human activities (with corresponding scores) and extracts features for the input video. It transforms the video into frames and loads pre-trained models by model names. This operator has implemented pre-trained models from VideoSwinTransformer and maps vectors with labels.
Code Example
Use the pretrained VideoSwinTransformer model ('swin_t_k400_1k') to classify and generate a vector for the given video path './archery.mp4' (download).
Write a pipeline with explicit inputs/outputs name specifications:
from towhee import pipe, ops, DataCollection
p = (
pipe.input('path')
.map('path', 'frames', ops.video_decode.ffmpeg())
.map('frames', ('labels', 'scores', 'features'),
ops.action_classification.video_swin_transformer(model_name='swin_t_k400_1k'))
.output('path', 'labels', 'scores', 'features')
)
DataCollection(p('./archery.mp4')).show()
Factory Constructor
Create the operator via the following factory method
action_classification.video_swin_transformer( model_name='swin_tiny_patch244_window877_kinetics400_1k', skip_preprocess=False, classmap=None, topk=5)
Parameters:
model_name: str
The name of pre-trained model. Supported model names:
- swin_t_k400_1k
- swin_s_k400_1k
- swin_b_k400_1k
- swin_b_k400_22k
skip_preprocess: bool
Flag to control whether to skip UniformTemporalSubsample in video transforms, defaults to False. If set to True, the step of UniformTemporalSubsample will be skipped. In this case, the user should guarantee that all the input video frames are already reprocessed properly, and thus can be fed to model directly.
classmap: Dict[str: int]:
Dictionary that maps class names to one hot vectors. If not given, the operator will load the default class map dictionary.
topk: int
The topk labels & scores to present in result. The default value is 5.
Interface
A video classification operator generates a list of class labels and a corresponding vector in numpy.ndarray given a video input data.
Parameters:
video: List[towhee.types.VideoFrame]
Input video data should be a list of towhee.types.VideoFrame representing video frames in order.
Returns:
labels, scores, features: Tuple(List[str], List[float], numpy.ndarray)
- labels: predicted class names.
- scores: possibility scores ranking from high to low corresponding to predicted labels.
- features: a video embedding in shape of (768,) representing features extracted by model.