Operator: video-decoder

Author: JunJie Jiang

Overview

Decode video by using https://github.com/NVIDIA/VideoProcessingFramework

• Users need to install the vpf package by themselves.

• Gpu decode only support h.264, h.265 and vp9, others will use cpu decode.

• 4% diff with cpu-decode.

Interface

__init__(self, gpu_id, start_time=None, end_time=None, sample_type=None, args=None)


Args:

- gpu_id: int >= 0

- start_time: float

- end_time: float

  decode video from start_time to end_time

-  sample_type: str
  
        uniform_temporal_subsample
        time_step_sample

-  args: dict
   sample_type is `uniform_temporal_subsample`
      num_samples: int

   sample_type is `time_step_sample`
      time_step: int
    

__call__(self, video_path: str)

Args:

• video_path: support local path and http/https url.

Returns:

• towhee.types.VideoFrame

Requirements

https://github.com/NVIDIA/VideoProcessingFramework

How it works

from towhee import pipe, ops, DataCollection

p = (
    pipe.input('video_file')
    .flat_map('video_file', 'frame', ops.video_decode.ffmpeg(gpu_id=0, start_time=10.0, end_time=15.0, sample_type='time_step_sample', args={'time_step': 1}))
    .output('frame')
)

DataCollection(p('./video.mp4')).show(limit=1)

from towhee import ops

d = ops.video_decode.ffmpeg(gpu_id=0, start_time=10.0, end_time=20.0, sample_type='uniform_temporal_subsample', args={'num_samples': 10})
for frame in d(video_path):
    print(frame)

print('#' * 50)

d = ops.video_decode.ffmpeg(gpu_id=0, start_time=10.0, end_time=20.0, sample_type='time_step_sample', args={'time_step': 1})
for frame in d(video_path):
    print(frame)

result: VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 10010, key_frame: 1 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 11078, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 12145, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 13280, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 14348, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 15482, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 16550, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 17684, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 18752, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 19887, key_frame: 0 ################################################## VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 10010, key_frame: 1 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 11011, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 12012, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 13013, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 14014, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 15015, key_frame: 1 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 16015, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 17017, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 18018, key_frame: 0 VideoFrame shape: (360, 480, 3), mode: RGB, timestamp: 19019, key_frame: 0

Reference

More Resources

• Chroma vs Pgvector | Zilliz: Compare Chroma vs. Pgvector by the following set of capabilities. We want you to choose the best database for you, even if it’s not us.
• Scalar Quantization and Product Quantization - Zilliz blog: A hands-on dive into scalar quantization (integer quantization) and product quantization with Python.
• Multimodal RAG with Milvus and GPT-4o: Join us for a webinar for a demo of multimodal RAG with Milvus and GPT-4o
• Building a Video Analysis System with Milvus Vector Database - Zilliz blog: Learn how Milvus powers the AI analysis of video content.
• Multimodal RAG with Milvus and GPT-4o: Join us for a webinar for a demo of multimodal RAG with Milvus and GPT-4o
• 4 Steps to Building a Video Search System - Zilliz blog: Searching for videos by image with Milvus