copied
Readme
Files and versions
2.8 KiB
#Japanese Image-Text Retrieval Embdding with CLIP
author: David Wang
Description
This operator extracts features for image or text with Japanese-CLIP developed by rinna Co., Ltd, which can generate embeddings for Japanese text and image by jointly training an image encoder and text encoder to maximize the cosine similarity.
Code Example
Load an image from path './teddy.jpg' to generate an image embedding.
Read the text 'スケートボードに乗っているテディベア。' to generate an text embedding.
Write the pipeline in simplified style:
import towhee
towhee.glob('./teddy.jpg') \
.image_decode() \
.image_text_embedding.japanese_clip(model_name='clip_vit_b32', modality='image') \
.show()
towhee.dc(["スケートボードに乗っているテディベア。"]) \
.image_text_embedding.japanese_clip(model_name='clip_vit_b32', modality='text') \
.show()
Write a same pipeline with explicit inputs/outputs name specifications:
import towhee
towhee.glob['path']('./teddy.jpg') \
.image_decode['path', 'img']() \
.image_text_embedding.japanese_clip['img', 'vec'](model_name='clip_vit_b32', modality='image') \
.select['img', 'vec']() \
.show()
towhee.dc['text'](["スケートボードに乗っているテディベア。"]) \
.image_text_embedding.japanese_clip['text','vec'](model_name='clip_vit_b32', modality='text') \
.select['text', 'vec']() \
.show()
Factory Constructor
Create the operator via the following factory method
japanese_clip(model_name, modality)
Parameters:
model_name: str
The model name of CLIP. Supported model names:
- japanese-clip-vit-b-16
- japanese-cloob-vit-b-16
modality: str
Which modality(image or text) is used to generate the embedding.
Interface
An image-text embedding operator takes a towhee image or string as input and generate an embedding in ndarray.
Parameters:
data: towhee.types.Image (a sub-class of numpy.ndarray) or str
The data (image or text based on specified modality) to generate embedding.
Returns: numpy.ndarray
The data embedding extracted by model.
2.8 KiB
#Japanese Image-Text Retrieval Embdding with CLIP
author: David Wang
Description
This operator extracts features for image or text with Japanese-CLIP developed by rinna Co., Ltd, which can generate embeddings for Japanese text and image by jointly training an image encoder and text encoder to maximize the cosine similarity.
Code Example
Load an image from path './teddy.jpg' to generate an image embedding.
Read the text 'スケートボードに乗っているテディベア。' to generate an text embedding.
Write the pipeline in simplified style:
import towhee
towhee.glob('./teddy.jpg') \
.image_decode() \
.image_text_embedding.japanese_clip(model_name='clip_vit_b32', modality='image') \
.show()
towhee.dc(["スケートボードに乗っているテディベア。"]) \
.image_text_embedding.japanese_clip(model_name='clip_vit_b32', modality='text') \
.show()
Write a same pipeline with explicit inputs/outputs name specifications:
import towhee
towhee.glob['path']('./teddy.jpg') \
.image_decode['path', 'img']() \
.image_text_embedding.japanese_clip['img', 'vec'](model_name='clip_vit_b32', modality='image') \
.select['img', 'vec']() \
.show()
towhee.dc['text'](["スケートボードに乗っているテディベア。"]) \
.image_text_embedding.japanese_clip['text','vec'](model_name='clip_vit_b32', modality='text') \
.select['text', 'vec']() \
.show()
Factory Constructor
Create the operator via the following factory method
japanese_clip(model_name, modality)
Parameters:
model_name: str
The model name of CLIP. Supported model names:
- japanese-clip-vit-b-16
- japanese-cloob-vit-b-16
modality: str
Which modality(image or text) is used to generate the embedding.
Interface
An image-text embedding operator takes a towhee image or string as input and generate an embedding in ndarray.
Parameters:
data: towhee.types.Image (a sub-class of numpy.ndarray) or str
The data (image or text based on specified modality) to generate embedding.
Returns: numpy.ndarray
The data embedding extracted by model.