logo
clip
repo-copy-icon

copied

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Readme
Files and versions

Updated 2 years ago

towhee

Image-Text Retrieval Embdding with CLIP

author: David Wang


Description

This operator extracts features for image or text with CLIP which can generate embeddings for text and image by jointly training an image encoder and text encoder to maximize the cosine similarity.


Code Example

Load an image from path './teddy.jpg' to generate an image embedding.

Read the text 'A teddybear on a skateboard in Times Square.' to generate an text embedding.

Write the pipeline in simplified style:

import towhee

towhee.glob('./teddy.jpg') \
      .image_decode() \
      .towhee.clip(model_name='clip_vit_b32', modality='image') \
      .show()

towhee.dc(["A teddybear on a skateboard in Times Square."]) \
      .towhee.clip(model_name='clip_vit_b32', modality='text') \
      .show()
result1 result2

Write a same pipeline with explicit inputs/outputs name specifications:

import towhee

towhee.glob['path']('./teddy.jpg') \
      .image_decode['path', 'img']() \
      .towhee.clip['img', 'vec'](model_name='clip_vit_b32', modality='image') \
      .select['img', 'vec']() \
      .show()

towhee.dc['text'](["A teddybear on a skateboard in Times Square."]) \
      .towhee.clip['text','vec'](model_name='clip_vit_b32', modality='text') \
      .select['text', 'vec']() \
      .show()
result1 result2


Factory Constructor

Create the operator via the following factory method

clip(model_name, modality)

Parameters:

model_name: str

​ The model name of CLIP. Supported model names:

  • clip_resnet_r50
  • clip_resnet_r101
  • clip_vit_b32
  • clip_vit_b16

modality: str

​ Which modality(image or text) is used to generate the embedding.


Interface

An image-text embedding operator takes a towhee image or string as input and generate an embedding in ndarray.

Parameters:

data: towhee.types.Image (a sub-class of numpy.ndarray) or str

​ The data (image or text based on specified modality) to generate embedding.

Returns: numpy.ndarray

​ The data embedding extracted by model.

wxywb fafed4f6fb fix the paper reference link. 13 Commits
file-icon .gitattributes
1.1 KiB
download-icon
Initial commit 2 years ago
file-icon README.md
2.6 KiB
download-icon
fix the paper reference link. 2 years ago
file-icon __init__.py
698 B
download-icon
Replace with towhee.models.clip 2 years ago
file-icon clip.py
2.4 KiB
download-icon
flatten the output vector. 2 years ago
file-icon requirements.txt
38 B
download-icon
add towhee requirements 2 years ago
file-icon tabular1.png
185 KiB
download-icon
update the readme. 2 years ago
file-icon tabular2.png
22 KiB
download-icon
update readme. 2 years ago
file-icon vec1.png
13 KiB
download-icon
update the readme. 2 years ago
file-icon vec2.png
13 KiB
download-icon
update the readme. 2 years ago