logo
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Readme
Files and versions

Updated 1 year ago

image-text-embedding

Image-Text Retrieval Embdding with LightningDOT

author: David Wang


Description

This operator extracts features for image or text with LightningDOT which can generate embeddings for text and image by jointly training an image encoder and text encoder to maximize the cosine similarity.


Code Example

Load an image from path './teddy.jpg' to generate an image embedding.

Read the text 'A teddybear on a skateboard in Times Square.' to generate an text embedding.

Write a pipeline with explicit inputs/outputs name specifications:


from towhee import pipe, ops, DataCollection

img_pipe = (
    pipe.input('url')
    .map('url', 'img', ops.image_decode.cv2_rgb())
    .map('img', 'vec', ops.image_text_embedding.lightningdot(model_name='lightningdot_base', modality='image'))
    .output('img', 'vec')
)

text_pipe = (
    pipe.input('text')
    .map('text', 'vec', ops.image_text_embedding.lightningdot(model_name='lightningdot_base', modality='text'))
    .output('text', 'vec')
)

DataCollection(img_pipe('./teddy.jpg')).show()
DataCollection(text_pipe('A teddybear on a skateboard in Times Square.')).show()
result1 result2


Factory Constructor

Create the operator via the following factory method

lightningdot(model_name, modality)

Parameters:

model_name: str

​ The model name of LightningDOT. Supported model names:

  • lightningdot_base
  • lightningdot_coco_ft
  • lightningdot_flickr_ft

modality: str

​ Which modality(image or text) is used to generate the embedding.


Interface

An image-text embedding operator takes a towhee image or string as input and generate an embedding in ndarray.

Parameters:

data: towhee.types.Image (a sub-class of numpy.ndarray) or str

​ The data (image or text based on specified modality) to generate embedding.

Returns: numpy.ndarray

​ The data embedding extracted by model.

wxywb 2f94969da3 remove dc2 10 Commits
folder-icon config update the operator. 2 years ago
folder-icon data/model update the operator. 2 years ago
folder-icon detector remove unused ipdb dependencies. 2 years ago
folder-icon dvl update the operator. 2 years ago
folder-icon uniter_model update the operator. 2 years ago
file-icon .gitattributes
1.1 KiB
download-icon
Initial commit 2 years ago
file-icon README.md
2.3 KiB
download-icon
remove dc2 1 year ago
file-icon __init__.py
729 B
download-icon
update the operator. 2 years ago
file-icon lightningdot.py
7.8 KiB
download-icon
refactor the readme. 2 years ago
file-icon requirements.txt
72 B
download-icon
add wget in requirements. 2 years ago
file-icon tabular1.png
22 KiB
download-icon
init the operator. 2 years ago
file-icon tabular2.png
176 KiB
download-icon
init the operator. 2 years ago
file-icon utils.py
1.8 KiB
download-icon
update the operator. 2 years ago
file-icon vec1.png
13 KiB
download-icon
init the operator. 2 years ago
file-icon vec2.png
14 KiB
download-icon
init the operator. 2 years ago