diff --git a/README.md b/README.md index 3c702f5..6277531 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ ## Description -This operator extracts features for image or text with [ALBEF](https://arxiv.org/abs/2103.00020) which can generate embeddings for text and image by jointly training an image encoder and text encoder to maximize the cosine similarity. This research introduced a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. This repo is an adaptation from [salesforce / ALBEF](https://github.com/salesforce/ALBEF) +This operator extracts features for image or text with [ALBEF](https://arxiv.org/abs/2107.07651) which can generate embeddings for text and image by jointly training an image encoder and text encoder to maximize the cosine similarity. This research introduced a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. This repo is an adaptation from [salesforce / ALBEF](https://github.com/salesforce/ALBEF)
@@ -35,8 +35,8 @@ towhee.dc(["A teddybear on a skateboard in Times Square."]) \ .image_text_embedding.albef(model_name='albef_4m', modality='text') \ .show() ``` -result1 -result2 +result1 +result2 *Write a same pipeline with explicit inputs/outputs name specifications:* @@ -54,8 +54,8 @@ towhee.dc['text'](["A teddybear on a skateboard in Times Square."]) \ .select['text', 'vec']() \ .show() ``` -result1 -result2 +result1 +result2
diff --git a/requirements.txt b/requirements.txt index e69de29..d9abd5a 100644 --- a/requirements.txt +++ b/requirements.txt @@ -0,0 +1,5 @@ +torch +transformers +timm +numpy +torchvision diff --git a/tabular1.png b/tabular1.png new file mode 100644 index 0000000..bd7262f Binary files /dev/null and b/tabular1.png differ diff --git a/tabular2.png b/tabular2.png new file mode 100644 index 0000000..eb7283b Binary files /dev/null and b/tabular2.png differ diff --git a/vec1.png b/vec1.png new file mode 100644 index 0000000..9eaf935 Binary files /dev/null and b/vec1.png differ diff --git a/vec2.png b/vec2.png new file mode 100644 index 0000000..e1ba624 Binary files /dev/null and b/vec2.png differ