logo
Browse Source

replace original images.

Signed-off-by: wxywb <xy.wang@zilliz.com>
main
wxywb 2 years ago
parent
commit
24543a09a5
  1. 10
      README.md
  2. 5
      requirements.txt
  3. BIN
      tabular1.png
  4. BIN
      tabular2.png
  5. BIN
      vec1.png
  6. BIN
      vec2.png

10
README.md

@ -9,7 +9,7 @@
## Description
This operator extracts features for image or text with [ALBEF](https://arxiv.org/abs/2103.00020) which can generate embeddings for text and image by jointly training an image encoder and text encoder to maximize the cosine similarity. This research introduced a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. This repo is an adaptation from [salesforce / ALBEF](https://github.com/salesforce/ALBEF)
This operator extracts features for image or text with [ALBEF](https://arxiv.org/abs/2107.07651) which can generate embeddings for text and image by jointly training an image encoder and text encoder to maximize the cosine similarity. This research introduced a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. This repo is an adaptation from [salesforce / ALBEF](https://github.com/salesforce/ALBEF)
<br />
@ -35,8 +35,8 @@ towhee.dc(["A teddybear on a skateboard in Times Square."]) \
.image_text_embedding.albef(model_name='albef_4m', modality='text') \
.show()
```
<img src="https://towhee.io/image-text-embedding/clip/raw/branch/main/vec1.png" alt="result1" style="height:20px;"/>
<img src="https://towhee.io/image-text-embedding/clip/raw/branch/main/vec2.png" alt="result2" style="height:20px;"/>
<img src="./vec1.png" alt="result1" style="height:20px;"/>
<img src="./vec2.png" alt="result2" style="height:20px;"/>
*Write a same pipeline with explicit inputs/outputs name specifications:*
@ -54,8 +54,8 @@ towhee.dc['text'](["A teddybear on a skateboard in Times Square."]) \
.select['text', 'vec']() \
.show()
```
<img src="https://towhee.io/image-text-embedding/clip/raw/branch/main/tabular1.png" alt="result1" style="height:60px;"/>
<img src="https://towhee.io/image-text-embedding/clip/raw/branch/main/tabular2.png" alt="result2" style="height:60px;"/>
<img src="./tabular1.png" alt="result1" style="height:60px;"/>
<img src="./tabular2.png" alt="result2" style="height:60px;"/>
<br />

5
requirements.txt

@ -0,0 +1,5 @@
torch
transformers
timm
numpy
torchvision

BIN
tabular1.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 448 KiB

BIN
tabular2.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 64 KiB

BIN
vec1.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

BIN
vec2.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

Loading…
Cancel
Save