logo
Browse Source

update the image for expansionnet_v2.

Signed-off-by: wxywb <xy.wang@zilliz.com>
main
wxywb 2 years ago
parent
commit
85d07be6ed
  1. 78
      README.md
  2. BIN
      cap.png
  3. 9
      expansionnet_v2.py
  4. BIN
      tabular.png

78
README.md

@ -1,2 +1,78 @@
# expansionnet-v2
# Image Captioning with ExpansionNet v2
*author: David Wang*
<br />
## Description
This operator generates the caption with [ExpansionNet v2](https://arxiv.org/abs/2208.06551) which describes the content of the given image. ExpansionNet v2 introduces the Block Static Expansion which distributes and processes the input over a heterogeneous and arbitrarily big collection of sequences characterized by a different length compared to the input one. This is an adaptation from [jchenghu/ExpansionNet_v2](https://github.com/jchenghu/expansionnet_v2).
<br />
## Code Example
Load an image from path './image.jpg' to generate the caption.
*Write the pipeline in simplified style*:
```python
import towhee
towhee.glob('./image.jpg') \
.image_decode() \
.image_captioning.expansionnet_v2(model_name='expansionnet_rf') \
.show()
```
<img src="./cap.png" alt="result1" style="height:20px;"/>
*Write a same pipeline with explicit inputs/outputs name specifications:*
```python
import towhee
towhee.glob['path']('./image.jpg') \
.image_decode['path', 'img']() \
.image_captioning.expansionnet_v2['img', 'text'](model_name='expansionnet_rf') \
.select['img', 'text']() \
.show()
```
<img src="./tabular.png" alt="result2" style="height:60px;"/>
<br />
## Factory Constructor
Create the operator via the following factory method
***expansionnet_v2(model_name)***
**Parameters:**
***model_name:*** *str*
​ The model name of ExpansionNet v2. Supported model names:
- expansionnet_rf
<br />
## Interface
An image captioning operator takes a [towhee image](link/to/towhee/image/api/doc) as input and generate the correspoing caption.
**Parameters:**
***data:*** *towhee.types.Image (a sub-class of numpy.ndarray)*
​ The image to generate caption.
**Returns:** *str*
​ The caption generated by model.

BIN
cap.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 10 KiB

9
expansionnet_v2.py

@ -82,7 +82,8 @@ class ExpansionNetV2(NNOperator):
max_seq_len=max_seq_len, drop_args=model_args.drop_args, max_seq_len=max_seq_len, drop_args=model_args.drop_args,
rank='cpu') rank='cpu')
checkpoint = torch.load('{}/weights/rf_model.pth'.format(path), map_location=torch.device('cpu'))
cfg = self.model._configs()[model_name]
checkpoint = torch.load('{}/weights/{}'.format(cfg,cfg['weights']), map_location=torch.device('cpu'))
self.model.load_state_dict(checkpoint['model_state_dict']) self.model.load_state_dict(checkpoint['model_state_dict'])
self.model.to(self.device) self.model.to(self.device)
@ -132,3 +133,9 @@ class ExpansionNetV2(NNOperator):
pred = ' '.join(pred).capitalize() pred = ' '.join(pred).capitalize()
return pred return pred
def _configs(self):
config = {}
config['expansionnet_rf'] = {}
config['expansionnet_rf']['weights'] = 'rf_model.pth'
return config

BIN
tabular.png

Binary file not shown.

After

Width:  |  Height:  |  Size: 90 KiB

Loading…
Cancel
Save