diff --git a/README.md b/README.md index 4ce0c37..ed12840 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,111 @@ # Image Embedding Operator with Resnet50 -Authors: name or github-name(email) +Authors: Kyle, shiyu22 ## Overview -Introduce the functions of op and the model used. +This Operator generates feature vectors from the pytorch pretrained **Resnet50** mode, which is trained on [COCO dataset](https://cocodataset.org/#download). + +**Resnet** models were proposed in “[Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385)”, this model was the winner of ImageNet challenge in 2015. The fundamental breakthrough with ResNet was it allowed us to train extremely deep neural networks with 150+layers successfully. Prior to ResNet training very deep neural networks was difficult due to the problem of vanishing gradients. ## Interface -The interface of all the functions in op. (input & output) +`Class Resnet50ImageEmbedding(Operator)` [source](./resnet50_image_embedding.py) + +`__init__(self, model_name: str)` + +**param:** + +- model_name(str): the model name for embedding, like 'resnet50'. + +`__call__(self, img_tensor: torch.Tensor)` + +**param:** + +- img_tensor(torch.Tensor): the normalized image tensor. + +**return:** + +- cnn(numpy.ndarray): the embedding of image. ## How to use -- Requirements from requirements.txt -- How it works in some typical pipelines and the yaml example. +### Requirements + +You can get the required python package by [requirements.txt](./requirements.txt) and [pytorch/requirements.txt](./pytorch/requirements.txt). In fact, Towhee will automatically install these packages when you first load the Operator Repo, so you don't need to install them manually, here is just a list. + +- towhee +- torch +- torchvision +- numpy + +### How it works + +The towhee/resnet50_image_embedding Operator implements the function of image embedding, which can add to the pipeline, for example, it's the key Operator within [image_embedding_resnet50](https://hub.towhee.io/towhee/image-embedding-resnet50) pipeline, and it is the red box in the picture below. + +![img](./pic/operator.png) + +When using this Operator to write Pipline's Yaml file, you need to declare the following content according to the interface of resnet50_image_embedding class: + +```yaml +operators: + - + name: 'embedding_model' + function: 'towhee/resnet50-image-embedding' + tag: 'main' + init_args: + model_name: 'resnet50' + inputs: + - + df: 'image_preproc' + name: 'img_tensor' + col: 0 + outputs: + - + df: 'embedding' + iter_info: + type: map +dataframes: + - + name: 'image_preproc' + columns: + - + name: 'img_transformed' + vtype: 'torch.Tensor' + - + name: 'embedding' + columns: + - + name: 'cnn' + vtype: 'numpy.ndarray' +``` + +We can see that in yaml, the **operator** part declares the `init_args` of the class and the` input` and `output`dataframe, and the **dataframe** declares the parameter `name` and `vtype`. + +### File Structure + +Here is the main file structure of the `resnet50-image-embedding` Operator. If you want to learn more about the source code or modify it yourself, you can learn from it. + +```bash +├── .gitattributes +├── .gitignore +├── README.md +├── __init__.py +├── requirements.txt #General python dependency package +├── resnet50_image_embedding.py #The python file for Towhee, it defines the interface of the system and usually does not need to be modified. +├── resnet50_image_embedding.yaml #The YAML file contains Operator information, such as model frame, input, and output. +├── pytorch #The directory of the pytorh +│   ├── __init__.py +│   ├── model #The directory of the pytorch model, which can store data such as weights. +│   ├── requirements.txt #The python dependency package for the pytorch model. +│   └── model.py #The code of the pytorch model, including the initialization model and prediction. +├── test_data/ #The directory of test data, including test.jpg +└── test_resnet50_image_embedding.py #The unittest file of the Operator. +``` ## Reference -Model paper link. +- https://pytorch.org/hub/pytorch_vision_resnet/ +- https://arxiv.org/abs/1512.03385 + + diff --git a/pic/operator.png b/pic/operator.png new file mode 100644 index 0000000..ba49963 Binary files /dev/null and b/pic/operator.png differ