Efficientnet Embedding Operator

Authors: kyle he

Overview

EfficientNets are a family of image classification models, which achieve state-of-the-art accuracy, yet being an order-of-magnitude smaller and faster than previous models[1], which is trained on imagenet dataset.

Interface

__init__(self, model_name: str = 'efficientnet-b7', framework: str = 'pytorch', weights_path: str = None)

Args:

• model_name:
  • the model name for embedding
  • supported types: str, for example 'efficientnet-b7'
• framework:
  • the framework of the model
  • supported types: str, default is 'pytorch'
• weights_path:
  • the weights path
  • supported types: str, default is None, using pretrained weights

__call__(self,  image: 'towhee.types.Image')

Args:

• image:
  • the input image
  • supported types: towhee.types.Image

Returns:

The Operator returns a tuple Tuple[('feature_vector', numpy.ndarray)] containing following fields:

• feature_vector:
  • the embedding of the image
  • data type: numpy.ndarray
  • shape: (dim,)

Requirements

You can get the required python package by requirements.txt.

How it works

The towhee/efficientnet-image-embedding Operator implements the function of image embedding, which can add to the pipeline. For example, it's the key Operator named embedding_model within image-embedding-efficientnetb5 and image-embedding-efficientnetb7 pipeline.

Reference

[1].https://github.com/lukemelas/EfficientNet-PyTorch#example-feature-extraction

More Resources

• The guide to instructor-xl | HKU NLP: instructor-xl: an instruction-finetuned model tailored for text embeddings with the best performance when compared to instructor-base and instructor-large.
• What is a Generative Adversarial Network? An Easy Guide: Just like we classify animal fossils into domains, kingdoms, and phyla, we classify AI networks, too. At the highest level, we classify AI networks as "discriminative" and "generative." A generative neural network is an AI that creates something new. This differs from a discriminative network, which classifies something that already exists into particular buckets. Kind of like we're doing right now, by bucketing generative adversarial networks (GANs) into appropriate classifications. So, if you were in a situation where you wanted to use textual tags to create a new visual image, like with Midjourney, you'd use a generative network. However, if you had a giant pile of data that you needed to classify and tag, you'd use a discriminative model.
• Comparing Different Vector Embeddings - Zilliz blog: Learn about the difference in vector embeddings between models and how to use multiple collections of vector data in one Jupyter Notebook.
• How to Get the Right Vector Embeddings - Zilliz blog: A comprehensive introduction to vector embeddings and how to generate them with popular open-source models.
• What is a Convolutional Neural Network? An Engineer's Guide: Convolutional Neural Network is a type of deep neural network that processes images, speeches, and videos. Let's find out more about CNN.
• Understanding Neural Network Embeddings - Zilliz blog: This article is dedicated to going a bit more in-depth into embeddings/embedding vectors, along with how they are used in modern ML algorithms and pipelines.
• What is approximate nearest neighbor search (ANNS)?: Learn how to use Approximate nearest neighbor search (ANNS) for efficient nearest-neighbor search in large datasets.
• Enhancing Information Retrieval with Sparse Embeddings | Zilliz Learn - Zilliz blog: Explore the inner workings, advantages, and practical applications of learned sparse embeddings with the Milvus vector database
• An Introduction to Vector Embeddings: What They Are and How to Use Them - Zilliz blog: In this blog post, we will understand the concept of vector embeddings and explore its applications, best practices, and tools for working with embeddings.