sql

author: junjie.jiang

Desription

Read data from sqlite or mysql.

Code Example

Example

from towhee import DataLoader, pipe, ops
p = (
    pipe.input('image_path')
        .map('image_path', 'image', ops.image_decode.cv2())
            .map('image', 'vec', ops.image_embedding.timm(model_name='resnet50'))
                .output('vec')

)


# table cols: id, image_path, label

for data in DataLoader(ops.data_source.sql('sqlite:///./sqlite.db', 'image_table'), parser=lambda x: x[1]):
    print(p(data).to_list(kv_format=True))

# batch
for data in DataLoader(ops.data_source.sql('sqlite:///./sqlite.db', 'image_table'), parser=lambda x: x[1], batch_size=10):
    p.batch(data)

Parameters:

sql_url: str

the url of the sql database for cache, such as '<db_type>+<db_driver>://:@:/'

sqlite: sqlite:///./sqlite.db

mysql: mysql+pymysql://root:123456@127.0.0.1:3306/mysql

table_name: str

table name

cols: str

The columns to be queried, default to *, indicating all columns

If you want to query specific columns, use the column names and separate them with ,, such as 'id,image_path,label'

where: str

Where conditional statement, for example: id > 100

limit: int

The default value is 500. If set to None, all data will be returned.

More Resources

• Zilliz attended VLDB Workshop 2021 - Zilliz blog: Applied ML & AI for Database Systems and Application
• Vector Database Comparison: Compare any vector database to an alternative by architecture, scalability, performance, use cases and costs.
• Database Evolution Journey from Rows and Columns to Vectors - Zilliz blog: From structured SQL and NoSQL and cutting-edge vector databases, this journey undertakes a significant transformation in data management strategies.
• Vector Database Stories: The Zilliz blog includes technical tutorials, customer stories, and industry developments related to the world's most advanced vector database.
• Choosing Between Relational and Vector Databases - Zilliz blog: Learn how to choose between relational and vector databases for your application by understanding the strengths of each.
• About Zilliz | Zilliz: Vector database