Semantic Search with SQLite

https://neuml.github.io/txtai/embeddings/query/

by txtai • 4 years ago

107 7 4 years ago

neuml.github.io

codazoda• 4 years ago

It's not clear to me if txtai reaches out to the internet for all these queries. I assume it does and processes much of this in the cloud. That probably makes it a non-starter for much of my work. I do wonder, however, since their API docs talk a little about cloud options (making me wonder if non-cloud is the default). But, it's not immediately obvious to me.

txtai3• 4 years ago

It's all local except for downloading the transformers models for vectorization. TRANSFORMERS_OFFLINE_MODE can be set and models downloaded manually for offline use.

sthatipamala• 4 years ago

Any examples of this being combined with full text search to get a hybrid index?

txtai3• 4 years ago

Sure thing. Examples can be found here: https://neuml.github.io/txtai/examples/

This one re-ranks the output from an Elasticsearch index - https://colab.research.google.com/github/neuml/txtai/blob/ma...

The next major release have more examples using the local BM25 scoring module.

pantsforbirds• 4 years ago

Interesting choice of a database. Duckdb seems like it is a much better fit for the type of query being done. Wonder why sqlite was chosen over duckdb.

txtai3• 4 years ago

Duckdb is on the roadmap. SQLite is the first implementation. The plan is to extend the interface for other database types, including Duckdb.

tackygeoduck• 4 years ago

If you’re using python the Jina Docarray package supports very similar workflows out of the box. Using an SQLite backend is possible as well in addition to many others. Currently using the SentenceTransformers package as well and you can have your own search engine running in a couple hours.

Semantic Search with SQLite

Related Stories