VectorDB: Vector Database Built by Kagi Search

317 93 3 years ago

Where does it save the data to? How is it persisted?

Is there any limitation to this? Does it work with text 500-1000 words? Does it work well with text that aren’t full sentences? Ie. Are just a collection of phrases?

tyingq• 3 years ago

The README.md example just has it in memory. Looking at the source, the Storage class uses a plain file and python's pickle module:

https://github.com/kagisearch/vectordb/blob/main/vectordb/st...

mathverse2• 3 years ago

Wonder why Crystal was not used.

freedomben• 3 years ago

Crystal is a super cool and underrated language. But why do you mention it here? Does kagi use crystal in other places?

mathverse2• 3 years ago

They use it for search afaik.

YellowSuB• 3 years ago

Yes it is made with Crystal

promiseofbeans• 3 years ago

Based on their job postings, they use it for most of their back-end: https://help.kagi.com/kagi/company/hiring-kagi.html#full-tim...

ipsum22• 3 years ago

This is a wrapper around FAISS, a vector search library. FAISS has a simple API, so it might be a better for your use case if you don't need heavyweight libraries that VectorDB requires, which include PyTorch, Tensorflow, and Transformers.

dmezzetti15• 3 years ago

This is possible but how would you encode your vectors?

sharemywin• 3 years ago

what's wrong with these?

https://www.sbert.net/docs/pretrained_models.html

ipsum22• 3 years ago

Bring your own embeddings. PyTorch and TensorFlow packages are 2GB+ each (don't quote me on that), which is unnecessary if you're making a network call to your favorite embedding service.

PossiblyKyle2• 3 years ago

I fail to understand why I should use it over a different embedded vector DB like LanceDB or Chroma. Both are written in more performant languages, have a simple API with a lot of integrations and power if one needs it

dmezzetti15• 3 years ago

To be fair, Chroma is also written in Python. And while LanceDB and others are written in Rust, that doesn't automatically give it super powers.

PossiblyKyle2• 3 years ago

Fair point, then you could claim it's similar to this DB with its reliance on Faiss. Despite that, Chroma at this point is more feature rich. I was mostly referring to this https://thedataquarry.com/posts/vector-db-1/

You are not wrong about the performance from Rust, but LanceDB is inherently written with performance in mind. SIMD support for both x86 and ARM, and an underlying vector storage approach that's built for speed (Lance)

6r17• 3 years ago

Python programmer for 15 years and i picked up rust to write an oAuth gateway not long ago ; i wrote it in python beforehand - rust DOES give you superpowers ; especially if you compare it to something like python that isn't nowhere as fast and has no typing

hantusk• 3 years ago

I thought the API here was quite neat. It's fairly simple to implement a lancedb backend for it instead of sklearn/faiss/mrpt as the source code is really simple.

This repo is basically just a nice api and the needed chunking and batching logic. Using lancedb, you'd still have to write that, as exemplified here: https://github.com/prrao87/lancedb-study/blob/main/lancedb/i...

mark_l_watson• 3 years ago

Same for me. I started using Chroma (about) a year ago, I am used to it, and if I am using Python I look no further.

When I use Common Lisp or Racket I roll my own simple vector embeddings data store, but that is just me having fun.

freediver2• 3 years ago

It is faster!

We needed a low latency, on premise solution that we can run on edge nodes with sane defaults that anyone in the team can whim in a sec. Also worth noting is that our use case is end to end retrieval of usually few hundred to few thousand chunks of text (for example in Kagi Assistant research mode) that need to be processed once at run time with minimal latency.

Result is this. We periodically benchmark the performance of different embeddings to ensure best defaults:

https://github.com/kagisearch/vectordb#embeddings-performanc...

andy995• 3 years ago

The two most interesting things to me in a "minimal" framework would be eliminating dependency on HF transformers and helping customize chunking.

Not a knock against this project, I see where it can be helpful.

dmezzetti15• 3 years ago

My assumption is that this gives you the ability to locally encode vectors. This is useful for those not using API services to build their vectors.

andy995• 3 years ago

Transformer inference is ~60 lines of numpy[0] (closer to 500 once you add tokenization etc). It would be nice to just have this and not all of pytorch and transformers.

[0] https://jaykmody.com/blog/gpt-from-scratch/

jsimian• 3 years ago

Yeah, I just tried this out (props to the devs, super easy to set up) and my main gripe is the chunking algorithms aren't great - could be alot more useful with a context option that gives surrounding results to search results. The sliding window chunking method always cuts off the start of sentences.

andy995• 3 years ago

I've found it works better to chunk by some logical sections in the document, e.g header h2 h3 h4 etc or 1.1 1.1.1 ... plus to be able to ignore some stuff (header and footer) plus other customizations.

At least for use cases where there are clusters of many similarly formatted documents, it would be cool to have a way of easily customizing chunking.

osigurdson• 3 years ago

What is being used to actually create the embeddings?

dmezzetti15• 3 years ago

The library is open source. Here's where you can see how they're creating the embeddings. https://github.com/kagisearch/vectordb/blob/main/vectordb/em...

simonw• 3 years ago

https://github.com/kagisearch/vectordb/blob/453bb658bb710838...

Looks like it uses one of these, depending on your settings:

Fast model: google/universal-sentence-encoder/4

Multilingual model: universal-sentence-encoder-multilingual-large/3

Normal model (Alternative): BAAI/bge-small-en-v1.5

Best model: BAAI/bge-base-en-v1.5

syntaxing• 3 years ago

I always run into the issue of loading existing embedding. For instance, I want to embed a folder which have five files and two are new, is there a way to only add the two new files to the stored embedding using this or chromaDB?

rasengan• 3 years ago

Keep track of the files and when you see new ones only add those.

conception• 3 years ago

https://github.com/wallabag/wallabag

No one has mentioned wallabag yet, so wanted to. Been working well for me - has apps and extensions. If you’re not excited to self-host - https://www.wallabag.it/en has been flawless with the exorbitant price of… 11 euro a year.

0x6c6f6c2• 3 years ago

This isn't even slightly related to a vector database. I like Wallabag but this comes off as a shameless plug.

edwinbalani• 3 years ago

Wrong article I think; there was another post about a bookmark manager on here.

politelemon• 3 years ago

If a team has an operational store already in Postgres, wouldn't it be best to just use the PGVector extension? So the data and the vector search functionality sit together, and there's one less moving part in the tech stack to manage?

dmezzetti15• 3 years ago

This recent thread discussed this - https://news.ycombinator.com/item?id=38416994

m3kw9• 3 years ago

It shouldn’t be legal to call your vectordb vectordb

andy995• 3 years ago

I prefer it to, say, vectr (such contractions used to be popular) or 'barbershop' or a similar single irrelevant word name as is popular now.

worldsayshi2• 3 years ago

Att least with barbershop it becomes much more searchable. "barbershop db" will very likely succeed.

LtWorf• 3 years ago

Someone decided to call a software "informatica".

A british recruiter was quite insistent to have a call… turns out because she read "informatica" on my profile (as in, laurea in informatica), asked me how many years of experience I had with the tool, I replied "I just heard of it 2 minutes ago when you first mentioned it". Then got mad at me for having written the word "informatica" on my profile.

arbuge• 3 years ago

Well, they do have the .com domain, so maybe an exception applies here.

bee_rider2• 3 years ago

Sometimes I wonder if Microsoft was actually on to something with their naming scheme.

Something like “Kagi vector search for databases” at least doesn’t leave anything up for misinterpretation.

richardanaya• 3 years ago

Kagi search is amazing, i've used it for the last few months. If this is what they are using to power it, i'm optimistic.

iTokio• 3 years ago

From the GitHub repo:

Thanks to its low latency and small memory footprint, VectorDB is used to power AI features inside Kagi Search.

stainablesteel• 3 years ago

super cool! looks like i can make a local search engine out of the massive datahoard of pdf books and articles i have

hubraumhugo• 3 years ago

Is there any kind of comaprison of the different vector DBs? What would you choose for different use cases? How do they differ?

dmezzetti15• 3 years ago

This thread from a few months ago is a good read - https://news.ycombinator.com/item?id=36943318

pknerd• 3 years ago

This or anyone suggest some other db/lib for local QnA like testing on Mac?

dmezzetti15• 3 years ago

Just posted this thread that gives an in-depth look at LLM frameworks (local ones included) - https://news.ycombinator.com/item?id=38422264

ianpurton• 3 years ago

https://bionic-gpt.com/blog/you-dont-need-a-vector-database/

freediver2• 3 years ago

Dev here. Thanks for submitting. To be fair this is not really a database, but a wrapper around few primitives such as locally-ran embeddings and FAISS/mrpt with a ton of benchmarking behind the doors to offer sane defaults that minimize latency.

Here is an example Colab notebook [1] where this is used to filter the content of the massive Kagi Small Web [2] RSS feed based on stated user interests:

[1] https://colab.research.google.com/drive/1pecKGCCru_Jvx7v0WRN...

[2] https://kagi.com/smallweb

VectorDB: Vector Database Built by Kagi Search

Related Stories