dmitrykan

On a Research Assistant: https://muves.io/

It helps to comprehend research papers (and not only papers - any document on any language) faster.

The tool is free to use, because we have credits from GCP. I guess at some point we'll need to introduce some level of subscription fee to keep it alive and useful, as it uses LLMs and vector search quite a bit.

Feedback is welcome!

Leo Boytsov, my guest on the Vector Podcast, made an honest and an eye-opening claim about vector search being intellectually rewarding, but professionally undervalued. What picked my attention was how he gives credit to people who actually deserve it, and how he speaks modestly about his own achievements. When professionally as a researcher, he accumulated over 1800 citations by now and helped to create the famous HNSW vector search algorithm.

My take on whether we've lost the category of vector databases.

I'm working on the tool, that includes AI. My original target is to test it on my https://www.youtube.com/c/VectorPodcast by offering something that Lex Fridman does for his episodes.

Current features: 1. Download from YT 2. Transcribe using Vosk (output has time codes included) 3. Speaker diarization using pyannote - this isn't perfect and needs a bit more ironing out.

What needs to be done: 4. Store the transcription in a search engine (can include vectors) 5. Implement a webapp

If anyone here is interested to join forces, let me know.

https://dmitry-kan.medium.com/

I write about vector search, ANN algorithms, neural search frameworks, search engines and algorithms in general and publish episodes of the Vector Podcast.

Nice post, thanks for sharing! Learnt about new dbs / solutions for vector search from it.

Vicki mentions this survey I wrote some time ago: https://towardsdatascience.com/milvus-pinecone-vespa-weaviat...

Hoping it'll be useful as well!

thank you, Doug! It is mutual. Had a fantastic discussion with you on the podcast: https://www.youtube.com/watch?v=Kpua1Euc-B8

It was great to discuss with Jo Kristian on these topics and more:

- History of Vespa - Tensor data structure and its use cases - Multi-stage ranking pipeline - Game-changing vector search in Vespa - Approximate vs exact nearest neighbor search tradeoffs - Misconceptions in neural search - Multimodal search is where vector search shines - Power of building fully-fledged demos - How to combine vector search with sparse search: Reciprocal Rank Fusion - The question of WHY (my favourite)

I wonder what topics are interesting to HN community -- it would help me focus on these topics / embed into my questions in new episodes.

Thanks for the article, I've learnt new search engines despite spending a couple of years recently in web scale search. I think you may consider https://usearch.com/ as another dimension in web scale search, where query log is learnt from the data, making it quite unique.

Great project! Elasticsearch / OpenSearch / Solr have their own learning to rank plugins. Have you considered integrating Metarank with such systems? Or is your vision to provide a reranker layer, that can be independent of the underlying search engine architecture?

This is fantastic work, Max! I like where you're going with this unit economy.

It is increasingly more and more common that companies need ways of productizing their deep learning models with tight SLA constraints and tight budgets. I think, one of the logical steps could be integration with vector databases / frameworks and measuring gains there in client setups.

5th episode of Vector Podcast with Tom Lackner, VP Engineering at classic.com explaning the choise of Qdrant vector DB for his projects.

thanks!

To learn about vector search and get practical with it, you can check out my blog as well: https://dmitry-kan.medium.com/list/vector-search-e9b564d1427...

I'm also hosting a podcast around these topics: https://www.youtube.com/channel/UCCIMPfR7TXyDvlDRXjVhP1g

11M+ Wikipedia articles indexed for similarity search.

From code to podcast to billion-scale ANN challenge and statistics

Thanks! One practical target I imagine having is: 1. write the program in Python. 2. translate it into Ć ("interlingua") 3. compile Ć into Go -- for instance there is a belief around me, that Python is not suitable for web scale loads, while Go is more suitable.

Go source code / binary in this case are of less importance for code readability, because they are meant for production deployments. Something that happened in GWT: write in Java, compile into JavaScript.

Thanks for sharing! The practical value of such languages could be to build an interlingua and support Python->Ć->Go. The question is: if Ć->Python works seamlessly, how easy it is to make the opposite translation? For instance, take some code that uses external library, like spacy.

If this becomes ever possible, it is going to be revolutionary.

A related discussion and a detailed study about vector databases can be found here: https://news.ycombinator.com/item?id=28727816

Vector Podcast is here to bring you the depth and breadth of Search Engine Technology, Product, Marketing, Business. In the podcast we talk with engineers, entrepreneurs, thinkers and tinkerers, who put their soul into search.

Depending on your interest, you should find a matching topic for you -- whether it is deep algorithmic aspect of search engines and information retrieval field, or examples of products offering deep tech to its users.

"Vector" -- because it aims to cover an emerging field of vector similarity search, giving you the ability to search content beyond text: audio, video, images and more.

"Vector" also because it is all about vector in your profession, product, marketing and business.

SoundCloud: https://soundcloud.com/vector-podcast-dmitry-kan/greg-kogan-...

Apple Podcasts: https://podcasts.apple.com/us/podcast/greg-kogan-pinecone-on...

sure!

Yes, for larger datasets of vectors, having the search do a linear scan will likely be slow. So you could take a look at KNN/ANN (K nearest / Approximate nearest neighbors), like https://faiss.ai/ But if you prefer to offload this complexity to a database, you can pick and evaluate one from my blog post.

You have 4 out of 6 open source DBs, and 2 commercial ones give you the managed service. All 6 can scale to quite large numbers of vectors.

My goal is to systematically study each DB through the lens of a specific search task, which will not be (only) text based.

If you think we could collaborate in some way on your dataset, that would be fantastic and probably a learning experience to both sides.

Thanks for more detail, this sounds quite interesting. So basically the 16-digit chromosome records you added to each document would work as nearest neighbors? Did this satisfy all your search needs for an algorithm?

btw, if would be cool to see some example of a genetic algorithm input and resulting vector? And how do you use the vectors in down-stream tasks?

Such a great question! I think vector search is designed to solve a specific task, like finding K nearest neighbors for your given vector embedding. If you have a few hundred thousand, a million, it might work fine with a linear scan. Again, this depends on your use case. But if you have tens of millions, hundreds of millions or even a billion of vectors, you will need something more robust to store, index and search over this data.

An example in NLP world is BERT-like NNs, that allow you to embed your text into a dense vector representation.

Hi! It is a metered post atm, will consider making it more public over time. If you want to get practical with vector search in Solr/Elasticsearch/Opensearch you can try out this repo: https://github.com/DmitryKey/bert-solr-search

Hi all, the blog's author here. I'm really curious to learn what is the perception of vector search in general, and if you had any exposure to vector databases of any sort / considering to try one out.

Neural Search in Elasticsearch: from vanilla to KNN to hardware acceleration

This is a third post in the series of learning vector search with Solr, Lucene and now Elasticsearch. In this post you will find different approaches and algorithms to implement vector search, including KNN (K nearest neighbours) and hardware accelerated KNN.

It will be great to collaborate, Max! So far I have the storage part figured out. And scoring is currently applied to the entire set of indexed documents. My goal is to work on some sort of nearest neighbours approach. So it'll be great to see if we could port KNN over to Solr.

Second post in the series about building Neural Search with BERT and Solr -- with the focus on practical aspects, speed, memory consumption and semantics. Will appreciate your feedback!