Elasticsearch Was Great, but Vector Databases Are the Future

https://thenewstack.io/elasticsearch-was-great-but-vector-databases-are-the-future/

by DISCURSIVE • 2 years ago

16 7 2 years ago

anon373839• 2 years ago

God, no. These tools serve fundamentally different purposes. Thinking like this is, I believe, one of the reasons it’s become so difficult to find what you’re looking for on Google when you know exactly what you’re looking for.

tdeck• 2 years ago

Can't wait to search server logs with a vector database :/.

defnotai• 2 years ago

Agreed. It’s worth pointing out that this is a sponsored post, so they’re gonna make their pitch regardless.

In my experience, semantic search systems are very hard to test and don’t always return great results when given a very small or vague query, which seems counter to how people are taught to think about search.

And then if you want to do a hybrid search with reranking, those network calls start to add up and degrade the UX.

And then each month, there’s a new post about some new architecture that promises another search accuracy or performance boost, but you’ll need to rebuild the entire search system to find out if it’s effective for your use case.

ing33k• 2 years ago

ES can handle vector search quite well. I have used it in prod and it's scaled well too.

patrickhogan1• 2 years ago

Question on this. Why is this true vs just having an LLM which knows your intent coming up with SQL queries or Elasticsearch queries?

I assume the LLM understands my intent through a well described prompt more than a direct input vector database would. Assuming this, why does the database matter? Just have a great db backend and then on top an LLM that helps you find the data you want?

glial• 2 years ago

For my work, ES's 'killer feature' was not the search, but rather complex aggregations over huge datasets (despite the crazy query syntax). Hard to see how vector databases would replace that.

codingjaguar• 2 years ago

The complex aggregations are indeed an important feature for sophisticated search products like e-commerce search with interactive filtering. There is probably no easy way for a vector db to catch up quick. But, - Most RAG and enterprise apps don't really need that level of sophistication in UX, what they need instead is simple and reliable infrastructure - Give vector db some time to catch up. Google Spanner didn't have any SQL capability at the early days, but they caught up after a few years, now with full SQL support. And SQL is probably more complex than Elasticsearch's aggregation.

Elasticsearch Was Great, but Vector Databases Are the Future

Related Stories