Serverless Semantic Search, Free tier only

https://qdrant.tech/articles/serverless/

by todsacerdoti • 3 years ago

91 61 3 years ago

llogiq6• 3 years ago

Author here. This was a fun exercise to produce a semantic search using only free-tier services (which in the case of Cohere means non-commercial use only and in the case of AWS Lambda is limited to one year).

It also marks my first foray into using cloud services for a project. I've long been a cloud sceptic, and doing this confirmed some of my suspicions regarding complexity (mostly the administrative part around roles and URLs), while the coding part itself was a blast.

rcme3• 3 years ago

I find serverless to be needlessly complex. I'd rather write an HTTP server and serve it off of t3.micro instance (also free-tier eligible). So much simpler for side projects.

llogiq6• 3 years ago

When starting this project I thought the same thing, but having done it I honestly cannot tell that much of a difference. Yes, there are two more steps in setting up the Lambda function, but in the end you still write an HTTP server and have them serve it.

kacperlukawski3• 3 years ago

It's a bit easier in Python if you use tools like https://www.serverless.com/. I'm not sure if Rust has something similar yet.

cebert2• 3 years ago

What about serverless do you find to be “needlessly complex”?

brigadier132• 3 years ago

Serverless is too broad a category to say things like "it's too complex". For example, if you already know docker, you can use google cloud run and just deploy the container to it. You then just say "I want to allow this many simultaneous connections, a minimum of N instances, a maximum of M instances, and each instance should have X vcpus and Y gb of ram".

joshstrange• 3 years ago

“Serverless” often has some upfront complexity but I greatly prefer it because once I have it running I’ve never had scaling issues or even had to think about them. To each their own and I’m sure that serverless isn’t the answer for everyone but for my projects (which are very bursty, with long periods of inactivity) is a dream.

numpad0• 3 years ago

Yeah it seems the premise of serverless is your code always restarts, which is exactly the same as cloud. The only difference is in front of trillion explosive gotchas in the giant 200GB free middleware called GNU/Linux are their employment in case with the serverless vs yours with the cloud.

UNIX is close to turning 50, and people are fundamentally paying as well as getting paid to make a written program loop to the beginning, instead of exiting. I think this is kind of wrong.

cddotdotslash• 3 years ago

It depends what you’re doing. I’ve run many side projects off a single Lambda function with the “public URL” config enabled. I pay $0 because of the free tier and updating the code is as simple as pushing a ZIP file. No SSH, no OS updates, nothing else to worry about. You start to get into trouble when you try to break your app into tons of microservices without using some kind of framework or deployment tooling to keep it straight.

foobarbecue2• 3 years ago

I find "serverless" is indeed more complex, because it's a higher abstraction layer. Often, I see people deploying containers lambdas or pods that are full unix environments, with permissions, network routing, filesystems etc. And then because it's "serverless" they use permissions (IAM), networking stuff (VPC, etc), filesystems (S3 etc), and other capabilities that they already have in a lower abstraction level (unix) and are sort of also using. So the complexity of a unix server is a unix server, but the complexity of "serverless" is a unix server plus all the capabilities you duplicated at a higher abstraction level.

Many other commenters replying to https://news.ycombinator.com/item?id=36693471 are interpreting "complex" as "hard for me to set up." I think that's neither here nor there -- no matter what's underneath, you can always rig something to deploy it with the press of a button. The question is: how many layers of stuff did you just deploy? How big of a can of worms did you just dump on future maintainers?

Roark66• 3 years ago

Cool, but I suggest to look into using terraform for "infrastructure as code" for creating all sorts of aws services/infrastructure and maintaining state.

It seems complex at first, but it is a lot more maintable and portable than creating aws infrastructure manually in the console. Once you leave your service to run for 6 months you will forget where stuff is, then in the worst possible moment if it goes down and you need to make some change you'll be franticly looking for aws docs... "can I create a synthetic canary and use the lambda I already have, or do I have to delete it and create it from Cloud Watch interface?" These kind of questions are the bane of Aws ops experience... And once you learn everything they "bring a new console experience"... So I prefer to learn terraform once and that's it.

Why terraform and not python with boto, cdk, cloudformation or ansible or something else? Because terraform is easy to port between providers (sort of), people who are not that good in python find terraform easier so you don't need "senior" people to maintain your code, finally it's a pretty "opinionated" about how w stuff should be done, so it's unlikely you'll open your project in a year and think "why in the world I did that!?", because all your tf projects will be very similar most likely. Also tf is mainly for infrastructure as code, there is no configuration management like in ansible... It is for one thing and it does it relatively well. (I have no relation to TF beyond being a happy user).

llogiq6• 3 years ago

Thank you for the suggestion! I actually thought about using terraform, but I wanted to keep the experiment somewhat minimal regarding technologies and as I had already added AWS Lambda and Rust to the tech stack, I wanted to stay as close to the metal as possible. Besides, this is not for commercial applications, so I don't think high availability is in scope.

andre-z2• 3 years ago

Made with Qdrant Vector Database https://github.com/qdrant/qdrant

Cohere embed API https://docs.cohere.com/reference/embed

and AWS Lambda

djbusby• 3 years ago

Those Cohere docs are awesome.

h1t35h• 3 years ago

Can't agree more..It is some UX done right.

victor106• 3 years ago

Agree,

Anyone knows what was used to generate the docs?

hn_205912495• 3 years ago

Am I being unreasonable to find it bizarre that the tutorial begins with subscribing to 3 different SAAS vendors?

Especially seeing these days you can run a vector store on-disk if you have less than 10 million records, pull any free embedding model straight from HuggingFace and run on consumer hardware (your laptop).

kacperlukawski3• 3 years ago

If you need semantic search locally then it's fine, but serving an embedding model might be still challenging. And if you want to expose it, your laptop might be not enough.

binarymax3• 3 years ago

Shameless plug: I built Mighty Inference Server to solve this problem. Fast embeddings with minimal footprint. Better BEIR and MTEB scores using the lightning fast and small E5 V2 models. Scales linearly on CPU, no GPU needed.

https://max.io

hn_205912495• 3 years ago

I've hosted embedding models on AWS Lambda (fair that this is a vendor, but 1 vs. 3), if you try an LLM with 1B+ parameters you will struggle, but if the difference between a light-weight BERT-like transformer and an LLM is only a few % of loss, why bother getting your credit card out?

Edit: another thought, skip lambda entirely and run the embedding job on the server as a background process, and use an on-disk vector store (lancedb)

bootsmann• 3 years ago

You serve the embedding model in a lambda and then run something like FAISS in the backend.

WinLychee2• 3 years ago

Doesn't match how things are run in production these days. As a vendor, you need to target the customer's environment as closely as possible. Even if it's theoretically feasible to serve off a single machine, you should have a cloud-native setup ready to go.

In principle you could totally run this on a single bare-metal node, but most will not be doing that in practice.

hn_205912495• 3 years ago

you should have a cloud-native setup ready to go

why is storing the file as a FAISS/LanceDB on-disk vector store not "cloud native"? I am running this setup in production across dozens of nodes, we migrated all of our infrastructure off Pinecone towards this solution and have seen 10x drop in latency, and the cost improvements have been dramatic (from paid, to totally free).

I have a bit of an axe to grind in the vector DB space, it feels like the industry has gaslit developers over the last year or so into thinking SAAS is necessary for vector retrieval, when low latency on-disk KNN across vectors is a solved problem.

vikp4• 3 years ago

This tutorial is very complex. Here's how to get free semantic search with much less complexity:

  1. Install sentence-transformers [1]
  2. Initialize the MiniLM model - `model = SentenceTransformer('all-MiniLM-L6-v2')`
  3. Embed your corpus [2]
  4. Embed your queries, then search the corpus

This runs on CPU (~750 sentences per second), and GPU (18k sentences per second). You can use paragraphs instead of sentences if you need more text. The embeddings are accurate [3] and only 384 dimensions, so they're space-efficient [4].

Here's how to handle persistence. I recommend starting with the simplest strategy, and only getting more complex if you need higher performance:

  - Just save the embedding tensors to disk, and load them if you need them later.
  - Use Faiss to store the embeddings (it will use an index to retrieve them faster) [5]
  - Use pgvector, an extension for postgres that stores embeddings
  - If you really need it, use something like qdrant/weaviate/pinecone, etc.

This setup is much simpler and cheaper than using a ton of cloud services to do embeddings. I don't know why people make semantic search so complex.

I've used it for https://www.endless.academy, and https://www.dataquest.io and it's worked well in production.

[1] https://www.sbert.net/

[2] https://www.sbert.net/examples/applications/semantic-search/...

[3] https://huggingface.co/blog/mteb

[4] https://medium.com/@nils_reimers/openai-gpt-3-text-embedding...

[5] https://github.com/facebookresearch/faiss

kacperlukawski3• 3 years ago

How would you host sentence-transformers model for free? You need it to vectorize each query so that has to be hosted somewhere. Is there any way to do it for free?

vikp4• 3 years ago

Just run it on CPU, on your own machine. That's the cheapest way. You could also rent a free/cheap VPS, and even parallelize across multiple machines/cores if you need it.

andre-z2• 3 years ago

TBH, it does not look "less complex", not at all. :) install, install, ... but where to install and run all of this? The topic is "serverless". This means you do not need to run anything, just need two cloud APIs and a Lambda Script.

vikp4• 3 years ago

I'd rather run ~10 lines of code locally than setup 3 cloud services and a lambda function, but to each their own...

hn_205912495• 3 years ago

15 minutes of install, install, install beats getting into the vicious SAAS vendor cycle of pay, pay, pay with heavy lock-in.

hiatus• 3 years ago

If you are going to shill for your project, the proper thing to do is disclose that.

https://github.com/azayarni is a contributor (andre-z on twitter).

zX41ZdbW• 3 years ago

I also recommend this approach when you want to understand every step. I recently did a presentation about this topic: https://www.youtube.com/watch?v=hGRNcftpqAk

It covers end-to-end, including ClickHouse as a vector database.

istjohn• 3 years ago

How does sentence-transformers compare to OpenAI embeddings? How long does it take to generate an embedding on a CPU?

freedmand• 3 years ago

I have proof-of-concepted[1] building an entirely static website offering client-side semantic search using transformers.js[2] and Semantra*[3] with MiniLM-L12[4]. It's usable on fast connections and decent CPUs but is not delightfully fast and can't handle huge document loads. Still could be useful for some embedded documents! I hope to incorporate an auto-builder for these experiences in the next big release of Semantra.

[1] https://twitter.com/dylfreed/status/1651572024488218627

[2] https://xenova.github.io/transformers.js

[3] https://github.com/freedmand/semantra *(author of this tool)

[4] https://huggingface.co/sentence-transformers/all-MiniLM-L12-...

lolive• 3 years ago

You absolutely NEEED to make it a plugin for Obsidian !!!!

brink• 3 years ago

I built out a Bible semantic search engine for myself last year using qdrant and Rust. It was a really fun experience, qdrant is solid, and Andrey was a nice guy to work with.

Serverless Semantic Search, Free tier only

Related Stories