Turbopuffer: Fast search on object storage

https://turbopuffer.com/blog/turbopuffer

by Sirupsen • 2 years ago

379 64 2 years ago

Having worked with Simon he knows his sh*t. We talked a lot about what the ideal search stack would look when we worked together at Shopify on search (him more infra, me more ML+relevance). I discussed how I just want a thing in the cloud to provide my retrieval arms, let me express ranking in a fluent "py-data" first way, and get out of my way

My ideal is that turbopuffer ultimately is like a Polars dataframe where all my ranking is expressed in my search API. I could just lazily express some lexical or embedding similarity, boost with various attributes like, maybe by recency, popularity, etc to get a first pass (again all just with dataframe math). Then compute features for a reranking model I run on my side - dataframe math - and it "just works" - runs all this as some kind of query execution DAG - and stays out of my way.

bkitano19• 2 years ago

+1, had the fortune to work with him at a previous startup and meetup in person. Our convo very much broadened my perspective on engineering as a career and a craft, always excited to see what he's working on. Good luck Simon!

snthpy• 2 years ago

Could you give an example of what you mean by _fluent "py-data" first way_ ?

You mean like a fluent API like `data.transform().filter()...` , that sort of thing?

cmcollier• 2 years ago

Unrelated to the core topic, I really enjoy the aesthetic of their website. Another similar one is from Fixie.ai (also, interestingly, one of their customers).

itunpredictable• 2 years ago

This website rocks

nsguy• 2 years ago

Yeah! fast, clean, cool, unique.

swyx• 2 years ago

what does fixie do these days?

sitkack• 2 years ago

They pivoted, but will probably pivot back to their original quest.

xarope• 2 years ago

Yes, I like the turboxyz123 animation and contrast to the minimalist website (reminds me of the zen garden with a single rock). I think people forget nowadays in their haste to add the latest and greatest react animation, that too much noise is a thing.

k2so• 2 years ago

This was my first thought too, after reading through their blog. This feels like a no-frills website made by an engineer, who makes things that just work.

The documentation is great, I really appreciate them putting the roadmap front and centre.

5-• 2 years ago

indeed! what a nice, minimal page... that comes with ~1.6mb of javascript.

bmar• 2 years ago

What's wrong with that?

bigbones• 2 years ago

Sounds like a source-unavailable version of Quickwit? https://quickwit.io/

pushrax2• 2 years ago

LSM tree storage engine vs time series storage engine, similar philosophy but different use cases

singhrac2• 2 years ago

Maybe I misunderstood both products but I think neither Quickwit or Turbopuffer is either of those things intrinsically (though log structured messages are a good fit for Quickfit). I think Quickwit is essentially Lucene/Elasticsearch (i.e. sparse queries or BM25) and Turbopuffer does vector search (or dense queries) like say Faiss/Pinecone/Qdrant/Vectorize, both over object storage.

fulmicoton• 2 years ago

Quickwit is targetting logs:

    - it does not do vector search. It can rank docs using BM25, but usually people just want to sort by timestamp.
    - its does not use an SSD cache. Quickwit reads directly into the object storage.
    - it is append-only (you can't modify documents)
    - it scales really well and typically shines on the 1TB .. 100PB range
    - it has a Elastic search compatible API.

vidar• 2 years ago

Can you compare to S3 Athena (ELI5)?

CyberDildonics• 2 years ago

Sounds like a filesystem with attributes in a database.

drodgers• 2 years ago

I love the object-storage-first approach; it seems like such a natural fit for the could.

eknkc4• 2 years ago

Is there a good general purpose solution where I can store a large read only database in s3 or something and do lookups directly on it?

Duckdb can open parquet files over http and query them but I found it to trigger a lot of small requests reading bunch of places from the files. I mean a lot.

I mostly need key / value lookups and could potentially store each key in a seperate object in s3 but for a couple hundred million objects.. It would be a lot more managable to have a single file and maybe a cacheable index.

jiggawatts2• 2 years ago

trigger a lot of small requests reading bunch of places from the files. I mean a lot.

That’s… the whole point. That’s how Parquet files are supposed to be used. They’re an improvement over CSV or JSON because clients can read small subsets of them efficiently!

For comparison, I’ve tried a few other client products that don’t use Parquet files properly and just read the whole file every time, no matter how trivial the query is.

eknkc4• 2 years ago

This makes sense but the problem I had with duckdb + parquet is it looks like there is no metadata caching so each and every query triggers a lot of requests.

Duckdb can query a remote duckdb database too, in that case it looks like there is caching. Which might be better.

I wonder if anyone actually worked on a specific file format for this use case (relatively high latency random access) to minimize reads to as little blocks as possible.

imiric• 2 years ago

ClickHouse can also read from S3. I'm not sure how it compares to DuckDB re efficiency, but it worked fine for my simple use case.

masterj• 2 years ago

Neither of these support indexes afaik. They are designed to do fast scans / computation.

tionis• 2 years ago

You could use a sqlite database and use range queries using something like this: https://github.com/psanford/sqlite3vfshttp https://github.com/phiresky/sql.js-httpvfs

Simon Willison wrote about it: https://simonwillison.net/2022/Aug/10/sqlite-http/

arcanemachiner• 2 years ago

That whole thing still blows my mind.

eknkc4• 2 years ago

Yep this thing is the reason I thought about doing it in the first place. Tried duckdb which has built in support for range requests over http.

Whole idea makes sense but I feel like the file format should be specifically tuned for this use case. Otherwise you end up with a lot of range requests because it was designed for disk access. I wondered if anything was actually designed for that.

cdchn2• 2 years ago

Is there a good general purpose solution where I can store a large read only database in s3 or something and do lookups directly on it?

I think this is pretty much what AWS Athena is.

tiew9Vii• 2 years ago

Cloud backed SQLLite looks like it might be good for this. Doesn’t support S3 though

https://sqlite.org/cloudsqlite/doc/trunk/www/index.wiki

canadiantim• 2 years ago

LanceDB

omneity3• 2 years ago

In 2022, production-grade vector databases were relying on in-memory storage

This is irking me. pg_vector has existed from before that, doesn't require in-memory storage and can definitely handle vector search for 100m+ documents in a decently performant manner. Did they have a particular requirement somewhere?

jbellis• 2 years ago

Have you tried it? pgvector performance falls off a cliff once you can't cache in ram. Vector search isn't like "normal" workloads that follow a nice pareto distribution.

omneity3• 2 years ago

Tried and deployed in production with similar sized collections.

You only need enough memory to load the index, definitely not the whole collection. A typical index would most likely fit within a few GBs. And even if you need dozens of GBs of RAM it won’t cost nearly as much as $20k/month as the article surmises.

yamumsahoe• 2 years ago

unsure if they are comparable, but is this and quickwit comparable?

hipadev23• 2 years ago

That’s some woefully disappointing and incorrect metrics (read and write latency are both sub-second, storage medium would be “ Memory + Replicated SSDs”) you’ve got for Clickhouse there, but I understand what you’re going for and why you categorized it where you did.

endisneigh• 2 years ago

Slightly relevant - do people really want article recommendations? I don’t think I’ve ever read an article and wanted a recommendation. Even with this one - I sort of read it and that’s it; no feeling of wanting recommendations.

Am I alone in this?

In any case this seems like a pretty interesting approach. Reminds me of Warpstream which does something similar with S3 to replace Kafka.

nh22• 2 years ago

$3600.00/TB/month

It doesn't have to be that way.

At Hetzner I pay $200/TB/month for RAM. That's 18x cheaper.

Sometimes you can reach the goal faster with less complexity by removing the part with the 20x markup.

TechDebtDevin• 2 years ago

I will likely never leave Hetzner.

AYBABTME• 2 years ago

200$/TB/month for raw RAM, not RAM that's presented to you behind a usable API that's distributed and operated by someone else, freeing you of time.

It's not particularly useful to compare the cost of raw unorganized information medium on a single node, to highly organized information platform. It's like saying "this CPU chip is expensive, just look at the price of this sand".

kirmerzlikin• 2 years ago

AFAIU, 3600$ is also a price for "raw RAM" that will be used by your common database via sys calls and not via a "usable API operated by someone else"

hodgesrm2• 2 years ago

It's not particularly useful to compare the cost of raw unorganized information medium on a single node, to highly organized information platform.

Except that it does prompt you to ask what you could do to use that cheap compute and RAM. In the case of Hetzner that might be large caches that allow you to apply those resources on remote data whilst minimizing transfer and API costs.

formerly_proven• 2 years ago

You seem to be quoting the highest figure from the article out of context as-if that is their pricing, but the opposite is the case.

$3600.00/TB/month (incumbents)
$70.00/TB/month (turbopuffer)

That's still 3x cheaper than your number and it's a SaaS API, not just a piece of rented hardware.

nh22• 2 years ago

as-if that is their pricing

No, that's not what I'm saying. Their "Storage Costs" table shows costs to rent storage from some provider (AWS?). It's clear that those are costs that the user has to pay for infrastructure needed for certain types of software (e.g. Turbopuffer is designed to be running on "S3 + SSD Cache", while other software may be designed to run on "RAM + 3x SSD").

I'm comparing RAM costs from that table with RAM costs in the real world.

The idea backed by that table is "RAM is so expensive, so we need to build software to run it on cheaper storage instead".

My statement is "RAM is that expensive only on that provider, there are others where it is not; on those, you may just run it in RAM and save on software complexity".

You will still need some software for your SaaS API to serve queries from RAM, but it won't need the complexity of trying to make it fast when serving from a higher-latency storage backend (S3).

yawnxyz• 2 years ago

can't wait for the day the get into GA!

cdchn2• 2 years ago

The very long introductory page has a ton of very juicy data in it, even if you don't care about the product itself.

zX41ZdbW• 2 years ago

A correction to the article. It mentions

    Warehouse BigQuery, Snowflake, Clickhouse ≥1s Minutes

For ClickHouse, it should be: read latency <= 100ms, write latency <= 1s.

Logging, real-time analytics, and RAG are also suitable for ClickHouse.

Sirupsen• 2 years ago

Yeah, thinking about this more I now understand Clickhouse to be more of an operational warehouse similar to Materialize, Pinot, Druid, etc. if I understand correctly? So bunching with BigQuery/Snowflake/Trino/Databricks... wasn't the right category (although operational warehouses certainly can have a ton of overlap)

I left that category out for simplicity (plenty of others that didn't make it into the taxonomy, e.g. queues, nosql, time-series, graph, embedded, ..)

arnorhs• 2 years ago

This looks super interesting. I'm not that familiar with vector databases. I thought they were mostly something used for RAG and other AI-related stuff.

Seems like a topic I need to delive into a bit more.

solatic• 2 years ago

Is it feasible to try to build this kind of approach (hot SSD cache nodes sitting in front of object storage) with prior open-source art (Lucene)? Or are the search indexes themselves also proprietary in this solution?

Having witnessed some very large Elasticsearch production deployments, being able to throw everything into S3 would be incredible. The applicability here isn't only for vector search.

francoismassot2• 2 years ago

If you don't need vector search and have very large Elasticsearch deployment, you can have a look at Quickwit, it's a search engine on object storage, it's OSS and works for append-only datasets (like logs, traces, ...)

Repo: https://github.com/quickwit-oss/quickwit

rohitnair2• 2 years ago

Elasticsearch and OpenSearch already support S3 backed indices. See features like https://opensearch.org/docs/latest/tuning-your-cluster/avail... The files in S3 are plain old Lucene segment files (just wrapped in OpenSearch snapshots which provide a way to track metadata around those files).

francoismassot2• 2 years ago

But you don’t have fast search on those files stored on object storage.

Turbopuffer: Fast search on object storage

Related Stories