Searchable field-level encryption on Supabase with CipherStash

https://supabase.com/blog/searchable-field-level-encryption-with-cipherstash

by dandraper • 7 days ago

48 51 7 days ago

tekacs• 4 days ago

I came here to say that this is presumably ORE/OPE (order-revealing/preserving) encryption, not FHE, but...

It is both remarkable and depressing how _little_ information is given, and how buried it is on the CipherStash website... _any_ information on what their security and/or threat model is, what is actually stored, how encryption and search works, or any trade-offs involved.

Just to list a few pages that tell you next to nothing:

https://cipherstash.com/docs/stack/reference/what-is-ciphers...

https://cipherstash.com/docs/stack/cipherstash/encryption/se...

https://cipherstash.com/docs/stack/cipherstash/encryption

I eventually found:

https://cipherstash.com/docs/stack/reference/security-archit...

Which... it sounds like 'searched without being decrypted' means... it encrypts your query against their fast KMS and uses that to compare against indexes that were also encrypted with the same KMS? And ORE/OPE is an optional mode when you want range support.

dandraper24• 4 days ago

Hey, CipherStash founder here: So we actually have 2 kinds of encryption: * standard encryption which can be decrypted (we call this "source" encryption) * what we call SEM: searchable encrypted metadata (cannot be decrypted)

Queries are performed by generating SEM which is "compared" to values in the table and the source encrypted values are returned to the caller (usually an application or agent) which does the actual decryption.

This is crucial because if you have a table of millions of rows, you don't want to decrypt every one to find the results. SEM works with built in postgres indexes (B-tree, GIN) so a query over millions of rows executes in literally milliseconds. Then the resultset (typically only 10s or 100s or rows) can is all that needs to be actually decrypted.

You're right that we use OPE and ORE. OPE is necessary on Supabase right now due to some limitations on their side but that's being addressed. ORE is more secure and almost as fast but requires "CREATE OPERATOR FAMILY" permissions in postgres to work.

We also use encrypted bloom filters and structured encryption (STE) for queries over JSON objects.

cipherjim2• 3 days ago

You basically reverse-engineered it correctly. I've written up the full threat model, components and per-index leakage in one place here: https://news.ycombinator.com/item?id=48967481

maxrmk2• 4 days ago

I’m having a hard time wrapping my head around what guarantees this does and does not make.

If you can run “select * where secret_col == 10”… why does it matter that the column is encrypted?

nullbio• 4 days ago

My thoughts as well. Seems like security theater.

aabhay• 4 days ago

you're probably only allowed a subset of the query language to talk to an encrypted table. E.g. only range queries that target a sample size > 5% of rows, etc., with exceptions for searches that hit an exact index such as looking up by id.

nicce2• 4 days ago

A lot is allowed: https://cipherstash.com/docs/stack/cipherstash/encryption/se...

nicce2• 4 days ago

It sounds like someone has discovered a side-channel, made a project around it, and forget/did not know what side-channel originally means. And then someone at Supabase who does not know crypto, is a victim of their marketing.

dandraper24• 3 days ago

lol yeah. Just a side chain that I found on a weekend. 8 years and $10m later. Couldn’t possibly be real now could it?

dandraper24• 4 days ago

CipherStash founder here.

If a column is encrypted using standard encryption (like AES-GCM) then the values are non-deterministic and fully randomized. That means that if you encrypt the same value twice, you'll get 2 different ciphertexts.

So the query: select * where secret_col == 10

Would actualy be: select * where secret_col == encrypt_aes(10);

And values in secret_col will never match (because the output of encrypt_aes will be different every time, even for the same input).

A common way around this is to use deterministic encryption which eliminates the randomization at the cost of a slightly weaker security model. What leaks is the ability to see if any 2 plaintexts are equal (because they have the same ciphertext) - what you actually want in the case of search.

You have to be careful implementing deterministic encryption though: don't use AES-GCM with a fixed nonce because the scheme completely breaks. You can use CBC mode but then you lose authenticity. We use AES-GCM-SIV (synthetic IV which retains authentication but is secure under a fixed nonce) and HMAC (keyed hashing).

But there are approaches to solving queries like: -- range/order SELECT * FROM foo WHERE x > 10; SELECT * FROM foo ORDER BY x; -- fuzzy text SELECT * FROM foo WHERE name ~ "dan";

These capabilities are all based on public research: For example, order/range uses: https://eprint.iacr.org/2016/612.pdf

Our docs are quite limited at the moment (fixing as quickly as we can!) but you can see the current list of supported queries here: https://cipherstash.com/docs/stack/cipherstash/encryption/qu...

hsbauauvhabzb2• 4 days ago

What does all this actually solve? Presumably SQL injection might still decrypt contents and order by/where clause enumeration would still be possible regardless. Keys must be stored in memory, on-disk or via a secret server meaning column encryption would not mitigate the impact of RCE/full shell compromise. Add in the cost of column level encryption when querying large volumes of data, this seems to be entirely angled at gold plated compliance security theatre rather than solving any actual problem.

MikhailTal• 4 days ago

the OP asked why, you more described the how.

What is the benefit of all this?

dandraper24• 4 days ago

Searchable encryption means you can have encryption and queries (from apps etc) still work for authorized users.

This is all started when I was the CTO of a health-tech and the engineers all had access to patient data. They needed DB access (data migrations, managing backups, reading non-sensitive fields) but not access to the sensitive values. 10m+ patient records - it made me nervous.

Row-level Security (RLS) was a nightmare, it was slow, easy to mess up and could be overridden by DBAs anyway. Plus the application connected to the DB using a service account so RLS was useless for end-user access control.

CipherStash solves the following:

* Hide sensitive data from direct database accessors (authorized or not!)

My original use-case. Direct access to the DB doesn't reveal sensitive data (unless the user also has permission to decrypt a specific record).

* Support or even replace application access controls

Decryption happens in the application layer and is based on end-user identity. That means a gap in application defenses is mitigated by the encryption. If the user can't decrypt the value then they can't access the data.

* Access auditing

Every sensitive data access is recorded (and by who) which is very useful for compliance and incident response. SQL query auditing is great for knowing what queries were run but it doesn't tell you what data was actually returned. Its also blind once data actually leaves the DB and can inadvertently leak sensitive data itself. Using key accesses with non-identifying value IDs to audit decryptions makes the auditing more reliable and super granular.

We have customers who use the tech to prove to their users (often large enterprise) that no internal staff can access the data.

* Agent access to data

The encryption shifts access control decisions down to the data layer on a per-value basis. This is incredibly useful for gating access to agents that need access to data. Every decryption requires authentication (e.g via Supabase Auth, Auth0 etc) and combining it with agent identities (creds issued to agents) means you can not only limit what agents can access, you can also audit exactly what _was_ accessed.

The searchable encryption is the enabler: value level encryption is what makes all of this possible. Searchable encryption means you can have encryption and queries (from apps etc) still work for authorized users.

Hope that helps! I'm so in the weeds I don't know if I explain things that well sometimes!

cvwright• 3 days ago

Don’t use deterministic encryption unless your data is already highly random.

Any adversary who knows the distribution of the plaintext data can guess what your encrypted values are, with accuracy that improves as your sample size increases.

There were a bunch of papers on this 10-12 years ago. (Full disclosure: I was an author on some of them.)

maxrmk2• 2 days ago

Thanks Dan, this is very interesting. From someone deeply involved in privacy I see the benefits of the auditability and risk mitigation in cases like SQL injection.

What does the risk profile look like in a full leak of the encrypted database?

If you had a column of integers are you able to order them without decrypting? Check that values are the same? Identify null values?

MattPalmer1086• 4 days ago

One would hope that only users who can already decrypt the data can perform the queries. In that case, it would give much faster query performance without allowing inference attacks on the data.

If not, then a lot of the data could be easily reconstructed.

dandraper24• 3 days ago

It is indeed the case.

dandraper24• 3 days ago

Because queries are encrypted, too. And you need to be authorized to encrypt a query term. That's the whole crux.

TBH, I'm so close to this now I forget that part isn't obvious! You literally can't do:

select * from blah where secret_col == 10

And get meaningful results. You can only do something like:

select * from blah where secret_col == Encrypt(10);

And in order to generate that encrypted query, you must be authorized. So an adversary can't simply do trial and error queries to guess records.

jph• 4 days ago

Anywhere you need direct database access outside your application, CipherStash Proxy provides a secure escape hatch.

This sounds like a back door. Is it?

To me, the whole article feels super-unclear about what exactly is involved.

Can HN folks who know more weigh in?

dandraper24• 4 days ago

Hey, CipherStash founder here.

Our model is that data is encrypted in the application before being saved to the database. In order to encrypt, decrypt or query, you need to use the SDK (@cipherstash/stack). Connecting via psql, pgadmin etc, you'll only ever be able to see encrypted values (those tools don't know how to decrypt).

Proxy connects to the database and then you connect to the proxy (with psql or whatever). Proxy can now perform queries and decryptions on your behalf so that you can still access data if you need - hence "escape hatch".

The proxy authenticates using your credentials (or an access key) and interacts with the key management server.

Hope that helps!

dandraper24• 4 days ago

Also the proxy runs in your infra not ours (docker container).

localhoster• 4 days ago

Looking at their website, reading through abit, and seeing the comments here. Guys, this website is entirely ai generated, which might indicate about their product some. Not saying that they are lieing or anything, but expect a wall of text that no one really read through, and expect to (pun intended) cipher out the details.

dandraper24• 4 days ago

CipherStash founder here: I'll cop it about the website - we're a small team so we lean on AI for marketing copy but this is a reminder that we need to do better.

The tech is the result of 6 years of work. We're a team of 8 engineers and have been working on it full time thanks to VC funding.

I'm obviously biased but its legit tech. We've invested in making it fast, secure and able to support most common query patterns.

Our github repos might be better than our website but I'll certainly admit that our docs/marketing needs some work!

Checkout out: https://github.com/cipherstash/stack https://github.com/cipherstash/encrypt-query-language https://github.com/cipherstash/proxy

DANmode• 3 days ago

I'll certainly admit that our docs/marketing needs some work!

Any chance you’re hiring for a related role? Technical marketing and technical documentation are my jam!

sorenbs• 4 days ago

We worked with the CipherStash team to build a Prisma Next extension that provides full type safety for their additional query shapes, and a fully managed experience for applying schema changes. It was a great experience working with the team, and we were particularly excited about this, as it helped us prove out the new extensibility mechanisms we have been working so hard for in Prisma Next.

Docs here: https://cipherstash.com/docs/stack/cipherstash/encryption/pr...

And a mention in our April update on Prisma Next: https://www.prisma.io/blog/prisma-next-roadmap-april-milesto...

If you want to give CipherStash on Supabase a try, using Prisma Next is the smoothest experience.

ethical• 4 days ago

Absolute snake-oil. Even IBM's version.

dandraper24• 3 days ago

Encrypted*.

cipherjim2• 3 days ago

James here, principal engineer at CipherStash. A few of you have (fairly) pointed out that the threat model is hard to find on our site, so rather than answer piecemeal let me lay the whole thing out. tekacs got most of the way there by reading between the lines, which is on us, not them.

The premise first: organisations shouldn't have to make a mutually exclusive choice between encrypting their data and being able to query it. That's the reason CipherStash exists, and everything below follows from it.

The threat model in one sentence: anyone with access to the database itself - dumps, backups, stolen credentials, SQLi, a hosting provider, an over-curious DBA - should learn as little as possible about sensitive values, and every legitimate decryption should require authentication and leave an audit record. We're explicitly not claiming to defend against an attacker who fully owns your running application with valid credentials. Nothing at this layer can, and anyone who tells you otherwise is selling snake oil.

The components:

* ZeroKMS: the key service. Root key material sits here, HSM-backed. Every encrypted value gets its own unique data key, but ZeroKMS never has it - keys are derived in your application from a seed we return plus a client key that never leaves your infrastructure. Two parties are needed to derive any key, so CipherStash cannot decrypt your data. Not "won't", can't.

* CTS: the token service. Exchanges tokens from your identity provider (Supabase Auth, Auth0, etc.) so decryption is bound to end-user identity rather than a service account. This is what makes the SQLi case boring: an attacker's session can only derive keys for data that user was allowed to read anyway.

* Auditing: because every decryption requires a key derivation, the key service sees every access - which value (by key ID, not content) and by whom. Query logs tell you what was asked; this tells you what was actually decrypted.

There are two ways to integrate, and you can run both:

* At the application layer, via the ORM adapters (Supabase, Prisma, Drizzle). Encryption and decryption happen inside your application process - the database only ever stores ciphertext and encrypted index terms, and plaintext exists nowhere outside your app. This is the tightest trust boundary and the mode the identity-bound stuff above is built for.

* At the SQL layer, via the Proxy - a Rust Postgres proxy that runs in your infrastructure. It encrypts and decrypts transparently, before results reach the client, so anything that speaks Postgres works unmodified: psql, BI tools, migration scripts, legacy apps you can't touch. Crucially, credentials through the Proxy are still tied to an individual identity, not a machine identity. There are two layers of auth in play: a machine credential identifies the Proxy instance itself, and the user's identity is tunnelled through it - so the engineer running psql derives keys as themselves, subject to the same access policy and audit trail as everything else. The trade-off between the two modes is where plaintext appears, not who's accountable. Escape hatch, not back door - a back door would be a bypass of the access controls - this isn't.

Key rotation is handled by the same re-encryption scheme that powers derivation: root key material can be rotated server-side without re-encrypting the data itself, because the stored ciphertexts never depended on a long-lived key sitting next to them.

On leakage, since that's the crux of the scepticism: yes, searchable encryption leaks. Equality indexes (HMAC) reveal when two values match, so frequency. Range indexes (ORE, Lewi-Wu 2016) reveal ordering. Text indexes (encrypted bloom filters) reveal probabilistic token membership. This isn't a bug we're hiding, it's the fundamental trade: for the database to use an index it has to compare something, and whatever it compares, it learns. The schemes that leak nothing (FHE, ORAM) are bullshit slow. What we've done is make the leakage bounded, documented, and opt-in per column, per query type. If the frequency or ordering of a column is itself sensitive - salaries being the classic example - don't index it. Encrypt it plainly and filter after decryption in the app. Most columns need no search at all, and the default is exactly that: no index, no leakage beyond ciphertext length.

The docs criticism lands and we're fixing it - the security architecture page (https://cipherstash.com/docs/stack/reference/security-archit...) now covers the primitives, key hierarchy and per-index leakage in one place, benchmarks are at https://github.com/cipherstash/benches, and the crypto that matters is open source and auditable: https://github.com/cipherstash/encrypt-query-language, https://github.com/cipherstash/ore.rs, https://github.com/cipherstash/proxy.

sigseg1v2• 3 days ago

Something here seems to not fit together to me on the assumptions this is based on, because you say

We're explicitly not claiming to defend against an attacker who fully owns your running application with valid credentials.

and dandraper says:

I was CTO [...] and the engineers all had access to patient data. They needed DB access [...] but not access to the sensitive values [...] CipherStash solves the following: * Hide sensitive data from direct database accessors

So putting these together is this implying that a developer who wants to test things on a large data set is given access to the database directly, but not access to debug the application? Surely for example if they could load the app in a debugger, step through it, and execute arbitrary code (eg. if they are developing on it) that includes "fully running the application"? So are you saying that scenario is not secure?

If so then I'm struggling to see how a developer would use this realistically. As a dev how am I supposed to work on an application without access to be able to change the code (assuming here then that "change the code" means I can also add a statement that dumps a decrypted value to the log)?

If the security level we want is something like "devs need to be able to test migrations work on the actual data" then why would I use this instead of just NOT giving direct db access and having an audited environment spin up a db copy and container and just output the logs of whether the migration worked, without access to query the db directly. If I do that then nobody needs direct db access right? But then why would I need CipherStash?

dandraper24• 3 days ago

This doesn’t have anything to do with being able to change the code. Devs can code and change things as they need. The database is also the same as it always is.

What CipherStash does is let you specify specific columns that you want to encrypt. The application (via our SDK), encrypts values for that field before it’s saved and optionally decrypts it again when it’s read. Decryption is performed when the user provides a valid credential (JWT).

This would allow you to give your devs full access to the production DB. You don’t have to give them permission to decrypt any or specific fields if they don’t need it.

So for your use case that would possibly save a lot of effort - no need to create a santized copy of the database (which in itself is fraught).

What James was referring to was the attacker threat model. Decryption happens in the application code. So an attacker who can steal encryption keys as well as be a valid JWT would be able to decrypt data for as long as the JWT is valid. That scenario is not something CipherStash can fully prevent.

What it does prevent is unauthorised access to data. Whether via the database directly or via the application.

What’s important here is that it’s the data itself that protected. If you move encrypted values from your prod DB to test or dev, the same rules apply. A user who can’t read “email” values that were encrypted in prod would still not be able to read those values if you moved them somewhere else.

Does that clarify things?

dandraper24• 3 days ago

Thanks for all the comments.

CipherStash isn't something that we just threw together on a weekend. We've spent years developing it. I've personally invested almost every waking moment for the last 6 years working on this (and the prior 2 preparing to quit my job so I could work on it full time).

I get that it seems like snake oil to some. But that's just because we don't do a good enough job of explaining what we've created. We're working on that - we're engineers, not marketers.

There have been prior attempts at making tech like this work, and maybe because those approaches were limited, its tainted how folks see encryption tech. We've studied many of the previous attempts and invested in the engineering required to overcome the problems.

Key management - we made a 2 party key system that is faster and has a better security model than AWS KMS alone. Not because we thought it was fun, not on a whim but because it was the only way to solve the real security challenges we faced when it comes to protecting data.

Searchable encryption - that is less secure than homomorphic encryption but FAST. Like under a few milliseconds for most queries (homomorphic would take literally minutes or worse). And a whole lot better than no encryption at all. It uses a key per value.

Encrypt query language - a framework that makes searchable encryption work in standard postgres. No exotic index schemes, no custom builds. Just SQL functions.

Typescript and Rust libraries to make this _almost_ drop-in for Prisma Next, supabase-js and Drizzle.

Identity integrations with Supabase, Clerk, Auth0, and Okta.

A postgres proxy that automatically encrypts and decrypts data for authorized requests - for when using the SDK isn't possible/convenient.

And an audit system that records every access and can send data to anything you can connect to AWS firehose or an S3 bucket.

This is real and IMHO is exceptional engineering...anyone know a great technical marketer?!

https://github.com/cipherstash/stack https://github.com/cipherstash/benches https://github.com/cipherstash/proxy https://github.com/cipherstash/encrypt-query-language

Searchable field-level encryption on Supabase with CipherStash

Related Stories