Beyond the SQLite single-writer limitation with concurrent writes

https://turso.tech/blog/beyond-the-single-writer-limitation-with-tursos-concurrent-writes

by syrusakbary • 10 months ago

80 70 10 months ago

fragmede4• 9 months ago

Can't use it locally (yet?) but it's definitely an interesting move in the space. For my personal projects lately I've been defaulting to sqlite in dev, and having a database wrapper layer to use something else in prod.

stronglikedan2• 9 months ago

Can't use it locally

I'm imagining some insane replication behind the scenes, where every write is happening concurrently on a different SQLite DB, and then merged together sequentially into some master DB.

glommer13• 9 months ago

why not? Turso is a fully local database, you can just download the shell and use it as you would use sqlite.

fragmede4• 9 months ago

More specifically, I didn't see any commits about multiple simultaneous writers having landed in main and it's not document as a feature in the release notes https://github.com/tursodatabase/turso/commits

neither the prerelease

https://github.com/tursodatabase/turso/releases/tag/v0.3.0-p...

nor their latest release.

https://github.com/tursodatabase/turso/releases/tag/v0.2.2

The core/mvcc directory does have commits though, so maybe you can.

https://github.com/tursodatabase/turso/commits/main/core/mvc...

frumplestlatz6• 9 months ago

I’m not in a rush to use a reimplementation of SQLite — particularly from startup bros that had a very public, one-sided, and unpleasant fight with SQLite over their contribution model.

D. Richard Hipp is a genuinely fantastic human being, and SQLite is a project developed at literally the level of planetary infrastructure given how broadly and everywhere it appears.

Forking his project while using the name to garner recognition and attention is poor form, and it is difficult to have faith in the results.

bitpush2• 9 months ago

I get where you're coming from, but isnt the whole idea of open source "if you dont like the approach, you're free to fork the code and do it the way you think is right?"

As long as the fork doesnt violate trademark (turso vs sqlite) it is working-as-intended?

I, for one, encourage this kind of behavior. We should have more forks. More forks = more competition = better results for everyone.

---

To make an analogy. Would you say the same thing if this were a for-profit company?

"I cant believe someone else is competing the same space as $x. $x is hugely successful, and so many people use it. I dont know why there's an alternative"

immibis2• 9 months ago

Both forking sqlite, and disliking the forking of sqlite, are allowed.

Plus, it's just technically bad. Most cases where you'd want to scale up sqlite are better served by a client/server database.

stronglikedan2• 9 months ago

Forking his project while using the name

The don't use the name. They use Turso. Even the HN title is wrong - the article title doesn't mention SQLite.

They refer to SQLite, but how could you not if that's what you forked from, and that's what has the functionality you're changing. That would be a very weird article if we didn't have that context.

frumplestlatz6• 9 months ago

They state, plainly on their home page:

“The next evolution of SQLite”

That is a material misrepresentation and absolutely trading on the SQLite name

Permik• 9 months ago

I'd still give them the benefit of the doubt as most (all?) of the contributors are Finns who you can almost guarantee to have a "no bullshit" type of mentality to essentially everything. And what I'd guess, the team most probably has quite an academic background because of the local culture.

glommer13• 9 months ago

Pekka is a Finn, and there are other Finns as well, but not all of us are Finns.

Our Finns are also not the standard Finns. I know Pekka closely for 15 years, and he smiled 3 times.

glommer13• 9 months ago

Author of Turso here. Couple of points

* You are right not to rush. You should keep using SQLite until Turso matures. Some use cases are more tolerant to new tech than others. It will take time for us to reach the level of trust SQLite has for broad use cases, but we are hoping to add value for some use cases right away. Never rush, tech matters!

* I have never met Hipp, but only heard great things about him.

* We never had a fight with SQLite over their contribution model (or about anything for that matter, I never even met Hipp or anybody else from SQLite). We just disagree with it - in the sense that we believe in different things. We don't think what they do is fundamentally wrong. Different projects take different paths.

* We are not using the SQLite name. We compare ourselves to SQLite because we are file and API compatible, and we do aspire to raise the very high bar they have set. It is hard to do this without drawing the comparison, but we are a different project and state it very clearly. I am not a lawyer (and neither you seem to be), but we believe we are doing is okay. If we ever have any valid reason to believe we crossed a line here, we will of course change course.

* We are not "startup bros". We spent 20+ years of our lives building databases and operating systems.

frumplestlatz6• 9 months ago

I wasn’t describing a legal dispute, and I haven’t made any claim about infringement. “Material misrepresentation” refers to the substance of the public messaging, not to a legal violation.

The issue I raised is that the phrasing and the way the fork has been presented create the impression of continuity and endorsement that doesn’t exist. That’s a reputational and ethical concern, not a legal debate. Calling it “the next evolution of SQLite” is, in practice, absolutely trading on the SQLite name.

There was a very public, one-sided disagreement about SQLite’s contribution model at the time, and you’ve been open about your criticisms of SQLite in the years since. That’s the context for my comment; it isn’t something I’ve imagined.

rednafi• 9 months ago

Awesome to see you chiming in here.

Just to set some context, I remember talking to Glauber and Pekka last year, right around when the Turso folks started toying with the idea of reimplementing SQLite in Rust.

It’s a moonshot, but if anyone can pull it off, it’s these people. If you don’t know their background, checkout ScyllaDB. It never ceases to amaze me how belligerent some HN folks can be - they won’t even take five minutes to do a Google search before calling someone a nobody or “SV tech bro.” Not that I’m saying you should do that to anyone.

hitekker2• 9 months ago

One ugly thing I remember was the insinuation that SQLite was somehow immoral. Glauber Costa attacked SQLite[1] with lines like:

We take our code of conduct seriously, and unlike SQLite, we do not substitute it with an unclear alternative. We strive to foster a community that values diversity, equity, and inclusion. We encourage others to speak up if they feel uncomfortable.

Sometime after the HN backlash[2], that paragraph was removed. Eventually, Turso released https://github.com/tursodatabase/turso without a code of conduct and hasn't had one since.

For me, the saddest part of the affair is that Richard Hipp and Glauber are both committed Christians. There should be no reason for envy, enmity, or memory games. But when business and money was on the table, it seems only one of them followed Jesus.

[1] https://github.com/tursodatabase/libsql/commit/3ac3ad263c0f0... https://web.archive.org/web/20221004144141/https://glauberco...

[2] https://news.ycombinator.com/item?id=33081159

glommer13• 9 months ago

What you are pointing to did happen. What you are forgetting to mention is that this was one phrase in a post that was otherwise singing their praises.

The charitable interpretation is that when people pointed this out, I have read it, and come to agree that the wording was too strong and not representative of what we wanted to convey. Therefore, it was taken down.

But why be charitable when one can just throw accusations around from the armchair towards people one barely knows ?

The one thing you are right about, is that I am failing to live up to the Lord. As much as I try, I keep falling short. It happens not only here but in all aspects of my life. If it wasn't for his grace I would be in Hell for sure.

tracker1• 9 months ago

Kind of cool to see work on this. I do hope that the final db file result is still binary compatible with SQLite 3 in whatever direction Turso moves towards though... Rust or not.

I've been advocating with several projects over recent years to get SQLite3 as an archive/export/interchange format for data. Need to archive 2019 data from the database, dump it into a SQLite db with roughly the same schema... Need to pass multiple CSVs worth of data dumps, use a SQLite file instead.

As a secondary, I wonder if it's possible to actively use a SQLite interface against a database file on S3, assuming a single server/instance is the actual active connection.

simonw2• 9 months ago

SQLite against S3 can work with some clever tricks. The neatest version of that I've seen is still this WebAssembly one: https://phiresky.github.io/blog/2021/hosting-sqlite-database...

I also got sqlite-s3vfs working from Python a few months ago: https://simonwillison.net/2025/Feb/7/sqlite-s3vfs/

Both of these are very much read-only mechanisms though.

glommer13• 9 months ago

simon, you will be pleased to know that the python package for turso is in good shape! (and if you do find any issues please scream at us)

huntaub• 9 months ago

As a secondary, I wonder if it's possible to actively use a SQLite interface against a database file on S3, assuming a single server/instance is the actual active connection.

You could achieve this today using one of the many adapters that turn S3 into a file system, without needing to wait for any SQLite buy in.

ncruces• 9 months ago

SQLite directly against S3 is workable if you mean querying a read-only database.

For example, from Go, you could use my driver, and point it to a database file stored in S3 using this: https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/readerv...

For read-write it's a terrible idea. Object storage assumes objects are immutable. There may be some support for appends, but modifying the middle of an object in place involves copying the entire thing.

What is on the verge of becoming viable is to use Litestream to do asynchronous replication to S3, and have read replicas that stream the data directly from S3. But what's stored in S3 isn't a database file, but a format created for the purpose called LTX.

jtarchie• 9 months ago

ncruces helped me with some code I made for VFS. It uses [zstd seekable](https://github.com/jtarchie/sqlitezstd) for reading a file. I thought it would be really well-suited for S3.

- Support for HTTP range queries - "Fast" read times - No disk required

I was wrong.

It turns out that for specific SQL queries, it might be fine, but not fast. For queries that do aggregations, like `COUNT`, sqlite loads the whole database anyway.

conradev3• 9 months ago

The single-writer limitation in SQLite is per-database, not per-connection. You can shard your SQLite tables into multiple database files and query across all of them from a single connection.

I agree that "the single-writer limitation isn't just a theoretical concern", but it's also solvable without forking SQLite. ulimit's the limit! If your goal is resource maximization of a given computer, though, Postgres is likely a better fit.

kgwxd• 9 months ago

Joins and Transactions are a pretty big part of SQL. I'm no expert, but if my quick search results are right, both are lost in the separate file per table scenario.

conradev3• 9 months ago

They work just fine!

Joins: https://simonwillison.net/2021/Feb/21/cross-database-queries...

Transactions: https://www.sqlite.org/atomiccommit.html#_multi_file_commit

avinassh• 9 months ago

You can shard your SQLite tables into multiple database files and query across all of them from a single connection.

You mean using ATTACH statement, right? If you use WAL mode, then you cannot get transaction safety / ACID with ATTACH [0]

If the main database is ":memory:" or if the journal_mode is WAL, then transactions continue to be atomic within each individual database file. But if the host computer crashes in the middle of a COMMIT where two or more database files are updated, some of those files might get the changes where others might not.

Moreover, ATTACH do not support more than 125 databases, so that limits the shards to 125. [1]

ATTACH does not solve the concurrency problems. That's why SQLite also has a BEGIN CONCURRENT experimental branch

[0] https://www.sqlite.org/lang_attach.html

[1] https://www.sqlite.org/limits.html

conradev3• 9 months ago

I would argue that BEGIN CONCURRENT doesn't solve the concurrency problems either, which is why SQLite also has an HCTree experimental branch :P

asfs4fsdfadf• 9 months ago

Can someone explain what "ecological niche" this new Turso DB occupies in between SQLite and Postgres?

SchwKatze• 9 months ago

TursoDB aims to be fully compatible with sqlite, so files you create with tursodb can be read by sqlite and vice-versa. Like sqlite, turso is in-process, so it runs alongside your application, which is quite different from postgres' client-server architecture.

The only think turso has in common with postgres is mvcc, which is a rather standard concurrency control model across modern databases. Idk if I answered your question :)

spiffytech• 9 months ago

Don't forget that the SQLite team is working on their own multi-writer mode that blows BEGIN CONCURRENT' out of the water: https://news.ycombinator.com/item?id=34434025

Though this stuff moves slowly (that announcement was almost 3 years ago!), so I'm glad to see Turso giving us options today.

simonw2• 9 months ago

Looks like that branch is still under actively development - seven new commits this month: https://sqlite.org/hctree/timeline

rvitorper• 9 months ago

Even in the “low-performing” zone of the single threaded SQLite and Turso, we are still talking about 50k rows per second and 1000 microseconds, aka 1 ms. It is insanely fast. 1ms is just the round-trip for my Postgres on RDS. It is amazing that SQLite is so awesome. I understand it is not for every use-case, but still awesome

crazygringo5• 9 months ago

While I understand that you could do this, I genuinely don't understand why.

SQLite and Postgres/MySQL/etc. occupy different niches. If you need massive concurrent writes, surely that's what Postgres/MySQL/etc. is for? Their engines are built around that from the ground up.

SQLite is built around a file that stores data for a single application, as opposed to being client-server with many clients. I've used it a ton for that, but the idea that your application would have so many threads needing to write concurrently that it would be slowed down by a file lock... just doesn't make sense to me.

What applications need this, and why wouldn't they use Postgres/MySQL/etc. if they need such a high level of concurrent performance? This feels like trying to adapt a small sports car to tow a semi-trailer. It doesn't seem like it's what it's meant for.

Quarrelsome3• 9 months ago

Agreed, Sqlite docs specifically state that its not designed for this, or rather that the solution isn't appropriate, that database as a service is appropriate when you want multiple clients writing.

The entire "problem" is a side-effect of using the wrong tool for the job.

glommer13• 9 months ago

Linux was designed to run in home PCs, and we keep running it in supercomputers. It works just fine. Tools evolve.

zeroq• 9 months ago

I'll give you an example of what I'm doing with SQLite.

I have an app that organizes my movie collection. I use IMDB public datasets as data source. Because versioning is hard, data changes over time (remember who directed Matrix?) it's just easier to rebuild the database every time.

A naive import of datasets is ~131M inserts. It takes 10 minutes (of which 100M takes exactly 6 minutes). Another 10 minutes to further clean up the data, set up indexes, optimize and vacuum.

It's fine for the use case, but still it would be nicer to be able to achieve that in half the time.

The app has no dependencies, database is a single file I can store on S3 and with a little bit of magic use it as a real database - meaning I can run a complex queries over http on that whole database and exchange 50kb.

The client is a standalone HTML file with some JS. No need of server for client, no need for hosting for server except of that S3 bucket.

Sure, it's definitely a niche application, but, at least for me, it makes perfect sense. Start your app with zero deps and add only what you really need. There's too much projects that start with google-esque architecture, with plenty of microservices, only to reach MVP with technical debt the size of US treasury.

CyberDildonics• 9 months ago

I don't understand what you are trying to say here, this doesn't seem to have anything to do with single writer limitations.

crazygringo5• 9 months ago

That's not concurrent writes. You would see zero speedup.

Your use case makes perfect sense for SQLite. It's a great example of what it's for. It won't benefit. That's my point.

grandfugue• 9 months ago

sqlite is embedded. I understand that there might be scenarios in which multi-threaded sqlite is beneficial when an application has many concurrent writers. But taking a look at the company's website makes me wonder what the project's motivation is. The company offers a database service, which is a completely different scenario from embedded dbs. If the intention is to offer a cloud service, evolving from "sqlite" seems odd. The only benefit I can think of is that the new db service helps existing sqlite users migrate. My issue is that if I choose sqlite to store data locally in my browser or my cell phone, why do I suddenly want to store it in the cloud?

Quarrelsome3• 9 months ago

My issue is that if I choose sqlite to store data locally in my browser or my cell phone, why do I suddenly want to store it in the cloud?

I don't think those solutions are necessarily that bad. There was one the other week that offered a means of guaranteed sync to a sqlite file in the cloud. Its nice to have the infra to allow users to hop around devices and have backups. What's weird to me, is trying to magic it into a performant multi-client db which the underlying technology was never designed to be.

solarkraft• 9 months ago

Why not take something that’s great and make it support more use cases? People are doing it the other way around too: https://pglite.dev/

nick_• 9 months ago

IMO this project kneecapped itself by being web-focused.

nextaccountic3• 9 months ago

SQLite is built around a file that stores data for a single application

A single application can need multiple concurrent writes

crazygringo5• 9 months ago

So that's what asking. When? Why? I've simply never encountered that, and can't imagine what a use case would be.

SQLite already supports lots of threads writing, they just all take turns. What application needs those writes to be concurrent?

jauntywundrkind• 9 months ago

Very excellent to see the embedded DB in the browser link at the bottom. https://shell.turso.tech/

I've been thinking Turso in the browser would be some eventual addition, is something the Rust rewrite would eventually unlock. And that would require considerable extra backing to make go. But here it is now!

glommer13• 9 months ago

It's very easy to do given the async model. It is one of the new features that is working the best so far.

adzm• 9 months ago

How does the mvcc / row versioning changes that accumulate get shared among multiple processes? Is it another file like the WAL file? Or is it just local memory as part of the client? I guess with everything having to commit or entirely roll back at the end, it doesn't need to be shared.

How does this compare with hctree then? It sounds similar, both using an MVCC model.

Really cool though, love the technical details and would be interested to see further. I feel like there is certainly a space for sqlite+fancier features. Though it would break backwards compatibility, I've always wondered what kind of performance gains could be had by using a different varint implementation. And enforcing or always using "strict" for smarter column layouts.

Also would be really cool to have indexed views.

And columnstore seems like a logical direction as well.

Beyond the SQLite single-writer limitation with concurrent writes

Related Stories