SQLite is all you need for durable workflows

https://obeli.sk/blog/sqlite-is-all-you-need-for-durable-workflows/

by tomasol • 2 months ago

720 387 2 months ago

obeli.sk

EGreg• 2 months ago

Files is all you need.

https://xkcd.com/378/

tclancy• 2 months ago

Post It Notes will do if you have a good system.

contingencies• 2 months ago

Those who don't understand Unix are condemned to reinvent it, poorly. - Henry Spencer .. via https://github.com/globalcitizen/taoup

jusonchan81• 2 months ago

Agree on this and it’s the architecture that backs our orchestration platform called https://unmeshed.io

orf3• 2 months ago

The caveat is that Litestream replication is asynchronous. A restore can miss the newest local writes if the SQLite volume disappears before they are copied. That is fine for many AI and experimentation workflows

In short: SQLite is not all you need, unless you’re just experimenting don’t actually care about durability, in which case you also need litestream + object storage.

Right.

bootsmann• 2 months ago

S3 is strongly consistent, if you need it anyways you can just use s3 keys to deconflict and store the workflow state.

orf3• 2 months ago

Yes, but directly using s3 as a key-value database is completely different from using SQLite + litestream.

gwking• 2 months ago

The suitability of Litestream for production disaster recovery is also an open question in my mind. I used 0.3.x for several years and when I tried to upgrade to the 0.5.x series there were runaway disk usage problems that would have caused downtime had they made it to prod. As far as I can tell these have not been entirely addressed, although recent bug reports suggest that they might be getting closer.

I want to love it, and I don't take open source projects like this for granted. But during my last production upgrade I chose to decommission Litestream in favor of a dumber, less granular solution using sqlite3_rsync and nightly backups because there is no point in using a backup system that is not rock solid.

0cf8612b2e1e3• 2 months ago

Postgres also does not synchronously replicate for free. You can setup both to get a confirmation write if you require that durability.

orf3• 2 months ago

postgresql also does not synchronously replicate

By default. Generally your primary database is in a completely different failure category than a kubernetes node running an ephemeral workflow pod.

paulddraper2• 2 months ago

Not for free, but without the needing additional software.

  synchronous_commit = on

paulddraper2• 2 months ago

"Durable workflows without the durability"

That's distributed workflows :)

dilyevsky• 2 months ago

i mean it's durable as long as nothing crashes or litestream has a data corruption bug which only happens every other release...

faangguyindia4• 2 months ago

try setting up replication/failover in postgres, it's much more work.

Xcelerate• 2 months ago

Haha, I just started doing this on my own. Found it helps the agents preserve state better. I typically ask them to design a DAG first based on a set of specifications and then execute it (each step stores something in a SQLite DB). Iteration is pretty simple then because I just ask for a tweak to one or two steps of the DAG, and then to re-run.

Funny how people are independently converging on similar patterns of "what works" here. Still feels like we're in the wild west with all these ad-hoc patterns of agent orchestration that people are coming up with.

zrail• 2 months ago

Same. The prompt was essentially, every checkbox in this PLAN.md should be task in SQLite.

sgloutnikov• 2 months ago

It's close enough that DBOS does support SQLite. [0] The default for prototyping is SQLite, but sure you can run it in production if you wanted.

Obligatory list of workflow engines and libraries because it's such a common need that a lot have rolled their own. [1]

[0] https://docs.dbos.dev/python/tutorials/database-connection

[1] https://github.com/meirwah/awesome-workflow-engines

oulipo22• 2 months ago

What would be the main differences between DBOS and Obelisk?

tomasol4• 2 months ago

Hi, I wrote a comparison blog post between Obelisk and the Java version of DBOS a couple of months ago: https://obeli.sk/blog/comparing-dbos-part-1/

bitexploder8• 2 months ago

I started setting up my workflows using Temporal. It deploys as relatively light weight local app. For an isolated local installation it uses SQLite. It makes the process of dealing with API retries and organizing workflows and tasks really simple. I recommend giving it a try. It is, philosophically, exactly what this article is suggesting, but it adds an incredibly rich and flexible interface for agents to work with. Additionally, the web UI makes it very easy to inspect workflows, review agent execution, etc. Temporal also encodes much higher reliability into your system, almost for free. Distributed and reliable systems are hard, don't reinvent the wheel IMO.

If you find yourself wanting things like an easy way to then introspect your SQLite database, figure out what is happening in the workflow, compose individual tasks, make workflows trivially callable, etc, give Temporal a look.

Alongside this, I have mostly moved away from files for agents. Markdown and JSON are great, but also feel like traps when building out smaller local apps. LLMs are great at SQLite and you can render anything you want out of it (Markdown, JSON, etc). It saves a lot of tokens when an agent can just query a specific row instead of having to fire up jq or grep through markdown. You get a nice portable self contained data management system that encourages agents to be more disciplined about how they structure their data than a bunch of files. It also continues to scale into MySQL/Postgres if your little local projects start to outgrow or become more formal, you already have schema and discipline around data.

jawns• 2 months ago

Could you give an example of a case where you'd use SQLite instead of jq or grep through Markdown?

fragmede4• 2 months ago

Honest answer is: whenever your markdown or json files get to be big enough that grep/jq takes long enough that you get bored waiting for it.

chaps• 2 months ago

The moment my JSON has any sort of depth and I need to write a parser for it and potentially account for unspecified behavior. JSON's nice when it's nice, but it's terrible when it's terrible. It's 100x easier to write SQL than writing jq and... dear god if I have to use grep -A or -B, I'm doing something wrong. Constraints are actually a good thing!

The underlying database isn't the most important thing. Just use SQL. Its namespacing (eg, through CTEs) is good and you're more likely to have colleagues who know SQL compared to jq.

phamilton• 2 months ago

My favorite lens on SQLite is that it is actually two things:

1. A robust durability implementation 2. A library of high performance data structure and algorithms

The fact this it's SQL is nice, but those two attributes are what make it great.

For example, I'm implement an in-process event log that I want to be durable. I started simple, but soon saw some edge cases and instead of playing whackamole I just swapped to using sqlite as an ordered kv store that gives me ACID.

Another example: ingesting multiple inter related datasets. Instead of a dozen hash maps in memory, I load them up into sqlite (no persistence) and then slice and dice as I need to.

It's a super useful tool.

gopalv• 2 months ago

an example of a case where you'd use SQLite instead of jq or grep through Markdown?

Usually we end up writing a script to incrementally refresh a data-set I'm analyzing (or have someone send me a copy after they pull it).

I've been using sqlite for anything which needs an UPDATE - modifying a row deep inside the data-set with jsonl is a pain.

My github is full of java programs which update sqlite3 files with threadpools and a single big lock around the UPDATE (& then I write or have an agent write code to analyze it).

DuckDB is slowly replacing it in the context of python, simply because of the ease of pushing a UDF into the SQL.

Also because I really like expressing things as LEAD/LAG with a UDF on top.

pokstad• 2 months ago

SQLite is more efficient for large data sets. A single markdown or JSON file needs to be streamed to locate a piece of data O(n). Updating an existing entry in a sequential file is even worse because you have to rewrite the file. SQLite has the data structures to quickly find data in O(log n) time.

peterson_lock• 2 months ago

This reads like an advertisement for Temporal :)

switchbak4• 2 months ago

I'm someone else who has inherited a bunch of ad-hoc orchestration systems and also used Temporal quite heavily. The latter does certainly come with some overhead (not so bad in the age of LLMs), but it also guides you along a well-trodden path of good practices. The latter being very important - it means that when you want to take on more advanced capabilities, you probably haven't painted yourself into a corner too badly and can take that on fairly easily. Think: retries, multi-tenancy, multi-lang, observability, etc.

pzduniak• 2 months ago

I can vouch for them too, being a super early adopter. One of the best early bets I've ever made. Awesome OSS product, glad the team decided to leave Uber to commercialize it.

baq• 2 months ago

Low key amazing tech, kinda like clickhouse - nobody is bragging it’s running their business

bitexploder8• 2 months ago

Well, just my experience. I installed it, had my agents configure it and it immediately solved problems I had with very little friction. Dealing with long running, long horizon agentic tasks that need very high reliability so I don’t have to babysit. I vibed the first version, realized I was reinventing reliable distributed systems. Stopped vibing and started surveying for something that fit :)

hedgehog• 2 months ago

It does, my experience has been that it adds code complexity, deployment complexity, and performance problems. There are some observability benefits, but other ways to solve that. It's possible there are workloads that fit it but not anything I've personally worked on.

dust42• 2 months ago

I just spent the last two weeks digging into workflow state engines and temporal was one of the candidates. It is a VC backed fork of Cadence. The got 0.3B funding and whatever positive I read about them on the net I take with a big spoon of salt. Just my 2 cents.

rick1290• 2 months ago

Interesting about the files vs db approach. I have been going back and fourth. I landed on db as well.

svara• 2 months ago

Word on HN is that you're either paying more money than you expected for temporal's managed solution or taking on substantial ops burden ultimately running their very heavy system yourself.

I wouldn't know, I've not done either, but I'd like to learn more from your or other's experience.

edumucelli• 2 months ago

Very heavy indeed, people will confuse the durability that Temporal provide with all the other properties a distributed system needs. They will then think that Temporal will solve all their problems.

bitexploder8• 2 months ago

I told an agent to set it up for me for some local stuff. It is written in Go. It has a painless path to run on a local SQLite DB. My agents use it to organize and coordinate workflows. It handles retries and long horizon tasks fine. As far as I can tell for the core workflows and tasks pieces it’s great. MIT license. Like anything it isn’t free to manage but it offers a lot in return. High reliability systems are hard. Temporal only solves some of it. It is far better than rolling it yourself.

I think a genuine problem right now is people are building agentic work flows and learning the hard way highly reliable agentic work flows are hard. Agents are unreliable. They are both not deterministic and not the backing APIs have pretty high error rates. Temporal has solved that pain for me and made it easy to diagnose problems.

I don’t have anything really large scale running. But big enough that it takes billions of tokens and high reliability to finish.

parthdesai• 2 months ago

use oban and call it a day: https://oban.pro/

kenforthewin• 2 months ago

Could you expand on the "substantial ops burden"? Let's say you're using a managed Postgres instance as the underlying data store, how substantial is the ops burden in that case? I understand that temporal is actually a set of 4 or so microservices on top of a data store, but if you're already running a distributed system backed by k8s or something like that, it doesn't seem like it adds significant incremental ops on top of that. But I could be wrong.

dtech• 2 months ago

Their managed solution is pricey and especially the linear scaling with how much you use it is very meh. It's comparable with AWS lambda which also isn't cheap. However it's minor on a typical cloud bill.

Self-hosting is very easy in my experience, I've done it for 2 years but management wanted to move to Temporal Cloud. They have a helm chart which just works including upgrades. This does assume you have the whole k8s shebang set up and working in your company. I never had to touch is outside upgrades which took maybe 30m including validation.

fcarraldo• 2 months ago

It sounds like you’re running this mostly on a single machine? Temporal gets much more complex with scale. Cassandra isn’t fun to manage. Ringpop and TChannel are hard to debug when things go wrong. The SQL backend support doesn’t support horizontally scaled replicas (just single instance) due to consistency requirements. Depending on how your code is written, modifying code baked into workflows becomes complex, as anything that modifies the history event ordering breaks determinism in already-deployed workers.

We use it heavily and everyone who started on it doing simple scripting/automation all love it, everyone who built real production systems on top of it all hate it. Possibly operator error, but my experience hasn’t matched the rosy picture painted in these comments.

bitexploder8• 2 months ago

Yep. Individual systems with yolo agents doing stuff in isolation. I could see how it can get complex. Most distributed systems are. No free lunches I guess. I am not sure what the alternatives are at scale.

kumulo• 2 months ago

Temporal feels massive, I tried it for a small workflow embedded on my system, and worked fine, but when thinking on scaling it, it just didn't made any sense for my use case.

I also have restate.dev on my reseearch list, which on paper should scale well and be definitely more lightweight and simple to setup, worth having a look.

inglor3• 2 months ago

In the last two years, we built (with a team of 15, now 100) a billion dollar business on top of Temporal that performs business critical applications for fortune 500 companies. We couldn't be happier with temporal.

Determinism sucks, you do have to work hard and make everything idempotent in activities like we would for durable software anyway. The language we used was incorrect (Go) and has a lot of boilerplate compared to alternatives we later investigated (Python and TypeScript). Visibility can be slow and misses information. We needed to write our own APIs to work effectively with Agents for root-cause analysis of failures.

With all the caveats - Temporal is amazing, it feels much better than previous orchestrators I used like Prefect or Airflow. 100% would adopt again.

taberiand2• 2 months ago

Depending on how your code is written, modifying code baked into workflows becomes complex, as anything that modifies the history event ordering breaks determinism in already-deployed workers.

I see this as Temporal surfacing inherent complexity of the domain in a way that forces the developer to consider it, rather than introducing extra complexity.

If it didn't make workflow determinism a strict requirement, the requirement would still exist - it would just hurt much worse in production when it's broken.

See also: Rust borrowing

chrisss395• 2 months ago

"gets much more complex with scale" feels like the crux of it. Pick the right solution for your intended scale

That said, I appreciate this is hard in practice. We need to start small to manage the development rabbit hole risk, while also wanting to dream big. There is a tension there that I find hard to balance.

cnlwsu• 2 months ago

We are a huge production setup where it’s absolutely critical successfully but we use temporal cloud. Hosting it yourself is what makes it miserable. https://temporal.io/resources/on-demand/netflix

linkregister• 2 months ago

Did I read your comment correctly that Temporal includes Ringpop in their product? Interesting choice to say the least.

bambax4• 2 months ago

My current client has a forest of 90+ SnapLogic pipelines that were badly written and maintained even worse; one of those was completely wrong, in that it generated wrong accounting data which could eventually have financial, fiducial and legal repercussions.

I rewrote the pipeline in Python (a correct version of it) with state management in SQLite and logs in plain old flat files, and everything has been running smoothly ever since. In fact this is the only data flow that has worked without errors or interruptions in the last six months.

Instead of replicating the db file with Litestream I do a remote backup with Restic before and after each run; it's not an exact replacement of Litestream as we could possibly lose a whole run if the machine died / disappeared at the end of a run, but it lets one restore any day very easily. In an ideal world I think we should have both (live replica + backups).

password43217• 2 months ago

It saves a lot of tokens when an agent can just query a specific row instead of having to fire up jq or grep through markdown

Just wanted to make sure no one missed this point in your comment because eventually users will be paying the full cost for tokens instead of VC's paying, with GitHub Copilot's pricing realignment leading the way.

spwa4• 2 months ago

What's wrong with looking up the algorithm called "work stealing" and just implementing it? Making web UIs for a working algorithm is something AI is really, really good at (and I'll actually trust it with that)

kubik369• 2 months ago

Meta comment: This is a domain under my countries TLD (Slovakia) and it is one of the handful of words that are a word with the TLD in my language (and coincidentally) also in English. Every now and then, I will check on the domains with a retrograde dictionary for domains that have this property and root of this particular domain had a roundcube email server on it (can be checked on archive.org). After further checking, the local company actually named themselves Obeli s.r.o. (s.r.o. is Ltd), presumably so that they could use a domain that is a real word when said together with the TLD. (EDIT:) Forgot to write the thing I wanted to mention in the first place: it appears the domain must have lapsed and/or the author bought it from the company that was using it.

Another fascinating fact: our countries TLD has been stolen Ocean's 11 style (I am not kidding). After Czechoslovakia split into Czech Republic and Slovak Republic, the newly created Slovak .sk TLD has been under the care of people from the local university. The university also had some offices that they were leasing out. Someone had leased this office space (EDIT: this is important as this means they had the same physical address), created a company that had the same name as the NGO that was taking care of the domain, so e.g. the NGO was named "My Company o.z." and the perpetrator created a "My Company s.r.o." (our countries version of the american Ltd). This person then wrote to ICANN to change the address to the "My Company s.r.o." presumably under the pretense that this was just an administrative error and from this point, they have functionally taken custody of the TLD. I was not able to find how they did it technically, but I presume they persuaded ICANN to then point to their servers instead of the real ones. After this happened, it seems that no one noticed for some time. When they noticed, they tried taking it back, but they weren't able to. For some inexplicable reason, the government during that time (Šuster era, early 2000s) gave the new company a contract that was functionally uncancellable from the government side. Later governments made this even more uncancellable and in 2017, then Minister of IT (and as of this day president!) Pellegrini made the contract literally uncancellable. As a result of this, we have one of the most expensive domains around (18e/year, rising each year for no good reason). (EDIT:) The company running our countries TLD is now a foreign entity that the whole thing has been sold to (multiple owners over time) and we as a country have no control over if I understand it correctly.

I might have gotten some details wrong as I am writing this from my memory of researching it a couple of years back, but you get the idea, crazy stuff. Here is an article in Czech [0] that tells the story a bit better, but you have to translate it.

[0] https://www.root.cz/clanky/pribeh-domeny-sk-aneb-kradez-za-b...

// EDIT: I have found that the article actually links the movement to return the TLD back [1]. It also has a story tab [2], so they have something much more precise than the paraphrasing I wrote.

[1] https://www.nasadomena.sk/

[2] https://www.nasadomena.sk/historia/

ymolodtsov• 2 months ago

That's a crazy story. National TLD is a weird business from the beginning.

shukantpal2• 2 months ago

SQLite is surprisingly performant for single node applications even when comparing to Postgres. Postgres consumes a lot more memory and requires IO to hop through IPC whereas you can keep everything in process in SQLite with a shared connection pool.

I've been testing different storage engines for my agent harness and I can get up to 7.5k concurrent sessions on a single vCPU with SQLite whereas Postgres crashes or runs out connections.

[0] https://github.com/impalasys/talon/pull/23#issuecomment-4577...

onlyrealcuzzo4• 2 months ago

SQLite is surprisingly performant for single node applications even when comparing to Postgres.

In the context of SQLite being understood to be a quite excellent piece of software - shouldn't we expect it to be?

In the context of a single-node, Postgres is overkill. It should not be expected to be competitive with SQLite.

This is almost like benchmarking an in-memory HashMap to Redis and being surprised that it performs well in ideal conditions.

shukantpal2• 2 months ago

Yes, agreed on SQLite/Postgres. But I'm going to benchmark RocksDB next and see what the performance characteristics are. I suspect the LSM tree storage engine of RocksDB might perform better since agents are so write heavy when running highly concurrent workloads. After all, you are streaming LLM tokens into disk and fanning them out to subscribed clients.

bob10292• 2 months ago

When used properly, SQLite is effectively an in-process method invoke. If the only remaining things in the way are your runtime, kernel, file system and a local NVMe storage device, you may find it massively outperforms hosted alternatives.

Leaving the current thread is where you lose the game in terms of latency. SQLite can work on timescales measured in microseconds if you don't force interthread communication.

themafia• 2 months ago

an in-process method invoke.

Pedantically it's an in process virtual machine for operating on structured data. Which is precisely where it shows it's weakness, in my experience, when you end up with complicated table structures and complex join mechanics you then need to start thinking ahead of the query planner and VM code a bit in order to maintain reasonable performance.

There are more than a few unusual things worth knowing:

https://sqlite.org/optoverview.html

levkk• 2 months ago

I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time.

This is a foundational principle of computer science. It seems to me that the "SQLite for everything" crowd is a little bit inexperienced.

teaearlgraycold2• 2 months ago

Well if you run a tiny single-threaded app then SQLite is a nice simplification over spinning up a separate machine for Postgres.

ai_fry_ur_brain3• 2 months ago

I use postgres for very simple apps. I have a Dockerfile I use in my boilerplate repo. It takes a single make cmd for me to build, start and run migrations. Its as simple as using sqlite.

eterm3• 2 months ago

Or you can run postgres on the same machine as the application, which lets you much more easily migrate if the time comes when you need to scale to multiple application servers.

There's a world between "local file" and "network DB server", running a DB server locally has lots of benefits from being able to easily query from outside if needed to forcing you to consider concurrency without the latency overhead of a network hop.

onlyrealcuzzo4• 2 months ago

It's almost as if Postgres isn't perfect, and one size shoe doesn't fit all.

Some people want some of the benefits you get from SQLite.

SQLite is obviously not perfect, but it's an incredible piece of software, and people regularly find good ways to make use of an excellent pieces of software.

refulgentis5• 2 months ago

I absolutely 100% do not understand it either. At all. Every time I try to over the last year or two I come away with the conclusion its something that sounds cool (to me too!) but is guaranteed to cause more problems than more obvious solutions.

That being said I'd kill for someone who used it and benefited to explain it to me in a practical sense. (specifically where syncing is involved, and syncing a subset of the SQLite is necessary. If it's "just" a document store thats treated like a blob for syncing/backup, that's familiar. If it's all in one storage but only local, that's familiar.)

Re: TFA, I guess it would have helped if I knew what Obelisk was, which is on me, and a more in-depth explanation of how this ties into AI/agents, which is on the industry/writer.

wat100003• 2 months ago

It's very likely that you have multiple SQLite databases in your pocket right now. It's one of the most widely deployed pieces of software on the planet. If your conclusion is that it's guaranteed to cause more problems than other solutions, then that's on you.

fragmede4• 2 months ago

So teach them. If you want to bring up computer science fundamentals, the question is where does SQLite sit with regards to the CAP theorem. Consistency, Availability, and Partition tolerance. SQLite isn't a distributed system, so there are no partitions to tolerate, so it's a CA system. Other databases make different tradeoffs. For systems that don't need concurrent writes, SQLite is pretty great! There are no users to manage, no permissions, no daemon to run, no server and port to mix up. Just open a file on disk using a library.

refulgentis5• 2 months ago

Strawman, no? "run an Obelisk server with a SQLite database", now we're distributed.

SQLite is a nice local store. It's this server stuff that I don’t grok, well, yet. :)

sevenzero3• 2 months ago

Isn't concurrency also limited by your machines disk speed for writes, what difference does it make if you write sequentially vs concurrently? Why does concurrency even matter for databases?

refulgentis5• 2 months ago

Isn't concurrency also limited by your machines disk speed for writes

Yes, in theory: given a large enough database, and a disk that can only do one operation at a time, and a large enough operation that touches enough of the database. In practice, in a SQLite single tenant scenario? No, not at all.

what difference does it make if you write sequentially vs concurrently. Why does concurrency even matter for databases?

As soon as your codebase involves reacting to events independently of a user taking action it becomes a practical concern. Generally, this is a broad question and has 1,000,000 answers.

EDIT: Originally I had "I think you understand generally, no?" appended but realized that's not helpful at all, if you did, you wouldn't be asking.

Something that may help is imagining what'd happen if a DB wasn't thread safe / didn't allow multiple writers. Ex. in SQLite's case, it allows multiple write operations to take place but there's a one-at-a-time queue. If we didn't have databases that were able to execute multiple writes simultaneously, you'd need a separate database for each concurrent writer you expect, and you'd effectively have a global lock. Orderly scaling would be ~impossible unless you did something crazy like have a single server per user

malisper• 2 months ago

Isn't concurrency also limited by your machines disk speed for writes, what difference does it make if you write sequentially vs concurrently? Why does concurrency even matter for databases?

For a simplified example, having three processes reading blocks X, Y, Z in parallel is much faster than having a single process read block X, wait for the read to finish, read block Y, wait for the read to finish, read block Z and wait for the read to finish.

doctorpangloss2• 2 months ago

sqlite is more like a file format than a database. it competes with .xlsx.

"SQLite for everything" crowd is a little bit inexperienced.

every time i see it in a real application, it becomes a huge focus of issues (for example: jellyfin, hermes, openwebui, comfyui)

fragmede4• 2 months ago

What kind of issues commonly arise?

O3marchnative• 2 months ago

This is a foundational principle of computer science

How exactly is this a foundational principle of computer science?

jph00• 2 months ago

You seem to have a rather limited understanding of what kinds of concurrency exist and how those needs are best met. Whether something is a server or not is not very relevant to this discussion.

SQLite is an excellent production db for many real world workloads, as has been widely documented. It is very different to Postgres, so requires learning a whole new skill set.

One way to think about it is that SQLite can work well for the parts of your system where there is naturally strong partitioning.

tasuki2• 2 months ago

SQLite can work well for the parts of your system where there is naturally strong partitioning.

Or the parts of your system that don't have big data and no need for massively concurrent writes. And that's the vast majority of systems!

leowoo91• 2 months ago

What additional skill set do you need to "learn" for SQLite? Copying files around?

bborud• 2 months ago

Computer science no more get its hands dirty with concrete software than physics primarily being about building bridges.

It is not «a foundational principle of computer science».

lanstin2• 2 months ago

I had very good results giving 1 SQL DB per go routine, so the accesses were serialized up front, on a very high volume (130K requests/second) service. Exact transactionality was not a product goal, and the SQLite was just to backup the in memory state. If we lost a little due to abend or something, that was ok (although for normal maintenance it caught SIGTERM and stopped the listen and then waited for in flight calls and then flushed the remaining changes to SQLite; then on startup it would read the SQLite into memory to populate before taking the listen; persistent storage across container runs, and never both reads and writes to the same file at the same time. (It also just closed the DB and opened a new one when it hit some limit of rows, so as not to fill the disk; the max size of the SQLite corresponded to the max size of the LRU map being served from in memory; then it just flipped A / B between "a full memory worth of data stored" and "the currently updating state." A lot easier than having to write out proto bufs to disk or whatever I would have done for transient (during restarts/maintenance) persistence.

petcat3• 2 months ago

Woof. That sounds very complicated. If you need that kind of write concurrency, use an unlogged table in postgres [0]. Then you don't have to invent a whole sharded thing yourself.

[0] https://www.postgresql.org/docs/current/sql-createtable.html...

peterspath• 2 months ago

That’s why there are billions of SQLite databases right?

SQLite is likely used more than all other database engines combined. Billions and billions of copies of SQLite exist in the wild. SQLite is found in:

Every Android device Every iPhone and iOS device Every Mac Every Windows 10/11 installation Every Firefox, Chrome, and Safari web browser Every instance of Skype Every instance of iTunes Every Dropbox client Every TurboTax and QuickBooks PHP and Python Most television sets and set-top cable boxes Most automotive multimedia systems Countless millions of other applications

https://sqlite.org/mostdeployed.html

mr_toad• 2 months ago

That’s a comprehensive list of single user devices.

pibaker• 2 months ago

GP calls out concurrency as a weakness of SQLite. Most of the examples here don't experience the same load even a moderately sized web service experience day to day.

And no, being a part of the python standard library doesn't means it is being used by the average python user. These days I'd say at least half of them are just there for machine learning.

ksd482• 2 months ago

levkk is talking about concurrency. The list you gave doesn't explain high concurrency requirements for usage.

petcat3• 2 months ago

sqlite is great for the contacts app on your phone, but that's it.

Hipp even said that it is not a replacement for a real multi-user, concurrent RDMS. Its primary competitor is "fsync".

slashdave3• 2 months ago

I mean... if you count flat files as "databases", there are a heck of a lot more

abtinf• 2 months ago

There are many cases where SQLite + concurrent front end (like a go net/http server) can handle all the load that a service might ever conceivably have to handle, especially if allowed to scale up hardware over time. You can trivially scale up SQLite to, what, hundreds of thousands of tps?

The only thing you really give up is HA/failover and DR. But there are solutions to deal with those. And single-server systems are generally surprisingly robust (since, in the absence of very complex control planes, uptime goes down with more systems).

kellogah3• 2 months ago

I was thinking of using SQLite on top of k3s/Longhorn to replicate it. Anyone do something similar? Folks mention light steam and aws but Jeff Bezos’s biceps are too much for me to handle.

ummonk2• 2 months ago

Why go through the trouble of shoehorning SQLite into a cloud database by getting solutions for HA/failover and DR, when you can just use Postgres off the shelf?

cortesoft3• 2 months ago

You also have the issue or normal maintenance (patching, OS upgrades, etc). You can’t do those without downtime if you are using SQLite.

BoredPositron• 2 months ago

I worked on an app that had sqlite databases per user... it was fine.

rpdillon3• 2 months ago

Sqlite is good for lots of stuff, but you're probably focusing your days on high-scale webapps that want sharding with networked DBs. That's one domain, and an interesting one, but there are lots of others.

I'm a big fan of re-evaluating prior "best practices" in light of technology changes, especially in ways that improve simplicity. Running my family's social media site off a single sqlite DB on a VPS is great. ~15 users, almost zero maintenance. I run my FreshRSS instance off of sqlite, as well as my "now" page. At work, I used sqlite for all kinds of things over the past decades: as an ad hoc job queue, as a quick way to ingest and query lots of logs locally, and present/filter in realtime with simonw's excellent https://github.com/simonw/datasette.

I don't think it's every "sqlite for everything" as much as it is "sqlite in lots of places you probably didn't think to apply it."

kentonv/Cloudflare's work on sqlite at the edge might have made this thinking a bit more popular, but it was always around. https://blog.cloudflare.com/sqlite-in-durable-objects/

I suspect being aware of all those little neat cases and wanting to leverage sqlite for them may be an indicator of experience, rather than the opposite.

droidjj• 2 months ago

Running my family's social media site off a single sqlite DB on a VPS is great. ~15 users, almost zero maintenance.

Details, please!

switchbak4• 2 months ago

I mean - I agree for the typical multi-user, SaaS webapp. But I don't think that's what these folks are proposing. If they are - yeesh, count me out.

If on the other hand they're talking about single-user, software in the small - hell yeah. In fact, I'd also promote DuckDB in this regard (mostly for analytics) - with the power of a single machine these days, you can do a surprising amount and never have to worry about distribution. Unless you know you'll have to, in which case you're probably just digging yourself a hole?

lunar_mycroft2• 2 months ago

The typical multi-user SaaS webapp doesn't have anywhere near enough users to overwhelm a single SQLite instance. Of the few that do succeed to the point where that's no longer true, a significant fraction can use techniques like sharding to stretch SQLite further.

nitwit005• 2 months ago

The reason the parent post is complaining that it doesn't make sense, is because people have indeed pushed the idea of using SQLite as an alternative for web apps like that.

pstuart• 2 months ago

Sure, SQLite doesn't solve every problem -- but in many cases it solves the need at hand with the reward of one less piece of infra required to support it.

I see obsessions with tooling/solutions constantly from experienced devs who fall in love with the original solution and think it's the only way to do things -- so the experience part cuts both ways.

MagicMoonlight• 2 months ago

How many production apps do you think have enough users to justify these huge DB servers?

mxey6• 2 months ago

Huge?

bastardoperator• 2 months ago

Are you one of my enterprise customers? What if your workload does not require write concurrency?

dboreham• 2 months ago

And of course there are now several responses proving your point.

faizshah• 2 months ago

Scale to zero is very useful.

emehex2• 2 months ago

I think you'd be surprised to learn how many real production apps are actually running on top of SQLite (by way of Cloudflare D1).

therealdrag04• 2 months ago

Many DB servers are built upon embedded DB primitives (like RocksDB), that doesn’t mean the primitives are sufficient on their own.

andersmurphy14• 2 months ago

Thing is SQLite scales better than both those network databases [1] if you're prepared to stick with one big machine (+ a standby).

This is even more obvious when you start doing transactions processing an row locks across the network limit you to 1-3k TPS that you cannot scale out of (Pareto distribution is merciless).

[1] https://andersmurphy.com/2025/12/02/100000-tps-over-a-billio...

lll-o-lll• 2 months ago

Seeing as I can get about 200K TPS from a networked DB in my environment, I have to question your setup here.

In the real world we are looking at things like RPO (recovery point objective) and RTO (recovery time objective). You need to consider HA and DR. It’s in these areas where SQLite does not scale.

That’s why I struggle to see the fit for SQLite in any sort of multi-user server environment. If you need the data to be durable, then the bigger DB’s have the tools. If you don’t need the data to be durable, just keep it in memory. I’m sure there are niches I am missing.

slashdave3• 2 months ago

You know you can host a database like Postgres on the same machine, right?

evdubs• 2 months ago

Wow, what an apples and aliens comparison. You add a bunch of transaction delays to your postgresql case because you can access a database over a network, but you use transaction batching for sqlite? Maybe just compare a local postgresql with/without batching to a local sqlite with/without batching to be much less misleading.

wat100003• 2 months ago

Someone with experience would know that concurrency isn't a universal requirement.

jpollock• 2 months ago

If your data is naturally sharded (users) with writes happening within a single shard, parallelism becomes easy. The request is routed to the shard hosting the user's data and reads/writes locally.

This makes scalability _much_ easier to reason about. It's cut-paste, cut-paste. Every N users needs another shard.

It does buy you a _different_ set of problems, like cross-shard querying (analytics) and how to do load leveling as users age out.

But it avoids the whole shared index scaling problems from inserts/updates with large user counts.

It becomes a hierarchical instead of a relational database.

rsalus3• 2 months ago

there is a difference between concurrency in a distributed environment and concurrency on a single machine across processes. SQLite is incredibly useful for the latter.

you seem like the inexperienced one to me..

zaptheimpaler2• 2 months ago

SQLite does not support concurrent writes at all (on a single machine), a single writer process locks the entire database.

slashdave3• 2 months ago

you seem like the inexperienced one to me

There is irony here

apatheticonion2• 2 months ago

For me, I have a use case that needs to support a few thousand users, probably a few hundred concurrently.

The combination of SQLite (libsql, a concurrent implementation of sqlite) and Rust means I can do so from a $2/m VPS and a single server instance.

Backups are done via a cron job that uploads to S3.

Does it pass the "Netflix scale" test? No

But it doesn't need to. I'm not profiting from the service and SQLite offers a path to scale if/when ready because... well it's just SQL and I can literally just swap `libsql::Connection` with `psql::Connection` in my repositories.

Plus upgrading from a $2/m VPS to a $10/month VPS quadripples the number of concurrent users I can support.

IMO, you can vertically scale extraordinary far with SQlite and an efficient server implementation.

I'd wager that 90% of forum websites, wordpress sites and online shops would be fine with SQLite.

Rohansi2• 2 months ago

The combination of SQLite (libsql, a concurrent implementation of sqlite) and Rust means I can do so from a $2/m VPS and a single server instance.

You can probably do it with regular SQLite, too. Being limited to a single writer isn't as devastating as it sounds when they get processed very quickly. Probably don't need Rust either but it'll be more efficient than the usual choices.

(Also, it looks like libsql is the same as SQLite? Only Turso has concurrent writes)

andersmurphy14• 2 months ago

Why do you need libsql? Single writer tends to scale better than concurent writes.

jrockway• 2 months ago

There is something appealing about "it's just a file" (it really isn't; it has locks and a WAL), but I agree with you.

I think people are afraid to read the documentation for postgres. You can start it up in milliseconds. Fast enough and light enough to run one copy for every test case in your test suite, or whatever you're using it for. (mkdir /tmp/whatever; initdb -D /tmp/whatever --no-instructions -A reject -c listen_addresses= --auth-local=trust --no-sync -c fsync=off -c unix_socket_directories=/tmp/whatever -U postgres --no-locale; postgres -D /tmp/whatever) Now you have a test database that behaves exactly like production because it's exactly like production. (OK, turning fsync off makes it a lot faster than production, so be careful.)

tucnak• 2 months ago

I think people are afraid to read the documentation for postgres.

Postgres may introduce a single-file embedded filesystem because what the hell, but the irony is all these guys won't even notice it. The same people that say Postgres backups are too hard.

zaptheimpaler2• 2 months ago

Personally I like Postgres for this reason too. Its extremely easy to run with Docker, I can dump data from all kinds of apps in there and I know it's not going to take any rearchitecting as soon as I need multiple concurrent writers.

I think docker is still super underappreciated so setting up any kind of server is seen as a chore. In my eyes it makes running tons of services like this very easy, so ill take the extra functionality, extensibility etc of postgres.

franga20005• 2 months ago

And I don't understand the obsession with server-based databases for single apps. Especially in containerised setups, every "app" gets its own database anyways, and if the app is further broken down into services, they usually communicate between each other and not with a shared database. So in those cases, what do you gain by pulling the database out of the "process" and onto the other end of a socket? In most cases, absolutely nothing. So why bother?

Don't get me wrong, I've worked with plenty of server-based databases, including proper dedicated database servers. It's great tech and often the best tool for the job. But not always and I'd argue not in the majority of uses.

stingraycharles3• 2 months ago

“Especially in containerised setups, every "app" gets its own database anyways, and if the app is further broken down into services, they usually communicate between each other and not with a shared database. “

You seem to be talking about a vastly different use case.

Containerized apps having their own database? What? Aren’t these types of containers stateless? I always very much try to keep state out of app containers.

What kind of data storage are we talking about?

HDThoreaun• 2 months ago

Every container gets its own database?

cortesoft3• 2 months ago

How do you do server maintenance or handle hardware failure if your database is SQLite? You are going to have to take downtime, even in the best scenario.

meszmate• 2 months ago

Most apps do not actually need the concurrency capacity that Postgres or MySQL are designed for.

arendtio• 2 months ago

SQLite for everything

is just wrong, and I don't think that the SQLite fans are that crowd. Taking a database server for everything is probably possible, but often unnecessary. With experience, one can properly judge when SQLite is sufficient and when it is not.

So arguing that the SQLite crowd is inexperienced feels weird, because inexperienced people have a much harder time judging when to use what and can just use the database server all the time (even when it is overkill).

bambax4• 2 months ago

SQLite is an embedded database

Yes, but that's not its main selling point. An SQLite database is also a single file, which makes it incredibly easy to replicate, backup, transfer, restore, etc.

mxey6• 2 months ago

SQLite in WAL mode which you want for server apps is multiple files.

Files which you cannot just copy while your application is running if you want a correct backup.

mxey6• 2 months ago

I think the SQLite website itself says it best:

SQLite does not compete with client/server databases. SQLite competes with fopen().

jappgar4• 2 months ago

It's touted by the people who use the word "just" a lot.

"Just use postgres" "Just use sqlite" "Juse use a monolith" "Just use sftp" "Just use an ec2 instance"

Usually these people have flunked out of the school of (distributed system) hard knocks. They couldn't hack it and are retreating to familiar.

The funny part is when one of those people fluke themselves into senior management when their saas takes off.

Inevitably they have to suck it up and hire experts in the same technologies that "no one needs".

yesb• 2 months ago

That may exist but the opposite type of irrationality is much more common.

Scalability = success. We need to be "scalable" because that means we're successful right? Scalability = real engineering. I'm a real engineer so I need to design everything to be "scalable" because I'm so smart

The funny part is when one of those people fluke themselves into senior management when their saas takes off.
Inevitably they have to suck it up and hire experts in the same technologies that "no one needs".

Sounds like they were the wise ones to build something simple that achieved a high level of success.

UltraSane2• 2 months ago

SQLite also gets really slow at around 50 million rows.

andersmurphy14• 2 months ago

Seems fine at billions of rows in my experience.

dzonga• 2 months ago

you also seem to underestimate the performance you can get on one machine.

localhoster• 2 months ago

Idk if this article was vibe written or the author just "got adjusted" but it's clearly is, and it's unreadable. Man this becomes anmoying

0x592• 2 months ago

Big complex data model with ambiguous query patterns? Postgres

Small, well defined, data model with known query patterns? Bespoke model

There probably is a place for sqlite and my project space so far hasn't yet well-aligned with it.

asdff• 2 months ago

Probably going to get some winces for this but I do everything with flat files. Maybe my data aren't massive enough, but I mean I can do the relational thing by just having these metadata in some column, and returning rows that contain my desired information in these columns. Even if the file were too big to fit into memory one could just subset chunks of it and chew through. All this can be done with no dependencies, just base libraries of a lot of languages.

0x592• 2 months ago

IMO flat files is a cromulent solution for a subset of the entire database space. "Everything" is too broad for me but different strokes for different folks. Happy dataing.

yokoprime• 2 months ago

If you're just doing workflows from a single node, i guess it can be ok as long as theres a single writer. But scaling across multiple servers it clearly is not all you need.

golem14• 2 months ago

Litestream releases 5.9 and newer have a bug that causes instances to sync an insane amount of data. a DB with <10K of data in it and practically no writes/reads causes something like 10GB of daily replication traffic. For my toy project that got needlessly expensive.

https443• 2 months ago

Is this bug logged?

irons• 2 months ago

Looks like https://github.com/benbjohnson/litestream/issues/1197. Still open as of now, with a potential cause noted in the latest comment.

Wilduck2• 2 months ago

I've been following litestream for a while, and it seems like the project has been hijacked by a vibe coder. I wouldn't trust it for critical tasks anymore.

netik• 2 months ago

Until you scale past one machine…

ChrisArchitect• 2 months ago

Building durable workflows on Postgres

https://news.ycombinator.com/item?id=48313530

momojo• 2 months ago

Surfacing for the thread:

Armin Ronacher's "Absurd Workflows: Durable Execution With Just Postgres" https://lucumr.pocoo.org/2025/11/3/absurd-workflows/

bze12• 2 months ago

Isn’t this very similar to cloudflare durable objects & workflows?

stephenlf• 2 months ago

Can’t wait to see the next iteration of this idea with “Logs are all you need for durable workflows.”

gchamonlive4• 2 months ago

Are logs all you need for durable workflows? I'm confused here. How'd persist and query nested or related data over logs? By logs I assume you mean something like elasticsearch or meilisearch?

wolttam• 2 months ago

Pretty much every durable system has an intent log of some sort. The log provides durability, the database system just integrates that log into a more queryable format.

fourside• 2 months ago

I read the parents comment as sarcasm and not a serious suggestion.

deathanatos• 2 months ago

I assume they meant a log like a WAL. A WAL should be (quite literally?) all you need for durable workflows.

A distributed WAL (to survive a machine death) would also probably be something I'd want, and … something I'm not sure you're getting directly from SQLite.

Rapzid• 2 months ago

Log as in the structure.

this_user• 2 months ago

Shortly followed by:

"Sockets are all you need for durable workflows" and then finally "Kernel primitives are all you need for durable workflows."

But seriously, part of being a professional is using the right tool for the job.

friendly_deer• 2 months ago

In all seriousness, I’d take a “s3 is all you need for durable workflows” and use it in data processing applications that move data from s3 -> s3 with no other dependencies.

_karie_• 2 months ago

Wait no further. It's already happening.

One reason why a "logs are all you need" solution may fail: untrusted-log-as-injection[1].

Check those SBOM, and don't forget to include their CICD pipelines[2].

[1] https://news.ycombinator.com/item?id=48315440

[2] https://github.com/jqwik-team/jqwik/issues/708#issuecomment-...

mrkeen2• 2 months ago

Yep. But we all know that one machine can and will fail (or be patched and restarted), so the log needs to be distributed.

Different workflows should probably go in different buckets or "topics" for clarity. Since it's distributed, the system must guarantee that the log items are stored in the same ordering ("offsets") among the nodes.

Not a bad way to do things.

password43217• 2 months ago

Pardon my ignorance trying to follow up on what is most likely sarcasm but is this not Kafka's claim to fame?

I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects. It doesn't seem like there are alternatives at the high end but instead the question is when other technologies (that are easier to manage, require less compute, etc.) max out.

mrkeen2• 2 months ago

Pardon my ignorance trying to follow up on what is most likely sarcasm but is this not Kafka's claim to fame?

Yes

I am joining a new project and need to know to what extent Kafka is still a part of the future for new big data projects.

It's not gonna win or die on merit.

If I were to sit here and propose that you mutate bank accounts in-place (which I'm more or less doing by analogy right now over on https://news.ycombinator.com/item?id=48339103), and you just need safe enough locking technology to do so, I'd be immediately and rightfully shouted down that no one does that and you write down the money transfers and derive the final balances from that.

Manage any other kind of state, however, by appending state-changes and rolling them into a derived state, and then you're chasing fads and doing resume-driven development. So I'm skeptical about the future of this way of doing things.

m2f2• 2 months ago

There's a wide gap from files to multipartition databases. Running databases in a container is not for me sorry whenever real production stuff is on the table.

Personally, lots of ETL can just be taken care of locally without involving enterprise databases. In such cases, DuckDB is 5x-10x better than SQLite and orders of magnitude simpler/faster than spinning up a dedicated Postgres database.

For general scripting, there's no match between a 20-lines awk script and a much cleaner, robust, maintainable equivalent SQL script based on DuckDB.

I just hope MotherDuck don't need to pump/dump for IPO - it would be sad losing that tool for the usual corporate greed.

szarnyasg• 2 months ago

Hello, DuckDB devrel here. First, thanks for the kind words :)

Second, it's funny you should mention the 20-line awk script. I was making a very similar argument yesterday at the Ubuntu Summit: at some point, using shell scripts with GNU coreutilus becomes impractical, while DuckDB SQL scripts scale better in terms of complexity and maintainability (and often also performance). My slides are here: https://blobs.duckdb.org/slides/duckdb-ubuntu-summit-2026.pd... (pages 32 to 36)

Third, MotherDuck develops a closed-source DBaaS on DuckDB. They build on DuckDB, and you connect to MotherDuck with DuckDB but they are a separate VC-funded company headquartered in Seattle. DuckDB is developed by DuckLabs, a bootstrapped (revenue-funded) company in Amsterdam. And the IP of the project is in a third organization: a Dutch non-profit called the DuckDB Foundation. For details, see https://duckdb.org/faq#how-are-duckdb-the-duckdb-foundation-...

infinet• 2 months ago

I use DuckDB and like it. Since many mentioned GB level json in this post, so they have large amount of data. Been column based, DuckDB uses more RAM as row count grows. It can be an advantage or disadvantage depends whether memory is constrained. Traditional row based DB such as SQLite can deal with large database with less memory.

orliesaurus• 2 months ago

Surprised no one has mentioned Turbopuffer yet [1] which natively supports dense vector similarity and BM25 keyword indexes out of the box

[1]. https://turbopuffer.com/

skybrian2• 2 months ago

Instead of "just use Litestream," I'd like to see a review of different object stores one could use and which ones work well with Litestream. Is there a nice object store I could run in another Linux VM? As a hobbyist, which services providing an S3-like API make the most sense?

chrsstrm• 2 months ago

Litestream is just the replication layer, it works with any S3 compatible storage, and all of these in their guide as well https://litestream.io/guides/#replica-guides

skybrian2• 2 months ago

No, I know that already. That's not a recommendation. Some storage layers have to be better than others on price, reliability, and so on?

Think Wirecutter, not install guide.

mburaksayici• 2 months ago

Agreeing on the point, I needed NoSQL version on the similar uses, I've used TinyDB : https://mburaksayici.com/blog/2024/09/21/easy-to-use-nosql-p...

gunnarmorling• 2 months ago

Related piece I wrote some time ago: https://www.morling.dev/blog/building-durable-execution-engi...

prmph• 2 months ago

Postgres ... is the right choice when you need higher availability, broader shared scalability, or other deployment properties that are better served by a network database. It is also the better fit when asynchronous replication to object storage is not the durability model you want... Many workflow systems do not need that on day one and should not start with more infrastructure than their state actually demands.

------

I see this kind of YAGNI thinking a lot, but in my view, it must be balanced against the effort you'd put into resolving any edge cases and adapting current architecture to your use case.

Imagine you deploy Sqlite, and thought it fine by itself, you keep running into some unforeseen challenges with the use to which you are putting. YOu'd need to sink valuable time and effort into addressing those. Then, when you have outgrown it, you'd beed to spend additional valuable times dping the same with Postgres.

This is why, when it comes to Architecture, I increasingly find my myself over-enigneering a bit. Assuming there is a good chance you might need to upgrade your architecture in the not too distant future, that approach is actually kind of very efficient. I find that I am able to uncover a lot of potential gotchas, which feeds back into the what the simplified current architecture should be, and helps me understand the roadmap I'm facing very well. I also avoid wasting too much time going too deep in directions that make sense now, but need a lot of plumbing to get right, when I can see that I'd likely have to throw it all out in a few years. Going from A -> B -C -> D, where each step is the optimal good-enough-for-now architecture but which requires a lot of work to stabilize and iron out the kinks of, is much less efficient than exploring D well enough to know whether you should build A, B, or C now.

Basically, some over-engineering, if done right, is not wasted. It cuts right to the heart of what you are dealing with, efficiently, and allows you to make (maybe) simpler but informed choices now as to how best to allocate your development resources now.

narnarpapadaddy• 2 months ago

My version of this is the “N+1” principle. Build for one more foreseeable use case than you currently have. The domain model will click in when you need to generalize a solution, and you’ll gain the ability to see your particular solution as one of several to the problem, and thus evaluate fit and tradeoffs more clearly.

Don’t do N+2. The goal isn’t to predict the future, nobody can do that. The goal is a durable understanding of the domain and the best fit implementation you can get with that current understanding and resources.

That said, SQLite passes that bar for me in most use cases.

Thaxll• 2 months ago

I started using SQLite for a home project after years of reading about it, I was shocked at the poor type system coming from Postgres. It is really inferior, not sure why it gets so much praise.

https://sqlite.org/datatype3.html

https://www.postgresql.org/docs/current/datatype.html

Working with date/time feels like using a 30years old database, nothing is enforced at insert. Really someone needs to explain why so many people like it.

RivieraKid• 2 months ago

Yes, this is basically my only issue with SQLite. SQLite with a strict type system would be great.

ThatMedicIsASpy• 2 months ago

it's a single file.

zimmi2• 2 months ago

You can use strict tables: https://sqlite.org/stricttables.html

pseudalopex4• 2 months ago

This could enforce dates are strings. They wanted to enforce dates are dates I thought.

chrismorgan• 2 months ago

I don’t like strict tables, because it conflates two concerns, with one somewhat good and one distinctly bad effect (in my assessments).

The somewhat good: it gets rid of most of the weak typing. It still coerces, in line with other SQL databases, but at least a column will only store values of one type. Personally I’d prefer to opt out of the coercion. And I don’t think most ways of writing SQL (in applications especially, but also manually) will ever actually trigger the strict differences. So it doesn’t feel like it’s actually particularly useful.

The distinctly bad: you’re limited to six datatype names. You may well now want external documentation or load-bearing comments in your schema, and your application code may be hobbled, if it liked to infer types based on the datatype name. For example, in sqlx, SQLite datatype BOOLEAN can automatically map to Rust type bool <https://github.com/transact-rs/sqlx/blob/75bc0487eb661da811b...>. Without that, you have to resort to a variety of less-pleasant techniques, such as selecting `done as "done: bool"` or overriding things in sqlx.toml.

I really, really wish they’d implement some form of CREATE TYPE and let that work with strict tables. If I could `CREATE TYPE BOOLEAN FROM INTEGER` and such, I’d be all in on strict tables.

grodes• 2 months ago

Read their docs

IshKebab• 2 months ago

It gets praise because of stuff other than the type system.

I agree it is disappointing, especially before strict tables.

You should check out DuckDB which is basically SQLite but with proper types. Although it is also OLAP (struct of arrays) rather than OLTP (array of structs) which may have worse performance for typical SQLite loads. In practice I doubt it matters if you have an application where you're considering either.

nickpeterson• 2 months ago

A lot of oltp databases have modeling conventions for making read only reporting tables. Do tabular dbs have an inverse for transaction heavy data, that later gets batched to a read optimized structure? I kind of think most databases (even oltp workloads) really are read dominated. I feel like DuckDB is really close to working as the ‘main db’ for such systems, but my lack of knowledge for how to handle quick mutations bothers me. It feels like some form of temporal data modeling would solve it but I don’t know.

formerly_proven2• 2 months ago

This is the fault/price of backwards compatibility. Most users of SQLite should just fire off a few pragmas on each connection:

    PRAGMA journal_mode = WAL
    PRAGMA foreign_keys = ON
    # Something non-null
    PRAGMA busy_timeout = 1000
    # This is fine for most applications, but see the manual
    PRAGMA synchronous = NORMAL
    # If you use it as a file format
    PRAGMA trusted_schema = OFF

You might need additional options, depending on the binding. E.g. Python applications should not use the defaults of the sqlite3 module, which are simply wrong (with no alternative except out-of-stdlib bindings pre-3.12): https://docs.python.org/3/library/sqlite3.html#transaction-c...

Also use strict tables. https://www.sqlite.org/stricttables.html

While it has bad ergonomics, you can also use CHECK constraints. For example, using sqlite's built in date support, it's possible but awkward:

    CHECK (
      date(my_date_col) IS NOT NULL
      AND my_date_col = date(my_date_col)
    )

The IS NOT NULL is needed because date returns NULL for invalid dates; the other check because it also accepts Julian days (date('2026') is sometime during year 4707 BC).

pseudalopex4• 2 months ago

The price of compatibility could be a pragma.

teravor• 2 months ago

if you have an application that needs to maintain state in a non-critical section or if you discover that using SQL is actually a good idea for some tasks (even in critical sections), SQLite is not only a good choice but it will save you a lot of time coming up with a brittle custom solution.

maintain an in-memory SQLite db and work it with SQL commands, and if you also want to preserve state across application restarts you can routinely save to disk or load from it: <https://www.sqlite.org/backup.html#example_1_loading_and_sav...>

this also happens to be the most convenient file-format (aka. application-format) I ever worked with.

PUSH_AX• 2 months ago

I went from using the various big player postgres clusters to SQLite, we have an MAU in 7 figures, all backed by SQLite durable objects. We have to think differently about the access patterns but the benefits have been worth it.

nodesocket2• 2 months ago

The biggest annoyance about SQLite for me is no ability to:

    ALTER TABLE users MODIFY COLUMN…

    ALTER TABLE users ALTER COLUMN…

    ALTER TABLE users ADD CONSTRAINT…

You have to create a new temporary table with correct schema, copy data into this new table, drop the old table, and then rename the temporary table.

simonw3• 2 months ago

They've been improving that recently:

2026-04-09 (3.53.0) - "Enhance ALTER TABLE to permit adding and removing NOT NULL and CHECK constraints"

I use my own sqlite-utils CLI/Python library to work around these limitations: https://sqlite-utils.datasette.io/en/stable/python-api.html#...

nodesocket2• 2 months ago

Ahh that's very nice. Unfortunately the default version of sqlite3 provided by Debian Trixie is 3.46.1.

lvl155• 2 months ago

And all you need is pen and paper to do calculations.

3dedb728-3f77• 2 months ago

Is this just a AWS ads?

fathermarz• 2 months ago

Excellent write up and inspired me for our next IA design run. After reading Fly’s Litestream work it makes me think this is a solid option.

flying_sheep• 2 months ago

Cloudflare durable object is implemented with SQLite (or some variant of it)

vatsachak• 2 months ago

Postgres doesn't cost any extra lol

faangguyindia4• 2 months ago

it does cost network stack

ai_slop_hater• 2 months ago

if you already trust your database, you do not need a separate orchestration tier

Wow. Really? I thought I needed to use an overpriced cloud SaaS for everything.

vultour• 2 months ago

The GitHub statistics for the project this website represents are insane. It has a sole author that has averaged approximately 20,000 lines of code every week in the past month. How do you even maintain that alone?

delduca• 2 months ago

No, mmap is all you need.

psanford• 2 months ago

I wrote a library[0] to let you concurrently update a sqlite db in s3 safely. It uses the little known sqlite sessions extension plus s3 compare-and-swap on a small metadata file to make this work reasonably efficiently and safely. I have been enjoying it for a bunch of small projects where I want a lambda function to have a db for state but I don't want to pay for a full database instance.

[0] https://github.com/psanford/s3db

password43217• 2 months ago

This is exactly what I had dreamed of except without being aware of the session extension... I put it off because I thought I would have to create a layer opening sequential partial sqlite dbs and somehow filtering to the most recent version of each record in-between compactions.

It sounds like sqlite sessions handles the hardest part and you've sanded down some of the rough edges while implementing the glue bits for s3 (generously licensed MIT); thanks for the heads-up!

vkaku• 2 months ago

Back in the day, I wrote a simple job queue with SQLite

https://github.com/guilt/squeue

It did the job, was fairly easy to use.

dannypdx• 2 months ago

All this SQLite hate from big vector db... leave SQLite alone!

faangguyindia4• 2 months ago

I've replaced all of these with Go + SQLite:

1. Intercom 2. Zendesk 3. Email marketing 4. Kanban 5. Todo 6. Our billing stack 7. Our issue tracker 8. Our forum 9. Uptime monitor 10. PagerDuty (clone)

I have dozens of products I sell, so I thought: why not build everything ourselves?

All of these run on the same server and use very little memory. I replaced all the SaaS tools we used with these.

I also moved to dedicated servers and dropped costs to about 1/10th of what we were paying for managed cloud solutions, while maintaining the same HA and even achieving lower latency (partly because noisy neighbors on VPSes were increasing tail latency).

We used to spend a ton on this stuff. These have now been in production for four months and have only needed minor updates.

Deployment is dirt simple. No Docker, no Kubernetes—just a systemd service and a binary built on the dev machine and deployed.

We also used to pay for services like MaxMind and IPData. I ended up hand-rolling my own IP geolocation service, which, in my tests, outperforms most existing solutions.

It all started with replacing Uptime Robot. Then I got more confident and replaced PagerDuty. After that, I replaced Intercom.

Finally, I had always heard people say, "Don't build your own billing stack." But I said YOLO, let me make that mistake myself. So I studied our existing billing solution, developed my own, and rolled it out. So far, we've had zero issues with it.

Caddy in front.

I found that we only use maybe 1–5% of the features most SaaS products offer, while the features we actually need keep getting buried deeper and deeper inside these "enterprise-grade" platforms, making our workflows more difficult.

I won't show my commercial products because our partners and clients probably wouldn't appreciate knowing how cheap I am—but I call it being resourceful.

I can show my free app, though, which has 20,000+ users and was launched recently: https://macrocodex.app/

It only uses the Zendesk clone. Email is handled through Cloudflare routing, so we pay almost nothing to run the app.

tonymet2• 2 months ago

I rolled my own uptime but how do you vary region or do residential testing ?

faangguyindia4• 2 months ago

I actually don't.

I just have uptime service hosted outside of our main infra. It connects to my service called Siren, which alerts me on my phone with an alarm on full volume with SWAT cat intro.

It's good enough for what we do, barely have any downtime. But it helped me figure out 6s downtime we would experience when our spot instances get knocked out, so it helped me increase health check frequency

6s downtime is a lot when you are getting hammered at 100 RPS.

kukkeliskuu• 2 months ago

I am doing approximately the same for some of these as well, plus CRM. For me, CRM is just a list of contacts with a state and some other fields, plus record who was contacted and when. But I can directly pull information that is relevant to me. Integrating to an external CRM is more work and arguably more risky than rolling your own.

imhoguy• 2 months ago

Thanks for your words of wisdom. I share same sentiment of simple apps. I self-learned coding thru 1990s and I am sick of herding distributed overgrown software at work.

Quick question: how do you inspect database contents of your deployed (micro)apps? I used SQLite in one production app and what I didn't like was to use terminal sqlite client on the server or having to copy DB to my laptop to query it with sqlitebrowser. With my Postgres server it is much simpler to just query by rich GUI client with SSH tunnel.

password43217• 2 months ago

This is the way, with "bus factor" as the one downside.

What happens when you get hit by a bus or someone with higher annual revenue tracks you down because of this comment and hires you away from this custom software stack with a bigger slice of the profits?

It sounds like there's also a bus factor for your server but I'm sure you're aware of that though it sounds like your clients aren't.

jedimastert• 2 months ago

Open source it all, I double dog dare you

FR10• 2 months ago

I'm also a Go+SQlitemaxxer. For my first SaaS I focused on keeping the list of external services very short, as I basically couldn't afford it. I only picked those services that were a) vital b) could be replaced with a homegrowned/FOSS alternative.

Besides keeping the costs down, I find that this approach makes it way more enjoyable to build and easier to manage than having 10 services/subscriptions.

*PS: I think that you could add a small QR Code for the iOS/Apple installation on your app's website.

Xeoncross2• 2 months ago

Thank you for sharing, if you don't mind can you share some Go+SQLite leanings as someone actually pulling this off?

1) How do you do backups? Do you use github.com/benbjohnson/litestream? CRON job backup with rsync?

2) Any issues with large databases and many clients? Is there a TPS or DB size where SQLite becomes problematic?

3) How do you deploy new binaries and safely shutdown the old instance? Caddy change to route to new binary + Go's HTTP server graceful-shutdown on old instance?

4) Do you use a pure-Go SQLite lib or one of the CGO libs?

zuzululu• 2 months ago

This is the way

simplestates• 2 months ago

Good framing. SQLite is often enough when the main problem is making workflow state durable, inspectable and easy to recover.

dnnddidiej• 2 months ago

Butchering the Beatles song again.

tomasol4• 2 months ago

sorry about that, I do love Beatles!

emodendroket• 2 months ago

SQLite is an underrated tool for how powerful it really is and probably people don't think of it often enough.

sharts2• 2 months ago

For something so widely deployed you’d think it’d be included in Claude/Codex/etc for

ryanisnan• 2 months ago

Sweet I get to tell my team we can move off of dapr workflows

aykutseker• 2 months ago

Storage never ended up being the thing we worried about. The painful bits started once a workflow could touch external systems. Replaying state is one thing. Replaying a charge or an email is another. How are you dealing with that?

shevy-java• 2 months ago

Hmmm. SQLite is great, but I remember years ago, at a university cluster, I had to populate a SQL database via tons of INSERT statements from genomic/meta-genomic workflows. Postgresql was so much faster just at that particular action (inserting data) that it convinced me that SQLite may be useful for many, many applications, but for "big data"(sets), Postgresql is simply better.

dev_l1x_be• 2 months ago

I start to think the SQLite is all i need to store data. When there is a chance of non-coordinated writes that I can distribute among servers (or even a range based ID) SQLite is my first idea. With durable storage backups this works amazingly well.

utopiah2• 2 months ago

The cycle of expertise :

- what is X, I just do Y

- wow I can see so many limits of Y, now I do X

- I use X for literally everything

- now that I properly understand the limits of Y but also the heavy constraints of X ... maybe Y is enough

- I use Y for literally everything

rinse & repeat. The thing is with actual usage and actual context one does learn and thus can get away with a lot more "basic" solution but it does require genuinely understanding the limits.

dmos62• 2 months ago

Persistence in folly leads a fool to enlightenment.

stingraycharles3• 2 months ago

Yup, when I look back at the silly stuff I did when I was somewhere in the middle (CQRS + event sourcing I’m looking at you), it’s interesting.

It is a source of expertise, because you really learn a lot from it. But when you become old (43 over here), you really learn to appreciate “boring” solutions.

utopiah2• 2 months ago

Boring is the new sexy. /wisdom

roryirvine2• 2 months ago

You also begin to recognise that the definition of "boring" changes over time, and - if you wait long enough - fashions begin to repeat themselves.

So, xBase was all you needed in the mid 80s. Then DBM was all you needed in the mid 2000s. Now, in the mid 2020s, we're told that it's SQLite that is all you need.

It was partly true then, and partly true now. But the full story's always been more complicated, so it's still worth considering a range of potential solutions rather than relying on simplistic rules of thumb or slogans.

(Wake me when nosql comes back into fashion, I'll be able to do a great "old man yells at clouds" routine about that one...)

trueno3• 2 months ago

behold, the bell curve meme

halamadrid• 2 months ago

Operators of Unmeshed here, which is basically a rewrite of Netflix Conductor. In this orchestrator we heavily use a uniquely scaled version of SQLite and also offers “managed” SQLite instances for managing user data. Combining the durable executions of Unmeshed and along with workflow primitives like sleep, workers, etc you can actually build complex systems with a lot less code than ever.

Check it out here: https://unmeshed.io

unnouinceput• 2 months ago

Quote 1: "DBOS recently argued that Postgres is all you need for durable execution...SQLite is all you need."

Quote 2: "SQLite State backed up to S3". Yeah buddy, if you think S3 isn't Postgres I'll eat my foot.

sgt• 2 months ago

Next one: Berkeley DB is all you need for durable workflows /s

bob10292• 2 months ago

This is especially attractive for AI agents and AI-generated workflows. Those systems are often bursty, experimental, and easier to reason about when each agent or tenant has a small self-contained unit of state.

I am finding that the most important thing is one big, consistent data warehouse that is updated with the state of the business as close to real time as we can get.

SQLite is not really great at this particular problem. Something like Postgres or SQL Server would be much more suitable for an OLAP data warehouse that can serve clients (AI agents) while simultaneously merging massive record sets from upstream business systems. These products also offer intricate permissions control. You can prove to an auditor that your AI solution will never see tables or rows it's not supposed to. SQLite doesn't even have a concept of a user, role or login.

The compute can stay cheap and disposable.

Again, hosted sql is better aligned. The alternative is DIY hosted sql (SQLite + some other magic) which immediately violates this rule.

freakynit• 2 months ago

SQLite backed with Raid-10 NVME disks and periodic backups to cloud storage is generally more than enough to run majority of the production workloads of startups.

Writes are single threaded, but, you can still easily do thousands per second.

DuckDB offers similar qualities, on the OLAP side.

This is not to say this is best combination.. but, when you consider the simplicity of setup, usage, operations, and backups, and cost element, this indeed offers one of the best, if not the best combination.

jackzhuo• 2 months ago

100% this. I used to default to Postgres for everything. But seeing SQLite handle concurrency so well now—plus having built-in BM25 search and vector support—it really is all you need for these kinds of architecture.

vixalien• 2 months ago

Has anyone actually used PGLite[0]?

[0] https://pglite.dev/

jessmartin• 2 months ago

Waiting for “JSONL is all you need for durable workflows.”

Xeoncross2• 2 months ago

So is https://github.com/obeli-sk/obelisk the Rust version of https://github.com/temporalio/temporal (Go)? Can you guys add a comparison between them on the site?

tomasol4• 2 months ago

Subscribe to this issue: https://github.com/obeli-sk/obelisk/issues/65

throwaway58670• 2 months ago

Holy sticky header... It takes literally half the screen on mobile. Shit like this makes me wonder: do you even look at your website? At least sometimes?

oulipo22• 2 months ago

Obelisk also supports Postgres. When you're using it with Postgres, what are the differences with DBOS? Are there any that would be significant

(I'm already using Postgres, so I don't really need a sqlite-based durable workflow engine, so looking to know how to choose between DBOS and Obelisk)

tomasol4• 2 months ago

Check out https://obeli.sk/blog/comparing-dbos-part-1/

bakrisolo• 2 months ago

The move toward local-first, durable workflows is honestly a breath of fresh air. I’ve spent the last few months building a document analysis tool and the amount of friction involved in trying to make cloud-based RAG pipelines 'durable' vs just keeping the state in local SQLite is night and day. Once you stop fighting the latency and privacy issues of the cloud you can build much faster.

sperandeo• 2 months ago

I concur. I keep waiting to hit the wall where I need Postgres and it just hasn't happened. PB ftw

sonthonax• 2 months ago

The thing that people forget about databases is that they are just really complicated ways of writing to a file.

The crux of all these “you only need sqlite” posts isn’t that you need SQLite, it’s that your architecture only ever needed flat appendable log file to begin with.

Once you realise that SQLite is just an aggregated view over this log it calls into question if you needed a third party query engine at all, suddenly the whole NoSQL vs SQL debate becomes a meaningless implementation detail.

Server based databases aren’t special beyond the fact they’re just distributed synchronisation primitives over files on a file system. But the utility of this is quite limited in practice, writing to a table will always require some sort of synchronisation, using SQLite vs Postgres is just a case of where you put the lock - in your own code or in a 3rd party server.

pdimitar2• 2 months ago

Once you realise that SQLite is just an aggregated view over this log it calls into question if you needed a third party query engine at all, suddenly the whole NoSQL vs SQL debate becomes a meaningless implementation detail.

You are correct on the premise, however my observation has been that teams who try to be lean and mean and minimize dependencies always start off believing they don't need a query engine... and they always end up needing one, often making things much worse by refusing to just migrate to any database due to sunk cost fallacy.

I fully agree with your argument that it's all just file operations in the end. To me SQLite wins for everything beyond persistence however: the SQL commands, extensions, backups, changesets / patchsets.

Can you, I, and many others write something that has 10% of SQLite feature set that serves us perfectly? Absolutely! But I don't want to, and in business settings nobody will allow you to hand-roll crucial infrastructure software (like databases) unless they're in full crisis mode and you're the mega-expensive consultant brought in to save the day.

SQLite is all you need for durable workflows

Related Stories