TigerBeetle is a most interesting database

https://www.amplifypartners.com/blog-posts/why-tigerbeetle-is-the-most-interesting-database-in-the-world

by todsacerdoti • 10 months ago

301 222 10 months ago

amplifypartners.com

itunpredictable7• 10 months ago

hope you all like this post as much as I enjoyed writing it!

jorangreef32• 10 months ago

Liked it almost as much as My Cousin Vinny! ;)

criddell• 10 months ago

Just to be a little pedantic for a second, isn't TigerBeetle a database management system (DBMS) whereas a TigerBeetle database would be actual data and metadata managed by TigerBeetle?

lioeters• 10 months ago

Often the term "database" is also used loosely to refer to any of the DBMS, the database system or an application associated with the database.

BrentOzar• 10 months ago

Because I'm sure other people will ask - no, it does not support SQL.

jorangreef32• 10 months ago

Joran from TigerBeetle here!

Yes, this is by design. SQL is a great general purpose query language for read-heavy variable-length string workloads, but TigerBeetle optimizes for write-heavy transaction processing workloads (essentially debit/credit with fixed-size integers) and specifically with power law contention, which kills SQL row locks.

I spoke about this specific design decision in depth at Systems Distributed this year:

1000x - https://m.youtube.com/watch?v=yKgfk8lTQuE

justinclift5• 10 months ago

which kills SQL row locks.

What's it like compared to MVCC?

big_hacker3• 10 months ago

What's it like compared to Redis or even KeyDB?

karmakaze• 10 months ago

It helps to know what kind of data TigerBeetle handles. The data committed by its transactions is an immutable Transfer of id:128-bit, debit_account_id:128-bit, credit_account_id:128-bit, amount:128-bit, ledger:128-bit, code:16-bit, flags:bitfield, timestamp:64-bit, user_data_128, user_data_64, user_data_32.

Transactions atomically process one or more Transfers, keeping Account balances correct. Accounts are also records, their core fields (debits_posted, credits_posted, etc).

This gives a good idea of what TigerBeetle might be good for and what it might not be. For anything where latency/throughput and accuracy really, really matters, it could be worth the effort to make your problem fit.

pbowyer2• 10 months ago

And yet some of the most popular OLTP databases in the world today are still highly dependent on a single node architecture.

Which databases? SQLite is the one I can think of, but it's designed for that use-case. Others start as single node but will replicate to other nodes, either as master-slave or master-master.

redwood3• 10 months ago

Postgres, MySQL

pbowyer2• 10 months ago

They both have replication, which allows them to be multi-node.

pjmlp6• 10 months ago

I am quite sure Oracle and MS SQL server do just fine in multi-node cluster based architectures, but maybe that isn't their target audience.

bob10292• 10 months ago

MSSQL has a full-blown Death Star deployment option now:

https://learn.microsoft.com/en-us/azure/azure-sql/database/h...

lomase• 10 months ago

The HN crowd does not talk much about those, but every single place I have worked used them or Postgres.

SchwKatze2• 10 months ago

Pretty much agree, I honestly think that TB team should invest some more effort on evangelizing the double-entry model for non-financial scenarios, like managing stocks, booking a show ticket etc. The 1000x API improvement has been done, now people must know how to use it

deadbabe2• 10 months ago

You could probably do something similar in Postgres.

jorangreef32• 10 months ago

You could, but TB brings a whole lot of debit/credit primitives out of the box (making it dead simple for e.g. 2PC money movement across third party systems, or transactions of thousands of transactions, to express complex financial contracts)… that are Jepsen-tested, which run multi-AZ, and which would probably take months if not years to replicate on PG. Even then, the internal concurrency control of PG’s storage engine would just get hammered by the contention, and would struggle to approach TB’s replicated performance, even if PG were single node.

SchwKatze2• 10 months ago

Similar, yes. But not near the same performance, TB is entirely built around this concept so they became pretty good at it. It's like comparing a lambo and a beetle (no pun intended).

But honestly, if double-entry really become a thing I foresee traditional DBMS agglutinating it just like they did with vector and object databases, getting the long tail of the market.

graemep• 10 months ago

It would be nice. I can think of things for which I want to process a financial transaction and something else in one atomic transaction (e.g. selling a ticket - something I am working on right now) and have both content (e.g. product descriptions) and finance in a single store.

CGMthrowaway• 10 months ago

It sounds awesome. I'm an a analytics guy using a lot of SQL - not a coder. Though I understand the OP writeup and the purported performance advantages. Can someone explain:

a) what Tigerbeetle data looks like in practice? Assuming it doesn't look like a regular table

b) how you use it, if you can't write sql queries?

c) separately, curious what double-entry would look like for stocks, tickets etc. E.g. I'm a venue and I have 1000 tickets in inventory & deferred revenue.. each time I sell a ticket I turn that inventory to cash and the deferred into a performance liability? Or no entries at all until a ticket is sold? Something else?

jorangreef32• 10 months ago

As one example, Rene Schallner made a pretty cool ticketing demo called TigerFans: https://tigerfans.io

justinclift5• 10 months ago

In less than a decade, the world has become at least three orders of magnitude more transactional. And yet the SQL databases we still use to power this are 20-30 years old. Can they hold up?

Errr yes. Without much sweat really.

Just because something started ~30 years ago doesn't mean it hasn't updated with the times, and doesn't mean it was built on bad foundations.

dakiol• 10 months ago

I disagree. If we are talking about distributed systems in which we have N different databases, then distributed transactions are left as an exercise to the reader (that’s why we have things like Sagas).

Within a single machine, yeah, relational dbs still work like a charm.

Normal_gaussian• 10 months ago

The other interesting thing to consider is that machines are a lot more powerful in all dimensions than they were 30 years ago - so many tasks that would have required distributed systems no longer do.

big_hacker3• 10 months ago

If it's gonna fit on one machine I'm picking Postgres, Mongo or Redis before TigerBeetle and let's be honest it's not the difference between the 4 that will make it not fit on one machine.

pjmlp6• 10 months ago

I was part of a project that successfully deployed them into production using Oracle, sometimes those license costs are actually worth it.

sgarland5• 10 months ago

Tbf, this is a natural consequence of companies doing premature optimization / cargo culting. There are many, many reasons to NOT go into microservices. Kelsey Hightower had some thoughts on this [0] back in 2020.

You can run a hell of a lot off of a small fleet of beefy servers fronted with a load balancer, all pointing to one DB cluster.

[0] https://changelog.com/posts/monoliths-are-the-future

pkphilip• 10 months ago

True. The older databases run very well even on hardware which was far less powerful than what is available now.

jorangreef32• 10 months ago

Joran from TigerBeetle!

Without much sweat for general purpose workloads.

But transaction processing tends to have power law contention that kills SQL row locks (cf. Amdahl’s Law).

We put a contention calculator on our homepage to show the theoretical best case limits and they’re lower than one might think: https://tigerbeetle.com/#general-purpose-databases-have-an-o...

tux32• 10 months ago

Traditional SQL databases hold locks across the network; under Amdahl's Law, even modest contention caps write throughput at ≈100–1,000 TPS

In fact large real world systems are not limited to 100-1000 TPS, or even 10 kTPS as the calculator tries to suggest. That's not because Amdahl's law is wrong, the numbers you're plugging in are just wildly off, so the conclusions are equally nonsensical.

There might be some specific workloads where you saw those numbers, and your DB might be a good fit for this particular niche, but you shouldn't misrepresent general purpose workloads to try to prop up your DB. Claiming that SQL databases are limited to "100-1000 TPS" is unserious, it is not conductive to your cause.

pluto_modadic• 10 months ago

So, the pitch for TigerBeetle is... "you can do database schemas wrong and we're performant enough"? (and also we don't have auth)

Xelbair3• 10 months ago

DNS still runs strong and it was ~~designed~~ released* in the November 1983.

It still holds up basically whole of internet.

in most cases SQL is good enough for 90% of workloads.

seemaze• 10 months ago

What would DNS look like if it were released in 2025?

slightwinder2• 10 months ago

Isn't DNS a catastrophic clusterfuck which also had to be patched up multiple times to not allow people burning down the whole world? I mean, yes, it's doing its job, but mainly because of people doing their job.

yomismoaqui2• 10 months ago

Are you telling me that new shiny things are not always better that established, time tested and boring tech?

Sometimes I feel like we software engineers have the worst memory of any engineers.

sgarland5• 10 months ago

I don’t even think it’s memory, it’s lack of experience. A lot of the battle-tested infrastructure is now abstracted away to the point that a lot of devs are hard-pressed to explain the difference between AWS’ ALB and NLB, much less name OSS versions of them.

Actually, load balancers are a great example. The number of times I’ve seen a team re-implementing HAProxy, but poorly, is entirely too high. The reasoning is always something along the lines of “we need this extremely specific feature.” OK, and have you even read the manual for HAProxy? Or if that feature truly doesn’t exist, did you consider implementing that separately?

causenad• 10 months ago

Kelsey Hightower gave an interesting talk on this - why understanding fundamentals will always matter more than the recent shiny new trend:

https://youtu.be/Jlqzy02k6B8

cmrdporcupine3• 10 months ago

Yeah, SQL is not the issue. At least not mostly.

The relational model has shown itself to be exactly the flexible and powerful model that Codd said it was, even in its relatively-debased from in SQL.

In fact the potential of it as a universal and flexible data model that abstracts away storage still remains to be fully unlocked.

Now as for existing SQL databases, yes, many of them were built on foundational assumptions about the nature of memory and secondary storage that no longer hold true.

Many of us in this industry still have our heads stuck in the past world of spinny spinny magnetic disks on platters with heads and cylinders and grinding noises. Real world hardware has moved on. We have a couple orders of magnitude higher IOPS generally available than we did just 10-15 years ago.

So I'm excited about products like CedarDB and others (Umbra before it, etc) that are being built on a foundation from day one of managing with hybrid in-memory/disk.

Throwing out SQL is not really the recipe for "performance" and the lessons of separating the storage system from the data model is still key, since the 1960s.

I am willing to grant that a specialized transaction/ledger system like TigerBeetle might have its place as an optimization strategy in a specific industry/domain, but we should not make the mistake of characterizing this as a general problem with the relational model and data storage broadly.

9rx4• 10 months ago

Throwing out SQL is not really the recipe for "performance"

We've discovered hacks to work around the limitations of SQL, so you can maintain performance with sufficient hackiness, but you have to give up "purity" to get there. Worse is better applies, I suppose, but if we were starting over today the design would be very different.

Throwing out SQL isn't strictly required, but like Rust isn't strictly required when you already have C, sometimes it is nice to take a step back and look at how you can improve upon things. Unfortunately, NoSQL turning into a story about "document databases" instead of "A better SQL" killed any momentum on that front.

websiteapi• 10 months ago

All of these apply to FoundationDB as well.

- Slow code writing.

- DST

- No dependencies

- Distributed by default in prod

- Clock fault tolerance with optimistic locking

- Jepsen claimed that FDB has more rigorous testing than they could do.

- New programming language, Flow, for testing.

You probably could solve the same problems with FDB, but TigerBeetle I imagine is more optimized for its use case (I would hope...).

AFAIK - the only reason FDB isn't massively popular is because no one has bothered to write good layers on top. I do know of a few folks writing a SQS, DynamoDB and SQLite layers.

itunpredictable7• 10 months ago

I am currently working on a post about DST with their team too haha!

dangoodmanUT5• 10 months ago

the one true database

oblio3• 10 months ago

AFAIK - the only reason FDB isn't massively popular is because no one has bothered to write good layers on top. I do know of a few folks writing a SQS, DynamoDB and SQLite layers.

I started writing this comment:

It seems interesting, but considering what it's for, why aren't the hyperscalers using it?

And while writing it I started searching for FoundationDB and found this:

https://github.com/<<apple>>/foundationdb

Ah, all right :-p

oblio3• 10 months ago

And a more serious comment, to separate it from the silly one below:

Interesting that they didn't release it with an SQL client, is there no way to make it compatible? Even with extensions to SQL, I imagine it would be great for a lot of use cases.

Edit: ah, it's more of a key-value store.

ksec2• 10 months ago

I thought they got bought by Apple and what happened to it after?

andrewl-hn2• 10 months ago

Apple bought them, took it down from the web, then quietly open-sourced it a few years later. They tried to make it popular, ran a conference for it, but the adoption was too minor for Apple to care afterwards.

It's still maintained by a sizable team at Apple, GH stats show that the activity is much lower now than it was 3 years ago, but there're about 10 people that contribute on a steady regular basis, which is honestly better than 99% of open source projects out there.

andrewl-hn2• 10 months ago

the only reason FDB isn't massively popular

The only reason is Apple. They liked the product that was released in 2013 so much they bought the whole company, and all other FoundationDB users were abandoned and were forced to drop it.

Who would trust a database that can be yanked out of you at any moment? Though a lot of products have license terms like this only a handful were ever discontinued so abruptly. It's under Apache license now but the trust is not coming back.

senderista3• 10 months ago

Well, all users except Snowflake.

wiradikusuma4• 10 months ago

We were considering TigerBeetle, but found blockers:

* We use Cloudflare Workers. TigerBeetle client app is not supported. It might work using Cloudflare Containers, but then the reason we use Cloudflare is for the Workers. --> https://github.com/tigerbeetle/tigerbeetle/issues/3177

* TigerBeetle doesn't support any auth. It means the containing server (e.g. a VPS) must restrict by IP. Problem is, serverless doesn't have fixed IP. --> https://github.com/tigerbeetle/tigerbeetle/issues/3073

vjerancrnjak• 10 months ago

But Cloudflare Workers or AWS Lambda setup would not work anyway with any db?

* spawning 1000 workers all opening a connection to a db,

* solved by service/proxy in front of db,

* proxy knows how to reach db anyway, let's do private network and not care about auth

dangoodmanUT5• 10 months ago

DBs like DynamoDB work great with these kinds of runtimes, they don't need a connection pool in front

wiradikusuma4• 10 months ago

Cloudflare has its own DB (D1, Sqlite-derived), but you can also connect with PostgreSQL using their adapter (Hyperdrive). I have used both, they're okay.

caust1c• 10 months ago

Wait, what? A database in 2025 doesn't support any kind of auth? A financial database? WTF?

C'mon folks, the least you can do is put a guide for adding an auth proxy or auth layer on your site.

Particularly since you don't use HTTP (cant easily tell from the docs, I'm assuming), then folks are going to be left wondering: "well how the hell do I add an auth proxy without HTTP" and just put it on the open internet...

wiradikusuma4• 10 months ago

Exactly my thoughts, and yes, it's not HTTP. Not a big deal, I guess, if they explain how to work around it...

jorangreef32• 10 months ago

Joran from TigerBeetle here!

TigerBeetle is our open source contribution. We want to make a technical contribution to the world. And we have priorities on the list of things we want to support, and support properly, with high quality, in time.

At the same time, it's important I think that we encourage ourselves and each other here, you and I, to show respect to projects that care about craftsmanship and doing things properly, so that we don't become entitled and take open source projects and maintainers in general for granted.

Let's keep it positive!

senderista3• 10 months ago

You are not entitled to anyone else's free labor.

jorangreef32• 10 months ago

You should chat to our solutions team, who'd be able to help you solve auth etc. at the logical level with end-to-end encryption, something we typically do for our customers.

wiradikusuma4• 10 months ago

I might not be able to afford paid support yet since it's just PoC, but if it's okay to be public, could they respond to the GitHub issue instead so it can benefit others? https://github.com/tigerbeetle/tigerbeetle/issues/3073

cakealert• 10 months ago

Wireguard would be an answer here, IP's are authenticated with ECC keys.

jorangreef32• 10 months ago

Yes, this is what we also recommend, amongst other solutions. The building blocks are orthogonal.

bell-cot• 10 months ago

Consider re-titling it "A very cool & interesting article about a very boring & reliable database"?

Since "interesting" is the very last thing that anyone sane wants in their accounting/financial/critical-stuff database.

itunpredictable7• 10 months ago

looks like the poster agrees

flir4• 10 months ago

They keep assertions enabled in production.

Never understood why we turn those off. An assert failing in prod is an assert that I desperately want to know about.

(That "never understood" was rhetorical).

mcherm• 10 months ago

I like to write assertions that aren't always easy to check. Like asserting that a list is sorted.

flir4• 10 months ago

That sounds easy to check. Can you expand on this, because I don't understand.

pjmlp6• 10 months ago

Then you want Design By Contract, dependent types, or formal verification.

However, several factors have to be taken into account regarding performance impact when they get cleverly written, thus in manys cases they can only be fully turned on during debug builds.

matklad• 10 months ago

Yeah, there's actually trickiness here. Another curveball is that, with simulation testing, you generally want more assertions to catch more bugs, but slow assertions can _reduce_ testing efficiency by requiring more CPU time per iteration! But then, if you know, at comptime, that the list is going to be short, you might as well do O(N^2) verification!

We captured these consideration in the internal docs here: https://github.com/tigerbeetle/tigerbeetle/blob/0.16.60/src/...

Cthulhu_• 10 months ago

On paper it's because assertions are only checks for developers to make sure they don't misuse an API. They still have values if it's user input, but at that stage, it would be better to change it to business logic like a proper error message returned to the end-user.

TZubiri4• 10 months ago

Because the very raison d'etre of asserts is to be a compile time/test time check. If you wanted them to run in production, you wouldn't use an assert, you would just run an if. Otherwise what's an assert to you? A mildly convenient sugar syntax alternative to if statements?

flir4• 10 months ago

A mildly convenient sugar syntax alternative to if statements?

Yup.

kevinqi• 10 months ago

right, it is just syntactic sugar, but if that wasn't helpful then why have it in dev either? I find it more confusing to have asserts be stripped, which creates an implicit dev/prod discrepancy

swiftcoder• 10 months ago

That a semantic we've all just sort of inherited from C - I don't know we're required to be bound by it forever. The word itself doesn't imply that assertions would be contextually disabled.

kstenerud• 10 months ago

It's historical: turning off assertions made the code run faster.

But nowadays an extra comparison is no biggie, especially if the compiler can hint at which way it's likely to go.

igtztorrero• 10 months ago

I really liked this article. I met the new kid on the block, the DBMS neighborhood. I also didn't know that Zig programming language existed. So many new things. Congratulations to TigerBeetle! I'm going to tell my team about it and try it out on an interesting project.

itunpredictable7• 10 months ago

thank you!

initramfs• 10 months ago

this has such great marketing zingers, I probably didn't know I needed TigerBeetle for something. I don't know yet, but I promise you, I'll find something to use it for.

self_awareness• 10 months ago

The most interesting database is the most boring one.

jorangreef32• 10 months ago

The problem with "boring", "popular" and "old" is that they are too often synonyms (and logical fallacies, i.e. as an appeal to popularity/authority).

Rather, I would say that the most interesting database is the fastest and safest.

Or as Jim Gray always put it: “correct and fast”.

ducklord• 10 months ago

What does TigerBeetle stand on the CAP theorem? Sounds like it's AP?

jorangreef32• 10 months ago

CAP defines availability physically: “requests to a non-failing node receive a response”. TigerBeetle ensures logical availability: as long as clients can reach a majority, requests complete with strict serializability, preserving safety and liveness under partitions.

cf. Durability and the Art of Consensus: https://www.youtube.com/watch?v=tRgvaqpQPwE

IshKebab• 10 months ago

Deterministic Simulation Testing (DST) feels like the most transformational technology that the fewest developers know about. It’s a novel testing technique made popular by the FoundationDB team

It's not novel. That's how hardware (ASIC) testing has been done forever. The novelty is applying it to software.

TigerBeetle’s VOPR is the single largest DST cluster on the planet. It runs on 1,000 CPU cores

Only if you exclude hardware, otherwise basically every chip design company has a cluster bigger than this.

bob10292• 10 months ago

Leading edge photo mask synthesis is easily in the top 5 most computationally demanding things out there. You need an actual supercomputer to solve these problems on meaningful (competitive) timescales.

https://en.wikipedia.org/wiki/Optical_proximity_correction

xnzm1001• 10 months ago

I work at a bank with old old monstrous sql queries. I thought I can make a use of tigerbeetle to simplify the system. But sadly, I just couldn't figure out how to make it work. Transaction requires lot of business logics but I couldn't convert that to make it work with combination of RDBMS + tigerbeetle. I wish there were some realworld example that I can get insight using tigerbeetle.

jorangreef32• 10 months ago

Appreciate that you created an anonymous account to be able to share this feedback nevertheless.

We typically hear from companies that TigerBeetle is the easier part of their stack. But to be fair, they may have a cleaner architecture.

Did you contact our solutions team for assistance with your business logic modelling? If not, please get in touch! solutions@tigerbeetle.com

sgarland5• 10 months ago

While I am a fan of TigerBeetle’s general stance on correctness, coding practices, and their desire to hyper-specialize, I have some critiques on the post.

I think the paragraph about multi-node is a bit misleading. Contrary to what cloud native folk will tell you, a single beefy DB, well-tuned and with a connection pooler, can serve a dizzying amount of QPS just fine. At a former employer, during a maintenance period, I once accidentally had all traffic pointed to our single MySQL 8 RDS instance, instead of sharing it between its read replicas. That was somewhere around 80-90K QPS, and it didn’t care at all. It wasn’t even a giant instance - r6i.12xlarge - we just had a decent schema, mostly sane queries, and good tuning on both ProxySQL and MySQL. At peak, that writer and two .8xlarge read replicas handled 120K QPS without blinking.

A DB hosted on a server with node-local NVMe (you know, what used to be normal) will likely hit CPU limits before you saturate its I/O capabilities.

For redundancy, all RDBMS designed for networked activity have some form of failover / hot standby capability.

My other mild criticism is in the discussion on TigerBeetle’s consensus: yes, it seems quite clever and has no other dependencies, but it’s also not trying to deal with large rows. When you can fit 8,190 transactions into a 1 MiB packet that takes a single trip to be delivered, you can probably manage what would be impossible for a traditional RDBMS.

None of this should be taken as belittling their accomplishment; I remain extremely impressed by their product.

justinhj2• 10 months ago

That a traditional modern RDBMS is very fast is not really useful here because the use case of Tigerbeetle includes work loads with high contention. Under that kind of load, in their own tests you can see in a recent conference talk, you can see the throughput drops dramatically in clusters where many transactions affect a single account.

edit: from the horses mouth is better https://news.ycombinator.com/item?id=45437046

ffsm82• 10 months ago

Huh, that as not been my experience at all.

Too be fair, its been something like 10 years iirc.

The database in question was MySQL 8, running on plain old enterprise ssds (RAID 10)

The workload was processing transactions (financial payments)

The database schema was ... Let's call it questionable, with pretty much no normalization because "it's easier when we look at it for debugging", hence extremely long rows with countless updates to the same row throughout the processing, roughly 250-500 writes per row per request/transaction from what I recall. And the application was a unholy combination of a PHP+Java monolith, linked via RPC and transparent class sharing

DB IO was _never_ the problem, no matter how high qps got. I can't quote an exact number, but it was definitely a lot higher then what this claims (something like 40-50k on average "load" days like pre Christmas etc)

Not sure how they're getting this down to ~250qps, it sounds completely implausible.

Heck, I can do single row non-stop updates with >1k qpm on my desktop on a single nvme drive - and that's not even using raid.

mike_hearn• 10 months ago

It's worth noting that other databases have features designed specifically to address this use case.

Oracle has lock free reservations on numeric columns: https://oracle-base.com/articles/23/lock-free-reservations-2...

parthdesai• 10 months ago

My other mild criticism is in the discussion on TigerBeetle’s consensus: yes, it seems quite clever and has no other dependencies, but it’s also not trying to deal with large rows. When you can fit 8,190 transactions into a 1 MiB packet that takes a single trip to be delivered, you can probably manage what would be impossible for a traditional RDBMS.

Isn't that the point? They're saying to separate out transactions workload from other workloads. They're not saying they'll replace your OLGP db, you remove transactionally important data into another db.

It's something similar that we see with another db: https://turbopuffer.com/

zenmac• 10 months ago

we just had a decent schema Glad someone mentioned it. In the dev ORM culture now days it is all about how to make the db look like the code. DB is essentially reduced to a object store. Often a proper decent schema can handle most of these load. You can always throw in redis layer later if your traffic is really getting to require that level of optimization.

porridgeraisin• 10 months ago

TigerBeetle looks interesting. A whole networked database using statically allocated memory. I usually use that technique for small keyboard-shortcut triggered applications.

One question in case folks who work there see this:

This is the most technically impressive zig project I've seen so far. Do you have a blog post detailing your perspective on zig? i.e what design decisions of ziglang helped you in massive way, what were the warts, and in general any other thoughts too?

ksec2• 10 months ago

Most teams write code fast. TigerBeetle tries to write code slow. Most teams treat testing as a necessary evil. TigerBeetle is built entirely on Deterministic Simulation Testing (DST). Most teams build their software on top of loads of other software. TigerBeetle has zero dependencies.

I believe that was more like the norm 25+ years ago. Before Google and Facebook brought in the move fast and break things mentality across software industry.

I hope TigerBeetle gets more recognition. Worth reading its Jepsen report as well. https://news.ycombinator.com/item?id=44199592

motoboi2• 10 months ago

Let's wait 25 years and we'll see if TigerBeetle becomes the next google or if it's engulfed by a less perfect (but faster) rival.

jorangreef32• 10 months ago

Give us 27 years—the company only turned 3 last month, and we designed for 30 :)

To be clear, TB's pretty young, only 3, but Jepsen-tested and already migrating some of the largest brokerages, wealth managements and exchanges in various countries. I'm excited to see what can be done in 27.

”Slow is smooth, and smooth is fast”

surajrmal• 10 months ago

Move fast and break things was the Facebook motto. I've never really felt it was a Google embraced concept. What Google's values have always lied with investing heavily on testing to ensure folks can be productive by knowing that if they would break something a test would catch it.

That said it's good to make sure you're building for requirements that exist. Engineers have a habit of inventing requirements and causing delays unnecessarily. Building something and placing it in the hands of users so they can give you feedback which you can react to is far more valuable than building products in a bubble.

dangoodmanUT5• 10 months ago

I really like the work Joran and his team have done around DST, distributed systems awareness, and performance practices. I especially love the craziness around no dependencies (well, could you consider an underlying OS a dependecy?).

But I've always felt the way they treat normal OLTP (they call OLGP) seems unfair. For example, comparisons using clear sub-optimal interactive SQL transactions for financial workloads, like locking rows rather than using condition checks at commit time, because "that's how OLTP was intended to be used when it was designed ~50(?) years ago".

In their cited https://tigerbeetle.com/#performance the lowest the slider can go is 1% contention. Do you think Stripe has 1% contention directly on an OLTP DB? Definitely not.

You can build systems that _expect contention_, and elegantly handle it at REALLY high throughput. These systems protect the DB from contention, so you can continue to scale. From talking to folks working on these systems, I roughly know the kinds of transactional (financial) throughput of DBs like Stripe's and other systems - they have _many_ more zeros behind them than their performance comparison page proposes they could possibly have at even 0.01% contention.

Their marketing largely ignores this fact, and treats everyone like they just slam the DB with junior engineer-designed interactive transactions. Most developers (I hope) are smarter than that if they're working at a payments company. There's even the title "payments engineer" for the kind of person that's thinks about scaling contention and correctness all day.

TigerBeetle is great, but I find the pattern of being quite misleading about other OLTPs off putting.

sgarland5• 10 months ago

Do you think Stripe has 1% contention directly on an OLTP DB? Definitely not.

Stripe runs on top of MongoDB, which is horrifying in its own right, but in any case comparing them to a shop running an RDBMS is apples to oranges.

redwood3• 10 months ago

Stripe delivers incredible uptime. This type of comment is beneath this platform.

xxpor• 10 months ago

well, could you consider an underlying OS a dependecy?

I've worked on systems that ran entirely in memory, and continued running during kexec. You can't do syscalls. So yeah :)

jorangreef32• 10 months ago

Hey Dan, appreciate the compliment.

Do you think it would be more fair to suggest that OLTP workloads have 0% contention? That debit/credit sets never intersect?

In practice, we're seeing that our customers are already some of the largest brokerages, wealth managements or exchanges, in their jurisdictions, even national projects, and all of them with decades of Postgres etc. experience, some even running installations with 200 Postgres/MySQL machines backing their sharded core ledger.

They're not junior engineers. For example, they know about stored procedures, but the problem of concurrency runs deeper, into the storage engine itself.

At least for our customers, for the seasoned payments engineers, OLTP contention is a killer. They're tired of scaling outside the DBMS, and the complexity of expensive reconciliation systems. Several of them are even leaving their Chief Architect positions and starting TB-related startups—I know of at least 4 personally, from fintech brands you will recognize, and probably use.

I hope the spirit of our talks, trying to get the big ideas across, to make people aware of contention and Amdahl's Law, is clear, and that you take it in good faith, to the extent that we show it ourselves.

dangoodmanUT5• 10 months ago

Oh I totally agree the world needs to be more aware of Amdahls law, and I appreciate that you make it known (I believe I even learned it from you). I just think that the communication is sometimes too much "it's our way or the bad way", if that makes sense?

I don't think they have 0% contention, and I agree that contention is the bane of a dev's existence, but I don't think it's as cut and dry as only looking at Amdahl's.

Specialized systems have done really well to handle contention: My go-to example is LMAX. I think I remember a talk where the devs said they were pulling 6M orders per second, and ofc the stock market has an aggressive power law. If you design around contention, you can still push silly high perf (as you've shown).

FWIW I think we both agree that Postgres is not the best DB for this either :P

n_u2• 10 months ago

comparisons using clear sub-optimal interactive SQL transactions for financial workloads, like locking rows rather than using condition checks at commit time

Could you give an example of how the same transaction could be written poorly with "locking rows" and then more optimally with "using condition checks at commit time"?

dangoodmanUT5• 10 months ago

subtract 50 if balance >= 50 is optimistic (the value at commit can be different than at read)

vs. "lock this row so it can't change while I subtract 50 from 50,000,000,000,000..." you get the point

fjfaase• 10 months ago

I wonder if the 'debits and credits' example is still realistic. Is it not that banks record all individual money transfers and keep them on record for years (often due to national laws that require this). Then the balance on a certain account is a simply a selection/view on the set of all transactions. For efficiency reasons you may want to store the result in some table. If there is a limit on withdrawals on a certain account, you do need to calculate the balance first and then a transaction mechanism is needed to assure that there are not several money transfer that individually do not surpass the limit but together do.

novoreorx• 10 months ago

Why hasn't anyone mentioned the game they made? It looks fantastic. The concept of letting players try to crash a system to demonstrate its robustness is brilliant.

jorangreef32• 10 months ago

Thanks!

For anyone who wants to play the walking sim (excuse the pun!): https://sim.tigerbeetle.com

oowoe2• 10 months ago

hiiiiiiiiii owo qwq pwp

oowoe2• 10 months ago

hiiiiiiiiiii owo pwp qwq

danielbarla• 10 months ago

TigerBeetle is plenty interesting, though if half of SpacetimeDB's [1] [2] claims are true, it's even more interesting for me. (The title of this post has "_a_ most interesting database", while the article has "_the_", hence this comment)

[1] https://spacetimedb.com/ [2] - https://www.youtube.com/watch?v=kzDnA_EVhTU&

kamranjon• 10 months ago

I’m somewhat confused by the distributed story for tiger beetle. In the docs here: https://docs.tigerbeetle.com/concepts/performance/#single-th...

It says: “TigerBeetle uses a single core by design and uses a single leader node to process events. Adding more nodes can therefore increase reliability, but not throughput.”

How does this work with multi-region? Does this mean that regardless of where in the world your users might live, they need to make a request to the leader node for a write to occur?

Is distribution purely for redundancy?

Voultapher• 10 months ago

The way I understand it yes. Latency will be dictated by going to the leader node and replication to enough followers. The CAP theorem dictates that if you want strong consistency the way they offer you have to talk to a quorum of nodes IIRC N/2 + 1. Paxos and Raft also work that way.

Apparently there is a game about it https://tigerbeetle.com/blog/2023-07-11-we-put-a-distributed...

mtlynch• 10 months ago

It's worth noting that the breathless worship of everything about TigerBeetle in this post is likely related to the fact that the author is an investor in TigerBeetle.[0]

[0] https://www.amplifypartners.com/portfolio/company

gregorvand2• 10 months ago

We did an episode with Joran on SEDaily which addresses some of the points raised in the discussions

https://softwareengineeringdaily.com/2024/09/12/building-a-f...

jorangreef32• 10 months ago

Gregor! Thanks again for having me there, and good to see you.

gregorvand2• 10 months ago

And you!

blenderob• 10 months ago

As nice as TigerBeetle is, do keep in mind that this article is written by an investment company that has invested in TigerBeetle: https://www.amplifypartners.com/blog-posts/our-investment-in...

noir_lord• 10 months ago

Yeah I'm not going to say what I think of the way it's written but suffice to say that was obvious from how it was written.

itunpredictable7• 10 months ago

I have a few of these posts I've written coming out over the next few months that I want people to discuss. Would you prefer I add a disclaimer at the top? Easy enough to add

gaudystead• 10 months ago

Personally, I'd say you should definitely add a disclaimer, otherwise a discussion that would likely gain traction is that the information might be being intentionally concealed.

cowpig2• 10 months ago

If you care about ethics of journalism then yes you should obviously disclose a financial interest in what you write.

sroussey• 10 months ago

My god, what do you expect?

It’s on said investment company’s website under the tag “Portfolio Spotlight”.

9rx4• 10 months ago

Keep in mind for what reason? The content doesn't change based on this knowledge. It either stands up for what it is or it doesn't. It wouldn't somehow be better if it were written by a random Joe user instead: It would be the exact same work. Attributing qualities of the author onto the work is a logical fallacy.

Use that precious mind space for understanding basic logic instead.

vrnvu• 10 months ago

I've been applying a lot of principles and suggestions from TigerBeetle style lately, mainly in Rust and Go and I can’t recommend it enough.

- single entry point, near-zero deps

- ci locally and tested, one command to runs tests, coverage, lint etc

- property/snapshot/swarm testing, I love writing simulations now and letting the assertions crash

- fast/slow split + everything is deterministic with a seed

- explicit upper bounds + pool of resources. I still dynamically allocate but it makes code simpler to reason about

Thanks to the TB team for the videos and docs they been putting out lately.

jorangreef32• 10 months ago

Such a pleasure to read this, that TigerStyle is making a difference for you. Appreciate the kind words!

mkleczek2• 10 months ago

While TigerBeetle way is very interesting I find lack of multicore processing support (or any kind of horizontal scaling) a blocker in terms real world deployment.

It is just too risky as there is a hard limit to scalability and while it might look it is high enough for foreseeable future, what am I supposed to do once I reach this limit? Financial database has to be planned with at least 15-20 years of growth in mind.

agos• 10 months ago

sorry if the question seems dumb, but are the clustering options not good for scaling?

mkleczek2• 10 months ago

No, they are only for HA. Only one cluster node is serving clients.

theYipster• 10 months ago

Looking at the Zig vs. Rust angle — an interesting takeaway from the article is that Zig makes it considerably easier to build statically allocated applications than Rust.

For all of Rust’s talk about “safety”, assuming the above assertion is true, than perhaps Zig is a better choice to augment or replace C in embedded, safety-critical software engineering, where static memory allocation is a core principle.

_1tem• 10 months ago

Sounds awesome but can I use it like a regular database for my Laravel/Rails/Django app?

gwbas1c• 10 months ago

OMG as I read this I became hyper-skeptical by the end.

Some points:

In general, in very tangible terms, what are the real benefits for choosing TigerBeetle over another distributed database? What are the target use cases? Most of the article is pontificating about academic details in a way that's putting the cart before the horse. (IE, all these details don't matter when a traditional database with traditional backups is "good enough" and comes with no technology risk.)

Who's using TigerBeetle? For what kind of applications?

https://news.ycombinator.com/item?id=45436926 states, "TigerBeetle doesn't support any auth". Poor security is an unprofessional oversight. For all the ambitious things TFA describes, it's shocking that they overlooked "secure by design."

The above-linked post is the only top-level post in this thread that claims real-world experience with TigerBeetle.

But, here are some quotes from the article that don't pass scrutiny:

Traditional databases assume that if disks fail, they do so predictably with a nice error message. For example, even SQLite’s docs are clear that:

SQLite is a file on disk. It's not something that would work in the same space as TigerDB. Furthermore, this statement needs proof: Do you mean to say that Oracle, MSSQL, MariaDB, Postgress, ect, can't detect when a file is corrupted?

All in all, you’re looking at 10-20 SQL queries back and forth, while holding row locks across the network roundtrip time, for each transaction.

These can often be solved with stored procedures. In this case, the problem lies somewhere between programmers implementing suboptimal solutions, and databases being too hard to work with.

Instead of investing in the technology of 30 years ago – when the most popular relational databases today were built

Don't assume that because technology is old, that it's bad. New databases come with higher, not lower, risk.

They say databases take a decade to build.

Prove it. No really, prove it.

Biodigital jazz.

It seems like someone is getting so obsessed with the code that they're forgetting the purpose and use case.

---

IMO:

Figure out the niche where TigerBeetle excels and traditional databases fall flat. Is it because your query model is easier to program with? Is it because, in a particular situation, TigerBeetle is more performant or cheaper? Is there a niche where existing databases corrupt data more than is tolerable.

Once TigerBeetle excels in a niche, expand outward. (This is the basic plan in "Crossing the Chasm." https://en.wikipedia.org/wiki/Crossing_the_Chasm)

Otherwise, this smells like an academic exercise to experiment with different programming techniques, but with no tangible deliverable that has a demonstrated market.

jorangreef32• 10 months ago

In general, in very tangible terms, what are the real benefits for choosing TigerBeetle over another distributed database?

Transaction processing at scale. The world doesn't need another string database. TigerBeetle is an integer database designed for (double-entry) counting, even under extreme write contention.

See also our 1000x talk going into what only TB can do (and why stored procedures still suffer from concurrency control in the internal storage engine): https://www.youtube.com/watch?v=yKgfk8lTQuE

Do you mean to say that Oracle, MSSQL, MariaDB, Postgress, ect, can't detect when a file is corrupted?

Yes. cf. https://www.usenix.org/conference/atc20/presentation/rebello and https://www.usenix.org/conference/fast18/presentation/alagap...

Also read the Jepsen report on TigerBeetle (to see the new storage fault injectors that Kyle Kingsbury added): https://jepsen.io/analyses/tigerbeetle-0.16.11

with no tangible deliverable that has a demonstrated market.

TigerBeetle is already being integrated into national payment systems. We also have a few enterprise customers. Some pretty large brokerages, wealth managements, exchanges and energy utilities. Granted, the company is only 3 years old, so we still have some market to demonstrate.

sunshine-o• 10 months ago

Anybody know of simple way/tool (local, single node) to have strict Debit/Credit consistency ?

Is there an extension for sqlite or simple command line tool that do that?

I know about the ledger cli tool but it a bit much since it is a full fledge double-entry accounting system.

jorangreef32• 10 months ago

You can also run TigerBeetle local, single node if you want.

12_throw_away• 10 months ago

The article covers the enjoyable experience of getting networked computers to all agree on a single monotonic clock time. That got me wondering - does there exist a commodity, high-precision, high-reliability hardware clock that can be plugged into a computer to give it super-accurate timekeeping?

(TBH, I'm pretty sure the answer is no - if you want super-accurate time, I think you buy your own very expensive NTP server [1].)

[1] https://www.masterclock.com/master-clock-products.html

dailyfin• 10 months ago

The post does a good job selling the “why” (hot-row contention + strict serializability) but it’d help the community to see a clearer “when.” A practical rubric would be: if X% of writes touch Y “house accounts” with RTT Z, OLTP throughput on MVCC/row-lock engines caps at ~N TPS; TB sustains ~M TPS—showing the apples-to-apples path from workload shape to DB choice. Framed that way, TB becomes a “consensus-backed integer ALU” you pair with a string DB, not a replacement for OLTP in general.

Two gaps I keep hearing in this thread that could unblock adoption:

1. Reference architectures: serverless (Workers/Lambda) patterns, auth/VPN/stunnel/WireGuard blueprints, and examples for “OLGP control plane + TB data plane.”

2. Scaling roadmap: the single-core, single-leader design is philosophically clean—what’s the long-term story when a shard/ledger outgrows a core or a region’s latency budget?

Also +1 to publishing contentious, real-world case studies (e.g., “fee siphon to a hot account at 80–90% contention”) with end-to-end SLOs and failure drills. That would defuse the “100–1,000 TPS” debate and make the tradeoffs legible next to Postgres, FDB, and Redis.

jorangreef32• 10 months ago

Much appreciated.

TigerBeetle is a most interesting database

Related Stories