james_cowling

I'm giving the first talk of the session and while I'll be talking about ideas we've actually built and tested in Convex, the point I want to get across is that I think the industry has gone in the wrong direction with exposing databases to clients and using an inexpressive declarative language to query them. Will be trying to avoid any shilling because I feel like this is a pretty important discussion to have.

Yes the readsets aren't based on lists of objects but rather the data ranges that a query touches, i.e., they don't suffer from phantom read anomalies.

If you were to attempt to fetch all items from an empty shopping cart, that query will automatically be invalidated if any item is added to the cart, even though there were no documents returned from the original query. Query intersection always works.

The aggregate example isn't a great one in Convex since it doesn't currently support built-in aggregates - a full-table aggregate involves a table scan and therefore the readset would be the entire index. We may add built-in aggregates that are incrementally computed if there's enough demand.

Convex is FSL licensed which means you can basically do whatever you want with it re running a service, other than directly competing with Convex, i.e., reselling the Convex platform. There are no limitations on you running a business on top of this and no obligation to pay anything.

After 2 years the code becomes Apache 2.0.

Note that if you're running the open source release you're on the hook for managing and scaling your own reliable infrastructure, as described in the open source readme.

I talk more about the philosophy re open sourcing convex in https://softwareengineeringdaily.com/2024/03/20/going-open-s...

(convex cofounder here)

Convex is in a very comfortable financial position but it is also open source: https://github.com/get-convex/convex-backend

Convex cofounder here. Yep the database transaction logic, timestamp assignment, concurrency control, etc is all custom. This is a pretty key requirement in our ability to perform dynamic subscriptions and automatic caching in real time.

Internally the very bottom of the stack on the hosted product is actually just a write ahead log on RDS and in the open source product it's just sqlite. We'll likely eventually build our own durable write ahead log, and have plenty of experience doing so, but this hasn't been needed so far.

Convex cofounder here. Initial focus has been low-latency OLTP database workloads, which is pretty tied to the user-facing end-to-end reactivity angle we're pushing. As a bonus feature we also have built-in strongly consistent full text search, but not vector search.

So far our answer for folks who want alternative storage/query engines is to use our streaming Airbyte source connector or write directly to Pinecone, Snowflake, etc. This should work great for most devs.

There are likely always going to be some developers who want to use a particular third party database in addition to Convex, but we plan to expand built-in support for most workloads over time so that Convex is a truly full-stack backend replacement.

(kmod was the founding engineer on the project and the guy who came up with the name!)

We should also clarify that Sujay and I are part of the old old team :) The current team have been doing an awesome job since then!

(a bunch of the early MP folks work at Convex now)

"It’s fairly easy to design a system with astronomically high durability numbers. 24 nines is a mean time to failure of 1,000,000,000,000,000,000,000,000 years. When your MTTF dwarfs the age of the universe then it might be time to reevaluate your priorities.

Should we trust these numbers though? Of course not, because the secret truth is that adherence to theoretical durability estimates is missing the point. They tell you how likely you are to lose data due to routine disk failure, but routine disk failure is easy to model for and protect against. If you lose data due to routine disk failure you’re probably doing something wrong."

https://medium.com/@jamesacowling/how-many-nines-is-my-stora...

We had a lot of requests for open sourcing MP but I don't think it would have been very useful. The code itself is interesting to look at but the system is designed for very large data sets, has a lot of moving parts to manage and configure, and would be very difficult to operate at anything below at least double-digit petabytes.

MP was designed for multiple exabytes across many data centers and geographic regions. A different system design (or just using S3) would be more appropriate for use at smaller scales.

Yeah we have a few talks about this and a chapter about this very issue in https://www.oreilly.com/library/view/seeking-sre/97814919788.... Totally agree that in a well designed system the sources of data loss are certainly not disk failures.

As far as I know Magic Pocket has had 100% durability, but that's obviously beside the point.

We designed and built the system almost completely ground-up, all the way down to the disk scheduler, but yeah we probably took the term OSD from Ceph.

Yep that’s me! I have my full name on here so wasn’t hiding but good point to be clear about disclosing.

Firebase has been a huge influence but with the rise of serverless there are a new generation of platforms people should be checking out instead, such as Convex, Supabase, etc.

Convex in particular is designed far more for end-to-end consistency rather than just a database to talk to: https://docs.convex.dev/understanding/convex-vs-firebase

We track the readset for any active subscription. When a new write transaction commits we compare the writeset for the transaction against any active subscriptions. If there's an intersection then it's likely that query will have been invalidated, so we rerun it.

I talk more about how we perform this comparison in this (now rather outdated) talk: https://youtu.be/iizcidmSwJ4?t=1218

Note that there are false-positives here, i.e., it's possible for the inputs to a query function to change without the outputs actually changing, but this is not a significant issue in practice.

Thank you! We have a really talented team working with us but we're very actively growing. If anyone is looking for a job in San Francisco: https://www.convex.dev/jobs

This is very nice feedback. The short answer is that we want to solve problems that matter for the people who need them solved.

Thanks for the feedback. We think React is the right community to target right now but the end state for Convex is certainly not limited to React.

The specific concern with regards to multiple round trips is scenarios where the client needs to send multiple serial requests to render a web view, e.g., having to fetch a list of posts first before knowing which post ids to fetch comments for. These request waterfalls can lead to high page load times for many web apps.

Consistency-wise Convex supports full serializability. The main point this article is making with regards to consistency however is providing tools to ensure not only that the database is consistent but that the data rendered to the end user is also a consistent view. I talk a little more about that here: https://youtu.be/B9aeddqwVas?t=446

Btw we are huge fans of Firebase and think it did a ton to advance the industry forward. Thanks for your work on it!

I'm not sure what your definition of smart polling is but there is no polling going on here - the query only reruns when the data dependencies change server-side due to a subsequent mutation.

Another key distinction is that "query" here doesn't refer to a database read, it could be a complex function containing multiple reads, relatively-arbitrary compute, etc.

Obviously depends on the kind of projects you're working on but in our experience we've been more productive in Rust compared to when we've built large systems in Python, Go, C and C++.

This is with regards to long-term productivity however. Generally it shows up in building things that work right the first time around, being able to refactor confidently, etc. Personally I still find myself faster in Go or Python if I have to throw a short-term project together very quickly but other members of the team still profess to being more efficient with Rust for these types of projects too.

I've been using Valora for a while now and even though I discovered it through the crypto world, I use it because it solves an actual problem for me. It's been a great solution for venmo-style payments with friends in different countries.

On/off-boarding of funds to local currencies is always a challenge in the crypto space but the app supports deposits from a debit card at least. Looking forward to exchange with fiat currencies getting more streamlined over time.

Thanks for the feedback! Glad folks are reading this stuff :)

Would you be able to give more details as to how you use disks as raw block devices, bypassing the filesystem? For example, the kind of abstraction layers you use, their interfaces and methods, disk layout of extents, metadata layout, consistency issues, fsync issues, and the various performance advantages from bypassing the filesystem?

We can definitely go into details on the raw block device. Will probably leave that to @jamwt to write a blog post on since there's quite a lot of content there. Most of the motivation for doing so was actually so we could exploit the write-zone layout of SMR disk drives, although we get some other minor benefits like a bit more "formatted" disk space and no filesystem bugs.

I was also wondering, I know you have two functional zones, but do you also keep an isolated timelapsed append-only backup of all data, in case of ultimate software error (something bad gets past the verifiers, or a delayed bug kicks in)?

Quick summary is that we write typically only data into at least two zones (regions), obviously replicated within these zones, and then we keep "trash" around in case there's a software bug that causes deletion. We also have a delete grace period where we keep data within the storage system for longer in case an application issued an erroneous delete.

The Dropbox filesystem layer actually checks that a file was added to MP before committing it to the metadata layer of the filesystem, so there's an extra check there that the write didn't just disappear.

We also log metadata about each put and every internal data transformation to a separate system so we can retrace our steps if there was ever an issue. This logging system is actually running on HDFS, so MP isn't the only storage system running at Dropbox, but it's by far the biggest one.

Yup we do this, and yes, the verifier has been broken before (during early development, not at production scale).

We do a number of things here, like taking down a sufficient number of storage nodes in a single region to make blocks appear "missing" in the region and force an automatic failover to the other region (this is transparent to users, apart from slightly more latency), or more direct/risky checks in our Staging cluster (we don't ever mess with data in our main production cluster).

In reality a large system like this regularly encounters timeouts or failures of sub-components which are masked by our multi-region redundancy but show up as spikes in the verifiers. These remind us that everything is working, in between more explicit DRT (Disaster Recovery Training) tests.

For sure. I hope we're all agreeing here :)

We've very strong believers that an effective replication strategy is just table stakes and that from there the real risks to durability are the "black swan" events that are much harder to model.

I gave at talk at Data@Scale recently where the main premise is about "Durability Theater" and how to combat it in a production storage system. In case you're interested: https://code.facebook.com/posts/253562281667886/data-scale-j...

From what I hear from other major storage providers Magic Pocket is definitely very high up on the paranoia/verification scale. There is of course a cost to all this verification traffic. Any sensible storage provider will be doing stuff like disk scrubbing and verification scans on their storage index however.

We haven't shared drive reliability stats before but I don't think we have any objection to doing so. Don't expect these in the next month or two but we'll likely get to this in the not-too-distant future.

Yeah to be clear my point from the article is that even though you can model an incredibly high durability number in theory, there's a lot more work required to see that in practice. I'm sure the S3 team also have extremely high internal durability numbers.

A lot of other factors go into an end-to-end durability numbers, including the possibility of software bugs, correlated failure, etc. Magic Pocket has an external durability claim of 12 9s which gives us a lot of headroom between the external figure and the internal model.

Fortunately both 11 or 12 nines is close enough to infinite as far as clients are concerned.

Hi folks, James from Dropbox here. Happy to answer any questions that come up so feel free to send them by way.

Thanks!

It's always a bit of a tradeoff. A 512MB bucket size would double the size of the Replication Table and we want make sure we keep the entire working set in the buffer cache. 1GB also has the advantage of filling up slower, which gives us more time to close full buckets and create new ones as needed.

That said, 1GB is a bit of an arbitrary number which just happens to work pretty well. 512MB might work just fine. The exact number isn't as significant since typically we lose a whole disk at a time and thus needs to recover a certain number of TB regardless of the bucket size.

This is a really great question since the vast majority of the work was in ensuring correctness and reliability: everything from testing discipline to fault injection to auditing. This also included hardware testing, like pulling out circuit breakers to test our power distribution, or overheating a rack to test graceful shutdown.

I'll give a slightly lazy answer here however and point you to a talk I gave about building durable systems, which covers a lot of this material: https://www.oreilly.com/events/velocity/devops-web-performan...

To be clear, we do need those durability and availability guarantees :)

We just benefit from our very-large scale and close alignment between hardware and software to tailor our infrastructure to our workloads.