mattrobenolt

Right, but I think you're kinda missing a lot of the tangible benefits here. This IMO is just reinforcing the idea of "unlimited" IOPS. You can't physically use the totality of IOPS available on the drives.

Even if you can't saturate them, even with low CPU cores, latency is drastically better which is highly important for database performance.

Having low latency is tangibly more important than throughput or number of IOPS once your dataset is larger than RAM no matter how many CPU cores you have.

Chasing down p95s and above really shine with NVMes purely from having whatever order of magnitude less latency.

Less latency also equates to less iowait time. All of this just leads to better CPU time utilization on your database.

You can simply place your database in the same AWS or GCP region and the same AZs.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-inst...

Not all AWS instance types support NVMe drives. It's not the same as normal attached storage.

I'm not really sure your arguments are in good faith here tho.

This is just not a configuration you can trivially do while maintaining durability and HA.

There's a lot of hype going the exact opposite direction and more separation of storage and compute. This is our counter to that. We think even EBS is bad.

This isn't a setup that is naturally just going to beat a "real server" that also has local NVMes or whatever you'd do yourself. This is just not what things like RDS or Aurora do, etc. Most things rely on EBS which is significantly worse than local storages. We aren't really claiming we've invented something new here. It's just unique in the managed database space.

It's more than just shutting down. You'd have to have an actual failure. Data isn't lost on a simple restart. It'd require 3 nodes to die in 3 different AZs.

While that's not impossible, the reality is that's very low.

So simply restarting nodes wouldn't trigger restoring from backup, but yes, in our case, replacing nodes entirely does require that node to restore from a backup/WALs and catch back up in replication.

EBS doesn't entirely just solve this, you still have failures and still need/want to restore from backups. This is built into our product as a fundamental feature. It's transparent to users, but the upside is that restoring from backups and creating backups is tested every day multiple times per day for a database. We aren't afraid of restoring from backups and replacing nodes by choice or by failure. It's the same to us.

We do all of the same operations already on EBS. This magic is what enables us to be able to use NVMe's since we treat EBS as ephemeral already.

Unlimited in this context just means you're going to be CPU limited before you hit limits on IOPS. It'd be technically not possible to be bottlenecked on IOPS.

That might not be 100% true, but I've never seen a RDBMS be able to saturate IOPS on a local NVMe. It's some quite specialized software to leverage every ounce of IOPS without being CPU bottlenecked first. Postgres and MySQL are not it.

We deal with this by always running 3 nodes in a cluster, one per AZ, and strong backup/restore processes.

So yes, the data per-node is ephemeral, but it is redundant and durable for the whole cluster.

It's still AWS/GCP, but it uses instance types with local NVMes.

Hi there, fundamentally this is a bit challenging against something vanilla Postgres or MySQL to bolt on since the connection itself is stateful.

Because of the Vitess layer we use, Vitess already has to maintain this with it's internal connection pooler and it knows which connections are bound in transactions on the underlying mysqld. And since Vitess is running "next" to mysqld on behalf of the client, the client is capable of being able to effectively pass around a session cookie with a reference to the in-progress session against the database.

So when the next request comes in, we're effectively saying "please use this bound session id", and vitess says "cool, we got a connection still going for you" and uses that reserved connection.

But tl;dr, it's fundamentally a design that Vitess enables us, and it's implemented effectively with bouncing a Session of state back and forth similar to a browser cookie.

Specifically this protobuf message: https://buf.build/planetscale/vitess/docs/main:vitess.vtgate...

We're a service provider. As a client and customer, you connect to us as a third party service. You don't control our uptime or connectivity. Nor do you control whatever network hops may be between.

Yup! There are a few caveats, but for the most part it'd be compatible.

https://planetscale.com/docs/reference/mysql-compatibility

Mostly agree here. I don't think in practice, you'd default want to use HTTP/3. I think as technology, it's still just very immature for reasons you mentioned.

In our case though, lots of our customers and lots of use cases do communicate over a WAN, and potentially large geographic distances. I think having this as an option is super interesting to see what we can do with it in the future.

In theory. In practice nothing implements this.

But in any case, even if your client did support this and the server supported it, we still need HTTP for other things. I don't think it's particularly a "gotcha". HTTP is also stateless, which has lots of benefits for us.

It helps some cases for sure and what I'd strongly recommend in practice. Connection pooling isn't always a viable option depending on the application though.

Connection pooling doesn't solve all the things we can improve by using HTTP as a base. We can be faster in just data transfer through compression, for example.

Using HTTP/3 starts to help tail latency that we can't solve with TCP. Unreliable networks with packet loss suffer greatly with TCP and not as badly with QUIC.

Also ever heard of gRPC? It's pretty popular and widely used. Whether you like it or not or whether it's the most optimal in theory, in practice HTTP is heavily heavily used for communicating between components.

To me, HTTP sits in a similar boat as JSON. Is it perfect? Is it good? Not necessary. But it's extremely extremely scrutinized and optimized due to its ubiquity in ways that other protocols and formats haven't been.

This was the entire point of this experiment and it's proved successful. The bias I wanted to challenge is exactly what you mentioned. Turns out, using HTTP as a transport and protobuf for encoding (which is basically gRPC) is comparable.

Check your bias. This was clearly communicated as an experiment with surprising results. In some contexts, we don't have a choice but to support HTTP. So simply wanted to see how it stood up to a traditional mysql client.

But you do you.

Much love. <3

Anything is possible with computers.

My apologies for not meeting your bar. I guess you missed the parts where it's faster in a lot of other cases too, and not slower in any.

To me, the fact that it's not slower at all is the big win. I didn't anticipate that the results of this are going to say "this is 5x better". The stereotype is that if it's over HTTP, it must be slower.

And by every measure, it's not slower. In cases, that may be edge to you, or don't care about extra latency, they're still improved. Why would you not want something that's generically better?

There are many other things that are beneficial with using HTTP as a transport that haven't even been discussed here since this was entirely focused on performance. Without at least matching in performance, not many of the other things would matter.

1) use persistent connections, let the OS handle them and tweak it to allow (both connecting server and mysql server). And never close the connection on the application side. (This could lead to potential deadlocks, but there are ways around it, like closing bad connections to clear thread info on mysql).

This is definitely ideal, but one thing that you can't entirely control is the server side or what's between. Sometimes your connections get interrupted, and it's not possible to maintain a connection forever. Yes tho, this is the ideal thing you should do with a connection pooler.

2) run the whole thing in a transaction, simply begin transaction or autocommit if allowed (same thing)

This shouldn't really help with latency. Being in a transaction doesn't reduce latency. If we're being pedantic, it would likely increase latency due to having to execute a BEGIN and COMMIT query, which is typically two more round trips, one per query.

I think what you're getting at is something like pipelining, where you can send multiple queries in one request, and get multiple results back. This is technically supported over the mysql protocol, but isn't very widely adopted in practice.

The tests cover both cases if you read.

But either way, yes, that's fundamentally a benefit of being able to use HTTP. We can multiplex multiple sessions over one underlying connection.

Sorry about that.

I'm not a PHP expert, so I don't know the landscape there fully. I do know our customer complaints and can say people care about cold start times in the PHP space and others.

So while it may be an edge case for you, it's not for others. It also doesn't discredit any of the other testing that doesn't focus on cold starts.

Edit since you edited yours after I posted:

I'm not going to argue the merits of what platforms people choose and it's not really our position as PlanetScale to do that. We serve our customers.

Maybe in your case, but not always. But I do explicitly call out that I intentionally wanted a test of a "cold start". This is extremely relevant for short lived applications and processes. Think PHP, and serverless environments, etc.

The other tests are measuring already a warmed up connection.

There's also reason why I intentionally coupled "connect + select 1" as the test, because I wanted to make it as close of a comparison as possible. If it was simply a "connect", our HTTP API would be even more favorable since connecting doesn't do authentication or anything like that like the mysql protocol does.

I have vaguely looked into it, but since we don't support it at PlanetScale and unlikely we will, it didn't seem worth it. It'd be hard to construct a practical test.

Similarly, since support is so low, it didn't make a lot of sense to double down and support it when we could do what works for us.

In our case, it's a bit more state than just a transaction ID, but the same concept applies.

And yeah, to mirror the other comment, this isn't particularly something I'd consider a security issue. If an attacker has access to your transaction (session) state, this is a similar risk vector on any web service that utilizes browser cookies or sessions. If someone hijacks your browser cookies, they can do anything they want into your account typically, ignoring 2FA and extra layers.

Same risk here. If someone were to intercept your request, which is strictly over HTTPS, and hijack your session, or you erroneously were logging the session and authentication credentials publicly, then you're exposing yourself to a lot bigger risk than someone adding a sleep() into your session. They could just dump everything from the database, because they are you.

A direct connection only changes this because it's a stateful protocol, and if the connection is severed, everything is lost/rolled back and nobody can intercept that.

But anyways, I'd hardly call a well established security model of HTTP sessions flawed in this regard.

If anything, it's possible for a client to do some dumb things, like, parallel queries within a single transaction, which you cannot do with a direct connection, which ultimately is just... going to be undefined behavior since they fundamentally get serialized into the database, just in an undeterministic order relative to the client.

Correct. I would not recommend doing that. This is analogous to a database driver. It's raw access to the database.

It's a good chunk of it, but is a lot more suitable for public consumption.

Yeah.

I would not consider myself a member of the JavaScript or TypeScript communities. I just build the infrastructure and platforms to support these kinds of things. :)

I'm going to use a big notice here: None of this is inherently safe to use as a client. This is similar security profile as a database driver. This API has no row level auth or anything like that. Unless this is explicitly what your product is doing, like within PlanetScale, this is what powers our Console, don't use this in a browser.

With that said, I personally have no real opinions on this topic. It's pretty far out of my wheelhouse. I think that's more something like Firebase and gang, but that's not something we're gunning for at the moment and aren't trying to replace that.