jordanthoms

Oh, I agree on this. People were never going to accept, nor IMO should they have, a massive reduction in their living standards. New technology is the way to make people's lives better while also reducing global warming.

I just got back from a off-grid island here in New Zealand - 20 years ago, generators were everywhere and as soon as it got dark you'd hear nothing but the buzzing of running them all around you. Now there is solar everywhere and it's completely silent.

Humans can be trusted to do the right thing, once all other possibilities have been exhausted.

Also interested in this - the Apple Watch for Kids setup seems a possibility, but it's only available in certain countries

This is dependent on the database you are using - if it's a key-sharded distributed database, you want to have insertions evenly spread across the key space in order to avoid having all the inserts go into a single shard (which could overload it)

Not quite any - in CockroachDB, all schema changes are non-blocking: https://www.cockroachlabs.com/docs/stable/online-schema-chan... . Yugabyte seems to be getting there with them also.

Still risks involved in migrations (mostly from the migration executing too quickly and creating high load in the cluster - the admission control system should have reduced this) and we have extra review steps for them, but it's been very useful to be able to migrate large tables without any extra application-level work.

Gmail was also experiencing issues: https://www.google.com/appsstatus/dashboard/incidents/shD5Vv...

That's kinda the point though isn't it? DownDetector is showing an early indication of a major outage in both of your examples. The issue may not be caused by the indicated service, but it's still a useful information source especially when we can correlate reports on there with what we are seeing in our internal monitoring.

We saw a big spike in latency and failures on the Google OAuth apis starting at the same time (15:21 UTC)

A single report on there is useless. A sudden flood of reports is a good sign that something interesting is happening.

It has false positives and noise for sure, but it's also very sensitive and shows issues very quickly.

I wouldn't trust it as a single source, but in a case like this where our internal monitoring shows a spike of issues with the Google APIs and we can see a huge spike in reported issues for Google on Downdetector starting at the same time, it's useful to confirm that the issues have an external source.

Your general point definitely stands - there is a pretty nice third party solution for google workspace though: https://github.com/GAM-team/GAM

I guess the problem there is pretty fundamental - in reality you'd be tensing muscles and shifting your weight etc before the snappy movement, but the game only knows that you want to move when you move the stick or push a button - so it either has to show that realistic motion after your button press and introduce latency, or sacrifice the realism in the animations.

It does seem like there would be a clean boundary between each school district, but actually there's plenty of sharing and collaboration on Kami that happens with users between districts, teachers and students move schools, parents can have children in different school districts, etc. Even a single classroom assignment can cross those, e.g. when someone external comes in to do a training session.

We could have modified our application layer to handle those cases, but it's a lot of extra complexity and room for error, and we'd have had to consider and solve for all of these cross-tenant situations as we add new functionality, so I was really keen to avoid that.

Also, there are some really big districts - NYCDOE has >1.1 million students and 1,800 schools. Even with them on a dedicated shard, it's quite possible that it'd get overloaded and we'd be spending more dev effort figuring out how to safely split them onto multiple shards.

When we looked at using distributed SQL database instead it was a clear win - from the application's perspective, it just looks like a really, really big PostgreSQL box, so we didn't need to change much. (the SQL support is very close to PG - The most annoying thing for us was the lack of trigram indexes, and Cockroach has now added those now). And in terms of the operational side, upgrading and maintaining CRDB has actually been easier than PG - version upgrades are easier to do without downtime, and schema migrations don't lock tables.

We moved took a large working PostgreSQL app and switched it over to CRDB and that doesn't match my experience. Our existing schemas and query patterns moved over nicely - latency for small indexed reads and writes did increase from ~1ms to ~3ms, but the max throughput now effectively unlimited since we could add capacity by adding new nodes into the cluster and letting CRDB automatically rebalance the workload. There was an increase in cost as it will need more cores, disk etc compared to a single-primary PostgreSQL, but that makes sense when you consider that every bit of data is getting stored on 5 different nodes and there are overheads to maintain the consistency.

For the highest throughput endpoints we did make some changes to be more optimal on CRDB so we could run a smaller cluster, but it didn't require anything close to a rewrite.

They would have to have many shards per city to keep up with the level of write traffic though. And what happens when a user from SFBA goes down to LA?

Clickhouse is a totally different use case - Cockroach is OLTP, Clickhouse is OLAP. We use both Cockroach and Clickhouse at scale and they are both great but not competing products - Cockroach is great for the types of reads and writes you do when serving user requests, processing transactions etc, but isn't optimal for analytics queries where you are going do things like read and aggregate data on a 50TB table. Clickhouse eats those kinds of aggregate queries for breakfast, and is fast for some types of small read queries too, but it's not built to handle random writes or frequently updating rows of data.

Sharding is anything but simple. A single shard per region wouldn't have enough write capacity so they'd have to be managing likely 100+ shards in each region - you'd have to build a lot of infrastructure to automate setting those up, rebalancing traffic to avoid hot spots and underutilized shards, in sync with schema migrations etc.

Even after that, now your applications using the DB have to be aware of the sharding - interactions between users who are housed on different shards etc could require a lot of work at the application layer. If your customers can be easily be split into tenants which never interact with each other this isn't so bad but for a consumer app like DoorDash there isn't clear tenant boundaries.

We looked at all this for Kami and realised that it would be much easier for us to move from PostgreSQL to CockroachDB (we had exceeded the write capacity of a single PostgreSQL primary) than to shard Postgres, and it'd make future development much faster. We could have made sharding work if we had to... but it's not 2013 any more and we have distributed SQL databases, why not use them?

These are also good points you've raised.

On scaling, yeah a single postgres server can handle a lot. For us we were well past the million user mark before running into serious issues. However, a lot of how we were able to keep postgres working for us as we grew was by shifting work from postgres to our stateless services like I alluded to before - e.g. making our SQL queries as simple to execute as possible even if it means more work for the client to piece the parts back together.

If everything had been running inside the database we wouldn't have had that option and we'd probably have hit scaling limits much earlier - I guess we could have split off the traffic to the highest traffic endpoints and have those handled by a separate service calling the PG db, but then you get into issues with keeping the authentication etc consistent.

Re security - yep, PG is already using C to parse untrusted inputs from the network, which is also scary, but it's (hopefully) well reviewed and mature code - and even so, I wouldn't want to expose PG's usual wire protocol port to the internet, so it's hard to imagine exposing HTTP from postgres to the wild west.

Ultimately it probably is just a question of the sort of project it's being used for - if it's for something that's not going to get need to get to larger scales, handle a lot of complexity over time, or pass security reviews and your main goal is simplicity, then maybe an approach like this is a good option. I've just found that things tend to start off looking small and simple and then turn out to be anything but, so I'd rather run `rails new` and point it at a standard PG server - which would be just as simple and productive when you are starting out, and can keep scaling as your customer base and team size grows up to the size of Shopify, Github, or Kami (shameless plug).

This is an interesting project, but I'd be unlikely to use it, for a couple of reasons:

- In my experience the performance of the stateful DB server has been the biggest bottleneck when scaling - it's much easier to scale the stateless application servers which sit between end user requests and your DB in a traditional architecture. So usually I'm wanting to move as much work as possible away from the DB in order to squeeze the most out of it before needing to shard the DB or move to a different solution, rather than moving more responsibilities into the DB.

- It's frankly pretty scary to load a C extension into postgres which is opening ports and parsing requests etc - bugs in it could crash the server or open security holes, and if you were able to exploit a vulnerability you'd be able to grab any of the data in the DB and easily exfiltrate it. This would be less of an issue if using this for an internal service which isn't directly exposed to the internet, but it still could make it easier for an attacker to escalate their access. (This isn't a 'it should be rust' comment really, even if this was in rust it would still be pretty worrying).

- Even if think you only need simple CRUD actions, over time you tend to need more and more logic around those actions. Authentication, verification, triggering processes in other systems, maybe you make schema changes and need to adapt requests from old clients, etc. It's really nice to have a more heavyweight application server where you can implement that logic - I'm pretty skeptical that row level permissions, triggers etc will be able to cleanly handle all those as you add new requirements over time. This applies also to other tools for more directly exposing your DB ( e.g. PostgREST ). IMO starting off using a tool like this is really just setting you up to have to do a pretty painful rebuild later on.

Am I missing something here, maybe I have misunderstood the intended use case?

Interesting - after the initial load it's substantially slower for me

If you want something which runs on the Raspberry PI and can do airplay/DLNA etc to a USB DAC without any hacks, try RoPieee XL: https://ropieee.org/xl/

Quite a few databases support time travel queries, in particular Oracle has for years and CockroachDB has them also. We can query the state of a table as it was at any point in the last 72hrs.

We switched to CockroachDB for some of our most heavily loaded tables a few years ago - and the scalability and operational simplicity has been great.

I don’t have a like for like benchmark, but at least for writes - we’re running with 5 replicas of each range so that’s 5x the disk write i/o just from that.

Add in some network overhead and running raft etc, and you are going to be needing a lot more resources than for PG to maintain a similar throughput - but adding resources when you can scale horizontally is so much easier that the tradeoff is worth it in my experience.

Yes, it’s removed on newer models, though not across all the products yet

It does exist, where laws allow for it - in NZ we have Snowball effect which does this [1].

As others have said though, when there's so much early stage capital around and you could instead raise from a few angels or a early-stage VC (who will have a larger stake and may be more able to help with later rounds) equity crowdfunding isn't that attractive for most founders.

There are good businesses which for some reason can't access that funding which might show up on a platform like this, but don't see it being all that common.

1 - https://www.snowballeffect.co.nz/

I guess it's all relative, our largest cluster scales up to around 3000 VCPU and there's plenty of much larger ones out there.

You might be able to get better utilisation of your VMs by running a smaller number with more cores on each - I'd try 16vcpu nodes.

Is this zero-downtime? pg_upgrade with the link option is pretty fast, usually only taking a few minutes even with our >10TB database, but it does require taking the server down while it's running which is a major issue for many modern apps.

I'm loving the cockroachdb upgrade process for our cluster - you just do a zero-downtime rolling restart to the next major version, with the ability to roll back to the old version if you see issues, and then when ready allow the database to apply it's migrations internally to upgrade the data structures, which is also zero-downtime.

We had less local restrictions than most other countries until delta hit 4 months ago - since then we've had a 3-month lockdown in our largest city and huge impact to small businesses, and we still have citizens locked out of the country. Deaths are comparatively low though.

I ran a Asus ac-68u at home at home for years and was generally happy, but the speed bump from upgrading to a ax-86u has actually been very substantial even for ac devices.

BTW I’m using a mikrotik router, but I’m not sold on Ubiquiti for home use even though I’ve deployed plenty of them at our office - my impression is that while the management on them is nice and they may be better for handling large numbers of clients, the more high end home aps have long range external antennas and seem better at providing high speeds to a small number of devices. It’s hard to find many objective tests of that though…

Did you try with a PWA? Discoverability can be hard vs having an App Store listing but I wonder if that would satisfy your user-base if you made it into a nice experience