ChatGPT outage for 3 hours – Ubuntu 22.04 had an automatic update to systemd

https://twitter.com/GergelyOrosz/status/1947745317237002441

by greatgib • 12 months ago

4 3 12 months ago

greatgib2• 12 months ago

For everyone here that doesn't understand the hate against Systemd. This is the usual kind of things with systemd, system randomly broken by bad opinionated code...

b3nto• 12 months ago

The Technology team’s post-incident investigation identified the primary root cause as a gap in environment controls. This control gap allowed an unsanctioned process to initiate an automated operating system update on production infrastructure where such updates should have been disabled.

source (link given in the twitter/X post): https://newsletter.pragmaticengineer.com/p/why-reliability-i...

The problem seems much more about operations and process than anything else. Letting a third party (your OS provider/default config) decides when and how your production systems are updated seems quite unreasonable (for many many reasons).

I am not sure to understand why systemd is faulty here ? If you do believe systemd is faulty, why ubuntu would not be as faulty at least ?

greatgib2• 12 months ago

From their point of view the issue is that automatic update happened without each update to be approved and tested.

But the upstream issue not under their control is that the systemd update or behavior during update should not have broken things like that. Especially not with stable releases.

And to be clear, accident can happen but it is not the first time. It is quite common.

Once they totally bricked systems by accidentally wiping the nvram partition of computers.

ChatGPT outage for 3 hours – Ubuntu 22.04 had an automatic update to systemd

Related Stories