A Null Pointer Exception Brought Down Mighty Google;7 Hours of Downtime

https://www.getpanto.ai/blogs/09-07-2025/how-a-null-pointer-exception-brought-down-mighty-google-7-hours-of-downtime-explained

by pavan_panto • 1 year ago

9 5 1 year ago

getpanto.ai

malux85• 1 year ago

Very surprising that it was not feature flagged, when I worked at Google we went to enormous lengths to support partial feature rollouts, supporting this for complex codebases often added weeks to development but was crucial because an outage of this magnitude was considered sacrilege

Metalnem• 1 year ago

This is a terrible article. It doesn’t cite its sources and, even worse, invents quotes. It's basically just an ad for some Panto AI tool.

Save yourself some time and just read the official incident report: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

Or the previous Hacker News discussion: https://news.ycombinator.com/item?id=44274563

gnat• 1 year ago

Ugh. Each of these points is a classic reliability precaution – yet all were missed simultaneously. As one analyst put it, Google had “written the book on Site Reliability Engineering” but still deployed code that could not handle null inputs. In hindsight, this outage looks like a string of simple errors aligning by unfortunate chance.

Yes, that's how major outages happen. By this stage of maturity any single failure generally doesn't break things dramatically. When things go this wrong, it's ALWAYS a combination of failures: failure of recovery system, omission in detection systems, gap in automated review, oversight in ...

The vacuous gotcha language is indicative of the low quality of the whole article. As Metalnem says in comments here, see the official incident report for a better writeup and more insight. https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

jaymzcampbell• 1 year ago

I really love the "Swiss Cheese model" for showing this in a very explicit way, it's easy to see how the most improbably thing could happen.

https://en.wikipedia.org/wiki/Swiss_cheese_model

KevinMS• 1 year ago

fitting for the company that invented the nil pointer exception

A Null Pointer Exception Brought Down Mighty Google;7 Hours of Downtime

Related Stories