l9i

Damien Miller

And I think Google has a backup IRC server on AWS, but that might just apocryphal.

They (>1) do exist but not on AWS or any other major cloud provider.

(Or at least that was the case two years ago when I still worked there.)

If you enjoy Organic Maps, please consider supporting them financially: https://organicmaps.app/donate/. Supporting their serving infrastructure for numerous users is surely costly.

Most outdoors activities are safer and more pleasant when there is some sunlight (think: walking through a forest, hiking, swimming in a lake, sports, having a picnic, etc).

Interesting that you are asking for the dirt given that DiRT stands for Disaster and Recovery Testing, at least at Google.

Every year there is a DiRT week where hundreds of tests are run. That obviously requires a ton of planning that starts well in advance. The objective is, of course, that despite all the testing nobody outside Google notices anything special. Given the volume and intrusiveness of these tests, the DiRT team is doing quite an impressive job.

While the DiRT week is the most intense testing period, disaster preparedness is not limited to just one event per year. There are also plenty tests conducted througout the year, some planned centrally, some done by individual teams. That's in addition to the regular training and exercises that SRE teams are doing periodically.

If you are interested in reading more about Google's approach to distaster planning and preparedness, you may be interested in reading the DiRT, or how to get dirty section from Shrinking the time to mitigate production incidents—CRE life lessons (https://cloud.google.com/blog/products/management-tools/shri...) and Weathering the Unexpected (https://queue.acm.org/detail.cfm?id=2371516).

A Gmail outage would be barely an inconvenience as Gmail plays a minor role in Google's disaster response.

Disclaimer: Ex-Googler who used to work on disaster reponse. Opinions are my own.

I found a comment that was factually incorrect and I felt competent to comment on that. Regrettably, I wrote just one sentence and clicked reply without providing any credentials to back up my claim. Not that I try to hide my identity, as danhak pointed out in https://news.ycombinator.com/item?id=28751644, my full name and URL of my personal website are only a click away.

I have replied to my initial comment with provide some additonal context: https://news.ycombinator.com/edit?id=28752431. Hope that helps.

I unfortunately cannot edit the parent comment anymore but several people pointed out that I didn't back up my claim or provided any credentials so here they are:

Google has multiple independent procedures for coordination during disasters. A global DNS outage (mentioned in https://news.ycombinator.com/item?id=28751140) was considered and has been taken into account.

I do not attempt to hide my identity here, quite the opposite: my HN profile contains my real name. Until recently a part of my job was to ensure that Google is prepared for various disasterous scenarios and that Googlers can coordinate the response independently from Google's infrastructure. I authored one of the fallback communication procedures that would likely be exercised today if Google's network experienced a global outage. Of course Google has a whole team of fantastic human beings who are deeply involved in disaster preparedness (miss you!). I am pretty sure they are going to analyze what happened to Facebook today in light of Google's emergency plans.

While this topic is really fascinating, I am unfortunately not at liberty to disclose the details as they belong to my previous employer. But when I stumble upon factually incorrect comments on HN that I am in a position to correct, why not do that?

The safe in question contained a smartcard required to boot an HSM. The safe combination was stored in a secret manager that depended on that HSM.

The engineer attempted to restart the service, but did not know that a restart required a hardware security module (HSM) smart card. These smart cards were stored in multiple safes in different Google offices across the globe, but not in New York City, where the on-call engineer was located. When the service failed to restart, the engineer contacted a colleague in Australia to retrieve a smart card. To their great dismay, the engineer in Australia could not open the safe because the combination was stored in the now-offline password manager.

Source: Chapter 1 of "Building Secure and Reliable Systems" (https://sre.google/static/pdf/building_secure_and_reliable_s... size warning: 9 MB)

That's not quite how it happened. ;)

<shameless plug> We used this story as the opening of "Building Secure and Reliable Systems" (chapter 1). You can check it out for free at https://sre.google/static/pdf/building_secure_and_reliable_s... (size warning: 9 MB). </shameless plug>

I can assure you that Google has a procedure in place for that.

If you find Andreas' work inspiring and/or useful, you may consider supporting him in pursuing his passion:

- https://github.com/sponsors/awesomekling/

- https://www.patreon.com/serenityos

- https://www.paypal.com/paypalme/awesomekling

(source: https://awesomekling.github.io/about/)

Thank you for letting us know Lucian! I shared your comment with our publisher (O'Reilly).

(disclaimer: I worked on the book)

Confirmed. I was hoping for the platypus but it turns out it was already taken anyway.

You can check out the whole O'Reilly menagerie at https://www.oreilly.com/animals.csp.

(disclaimer: I worked on the book)

In a somewhat snarky reply, I can assure you that the book release was planned long in advance, unlike the outages. ;)

(disclaimer: I worked on the book)

This book should be suitable for software engineers without security background. There are some sections that might require some knowledge but they are explicitly marked as Deep Dive.

(disclaimer: I worked on the book)

  Le*v*andowski != Le*w*andowski

(I'm Piotr Lewandowski.)

Most of the design principles covered by the first part of the book are quite generic and can be applied to a broad range of systems. If you are working on embedded systems, the Design for Recovery chapter might be a good start since it contains examples related to constrained enviornments.

(disclaimer: I worked on the book)

Thanks for pointing this out, we're on it (other formats have a similar problem).

(disclaimer: I work at Google)

We were very much aware that not all companies can afford to staff a dedicated security team. We tried to do our best to make sure that the book is applicable to a wider audience: from startups, to big corporations.

(disclaimer: I work at Google)

We did our best to highlight the intersection of security and reliability. I admit there might be more emphasis on security in some chapters, especially those that overlap with the previous two SRE books (that were exclusively about reliability). We wanted to avoid repeating ourselves.

(disclaimer: I work at Google)