Why prod is down

https://www.gruntwork.io/why-prod-is-down/

by napolux • 9 years ago

54 12 9 years ago

gruntwork.io

fhood• 9 years ago

My goodness, this is a work of art.

davidspiess• 9 years ago

It's scary how relatable it is

greatNespresso• 9 years ago

Is that real ?

dsr_• 9 years ago

Yes, those are real words written by a real person, stored on a real server and delivered to you by a real Internet.

A better question might be: "Is this really a copy of a letter written by a particular person to another particular person describing a particular incident?" to which the answer is "No."

An even better question might be: "Do the problems outlined in this document really occur?" to which the answer is "Yes, frequently."

gozur882• 9 years ago

Worst of all, our database is not backed up, so we lost our data backed up, but the backups don't work, so we lost our data sharded, so we lost our data twice NoSQL, so we're webscale, and we've been losing data for years

Heh. What I keep running into at various companies: "Backed up, but for some reason the restore is going to take three days (if it works), and it will be faster to tap original sources.

Why are we doing backups, then? Company policy.

itsjustjoe• 9 years ago

Because backing up stuff is super easy, but no one actually tests restoring data until they need to do it live.

apple4ever• 9 years ago

We'll do it live! We'll write it and we'll do it live!

gozur882• 9 years ago

It adds to the excitement.

Sgoettschkes• 9 years ago

No one ever got fired for doing backups. It's "industry standard" so if you don't do it, you'll be out of your job faster than the server which went down. Backups being actually restorable in a reasonable amount of time isn't "industry standard", so nobody cares.

sdfjkl• 9 years ago

The underlined placeholders aren't inputs and cannot be filled out.

Also missing are a few options on what to do, including: "Let it burn", "Throw more gasoline at it", and "Quit while you still have some sanity left".

agotterer• 9 years ago

Somewhat related, has anyone used gruntworks? Thoughts?

MarkMMullin• 9 years ago

We failed over but there was nothing there so we can't fail back -- eek, we're dead :-)

Why prod is down

Related Stories