Why S3 went down

http://status.aws.amazon.com/s3-20080720.html

by slackerIII • 18 years ago

144 36 18 years ago

status.aws.amazon.com

sanj• 18 years ago

"By 11:05am PDT, all server-to-server communication was stopped...

By 2:20pm PDT, we'd restored internal communication..."

I wonder why it took 3 hours to clear state.

cperciva2• 18 years ago

I wonder why it took 3 hours to clear state.

It didn't -- it took 3 hours to get all the machines talking again after the state was cleared.

This is probably mostly due to two factors: 1. When doing a "clean restart" Amazon almost certainly had each S3 node look at the data it had stored to make sure that they had correct metadata; and 2. After each node had been restarted, Amazon probably had to relink them gradually rather than all at once -- most self-organizing network protocols have limits on the rate at which nodes can join or leave the network.

curiousgeorge• 18 years ago

Data integrity testing?

mixmax• 18 years ago

They get a lot of points in my book by providing a reasonably detailed description of the event instead of delivering the usual non-informative yada-yada.

sgrove• 18 years ago

I think amazon is approaching this in the right fashion - when you have a major outage, show that you understand exactly what the issue is, and why it won't happen in the future. That's quite a while to be down, but their openness, and what I perceive as honesty, helps me trust that they know what they're doing. And like mixmax said, their detailed description and timeline scores them extra points, instead of the usual corporate nothingness. We're still planning on moving over to "cloud-computing", or "infrastructure-outsourcing", and AWS still seems to be the leader in the field.

gaika• 18 years ago

That's why git is using checksums as commit labels. It forces consistency checks at every step.

mechanical_fish• 18 years ago

On Sunday, we saw a large number of servers that were spending almost all of their time gossiping and a disproportionate amount of servers that had failed while gossiping.

This is my favorite sentence of the day.

Now every time I see that damned whale on Twitter I'm going to find myself involuntarily crying out, "Oh, no! Failed while Gossiping!"

alabut• 18 years ago

The converse is true too, when Twitter is up and running just fine. I'm going to think of that every time I tweet - I succeeded at gossiping and failed at working.

ROFISH• 18 years ago

A single bit was flipped? Cosmic radiation comes back and us all in the rear! (Of course there's more reasonable reasons, but cosmic radiation is always my favorite. :D)

DougBTX2• 18 years ago

That's a great bug, very nice "illness"-like behaviour. The spread of a corrupt bit, infecting and taking down the host.

jacobbijani2• 18 years ago

Yeah, it almost read like a description of a virus.

briansmith• 18 years ago

That is why gossip protocols are also called "epidemic."

tdavis• 18 years ago

Loose lips sink ships!

(sorry, I had to...)

brfox• 18 years ago

What was Amazon's response when their storefront went down a couple months ago? Were they this upfront about it?

sh1mmer• 18 years ago

I think this is possible with developer relations.

Trying this approach with consumers could backfire horribly if someone misunderstands the details or tries to twist them out of context.

That's a lot hard to do with a developer audience.

beaudeal• 18 years ago

i agree with mixmax in that i really appreciate them being open and forthcoming about the problems, and giving detailed explainations -- on the flipside, im wondering what the outage will do to potential customers and those who are currently using the service...its a bit scary to think that they are hosting some very serious businesses and everyone went down for many hours

holygoat• 18 years ago

This is exactly the kind of RFO I would like to get from our telephony providers after an outage. Detailed, honest (no point lying -- you just had an outage!), and taking steps to improve matters in the future.

Everyone has outages: it's what you learn from them that counts.

marijn• 18 years ago

Downtime? It was only an 'Availability Event', thank you very much. But yes, I like that they provide details.

charlesju• 18 years ago

I think that while it is unfortunate that Amazon went down, they handled the situation properly. I'm sure that within a couple of years they'll hammer out all the kinks in the system.

newton• 18 years ago

Can I point out that this explanation did not identify a root cause? How was the corrupt message originally produced?

If they know how it happened, it's not reflected in this article. The solution they describe addresses the detection of and recovery from future mysterious occurrences rather than identifying, understanding and eliminating whatever bug or condition caused this one.

attack• 18 years ago

Memory corruption.

jordyhoyt2• 18 years ago

Right, it likely wasn't a bug if it is a single bit and this only happens once in a blue moon. A machine can only transmit perfect 1's and 0's for so long before getting one wrong.

pjf• 18 years ago

Is it only me reading the message as "we don't actually know the reason, but in an act of desperation we rebooted all of our servers and hopefully this wont happen again"?

kaens• 18 years ago

FTA: "We use MD5 checksums throughout the system, for example, to prevent, detect, and recover from corruption that can occur during receipt, storage, and retrieval of customers' objects. However, we didn't have the same protection in place to detect whether this particular internal state information had been corrupted. As a result, when the corruption occurred, we didn't detect it and it spread throughout the system causing the symptoms described above."

drawkbox• 18 years ago

Someone checked in an infinite recursive loop.

Why S3 went down

Related Stories