Amazon goes down; takes S3, Salesforce, Target, and others with them.

http://www.techcrunch.com/2009/12/23/amazon-down/

by Sam_Odio • 17 years ago

78 46 17 years ago

techcrunch.com

Sam_Odio• 17 years ago

It looks like a DNS issue, since while http://amazon.com isn't working for me http://72.21.207.65 is (kind of).

EDIT: "No A records were found for amazon.com" http://www.zoneedit.com/lookup.html?host=amazon.com&type...

ggrot• 17 years ago

I wonder how much money amazon loses per minute 2 days before christmas. Ouch.

alain94040• 17 years ago

At $20B annual, assuming Christmas is a big chunk, I'm guessing $2B (10%) in the last two weeks, 2 minutes would then be worth $200,000?

I think my 10% number is too low, so maybe $500K for two minutes? But this is not profit, just revenue.

wglb• 17 years ago

Do you think it might catch up once it is available again? It is a pretty serious shopping destination, and possibly people retry.

rlpb• 17 years ago

I don't think that Amazon itself will lose much, since they're well-known enough that most people who fail to reach them will retry later.

I think the small fry using S3 and EC2 will be the ones who are actually hit by this.

cookiecaper• 17 years ago

I don't know, I don't think it'd be as much as 4 - 7 days before Christmas. Most people know that it's too late to order from Amazon or any other online-only store by the night of Dec. 23. It's probably more money than they'd lose normally, but maybe not that significant?

I have no data to back that up, it's pure speculation.

justinsb3• 17 years ago

It is DNS. If you put the EC2/S3 address into /etc/hosts, the services work fine. Affecting lots of other big websites as well apparently (target, salesforce) because they all outsource to UltraDNS

shaddi• 17 years ago

NANOG chatter confirming it is an issue with UltraDNS. Seems to be west coast related.

EDIT: Potentially a DOS attack. From NANOG: "We have some DNS providing type customers (not UltraDNS) receiving a few million packets/sec of UDP/53 DoS traffic, starting at about the same time as the UltraDNS problems. No clue if it's related, but it certainly sounds suspicious. :)"

aristus• 17 years ago

I'm very surprised that they rely on a single vendor. But I guess DNS is one of those things you don't think about until it fails.

metachor• 17 years ago

The point of using UltraDNS is that it they provide fast "real-time" failover of DNS routing. This is used, for example, for fault tolerance where the IP address responding to a domain name might change due to failure scenarios or load balancing (where a different server is now primary responder to the domain name, maybe located in a different data center or country). Infrastructure-wise, UltraDNS is kind of like the Akami of DNS, instead of content distribution.

cperciva6• 17 years ago

And people told me I was crazy for hard-coding IP addresses for s3.amazonaws.com and sdb.amazonaws.com into the Tarsnap code... :-)

justinsb3• 17 years ago

Awesome ... was that for security or for redundancy? Did you have a solution for what would happen if the IP address changed?

cperciva6• 17 years ago

Mostly security -- DNS is one of the worst offenders when it comes to protocols with security problems, both in terms of protocol issues and bugs in implementations.

Amazon doesn't move endpoints very often, and I round-robin requests to several endpoints; so in the rare cases an AWS endpoint changes I see a slight decrease in Tarsnap performance and can make the Tarsnap server stop using the dead endpoint before any users are likely to notice.

akl• 17 years ago

You are crazy for doing that - one instance of downtime doesn't justify ignoring all the advantages DNS brings.

cperciva6• 17 years ago

When I made that design decision I wasn't considering the possibility of DNS outages at all; I was just thinking in terms of "there's a huge number of places between me and ultradns where someone could insert a spoofed DNS response".

derefr• 17 years ago

Are there DNS servers that support versioning? The best solution I could imagine would simply be to set Tarsnap to normally use the current DNS records for S3, but be able to rollback to a valid zone record if they encounter an update that makes the servers stop resolving.

baq• 17 years ago

you can expect such downtimes every Christmas. why? ask yourself how much protection money UltraDNS was told to pay.

justinsb3• 17 years ago

I thought DNS was supposed to try backups servers automatically... any DNS experts able to explain what's going on? Some of the ultradns servers are returning (correct) values, others simply not responding.

lsc3• 17 years ago

yeah, uh, if you are smart, you have a secondary DNS provider. But that really requires you managing it yourself. the problem was that many people outsource, which usually means going with only one provider. (now, ultradns does have a good setup, they probably aren't a bad choice for a provider, but having only one is just plain stupid.)

tinio• 17 years ago

It looks to be back up now.

notmyname• 17 years ago

surely they mean "Amazon goes down and takes the Internet with it"

slig• 17 years ago

Only if TC was hosted there.

newhouseb3• 17 years ago

Why wouldn't such a company run their own name servers? I understand it's "yet another thing to maintain," but I've set up bind before... didn't seem that bad.

rbranson2• 17 years ago

It's relatively trivial to outsource. It's one of those services that's easy to measure, quantify, and manage (from an outsourced perspective). There's also a bit more to it then that. The Anycast routing can be quite difficult to setup and maintain. It's virtually useless outside of a very small set of protocols (DNS being one of them), so it wouldn't make sense for Amazon to bring that kind of talent in-house for something like DNS.

lsc3• 17 years ago

the mistake was outsourcing to only one provider. it's easy enough to setup a BIND slave elsewhere that automatically transfers the zone from your primary provider.

kierank• 17 years ago

The Anycast routing can be quite difficult to setup and maintain.

Anycast's not particularly difficult to setup and maintain.

evgen• 17 years ago

This is one of those services that a dedicated provider can sometimes do better than internal IT. Ultradns, for example, has secure secondaries Colocated with large ISPs so you get some good protection against cache poisoning attacks. Everything they do you could do yourself, but it would cost you a lot more than what they charge. (full disclosure: I am a customer and until this evening I was quite happy with their service and reliability)

newhouseb3• 17 years ago

"Colocated with large ISPs so you get some good protection against cache poisoning attacks"

Ah yes, that's not something I factored into my equation. Thanks!

jseifer• 17 years ago

It doesn't say in the TC article and I can't really tell from this thread. How long was Amazon actually down?

boredguy8• 17 years ago

Never let truth get in the way of a good headline.

artagnon• 17 years ago

Haha.. true. What else can we expect from Techcrunch?

tybris• 17 years ago

Internet can be so fragile.

mattiss• 17 years ago

When Rackspace goes down and takes TechCrunch with them, the whole INTERNET goes down. When Amazon goes down, Salesforce, Target, and others...

Amazon goes down; takes S3, Salesforce, Target, and others with them.

Related Stories