Amazon Web Services are down

http://status.aws.amazon.com/?a

by yuvadam • 15 years ago

560 328 15 years ago

status.aws.amazon.com

yuvadam2• 15 years ago

Current status: bad things are happening in the North Virginia datacenter.

EC2, EBS and RDS are all down on US-east-1.

Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent issues.

samratjp• 15 years ago

http://status.heroku.com/ for those on Heroku

mike-cardwell4• 15 years ago

Is it just me, or is their status page down?

guynamedloren2• 15 years ago

Indeed they are. Right before the issues began, I pushed a bad update to one of my Heroku apps, causing it to crash. A minute later I fixed the bug, re-pushed the git repo to Heroku... and nothing. I've been stuck with an error message on my website for hours. Unfortunate timing!

jrnkntl• 15 years ago

Staging servers are an easy thing on Heroku :)

tlholaday• 15 years ago

You have my sincere sympathy.

potomak2• 15 years ago

Also Cuorizini (http://cuorizini.heroku.com) is down!

wmoxam• 15 years ago

Not all EC2 & EBS instances are down. I have several in US-east-1a and 1 is down, while all of the others are working.

jedberg31• 15 years ago

Same with us. About 10% of our 700+ volumes are having problems right now.

It's hard to tell for sure since there isn't any load.

nostromo• 15 years ago

We're fine on EC2 -- but everything on RDS seems to be giving us big problems. We started a few backups before we new it was systemic, and all of them are stuck at 0%. We also tried spinning up new instances -- and they're all stuck in booting.

agotterer• 15 years ago

We had about 45 min of downtime around 4am EST. Our RDS instances, EBS backed and normal instances all returned without problems. We are in Virginia us-east-1a and us-east-1b.

obfuscate4• 15 years ago

I think which physical data center "us-east-1a" etc. corresponds to differs from user to user, to load-balance given that people will probably be more likely to use 1a than the other zones.

antonioe5• 15 years ago

Foursquare is up now. Quora not showing the 503 anymore, Reddit still down.

antonioe5• 15 years ago

I stand corrected. 4sq down again.

jedberg31• 15 years ago

We rely heavily on EBS still, so this is hurting us more than most others. Hopefully they'll have us back up soon.

bad_user5• 15 years ago

Foursquare is up, but Quora is still showing the 503.

powdahound• 15 years ago

I'm seeing 1 EBS server out of 9 having issues (5 in one availability zone, 4 in another). CPU wait time on the instance is stuck at 100% on all cores since the disk isn't responding. Sounds like others are having much more trouble.

vnorby• 15 years ago

From EngineYard: "It looks like EBS IO in the us-east-1 region is not working ideally at this point. That means all /data and /db Volumes which use EBS have bad IO performance, which can cause your sites to go down."

kylec• 15 years ago

I guess this is one Reddit outage that can't be blamed on poor scaling

thezilch• 15 years ago

Looks like troubles in only one availability zone.

mechanical_fish2• 15 years ago

That seems to be incorrect. We have problem children in us-east-1b, -1c, and -1d.

justinsb4• 15 years ago

AWS have now confirmed that this affects multiple availability zones. From the status page: "..impacting EBS volumes in multiple availability zones in the US-EAST-1 region"

StavrosK3• 15 years ago

Apparently most of their problems are caused by bad EBS writes/performance, or at least so they said a few weeks ago after some particularly bad downtime.

It looks like EBS will randomly decide to switch to a few bps of performance from time to time. I would use Amazon for my startup, but these issues really make it hard to justify.

muppetman6• 15 years ago

Slightly offtopic, but wasn't that post by an ex-employee as a comment? Not that the technical aspect of it wasn't fantastic, because it was, but I don't think Reddit said anything publically did they?

valisystem• 15 years ago

EBS seems to be the main problem here, I'll cite a former reddit employe (first comment on the blog that talked about EBS problems).

I don't work for reddit anymore (as of about a week ago, although I didn't get as much fanfare as raldi did), but I can tell you that they're giving Amazon too much credit here. Amazon's EBSs are a barrel of laughs in terms of performance and reliability and are a constant (and the single largest) source of failure across reddit. reddit's been in talks with Amazon all the way up to CIOs about ways to fix them for nearly a year and they've constantly been making promises that they haven't been keeping, passing us to new people (that "will finally be able to fix it"), and variously otherwise been desperately trying to keep reddit while not actually earning it.

Source: http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...

cache: - scroll down for comment - http://webcache.googleusercontent.com/search?q=cache:cfbs-sp...

crb3• 15 years ago

EC2 instances only have one network interface. The public IP address you have pointing to your instance is a DNAT done somewhere further up the chain.

If you get a large network load to your instance - say, a DDoS attack - you can find you no longer have enough network capacity to talk to your EBS disks.

This is what happened to Bitbucket in 2009: http://blog.bitbucket.org/2009/10/04/on-our-extended-downtim...

jedberg31• 15 years ago

Thankfully, no. :)

But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be able to come back quickly.

thirtysixred• 15 years ago

SHUT. DOWN. EVERYTHING.

Good plan though. :p

pvh• 15 years ago

Hey Jedberg, if you guys aren't already rolling your own, check out fdr's WAL-E tool. It bounces postgres write-ahead logs off S3 and goes great with the new PG9 replication.

https://github.com/heroku/WAL-E

stef254• 15 years ago

Amazon is really not being kind to you guys; I sort of hope you'll find an alternative solution fast!

hysterix2• 15 years ago

Sup jedberg, I obviously don't have nearly the level of knowledge with the intricacies of reddit, but coming from a strictly "business" standpoint, the amount of downtime reddit receives due to amazon issues is astounding. Perhaps it's time to look for alternatives?

Anyway, thanks for your time.

blhack3• 15 years ago

Will you guys ever do something like this:

http://cache-scale.appspot.com/c/www.reddit.com/

but officially supported (and paid for) by you?

Joakal• 15 years ago

How does Reddit display the 'offline' page if it's down?

jedberg31• 15 years ago

The server is still up, so we can serve it right out of the load balancer.

clistctrl• 15 years ago

I was to be under the impression a great deal of Reddits issues were linked to Amazon.

mcritz• 15 years ago

This feels the same way as hearing that the whole Internet just got shut down.

jedberg31• 15 years ago

Yes, they are. :(

griosa2• 15 years ago

no reddit at work today!

jedberg31• 15 years ago

Believe me, the last thing I want is to be up at 3am working on this. I'd much rather be sleeping and letting you not work.

athesyn• 15 years ago

My four-day weekend is already off to a bad start(UK here).

jedberg31• 15 years ago

Why do you get a 4 day weekend?

taitems• 15 years ago

It could be worse/better, you could have an Australian 5-day weekend.

G_Morgan4• 15 years ago

Mine is worse. I booked Tues-Thurs off. I only have internet in work at the moment. I'm going to miss reddit now and be without internet until I return to work on the 3rd of May. Stupid Sky and their stupid take forever switch overs.

mikecane• 15 years ago

Is it a 4-day holiday in Norway too?! Opera Support Forums have been sketchy for hours and if they won't come back until Tuesday, I'm SOL with my Opera problems.

hendi_5• 15 years ago

Yay for relying on the cloud \o/

jedberg31• 15 years ago

That's like saying "yay for relying on a datacenter"

hendi_5• 15 years ago

Note that I've written "relying", not "hosting", meaning I'm well aware of the fact that using "the cloud" as an additional layer of capacity can be a good thing.

But relying on the cloud means that you're giving up control.

Power outage in your data center? Buy diesel generators next time! Faulty network card? Put in two redundant ones!

AWS down? Wait and pray!

muppetman6• 15 years ago

Yes! It's like those dumb people that rent out 5 floors of a 100 story building, instead of building their own SkyScraper. Then, horror of horrors, 2 of the 8 lifts breaks down for 12 hours.

Such idiots for paying rent, even though building SkyScrapers isn't their core business.

jedberg31• 15 years ago

Thank you. I'm totally stealing this next time someone uses that "cloud is for idiots" line.

This is an excellent metaphor.

hendi_5• 15 years ago

Poor analogy; if the lifts break, just use the stairs.

Note that I've written "relying", as in "depending on the cloud without any alternative backup". Depending on the cloud means depending on something you cannot influence.

mootothemax• 15 years ago

What do you do when a building contractor cuts through your datacentre's link? For example, this happened in London a while back.

My point is: things go wrong. Blaming this on the cloud is silly.

elliottcarlson4• 15 years ago

It happened to ServInt back in 2004 - the VA Department of Transportation cut the MCI OC-48 and OC-12 fiber strands running to the ServInt datacenter. ServInt handled it amazingly well and learned from their lessons moving forward and if I recall correctly gave a comp'd month to it's customers. They followed up by adding additional backbones in addition to the MCI one and have always been solid since.

G_Morgan4• 15 years ago

It isn't about relying on the cloud. It is more about incompetent cloud providers.

oceanician• 15 years ago

all is fine on that service page now.

astrodust• 15 years ago

Does seem harsh to brand Amazon as incompetent. Things like this probably happen to Google's server farm all the time, but since they're the only customer they can and do reconfigure everything massively to adapt.

Amazon can't do that to your application that runs on a handful of systems.

mathrawka4• 15 years ago

I think this is a good example of how the "cloud" is not a silver bullet to making your site always up. AWS provides a way to keep it up, but it is up to each developer to ensure that they are using AWS in a way to make sure their site can handle problems in one availability zone.

I think we will see more of a focus from big users of AWS about focusing on how to create a redundant service using AWS. Or at least I hope we will!

tybris2• 15 years ago

NetFlix should sell their chaos monkey as a commercial product.

justinsb4• 15 years ago

Don't let AWS hear that, or they'll charge us for their failures by rebranding it as a feature.

jedberg31• 15 years ago

This outage is affecting all AZ's in the East. So even a multizone setup wouldn't help for this one. Only a multiregion setup.

This outage is a lot like having your entire datacenter lose power.

mathrawka4• 15 years ago

I thought AZs were supposed to be different physical data centers.

If that is not the case, then having a multi-region setup would be a necessity for any major sites on AWS.

Perhaps there will be a time where to truly be redundant, one would need to use multiple cloud providers. Which would be a _huge_ pain to do now I imagine, with all the provider lock-ins we have.

6ren• 15 years ago

If "multi-region" means North America, Europe and Asia Pacific, doing so would also improve world-wide latency (e.g. here in Australia...).

Could you use this outage to justify switching to multi-region?

justinsb4• 15 years ago

All well and good, but the elephant in the room is that multiple availability zones have failed at the same time. It looks like AWS have a single point of failure they weren't previously aware of.

potomak2• 15 years ago

Quora says: "We'd point fingers, but we wouldn't be where we are today without EC2."

olegp3• 15 years ago

Nice way to point fingers while saying you're not.

stevepotter• 15 years ago

Yes, that was by far my favorite comment. Well played.

smtm• 15 years ago

AWS/S3 has become the new Windows - great SPOF to go for if you want to attack. This space needs more competition.

dangrossman3• 15 years ago

Two years ago TechCrunch was publishing an article every time Rackspace went down listing all the hot startups down along with it. AWS is no more a SPOF than any other major hosting provider.

helium2• 15 years ago

I just launched a site on Heroku yesterday and cranked up the dynos up in anticipation of some "launch" traffic. Now, I can't log in to switch them off. Thanks EC2, you owe me $$$s

mike-cardwell4• 15 years ago

Isn't that a design flaw in Heroku? Shouldn't you be able to log into Heroku and change stuff like that even if the entire of Amazons cloud service is down?

Maxious2• 15 years ago

> Nothing special-case here: we deploy with git push, just like any other Heroku user. Dogfooding is good for you. http://blog.heroku.com/archives/2009/4/1/fork_our_docs/

SandB0x• 15 years ago

Agree. You can delegate work but not responsibility.

random423• 15 years ago

Actually, I'd expect Heroku to not charge for when the site was down, as they are clearly not available, it does not sound fair if they charge for it.

Am I expecting too much from them?

helium2• 15 years ago

My app on heroku is running, it's just that I can't log in to their management console to de-allocate resources that I am paying for by the hour.

dsl• 15 years ago

4/21/2011 is "Judgement Day" when Skynet becomes self aware and tries to kill us all. http://terminator.wikia.com/wiki/2011/04/21

I am just a little freaked out right now.

jedberg31• 15 years ago

Don't worry. If skynet is in EC2, we'll be fine.

HelloBeautiful• 15 years ago

You don't understand, Skynet is using all Amazon resources, hence the outages ;-)

Amazon have stated many times that amazon.com itself runs mostly on AWS platform, but it works fine now ...

crb3• 15 years ago

Do we all regret letting GLaDOS reboot yet?

crb3• 15 years ago

Sigh, Portal jokes were so much more popular on Reddit. :)

iamwil• 15 years ago

I wonder if it's a virus or worm whose activation date was today due to the fact above, found its way into the amazon servers.

Probably just a coincidence.

oomkiller2• 15 years ago

They better start writing their explanation now. Multiple AZ's affected?

jedberg31• 15 years ago

They better be cutting me a check too.

ig13• 15 years ago

As an advertiser on Reddit, you guys better be cutting me a check too :)

(although my app is also dead right now so not as if the advertising would do me any good...)

frekw• 15 years ago

It's a bit ironic that Amazon WS has become a SPoF for half the internet.

ig13• 15 years ago

A couple of hours into the failure, and no sign of coverage on Techcrunch (they're posting "business" stories though). It shows how detached Techcrunch has become from the startup world.

Edit: I tweeted their European editor about it and he's posted a story up now.

dwc3• 15 years ago

Perhaps this isn't really news. These days it's normal.

dwc3• 15 years ago

It's ugly, but true enough. You don't have to like it to acknowledge it. It's just another cloud outage bringing down one or more high profile sites. It's a "dog bites man" story.

mathrawka4• 15 years ago

So do we get some credit on our AWS accounts? I haven't really read their SLA for EC2.

elliottcarlson4• 15 years ago

"In the event Amazon EC2 does not meet the Annual Uptime Percentage commitment, you will be eligible to receive a Service Credit as described below."

http://aws.amazon.com/ec2-sla/

dpogg12• 15 years ago

I did the math on it, with the downtime so far it's almost approaching the magic 99.95% barrier where everyone gets a 10% bill credit. Wouldn't that be something to light a fire under the collective arses of those in charge of keeping EBS stable.

lylejohnson• 15 years ago

> Wouldn't that be something to light a fire under the collective arses of those in charge of keeping EBS stable.

Yeah, the only possible explanation is that Amazon's put a bunch of lazy guys in charge of EBS. How hard a job could it be?

antonioe5• 15 years ago

To receive a Service Credit, you must submit a request by sending an e-mail message to aws-sla-request @ amazon.com.

espeed2• 15 years ago

Quora is down, and evidently "They're not pointing fingers at EC2" -- http://news.ycombinator.com/item?id=2470119 -- I was going to post a screen shot, but evidently my Dropbox is down too.

jwr• 15 years ago

It's really mostly EBS failures, so the title is overly dramatic. And EBS has been known to have issues.

dogas• 15 years ago

This is incorrect. EBS is part of the issue, as well as intermittent connectivity failures for EC2 instances.

We're also seeing connectivity issues with our elastic load balancers.

bad_user5• 15 years ago

I've got 2 EC2 instances with EBS root devices -- both are unreachable. My RDS instance is responding, but it is very slow -- struggling to make a backup.

espeed2• 15 years ago

http://www.information-age.com/channels/the-cloud-and-virtua...

olegp3• 15 years ago

Assuming the problem is indeed with EBS, I would say this should be a warning sign to anyone considering going with a PaaS provider, which Amazon is quickly becoming, instead of an IaaS provider like Slicehost or Linode.

The increased complexity of their offering makes it more likely that things will break, leaving you locked in.

I did a 15 minute talk on the subject, which you can check out here: http://iforum.com.ua/video-2011-tech-podsechin

EDIT: here are the slides if you can't bother watching the video http://bit.ly/eqDNei

ghshephard• 15 years ago

Every time someone makes the claim that downtime should be a warning sign about going with a PaaS provider (or, indeed, an IaaS provider, or in some cases, people even make this claim about going with someone else's Data Center) - I always respond: "And why do you believe that you would do any better?"

Every environment I've been involved in as an operations professional for the last 15 years has experienced downtime regardless of how much we invested in staging, testing, change control procedures, redundancy and ITIL methodology.

While it is true that there are some environments in the world that experience no, or close to no downtime (The NYSE, google.com, the Space Shuttle) - it's usually not worth the investments those organizations make in ensuring 100% uptime.

I'm more interested in seeing what the monitoring methodology is, what the response protocols are, and what the reported downtime has been over the last year has been, than I am about getting too concerned about the occasional outage.

bad_user5• 15 years ago

Lets hope they don't lose any of my data.

olegp3• 15 years ago

The problem with Amazon is that despite touting an open API, their infrastructure internals and practices are a trade secret, so the likes of Eucalyptus are having to play catchup. In other words, I cannot replicate their infrastructure in my own data center, even I had the money to pay them. I suspect this is the main reason that Heroku didn't move off Amazon, and not the fact that Amazon was providing them great value for money.

There is definitely value in platforms, but those platforms should be built on completely open standards with no vendor lock in or influence, as is the case with OpenStack for example: http://www.theregister.co.uk/2011/02/10/rackspace_buys_opens...

If you didn't watch my talk, I should point out that I'm working on Akshell (http://www.akshell.com), which is itself a platform provider, so I am in fact agreeing with you.

billswift• 15 years ago

Maybe that's true for an individual, but from a broader (societal/economic) view this is a bigger problem, because it affects many more people.

mechanical_fish2• 15 years ago

Yep.

The thing to remember is that even if you could do better -- and it is certainly possible -- your customers might not pay you for it. It will cost you, a lot, to build a significantly better system than AWS. Reliability is about redundancy, and redundancy is synonymous with paying for things that sit idle for years at a time, and paying for the hardware and personnel to run disaster drills over and over, and customers hate that. Reliability is also about avoiding excessive complexity and limiting the rate at which you change things, and customers hate that too. If they wanted highly reliable, slow-changing established tech they'd be using land-line phones, not Quora or Reddit.

lobster_johnson• 15 years ago

I agree with your sentiments, but this:

> I always respond: "And why do you believe that you would do any better?"

…is an apples-to-oranges comparison that implies that the thing that Amazon is currently doing is what you would be doing in an Amazon-less scenario. It's not.

Amazon hosts thousands of customers and needs to service all of them in the same infrastructure, which coincidentally is also shared between every customer by virtue of being virtualized. It's a complex structure in which every resource -- CPU, RAM, disk, network -- is shared, which means that even a minor EBS problem could potentially have a butterfly-effect propagating through multiple customer's hosts. There are limitations imposed by the structure; for example, the only way to access large amounts of block storage with local-like performance is through EBS. But EBS has been shown to be high-latency and flaky, and Reddit's chronic problems with EBS is a good example why it one probably should stay away from it entirely. Network latency on EC2 is also pretty horrendous compared to classic non-virtual setups; I have set up HAProxy clusters where the connection setup time to backend hosts would be measured in tens of milliseconds.

Now, if you hosted your own stuff, you would be the only customer, and you would not be sharing resources with anyone else, and you could design your hardware exactly to your specifications (fast local disks with low latency, fast networking with low latency and so on). The difference in complexity is significant. There are tons of challenges and costs involved in hosting your own stuff and maintaining uptime, but the complexity equation is different.

justincormack2• 15 years ago

I dont see EBS as a very PaaS type service. It is a virtualised network storage volume with snapshotting, just what you might expect from an IaaS provider. Clearly it is one of the more complex parts of the Amazom infrastructure though (other providers dont provide anything similar). If you dont use EBS many applications would have to implement a lot of its functionality themselves, and I suspect this would lead to some data loss instead, when drives fail or instances fail leaving drives unavailable.

You can avoid EBS if you use local drives and s3, it is just difficult to run a database like this if it was not built that way.

antonioe5• 15 years ago

Had our blog go down. Didn't realize it was AWS wide..did a reboot. Now I am in reboot limbo. Put an urgent ticket into Amazon. They just said they are working urgently to fix the issues. Let's see how long this goes.

xlevus• 15 years ago

Ruh Roh. The service I'm using to acquire accommodation seems to be dependent on AWS. Guess I'm going to be homeless tomorrow if it doesn't get fixed. :X

Note to self. Don't ever build a service reliant on AWS.

crashpadder• 15 years ago

If its us (crashpadder.com) - sorry for the inconvenience. Email us hello [at] crashpadder.com and we'll do all we can when we're back up.

We'd moved to AWS from an even worse host a few months ago, and until this morning had been pretty impressed...

asymptotic3• 15 years ago

Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region.

Amazon's EBS SLA is less clear, but they state that they expect an annual failure rate of 0.1-0.5%, compared to commodity hard-drive failure rates of 4%. Hence, if you wanted a higher level of data availability you'd use more than one EBS volume in different regions.

These outages are affecting North America, and not Europe and Asia Pacific. That's it. Why is this even news? Were you expecting 100% availability?

stef254• 15 years ago

Reddit's been down for several hours today, I'm sure they are already way lower 99.95%.

JonnieCache6• 15 years ago

0.05% of one year is 4 hours, 22 minutes and 48 seconds.

asymptotic3• 15 years ago

If the Reddit web server admins took availability seriously they would have chosen to deploy across more than one region. Do you disagree? Why do you disagree? I'm being honest, no snark involved in my questions.

tedjdziuba3• 15 years ago

Reddit being down is not news.

blhack3• 15 years ago

http://cache-scale.appspot.com/c/www.reddit.com/

That's a cache of it. I really wish that the admins at reddit would implement something like this themselves, then link to it when downtime like this happens.

hoop5• 15 years ago

    Amazon's EC2 SLA is extremely clear -
    a given region has an availability of 99.95%.
    If you're running a website and you haven't
    deployed across across more than one region then,
    by definition, your website will have 99.95%
    availailbility. If you want a higher level of
    availability use more than one region.

Good point.

let P(region fails) = 0.05% and let's assume (and hope) that the probability of failure of one region is independent of the state of the other regions.

P(two regions fail) = P(one region fails and another region fails) = P(region fails) * P(region fails) = 0.05% * 0.05% = 0.0025%

Making your availability = 100% - 0.0025% = 99.9975%

Ultimately it's more of a business decision if you want to pay for the extra 0.0475% of availability. I would think (or hope) that most engineers would want it anyway.

The numbers at this size appear insignificant. How would one (say an engineer) convince "the management" that the extra 0.0475% of availability is worth the investment/expense?

gphil• 15 years ago

> The numbers at this size appear insignificant. How would one (say an engineer) convince "the management" that the extra 0.0475% of availability is worth the investment/expense?

In businesses where physical goods are sold to customers, "the management" is generally very motivated to avoid stock-out situations in which sales are lost due to lack of inventory (even if it's only a very small percentage.) The reason for this is because they are concerned about the potential loss of customer goodwill. It seems that the same applies in this situation.

bartonfink• 15 years ago

Why wouldn't a simple expected value calculation work? You've shown that you can calculate the extra availability that subscribing to another region provides. Simply multiply the cost of an outage by the extra availability provided by an additional region that would have prevented that outage.

If expanding to another region costs more than just taking the outage, then it's categorically not a good option. If management still says no in the face of numbers that suggest yes, then that tells you that you're missing a hidden objection, and how you proceed will depend on a lot of factors specific to your situation.

gairbheil• 15 years ago

No, your decimal point is off. 0.05% * 0.05% = 0.0005 * 0.0005 = 0.00000025, or 0.000025%. It works out to an expected downtime of 8 seconds per year, instead of over 4 hours for one location.

Of course, redundancy doesn't set itself up, so there are added costs on top of Amazon.

yatima2975• 15 years ago

Minor nitpick, but the availability should be even better, since 1% * 1% = 0.01% the availability becomes 99.999975% - six nines, anyone?

jerf• 15 years ago

"let's assume (and hope) the probability of failure of one region is independent of the state of the other regions."

In practice, that's not true, and it's not true enough to ruin the entire rest of your calculations. For Amazon regions to function independently, they'd have to be actually, factually independent; there is no interaction between them. The reaction to one node going down is never to increase the load on other nodes as people migrate services, etc. There's fundamentally nothing you can do about the fact that if enough of your capacity goes out then you will experience demand in excess of supply.

If you want true redundancy you will at the very least need to go to another entirely separate service that is not Amazon... and if enough people do that, they'll break the effective independence of that arrangement, too.

(This is a special case of a more general rule, which is that computers are generally so reliable that the ways in which their probabilities deviate from Gaussian or independence tends to dominate your worst-case calculations.)

bdonlan• 15 years ago

Note also that 0.1-0.5% refers to irrecoverable data loss, not temporary unavailability.

jws• 15 years ago

Silver lining: Hopefully I can test my "aws is failing" fallback code. (my GAE based site keeps a state log on S3 for the day when GAE falls in a hole.)

G_Morgan4• 15 years ago

This code should be well tested by now. Amazon is doing you a failure by being rubbish.

Jun8• 15 years ago

This is like witnessing your parents having sex while a kid: you sort of knew this is a possibility but it is a devastating blow to your belief system nevertheless.

The amount of services I use that depends on Amazon is amazing. They have really become the utility company of the Web.

smackfu2• 15 years ago

So when big sites deal use Amazon Web Services for major traffic, do they get a serious customer relationship? Or is it just generic email/web support and a status page?

mleonhard2• 15 years ago

There are subscriptions for various levels of support, from $50/mo for 12 hour response time to $15k/mo for 15 minute response time.

http://aws.amazon.com/premiumsupport/

smackfu2• 15 years ago

Interesting pricing. The Platinum seems priced to have no one use it, considering how much of a jump it is over Gold.

paraschopra• 15 years ago

http://venuetastic.com/ - feel bad for these guys. They launched yesterday and down today because of AWS. Murphy's law in practice.

tedjdziuba3• 15 years ago

They are scaling in the cloud, at least.

hysterix2• 15 years ago

Wow. I can only imagine the intense frustration the site owner must be feeling right about now. Makes you really stop and question the whole "cloud" based service. Or at least should make you realize you need fall-backs other than the cloud service itself.

swedegeek• 15 years ago

In case anyone is late to the party and missed the non-green lights on the AWS status dashboard, here is the page as of about 9:30 EDT...

http://screencast.com/t/p69xAoDJRSer

ck2• 15 years ago

So what percentage of the top 1000 sites are now crippled by this?

dmuth2• 15 years ago

Being unable to get much done here, my co-workers have found other things to do in the office: http://www.youtube.com/watch?v=u1-oGxDHQbI :-P

jedberg31• 15 years ago

What did I just watch and why?

tybris2• 15 years ago

All hosting services go down occasionally. If you want to stay up you need to build a fault-tolerant distributed system that spans multiple regions and potentially multiple providers.

Also, Amazon should fix EBS.

kennethologist• 15 years ago

Thankfully, my major clients are using the Asia EC2 Instances!

timf• 15 years ago

Some quotes regarding how Netflix handled this without interruptions:

"Netflix showed some increased latency, internal alarms went off but hasn't had a service outage." [1]

"Netflix is deployed in three zones, sized to lose one and keep going. Cheaper than cost of being down." [2]

[1] https://twitter.com/adrianco/status/61075904847282177

[2] https://twitter.com/adrianco/status/61076362680745984

campnic• 15 years ago

"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.

jedberg31• 15 years ago

At their level of income, this is true.

For us, we are just now staffing up to the level where we can make the changes necessary to do the same thing.

darrenkopp• 15 years ago

Also, when there are service interruptions, they send out credits to customers.

xsmasher• 15 years ago

It's a strange algebra though; doesn't it mean the WORSE Amazon's uptime is, the more money you should give them?

nateberkopec• 15 years ago

Every decision in a business is like this - measure the cost of action A versus the cost of not-A. It's just rare that in this case, those costs are easily quantifiable.

dfranke• 15 years ago

I'm actually surprised if incurring 50% extra hardware costs really is cheaper than the cost of being down. If Netflix is down for a few hours, then it costs them some goodwill, and maybe a few new signups, but is the immediate revenue impact really that great? Most of Netflix's revenue comes from monthly subscriptions, and it's not like their customers have an SLA.

elliottcarlson4• 15 years ago

Are they only in three zones, or three regions? Three zones would not have helped them in this particular scenario and they would have still been at risk.

And if they do mean three regions - can that cost of spanning various regions be quantified for different companies. The money spent vs money earned for Netflix may be very different compared to Quora and Reddit. At the same time, the data synchronization needs in between regions may also vastly differ for different type of companies and infrastructures thus leading to varying amount of cost to maintain a site on multiple regions.

alexpopescu2• 15 years ago

More comments coming from Adrian Cockcroft:

1. See slides 32-35 of http://www.slideshare.net/adrianco/netflix-in-the-cloud-2011

2. "Deploy in three AZ with no extra instances - target autoscale 30-60% util. You have 50% headroom for load spikes. Lose an AZ -> 90% util."

https://twitter.com/#!/adrianco/status/61089202229624832

prakash• 15 years ago

Here's the 24h latency data on EC2 east, west, eu, apac: http://dl.dropbox.com/u/1898990/EC2-multiple-zones-24h.png

Last 60 minutes comparison data: http://dl.dropbox.com/u/1898990/EC2-multiple-zones-60m.png

time in GMT.

A study we (Cedexis) did in January comparing multiple ec2 zones and other cloud providers: (pdf) http://dl.dropbox.com/u/1898990/76-marty-kagan.pdf

jjm4• 15 years ago

Pure opinion: That convergence might show that Amazon tried to do a failover on a DC level. Once they figured that wouldn't work or east was down for the count they just let it cycle to the ground under latency.

olahav• 15 years ago

Yes - It is all business decisions. As someone said already an instance on AWS can cost up to 7X a machine you own on co-location. here is how outbrain manages it's multi datacenter architecture while saving on Disaster recovery headroom. http://techblog.outbrain.com/2011/04/lego-bricks-our-data-ce...

ig13• 15 years ago

Given that Heroku's parent company (Salesforce) owns a cloud platform, it seems kinda inevitable now that Herkou will perhaps sooner-than-later switch back-ends (or at least use both)

greaterscope• 15 years ago

Wish were able to download our ebs snapshots, which are supposedly hosted on S3. What does everyone else do?

dangrossman3• 15 years ago

I take the snapshots. I also have servers send backups to each other each night. I also have a nightly cron job run and rotate backups of the most critical databases to an external drive on my home network. A Tonido Plug does that job (Ubuntu on a tiny ARM server in a plug that costs virtually nothing to run).

Now, some of the databases are simply too large or under too much load to take a live backup while the sites are running. Those I run on Amazon RDS with the MultiAZ feature enabled. There should be two copies of the database running at all times, both servers keeping a 3 day binlog for point-in-time backups, and making a nightly snapshot to S3. I have to rely on Amazon for that.

But I still take daily home backups of the most valuable individual tables off those servers, like user registrations and payment records. Even if I can't have off-site backups of the whole database, I'll have off-site copies of the part I'd need most in case of an Amazon-entirely-offline catastrophe.

singlow• 15 years ago

It's definitely a limited outage. My three instances seem to have operated all night with no problem. Two of them are EBS instances.

alexpopescu2• 15 years ago

Instead of enumerating who's down, I'd be more interested to hear about those that survived the AWS failure. We could learn something from them.

dmuth2• 15 years ago

Holy crap. An Amazon rep actually just posted that SkyNet had nothing to do with the outage:

https://forums.aws.amazon.com/message.jspa?messageID=238872#...

piramida• 15 years ago

Today, April 21st 2011, according to the "Terminator", Skynet was launched... No wonder AWS is down

mtodd3• 15 years ago

Why is ELB not mentioned at all on the Service Health Dashboard?

We're experiencing problems with two of our ELBs, one indicating instance health as out of service, reporting "a transient error occurred". Another, new LB (what we hoped would replace the first problematic LB), reports: "instance registration is still in progress".

A support issue with Amazon indicated that it was related to the ongoing issues and to monitor the Service Health Dashboard. But, as I mentioned before, ELB isn't mentioned at all.

jen_h• 15 years ago

We've got a single non-responsive load balancer IP in one of our primary ELBs (others have been fine for several hours now), so while everything else for us is up & running, still have transient errors for folks that get shunted to through that one system.

The interesting thing about the ELB in a situation like this is that I believe it may, in many instances, be better to hobble along and deal with an elevated error rate if at least some of your ELB hosts are working than to re-create the entire ELB somewhere else, especially if you're a high-traffic site where you may hit scaling issues going from 0 to 60 in milliseconds (OMMV, but we've been spooked enough in the past not to try anything hasty until things get back to normal).

mtodd3• 15 years ago

We have an identical load balancer to one that is causing problems so we're lucky enough to reroute traffic through that one instead to get to the same boxes. (The boxes serve two different APIs through two different DNS CNAMEs so we split the ELBs for future and sanity). In this case, it's helped us out. Alternatively, we would've just routed all traffic to our west coast ELBs.

mtodd3• 15 years ago

Quote from the AWS support rep: "I can confirm that ELB has been affected by the EBS issue despite the lack of messaging on the AWS Dashboard".

hcentelles• 15 years ago

It seems like availability zone us-east-1c it's working, i can launch a EBS backed instance right now.

marjanpanic• 15 years ago

Amazon down - just one more reason to try out BO.LT and their amazing CDN and page sharing services...

Just launched: http://techcrunch.com/2011/04/21/page-sharing-service-bo-lt-...

antonioe5• 15 years ago

1:23EST and Reddit is back up. Quora/4SQ still down. My site still down.

jjm4• 15 years ago

Everyone talks about SLAs but I believe it doesn't consider the fact that the EBS vols are still up (not on fire, and available) and are phantom writing or that the network is queued up the wazoo so writes don't even happen in a timely manner as you'd expect.

elliottcarlson4• 15 years ago

I'm not sure that just because they are up, yet unusable, would negate an SLA.

You could have a dedicated server in a datacenter - if the network goes out, your machine is still up and happily waiting to serve requests - but it's still unusable and not actually in service.

pextris• 15 years ago

reddit.com is down, but luckily http://radioreddit.com is not.

Amazon Web Services are down

Related Stories