AWS is down due to an electrical storm in the US

http://status.aws.amazon.com

by aritraghosh007 • 14 years ago

239 161 14 years ago

status.aws.amazon.com

jaequery• 14 years ago

with storms getting worse each year, i'll likely be choosing either central or westcoast datacenters ... from now on.

freditup• 14 years ago

Tornadoes in the midwest and earthquakes in the west. You're doomed.

Seriously though, as horrible as downtime is, I think most internet users aren't terribly surprised when they can't go to a specific website for a short period of time.

jser• 14 years ago

Any press is good press? Can't count the number of times I read about the Twitter fail whale.

robryan• 14 years ago

For ecommerce, people will just got buy somewhere else. So not to big a deal for them but a really big deal to your bottom line. Especially when you are still paying for traffic acquisition, sure you can turn Adwords off in minutes but other traffic sources may take 24 hours.

SomeCallMeTim• 14 years ago

If you're willing to go outside AWS...Denver. We occasionally see tornadoes out here, but Denver is outside "Tornado Alley" [1], so they're pretty rare. On the WikiPedia tornado page they set the odds of a tornado hitting the downtown area of a big city at around one in a thousand per year.

If you want to be extra paranoid, you could always pick a secondary data center location where it would be safer in the summer, since Denver would be safe from tornadoes in the winter. [2] Or you could just put one on the West Coast and hope that there's not a 1-in-a-thousand tornado within a week of a big earthquake.

[1] https://en.wikipedia.org/wiki/File:Tornado_Alley_Diagram.svg

[2] http://www.weather.com/outlook/weather-news/severe-weather/a...

sliverstorm• 14 years ago

Tornadoes in the midwest and earthquakes in the west. You're doomed.

Yup, everywhere has got its natural disasters.

codex• 14 years ago

Pardon my rant, but I am frustrated. It seems there is always an excuse with Amazon cloud. Is Google similarly disabled?

kroo2• 14 years ago

Nope.

jordanthoms2• 14 years ago

I wonder how much of Google's reliability comes from service failover and redundancy and how much from very reliable datacenters etc. I'd find it hard to believe that their platform and DCs aren't better than what we have seen from AWS, which could make compute engine a very attractive product.

batista2• 14 years ago

If you mean GAE, its even worse...

jonny_eh• 14 years ago

I thought App Engine was much more reliable when they switched over from the Master/Slave Datastore to the High Replication Datastore.

icoloma• 14 years ago

I can only talk about my experience, but I've had zero downtime with GAE since we migrated to the HR datastore. Development is x1000 harder, but then everything Just Works.

RegEx3• 14 years ago

The status page seems to really underplay the severity of the situation. Netflix and Heroku are down, yet these are just side effects of 'performance issues' instead of a 'service disruption'. I wonder what it would take to cross that threshold.

adrianpike2• 14 years ago

AWS has historically been both slow to update and heavily optimistic with their status page.

When I got the frantic texts when EC2 first dropped offline, sure enough, the AWS status page was all green, but twitter was alight with people talking about it.

I suspect a service disruption would have to be Godzilla.

RegEx3• 14 years ago

According to other HNers, their RSS feed seems to be fairly accurate (someone please verify this), which makes the whole thing that much weirder.

joezydeco• 14 years ago

You called it. I had my money on the volcano.

http://www.youtube.com/watch?v=tVEPvXBEOSE

rmc2• 14 years ago

When our EC2 hosted website seems offline and AWS status page is all green, the first thing I do is check twitter. Usually you can see people asking about outage before amazon acknoledge it.

zacharyvoase• 14 years ago

By what stretch of the imagination is this icon suitable for representing a total loss of availability due to a power outage?: http://status.aws.amazon.com/images/status2.gif

Is this not a 'service disruption' situation? At the bottom of the page, the yellow icon is associated with 'performance issues'.

If there's one thing that's shocked me about AWS, it's the total failure to acknowledge the severity of service disruptions. Like the above case, or the fact that a 3-hour loss of connectivity is displayed on the service history as a green tick with a small 'i' box: http://oi46.tinypic.com/x5qtch.jpg

hristov• 14 years ago

It is obvious we are dealing with the imagination of a marketing exec here. And that is a sick cynical place.

flyt• 14 years ago

Or that there is absolutely no way to deep link to an ongoing outage, and users must reload, then expand the link every single time, or subscribe to an RSS feed.

AWS needs to blantantly copy Heroku's status system, which is worlds better for people needing fast updates on their infrastructure.

https://status.heroku.com/ vs http://status.aws.amazon.com/

droithomme• 14 years ago

Wow, that is so beautiful! I am in awe.

phillmv• 14 years ago

What's the deal with the % measuring? I know going below three nines is seen as "bad service", but three hours of outage in a month gives you 99.5%, not 99.97%

jdub• 14 years ago

They've reporting a power issue with a single US East availability zone. There are four EC2 availability zones in US East. Strange that they would cite orange as "performance issues", but it's certainly more appropriate than suggesting a complete service disruption.

freerobby• 14 years ago

It is absolutely affecting more than one availability zone. Just because people didn't lose boxes outside of one AZ doesn't mean there weren't service disruptions outside of it. I know multiple people, myself included, who STILL cannot launch new instances anywhere in US-East. The EC2 web console and API have also been down intermittently and throwing errors for hours.

underwater• 14 years ago

It's a terrible choice, but seems inspired by the triangle shape used for warning signs.

astrodust2• 14 years ago

Clearly Amazon hired Baghdad Bob as their PR guy when he was looking for a new gig.

Apart from Apple's legendary secrecy, Amazon's EC2 is a solid #2 in terms of impenetrability.

vl• 14 years ago

This is not fair, Amazon publishes surprisingly detailed postmortem reports for AWS problems.

datasage3• 14 years ago

They do have a red icon if an event impacts an entire region. If a customer is correctly utilizing multiple availability zones, a failure in one zone should only impact the customer until they can fail over (Should be within minutes if they are automated).

kajham• 14 years ago

Aren't multiple availability zones impacted though? I don't get how Heroku can be so widely impacted if its not for the fact that multiple Availabiltiy Zones have been impacted.

talonx• 14 years ago

But the icon is meant to display AWS status, not the customer's. Friday's outage did warrant a red icon - unreachable instances, corrupted EBS volumes, failed API calls.

rwl4• 14 years ago

Hey! At least Amazon.com is up!

zvrba• 14 years ago

An upwards triangle is used as a danger/warning traffic sign.

1SaltwaterC• 14 years ago

We had 0 downtime. The only thing that's screwed up is a read replica of a multi-AZ MySQL on RDS deployment. Amazon did not send any notification. Kinda annoying.

adrianpike2• 14 years ago

There's another comment thread going over at (http://news.ycombinator.com/item?id=4180339), if, like me, you got extremely lucky and picked today's lucky availability zones, and have time to read HN instead of scramble to get things back up.

Good luck, friends.

molecule• 14 years ago

reading the linked page, "AWS is down" means "some N. Virginia AWS services are down"

jamespcole2• 14 years ago

unfuddle is down also, can't access any of my repos. I was going to spend the day working too - oh well time for a long lunch.

This isn't the first time this has happened to AWS - we moved our app to linode last year after this happened to us and it seems to affect AWS more than any other hosting i've ever used, i'd be interested to know how their infrastructure is set up because it doesn't seem particularly robust.

lsb• 14 years ago

Interestingly, this is a great time to see which of your favorite websites are rock-solid and which are kind of shaky.

I've been thinking about building a site with a Parse backend, and they're up, which is good to discover.

jared314• 14 years ago

It's like looking for a house in the rain, so you can see where the water drains.

sehugg• 14 years ago

Be careful. Nothing is as it seems right now. Do not trust any API output, nor should you do any API operations that are non-recoverable. Things are up that are reported down and vice-versa.

Wait for the dust to settle. We're all just going to be a bunch of Fonzies here.

EDIT: Looks like API access has been restored, so I'm cautiously optimistic about things working. Note though that some instances may have rebooted or be otherwise impacted so check your error logs.

EDIT2: Nope, ELB is still hosed. Continue to be skeptical.

somesaba• 14 years ago

We're soooo screwed

Aloisius• 14 years ago

Just pay the extra money and get off US-East people.

danryan• 14 years ago

us-west-2 is the same price as us-east-1. Price is no excuse anymore.

rdl6• 14 years ago

Latency to Europe kind of is, if you want to be in a single site with acceptable ping times to Europe, US East, and US West. But I'd rather just solve replication and be in each.

maybird• 14 years ago

Acts of God can happen anywhere.

nakkiel2• 14 years ago

You certainly mean bad weather, don't you?

joelhaasnoot• 14 years ago

% of overhead power lines in the US are much greater than say in Amsterdam...

rmc2• 14 years ago

EU-West had similar levels of outage last year due to a lightening strike. Twas out for several hours and took a few days for everything to be back to normal.

spullara• 14 years ago

Ok startup people. It is worth it to host in a different zone than US-EAST.

k-mcgrady• 14 years ago

I host in Europe (Dublin I think). I'm from here and so are my customers so there's no latency worry for me.

philip12092• 14 years ago

I commented along the same lines during the last AWS/Heroku outage, but Rackspace still is giving me amazing value and uptime, and every time I try to move away (as I did this week with my lastest project, on Heroku) I get hit with a massive service disruption that pushes me back to Rackspace.

shawnps2• 14 years ago

Hi, I work at Rackspace. If you don't mind me asking, what makes you initially want to move away?

philip12092• 14 years ago

Complete curiosity. I love Rackspace, but am always looking to try new things - new languages, new databases, new methods of hosting, new DNS, new foods, new music - I'm just a curious person. It's not problem with Rackspace; all of my mission-critical stuff is still hosted with you. But I use random projects of mine to to explore in multiple ways (including Black Sands by Bonobo, which I discovered last week - possibly my favorite coding album now).

halefx• 14 years ago

He wants to play downtime roulette and Rackspace just won't let him.

Fizzer• 14 years ago

Funny to post this here, but I'm actually planning on migrating from Rackspace to AWS purely because of price.

Rackspace's prices are insane. $1,314/mo for a cloud server with 30gb of ram, compared to $657/mo for 34gb on AWS.

Plus with AWS you can use reserved instances to get that cost down to $286/mo. Rackspace has no way to get the cost down.

That makes Rackspace cost over 4.5x more when comparing based on ram.

nemesisj• 14 years ago

I'm not sure if others have noticed, but Rackspace London is probably the worst host I've ever used in my entire life. I'm not sure if this is because they were acquired or what, but it's genuinely frightening how awful the service is and how rude, irresponsible, and incompetent the staff are.

Terretta• 14 years ago

I moved because Rackspace was unable to resolve performance problems and outages with the interaction of the LBs in front of Cloud Sites and the Cloud Sites PHP nodes in DFW.

jordanthoms2• 14 years ago

Google Compute Engine anyone?

msie• 14 years ago

Windows Azure anyone?

locusm2• 14 years ago

Their current VM performance is incredibly bad. To be fair they are in some preview mode.

jc4p• 14 years ago

HP I-forget-their-catchy-name-for-their-service anyone?

raverbashing• 14 years ago

RedHat OpenShift?

Or they're just like Heroku and sit on top of AWS?

Jare• 14 years ago

I just got a promoted tweet by VMWare Cloud or something like that. Considering the rest of my feed was full of #EC2 #fail, it was a good choice.

charliesome• 14 years ago

Dedicated servers anyone?

bifrost• 14 years ago

I signed up for access, but no love yet.

suninwinter• 14 years ago

It looks like this is affecting iTunes Match, possibly. I have two tracks just sitting there, waiting to upload and running lsof -i shows iTunes with a connection to an AWS machine.

gee_totes2• 14 years ago

Do we know this is due to an electrical storm? Today had a leap second as well (The minute of midnight, June 30th lasted a second longer than normal).

dfc2• 14 years ago

The leap second has not happened yet[1]:

                                   UTC TIME STEP
                            on the 1st of July 2012
                      
  A positive leap second will be introduced at the end of June 2012.
  The sequence of dates of the UTC second markers will be:		
		
                          2012 June 30,     23h 59m 59s
                          2012 June 30,     23h 59m 60s
                          2012 July  1,      0h  0m  0s

[1] http://hpiers.obspm.fr/iers/bul/bulc/bulletinc.43

icefox• 14 years ago

At least for me Netflix died at exactly midnight....

jaredbeck• 14 years ago

8:40 PM PDT We can confirm that a large number of instances in a single Availability Zone have lost power due to electrical storms in the area. Amazon Elastic Compute Cloud (N. Virginia)

dangrossman• 14 years ago

This is the kind of weather conditions that spawn very electrically active storms. I don't doubt they could cause the issues. Last night was probably the most electrically active storm I've ever seen up here -- virtually non-stop lightning strikes for an hour or two, and there's another just like it over Virginia right now.

http://i.imgur.com/d5pEP.png

gee_totes2• 14 years ago

So how would an electrical storm take down AWS? Don't the data centers have backup generators?

kylebrown2• 14 years ago

Would it help if they designed the building as a Faraday cage?

hendler• 14 years ago

If you have a load balancer you may have balanced across availability zones (Not regions) you'd still be up. So US-EAST didn't all go down, just one AZ.

genwin• 14 years ago

But many people are saying that despite paying for multi-AZ for RDS, they were still down. Do you think they didn't also load-balance across AZs for their webservers?

paulsutter3• 14 years ago

AWS is not down. Only US-East. If your app is down, it's only because you don't care about the availability of your service.

It's pointless to complain. We've all seen before that Amazon can't keep whole regions up. If you rely on a region being up, you will have downtime and it's your fault.

haberman• 14 years ago

According to the AWS status page, only one availability zone within US-East is down, not the whole US-East region. Running a highly-available service exclusively from US-East is a reasonable strategy as long as you're spread across multiple availability zones.

I'm not an AWS customer, just reading their docs; please correct me if I'm wrong about any of this.

dragonstyle• 14 years ago

We're across availability zones and definitely ran into an outage across zones tonight.

sseveran• 14 years ago

The idea that a single data center can host a highly available service is just wrong.

oconnore• 14 years ago

Robust systems aren't hip. Get back to work and ship, ship, ship.

narsil• 14 years ago

That's how I feel about most hackathons. There isn't enough work done on existing projects, but quick hacks the demonstrate concepts are the norm. It would be great to see a hackathon where the primary purpose was to contribute to an existing open source project or something similar.

beedogs• 14 years ago

> If your app is down, it's only because you don't care about the availability of your service.

That is absolutely absurd. At what point did the common-sense solution to "unacceptable downtime on AWS" become "buy two of everything"?

NyxWulf2• 14 years ago

If you operate a service where uptime is critical, two of everything isn't enough. You have to ensure they will be separated geographically and logically so that when things like this happen, you don't lose your systems.

We operate systems that sit on the pages of the top e-commerce companies in the world. We have 10 separate segments of clusters. Operating in four AZ's in East, three AZ's in West-1 and two AZ's in West-2. When this outage happened, the servers that were impacted in east were removed from our DNS and within 9 minutes the impact of this event on their sites was eliminated.

lallysingh• 14 years ago

If you want reasonable uptime, you don't have any single points of failure. So 2 of everything.

marcamillion• 14 years ago

I am getting tired of all of these outages.

I know outages happen all the time at hosts, and maybe as a result of either a) news is more accessible now, or b) Amazon is bigger than most other hosts....I feel like Amazon & Heroku are going down WAYYYYY too much.

I am starting to wonder if this "tell all" policy is really best.

dakrisht• 14 years ago

Is this the same EC2 zone that went out just 3-4 days ago??

Second or I believe third power outage/loss of service for AWS in the past 10-days if I'm not mistaken.

This is wild. I wonder what's going on at Amazon and if they're capable of handling this much usage in addition to having power issues, etc.

Instagram and Netflix servers are down from what I hear and have been down for a few hours. Now it makes sense that they're being hosted on AWS.

jack7890• 14 years ago

Would it be expected that Amazon will issue substantial refunds (e.g. no charges for June hosting for impacted users) due to the problems today?

marcuspovey• 14 years ago

Cloud taken out by a cloud.

tathagatadg2• 14 years ago

So cloud is not ready for storms? (troll face)

tathagatadg2• 14 years ago

So cloud is not ready for storm? (troll face)

bconway• 14 years ago

If you're interested in information on the storms themselves and the destruction they caused in West Virginia, there's good coverage here: http://www.foxnews.com/weather/2012/06/30/state-emergency-de...

morsch• 14 years ago

Gov. Earl Ray Tomblin in a statement: With temperatures near 100 degrees expected this weekend, it's critical that we get people's power back on as soon as possible.

So let me get this straight: the critical issue with not having electricity after a huge storm is that the A/C isn't working? And 100F/38C isn't even that hot, right?

vidarh• 14 years ago

High temperatures pretty much always lead to increased mortality for the old and sick, so yes, that is one of the critical issues. Most other places where power is critical (e.g. hospitals) have backup generators and diesel supply contracts.

ckrailo• 14 years ago

Yes it's that hot. Google about people dying in homes at temps < 100, even.

cowkingdeluxe• 14 years ago

100F in a non-dry climate is very hot as it feels like 110F. Heat sickness is a very real concern, along with food spoilage and pets. Additionally, sick people without access to A/C in 100F weather can become even more sick very quickly.

smithian• 14 years ago

100F with high humidity will result in a number of preventable deaths due to heat exhaustion, heat stroke, dehydration, etc. So yes, it is the most critical thing, along with power to hospitals and emergency services.

kryptiskt• 14 years ago

According to Colin Percival on Twitter[1][2], the US East-1 AZ has more IP addresses, and thus probably other resources, than the rest of AWS put together. It casts comments about "limited to one availability zone" into some relief.

[1] https://twitter.com/cperciva/status/219067641023840257 [2] https://twitter.com/cperciva/status/219067963356098561

rabbitfang• 14 years ago

> the US East-1 AZ

us-east-1 is a region, containing multiple AZ's.

streeter• 14 years ago

You can use the EC2 API and ec2-describe-availability-zones to find out which availability zone is having issues: http://alestic.com/2012/06/ec2-outage-availability-zone

dustingetz• 14 years ago

how is it that amazon.com itself is never, ever, impacted?

edit: so basically, the businesses suffering outages (heroku, netflix, etc) don't value uptime to the same extent that amazon does. they got what they paid for.

dkulchenko• 14 years ago

This outage only affected us-east-1. Considering an Amazon.com outage would cost them $51k in lost sales every minute, I seriously doubt they put all their eggs (servers, that is) in one basket.

sofuture2• 14 years ago

Amazon.com does not run on the same EC2 that you and I use. It runs on a nearly identical system that is isolated and private to Amazon. I wouldn't be surprised if they were in entirely different physical locations.

16s2• 14 years ago

Must be the same storm that took several of my trees down (east coast Virginia USA) last night. It was a violent storm. 90 MPH winds. Made 80 foot tall oaks bend like straws and they were almost touching the ground. I spent the morning running the chainsaw just to clear the downed trees from the driveway.

AEP (local power company) says about 65% of customers in this area are w/o power. May be days before it's fully restored. Hope no one from the HN community got hurt.

Edit: I posted this from a computer in town. No power at my place so I can't respond to follow-up posts.

gcb• 14 years ago

Isn't that bastard operator from hell excuse #74?

AWS is down due to an electrical storm in the US

Related Stories