Majority of AWS us-east-1 down/having issues, Heroku included

https://news.ycombinator.com/item?id=10247065

by robbiet480 • 11 years ago

32 13 11 years ago

news.ycombinator.com

DynamoDB has been down for about 25 minutes now and has taken other services with it, like Lambda, Cloudwatch, EC2, ASG, SQS. I'm assuming the 503's on Medium, TheNextWeb and The Guardian are related. Lots of people on Twitter confirming.

AWS has updated their status page to say they are looking into increased error rates on DynamoDB and increased APIs errors on EC2 as of 3AM PDT.

Full list of services affected:

* ASG

* CloudFormation

* CloudTrail

* Cloudwatch

* CodeCommit

* CodePipeline

* Config

* Directory Service

* DynamoDB

* EC2

* Elastic MapReduce

* Lambda

* Mobile Analytics

* OpsWorks

* RDS

* Redshift

* SES

* SNS

* SQS

* SWS

Heroku is now also reporting they are having issues because of AWS: https://status.heroku.com/incidents/811#update-2808

Status page is http://status.aws.amazon.com/

RyanGWU822• 11 years ago

We're seeing the same problems with DynamoDB... it started around 2:20 AM PDT (1 hour ago). We're not having any problems with EC2—what kind of problems are you seeing? I hope it's just affecting the API and not running instances.

Side note: Medium posted an article last week about how most of their data is stored in DynamoDB, so I'm sure the Medium outage is related.

robbiet480• 11 years ago

Right now I can't load Lambda, CloudWatch or SQS. People on Twitter reporting massive ASG issues that caused their deployments to scale up massively suddenly.

donwb2• 11 years ago

Same here... PagerDuty started blowing up a little while ago only to discover we're seeing issues in SQS and Dynamo (plus ASG).. Damnit

donwb2• 11 years ago

....and now Dynamo has gone to full-blown "RED" status on the AWS status page. :sigh:

yuvadam2• 11 years ago

I'll take the opportunity to once again complain about how AWS's status page is a UX clusterfuck.

Pretty much the entire us-east-1 region is unreliable, that needs to be glaring from the page. Instead, I need to browse hundreds of happy green checkmarks to find out what is/n't broken by their unreasonable standard.

What a fucking joke.

deskamess• 11 years ago

Not even consistent...

Amazon DynamoDB (N. Virginia) Increased API error rates : Red icon.

Amazon Redshift (N. Virginia) Increased API Error Rates : Green icon with check mark.

Two recommendations:

- No green with check mark (info). Change that to yellow/orange. At this point you are not fooling anyone that green with a check mark is not yellow. There can be no green if it is service impacting (reduced availability, performance, broken, etc - anything other than the norm)

- Default sort should be by status : sort order is red-yellow-green. Need to know whats broken up-top.

yuvadam2• 11 years ago

Goes without saying that Heroku is a great example of how it should be done -

https://status.heroku.com/

vijaydev• 11 years ago

Wrote a simple script to hide all the green statuses https://gist.github.com/vijaydev/c0a670e79dcf7657cb2e

RyanGWU822• 11 years ago

Just spoke to AWS Premium Support around 4:35 AM. They don't have any more information than what's on the status page, basically. I was hoping to get an ETA for a resolution, but no such luck.

aps-sids• 11 years ago

AWS Premium support is a joke. I never got a reply under 1 hour time, like they promised.

BilalBudhani• 11 years ago

Our app is mostly dependent on AWS's DynamoDB and we're facing problems with it. I can confirm the outrage of it.

Majority of AWS us-east-1 down/having issues, Heroku included

Related Stories