Heroku is down again

https://status.heroku.com/

by ardakara • 14 years ago

203 126 14 years ago

status.heroku.com

dnsco• 14 years ago

Apparently it's AWS East.

KenCochrane2• 14 years ago

Don't think it was just heroku, lots of other sites were down as well, netflix.com, etc. most likely another AWS issue.

devth• 14 years ago

My app with 2 dynos is down.

Their status site is running fine altho it's not reporting errors: https://status.heroku.com/

Their Helpdesk is down: https://api.heroku.com/helpdesk/login?timestamp=1341025835&#...

Devcenter is down: http://devcenter.heroku.com

AWS isn't reporting any errors: http://status.aws.amazon.com

shuzchen• 14 years ago

That's standard. You'd expect them to run everything they have on their own service except for the status site.

KenCochrane2• 14 years ago

Amazon posted an update:

8:21 PM PDT We are investigating connectivity issues for a number of instances in the US-EAST-1 Region.

stbullard3• 14 years ago

At 11:25 Eastern, https://status.heroku.com/incidents/386 was posted: "We're currently experiencing a widespread application outage. We've disabled API access while engineers work on resolving the issues."

jc4p2• 14 years ago

It's been fifteen minutes since our site in US-East went down and AWS Status hasn't said anything yet.

heliostatic• 14 years ago

As of 10:16 CDT, I can't reach Netflix or Heroku, although AWS status (http://status.aws.amazon.com/) is not yet reporting any current outages.

usaar333• 14 years ago

Ugh, the EC2 administration console is done. Being in other availability zones won't save you..

sehugg2• 14 years ago

There are US-East problems, looks like:

  - One AZ is down
  - API commands are spotty and may return incorrect results
  - ELB looks screwed
  - IP reassignments don't seem to be working
  - Who knows what the fuck else is broken

dragonstyle• 14 years ago

We're in AWS East and definitely fighting some issues here, though we're trying to understand what is happening.

mthreat• 14 years ago

AWS should call them Unavailability Zones.

alanh• 14 years ago

I had the same thought, but it’s not HN material.

robbiet480• 14 years ago

MASSIVE Storms in VA area where us-east-1 is. 326,000 customers without power already, worst lightning I have seen in my 20 years of life. Sky is intense blue/green/purple. This is most likely what the issue is

prezjordan2• 14 years ago

No matter how powerful we become as a species with our technology, we are still at the mercy of the clouds. Pretty cool if you think about it.

tyrmored• 14 years ago

The cloud is no match for the cloud!

MrMatters• 14 years ago

Well, until we figure out plausible ways to control weather reliably on a large enough scale, at least. Without killing the atmosphere or our species or anything like that.

oldstrangers• 14 years ago

Or if we just built our power grid underground like rational people.

meskyanichi5• 14 years ago

Or as the religious people call it, "The Act of God". No seriously, people actually write that shit in their terms. Hilarious.

wavephorm2• 14 years ago

Hopefully one day it will be feasible to host our computing architecture in space and avoid all these terrestrial obstructions.

jaequery3• 14 years ago

this is pretty bad, really bad ...

vibrunazo• 14 years ago

I'm completely ignorant here. But aren't these outages usually solved by having backup servers in different locations? As many datacenters do, and as I imagine something as huge as heroku would?

wensing• 14 years ago

Stormpulse.

tomjen32• 14 years ago

Not cool at all.

All it means is that humans are not yet powerful enough to make the environment work as it should (ie serve humans).

bsg75• 14 years ago

I see what you did there.

shadowz• 14 years ago

Seems pretty severe actually. Washington Post has a live blog going on about it:

http://www.washingtonpost.com/blogs/capital-weather-gang/pos...

douglasisshiny• 14 years ago

I live in DC. It was an amazing storm. A transformer in my area went down fairly quickly. Fortunately, I live near a large hospital.

jberryman• 14 years ago

Was watching a movie in a big 20-screen theater in Richmond, and they told everybody to just leave (incidentally, not through the emergency exits, instead they funneled 100s of people into the lobby all at once :/)

Klinky• 14 years ago

Saw this post here on HN, pulled up www.chart.state.md.us to watch the live traffic cams in the area. Clicked through a couple, some of which showed heavy rain, wind & lightning. Then the stream froze and now the site is completely unresponsive.

hristov• 14 years ago

In other words, Amazon's power backups failed again. On the bright side at least they are not running a nuclear reactor.

ultrasaurus2• 14 years ago

Yup, a lot of services served by AWS are having issues. We're seeing a huge spike in incidents being triggered in PagerDuty.

(fyi: our customers are still being alerted)

scorpion0322• 14 years ago

What infrastructure do you use for alerting your customers?

drewvanstone• 14 years ago

We have dozens of servers that are unavailable at the moment (US-East). Obviously AWS is having major issues.

ricardobeat• 14 years ago

Please link to https://status.heroku.com/, pointing to a broken URL is pointless.

sneaky_weasel• 14 years ago

Everything still down for me. Would have expected some redundancy...

MicahWedemeyer• 14 years ago

Can we update the title to something like "AWS US-east-1 is down" instead of just Heroku?

sofuture2• 14 years ago

> informative titles get changed

> uninformative titles left intact

hn 2012

jdcryans• 14 years ago

Then you'd have to change the link too?

rplnt• 14 years ago

Appropriate title might be that "Heroku is down due to AWS outage which is down due to power failure which happened due to storms caused by moist winds colliding with hot air that was heated over the continent by sun that....". It really doesn't matter. Heroku is down. Customers don't care.

Apreche2• 14 years ago

Why does the AWS dashboard show all green when that is most definitely not the case?

http://status.aws.amazon.com/

prezjordan2• 14 years ago

The red check marks are in VA and can't be displayed.

jc4p2• 14 years ago

If only they did multi-AZ hosting like they keep telling us to do when there's outages :)

ultrasaurus2• 14 years ago

I strongly doubt that's the case, at PagerDuty we're seeing ~100x the regular traffic. I think it's been having issues for at least 15 minutes.

Apreche2• 14 years ago

Even now that it is updated it has yellow triangles for "performance issues" instead of red circle for service disruption. Seems like they are in denial.

oakwhiz• 14 years ago

Power went out = service disruption.

It's underhanded to call it a "performance issue," if not an outright lie, albeit a small one.

erichmond2• 14 years ago

This was the disappointing thing for me as well. Our connectivity died around 8PM EST-ish, and I immediately went to status.aws and it said everything was normal. I then proceeded to waste half my night looking at our internal infrastructure trusting that page was accurate.

I've learned my lesson.

fosk• 14 years ago

It's not Heroku that is down, AWS is down.

coryshaw• 14 years ago

Ahh I was just using crunchbase and its now down, must be related.

momoro• 14 years ago

Aws East connectivity issues, 8:21pm:

http://status.aws.amazon.com/rss/ec2-us-east-1.rss

RegEx• 14 years ago

Loadbalancers are down for me: Getting 'Response contains invalid JSON' upon attempted termination.

darrenkopp• 14 years ago

Think that netflix is down too. so much for chaos monkey?

blantonl• 14 years ago

Yup, I can confirm from here in Montana that both Netflix and Herkou are down.

However, I have 20 instances on us-east. And haven't seen any problems, even during yesterday's outage on AWS.

Edit: that doesn't mean this isn't an AWS outage.... It almost certainly is.

sofuture2• 14 years ago

Yeah, my ~20 instances are okay. I think we had a hiccup with our (multi-az, whooo!) RDS though.

zenogais• 14 years ago

Confirmed in California as well.

undergroundhero• 14 years ago

Confirmed in Missouri.

jaequery3• 14 years ago

i think netflix ceo just signed up for a rackspace account

justinsb2• 14 years ago

You're joking, but I bet Netflix's Asgard system gets support for OpenStack pretty quickly now...

superuser22• 14 years ago

Rackspace got hit by a truck last year and went down for a while too. Not cloud != perfectly reliable.

theoutlander• 14 years ago

No wonder my wife was just asking me why Netflix is down on Xbox live here in Seattle.

pud• 14 years ago

This is the motivation I needed to spread my EC2 instances across multiple availability zones. When the power comes back.

(fandalism is down)

sehugg2• 14 years ago

Don't assume that'll save you.

edouard12345672• 14 years ago

idea for heroku : allow customers to host a "my app is down page for blah blah reason" where they host their status page (rackspace I guess?). Who think this would be useful? My users see a blank page right now when they go to ZeTrip, I'd rather show them a static page saying : "our site is down due to amazon lack of redundancy."

devicenull• 14 years ago

Cloudflare lets you do this afaik. I'm not sure I'd trust a service to show a proper 'this site is temporarily down' page when something very bad has happened.

reustle• 14 years ago

I can't get to netflix.com. That is no good. Luckily all of my 75+ east servers seem to be ok.

rorrr• 14 years ago

netflix.com is down here as well

stbullard3• 14 years ago

EC2 status:

8:21 PM PDT We are investigating connectivity issues for a number of instances in the US-EAST-1 Region.

8:31 PM PDT We are investigating elevated errors rates for APIs in the US-EAST-1 (Northern Virginia) region, as well as connectivity issues to instances in a single availability zone.

8:40 PM PDT We can confirm that a large number of instances in a single Availability Zone have lost power due to electrical storms in the area. We are actively working to restore power.

joelg87• 14 years ago

8:49 PM PDT Power has been restored to the impacted Availability Zone and we are working to bring impacted instances and volumes back online.

stbullard3• 14 years ago

9:20 PM PDT We are continuing to work to bring the instances and volumes back online. In addition, EC2 and EBS APIs are currently experiencing elevated error rates.

edouard12345672• 14 years ago

"heroku status" command returns : All Systems Go: No known issues at this time.

aquark• 14 years ago

Just started up try using filepicker.io and it seems to be down too.

Beginning to feel pretty lucky though -- this is at least the 4th AWS-East outage that has made enough of a splash to notice but missed my instances. Upgrading to multiple availability zones was scheduled for Monday anyway.

moskie• 14 years ago

Comcast's login server looks like it's down too: http://login.comcast.net. Prevents me from logging into HBO Go. No Netflix either. :(

apawloski2• 14 years ago

Reddit also seems to be experiencing some difficulties. Are they still on AWS?

rgarcia• 14 years ago

How many times does this have to happen before heroku spreads across multiple regions?

cardmagic• 14 years ago

Like http://appfog.com/ has?

fragsworth• 14 years ago

You say this like they can just snap their fingers and provide regional services.

inopinatus• 14 years ago

Well, nothing of scale happens overnight.

But anyone deploying a critical application to AWS makes a point of cross-region data replication. Heroku have long known that they lose potential customers to, say, Engine Yard as a result of only hosting at US-East.

One can only conclude that this is a clear business decision on their part. I can hardly believe that Heroku's engineers are incapable of it. Indeed I would be very surprised to learn that they haven't brought up an instance of their platform at, say, US-West, for testing or proof-of-concept purposes.

Of course, productising that is a different matter. Extending the control plane, front end, and pricing/billing systems might have considerable associated project cost. Perhaps they have concluded that the costs outweigh the additional revenue. Or, just haven't got around to it yet.

tomjen32• 14 years ago

If not, why pay them rather than go to amazon directly?

mechanical_fish• 14 years ago

It's a hard thing to engineer, especially after the fact, and especially when you are trying to hide it behind an abstraction layer. (Which is to say: You can't expect your customers to engineer their apps with multiregion in mind, or to take it kindly when you raise rates to support additional redundant hardware and bandwidth.)

e.g. because region-to-region data transfer is not free, and trans-region latency is ugly, you can't just relaunch half your instance farm in another region and expect happiness. There are also routing issues: Internal IPs don't work across regions, elastic IPs don't transfer across regions...

hpatel• 14 years ago

Even if they can't do it right away, they should communicate a plan for how they are going to tackle this recurring issue. That's the whole point behind status.heroku.com / trust.salesforce.com. They are part of a publicly traded corporation with a lot of resources.

Extremely nerve wracking for new startups like ours.

scorpion0322• 14 years ago

I guess, it is always a good plan to have an instance deployable on Linode.

jaequery3• 14 years ago

did the datacenter get flooded or what? this is just "major" downage.

apawloski2• 14 years ago

It's likely related to the power outages across VA

knodi• 14 years ago

Come on not another power issue, what happened to the generators... and the back up generators that they fixed few weeks back.

excuse-me3• 14 years ago

The little red ribbon that you pull to get the AA batteries out is stcuk underneath - they are looking for a pen to flick the battery but since everyone switched over to Fire tablets there aren't any pens.

jordanthoms2• 14 years ago

Heroku's uptime for June is going to be.. not so hot.

bohara• 14 years ago

For shissle.

Had to move off Heroku for my latest app. That amount of downtime would put me out of business.

To bad. I really like the Heroku platform

_nato_2• 14 years ago

what a winner! And they charge!! FUX.

_nato_2• 14 years ago

I have 3 customers -- what the are they going to do, those poor souls!!

philip1209• 14 years ago

Cloudflare Always-Up isn't showing on my page - is Cloudflare affected too?

ahmedaly• 14 years ago

I can't believe that guys at heroku are not ready for such situations!

They rely ONLY on virginia's instances because its the cheapest, without caring about customers.. or thinking of replicating their services in multiple locations for such issues!

technotony• 14 years ago

I just lost a potential hire because of this, was demoing my app to someone and it wasn't working, she thought it was because of the product. Damn you heroku!

bsaul• 14 years ago

Google Appengine's just fine. Dont't know how many AZ i'm on and don't want to know :)) The more i see about amazon failures the more i think VM are just not high enough for me in the abstraction layer...

manishm2• 14 years ago

Simple solution to this is to have a backup or failover to a non-AWS Datacenter too, basically don't be just dependent on one Datacenter. E.g. MS Azure/Google/Rackspace This not only spreads your risks but keeps your customers happy.

digeridoo• 14 years ago

There's nothing simple about that. :)

manishm2• 14 years ago

Yea, but it should be done by providers like Heroku who can talk to multiple datacenters :)

kevinprince• 14 years ago

We use Heroku for our event tools, thankfully we have nothing live this weekend or this would be a disaster.

Heroku is down again

Related Stories