Heroku is down

http://www.heroku.com/?test

by jackmoore • 14 years ago

117 98 14 years ago

heroku.com

trebuch3t• 14 years ago

I was going to get details from status.heroku.com but that's down too.

wilfra3• 14 years ago

It is up for me:

Potential Platform Issues Jun 7, 2012 15:55 UTC - Update

We have confirmed widespread errors on the platform. Our engineers are continuing to investigate. Posted Jun 7, 2012 15:58 UTC Issue

Our automated systems have detected potential platform errors. We are investigating. Posted Jun 7, 2012 15:55 UTC

skram• 14 years ago

status.heroku.com is up for me. My apps aren't though.

pilap82• 14 years ago

it's up sometimes :)

Potential Platform Issues 5m+ We have confirmed widespread errors on the platform. Our engineers are continuing to investigate.

this is the message for Production and Development

AntonTrollback• 14 years ago

http://www.nooooooooooooooo.com/

jboggan2• 14 years ago

What's their usual outage duration? I'm supposed to demo a project in a few hours.

wilfra3• 14 years ago

They keep a historical archive of incidents at https://status.heroku.com/past

Last time I can recall this happening to us it was down for several hours.

jboggan2• 14 years ago

502 Bad Gateway . . . is there a page that displays the status of the status page?

garindra3• 14 years ago

Status update on https://status.heroku.com/incidents/372 :

"We have confirmed widespread errors on the platform. Our engineers are continuing to investigate."

jt2190• 14 years ago

Also can follow @herokustatus on twitter: https://twitter.com/#!/herokustatus

Edit: Also IRC #heroku

Edit 2: heh... I just noticed the "subscribe to notifiations" link on the incident page: https://status.heroku.com/incidents/372

splatcollision• 14 years ago

Heroku: Awesome when it's up, but when it's down, it's really really down...

andQ• 14 years ago

https://status-old.heroku.com/ seems to work

mikejarema• 14 years ago

Interesting to see that www.heroku.com runs on their own platform, and is just as susceptible to platform downtime.

On the other hand its surprising to me that when heroku goes down, it goes down hard, namely that all hosted sites including their own are unavailable.

mjackson• 14 years ago

Totally agree with this assessment. I don't know how their routing mesh works exactly, but it seems that they're coupled too closely. Either everybody is mostly working fine, or everybody's down.

asparagui• 14 years ago

and it's back up....

just a glitch in the matrix. it occurs when they switch over local control.

wilfra3• 14 years ago

We are still down: http://warsocial.com

Edit: Came back for a second. Went down again. Came back. Went down.

joshcrews2• 14 years ago

I checked 5 sites, all single dyno: 3 down, 2 up

joshcrews2• 14 years ago

update: all 5 down

bad_user3• 14 years ago

I have 4 apps on 20 paid dynos. All of them down.

masonhensley• 14 years ago

2 of 3 up, all single dyno staticish sites; but it could be cloudflare propping them up with their cache.

bdesimone2• 14 years ago

When will heroku have a multizone offering?

I know most sites hosted on heroku are pet projects with no real need that type of uptime, but this type of downtime makes it impossible to use them as enterprise customers.

chao-2• 14 years ago

Agreed. We are finally scaling from "small scale production" to having actual customers we want to take care of. Worse, we began a huge marketing push this week and had been seeing promising results. Now we'll probably miss a boatload of potential customers.

Heroku had been a joy to work with until now. I ask of you all (as most of you are far more experienced in these matters than I), what is the traditional practice to mitigate this sort of risk? Paying for hosting with two separate companies?

EDIT: I understand the benefits of cloud hosting over hiring a sysadmin. At the same time, I'm interested to learn about what possible solutions there are. I'm at the point where I don't know what I don't know, and even the name of a topic or technology would be a huge help.

bad_user3• 14 years ago

On June 1st (this month) they've had routing errors. On June the 5th (the day before yesterday) they again had routing errors.

My apps triggered errors left and right and I was bashing my head against the keyboard because I noticed the errors earlier than their status update and I believed it was our fault.

Plus this last week the latency of the requests have been incredibly high, while the traffic to my apps have not increased. Again I assumed it was my fault.

I'm contemplating moving back to EC2 or Linode.

kennystone• 14 years ago

Heroku is really good. You will probably have more downtime than them unless you hire a whole team of operations and admins and pay a lot of money for hosting. It may also take a while to build out the deployment and development tools they have.

switz• 14 years ago

If status.heroku goes down due to 500 errors (overloading) use https://status-old.heroku.com/

riffraff• 14 years ago

the incident specific page is up often when the main page is not, in this case

https://status.heroku.com/incidents/372

soulclap• 14 years ago

It seems like they are quick and post a lot of updates on this status page each time when something's wrong. (I am not using Heroku but had the 'joy' of trying to run a site on Dreamhost for a while, so I have seen the worst.)

namm• 14 years ago

my demo site is still down.

RoyceFullerton• 14 years ago

Confirmed. One of my apps is down. I hope they resolve it soon, but I don't pay anything at the moment so I am not too upset.

instakill• 14 years ago

What's crap about these outages is that custom 500 error pages that you've created aren't shown.

bitsweet2• 14 years ago

What would really help in these situations is a failover to our maintenance pages or some other static page we could provide instead of showing the Heroku "Application Error".

This is probably harder then it seems, especially when the outage is related to their routing infrastructure.

mcs• 14 years ago

Well, there is an option to have error pages on your heroku domains, but that is probably dependent on at least a minimum level of the routing layer working.

https://devcenter.heroku.com/articles/error-pages

bitsweet2• 14 years ago

Yeah, I have that setup but its not being used at the moment with this outage - if full redundancy with multiple regions is a huge challenge, perhaps some interim basic routing redundancy would go a long way so we could at least display a branded error page during an outage.

pardner2• 14 years ago

I agree. There is a very simple way to do this which I just posted (since it apparently hasn't occurred to Heroku). All they need to do is change the default app error message once they have confirmed they have a platform issue. So within a few minutes of an outage, a better message is displayed to our customers that does NOT make it look like we screwed up. http://blog.pardner.com/2012/06/dear-heroku-dammit-quit-blam...

dholowiski2• 14 years ago

How would that help?

zsherman• 14 years ago

My apps are finally back up, how's everyone else doing?

kunalmodi• 14 years ago

things seem to be working for me, but Heroku seems a little off still. I got an error when I tried to view my app logs though, which is a little scary

Cushman• 14 years ago

They went down literally the moment our demo thing started. I swear this is because of me.

loceng• 14 years ago

What luck.

jszielenski• 14 years ago

Same here! This happened last year too! Annoying.

g0atbutt• 14 years ago

As a newbie, I've really enjoyed deploying on Heroku. However these outages really terrify me. Is there an easy way to host a redundant version of your app outside of Heroku?

philip12092• 14 years ago

If you use Cloudflare it will serve a static version of the site until it is back online.

garindra3• 14 years ago

I always use Cloudflare in front of Heroku; mainly because of that and the cheap & simple SSL set-up that they have.

patio11• 14 years ago

Ask a simple question, get a simple answer: No. This is no knock on Heroku, either. No hosting solution can make that easy: it is an enterprise requirement which implies six figures of investment and a dedicated ops team with no newbies on it.

jsprinkles8• 14 years ago

I'm sorry, but that's completely untrue. I can't speak for using Heroku specifically, but in the end hosting is just running an app. It's fairly trivial to make an app multi-provider these days through a plethora of methods. DNS and a low TTL is on the easiest-to-approach end if your app is designed for it and aware of the complications, which implies thinking about your database and other supporting architecture from the perspective of a multihomed setup. If your app isn't idempotent nor designed to be multihomed, it'll be harder, but six figures of investment is insanely high even in that awful case.

Perspective: I could throw an app on Rackspace and Amazon, with a replicated database, for under $200, in about a day.

freditup• 14 years ago

What kind of service guarantees does Heroku have?

bretthoerner• 14 years ago

http://www.whoownsmyavailability.com/

lemieux• 14 years ago

What is it suppose to do?

yourcelf• 14 years ago

I think it's similar to http://isitchristmas.com/ . It's a simple answer that's always the same, to drive the point home: you are always the one responsible for your uptime, no matter whether you choose dedicated hosting, the cloud, your own closet, etc. You can't outsource responsibility.

grandalf3• 14 years ago

Heroku is a great platform and has come a really long way -- but it could really use some competition. Some of the main issues that would be resolved quickly with competition are:

- it would be possible to have the non-DB part of your heroku app be spread across multiple availability zones.

- worker pricing would be much lower or based on actual CPU cycles used.

- add-on providers would be vetted more thoroughly before getting a spot in the add-on store.

- they'd reopen the #heroku irc channel for informal support

ricardobeat4• 14 years ago

you mean like AWS, EngineYard, appfog, dotCloud?

bdesimone2• 14 years ago

IaaS != PaaS

grandalf3• 14 years ago

I haven't looked at DotCloud, but the others are not really competitors.

believeUme• 14 years ago

Goes to show that the five nines is a bunch of crap. More like two nines.

patrickgzill2• 14 years ago

5 nines is 5 minutes of downtime per year. 4 nines is 52 minutes of downtime per year.

See: http://en.wikipedia.org/wiki/High_availability#Percentage_ca... for a handy chart and follow along with any of your favorite *aaS providers!

grandalf3• 14 years ago

RedHat was timing its blog postings in anticipation of this:

https://openshift.redhat.com/community/blogs/new-openshift-r...

benmccann• 14 years ago

There's no big announcement there that I see. They release a blog post like this at least once a week.

pardner2• 14 years ago

still getting sporatic errors

fragsworth• 14 years ago

Eventually, they need to offer multiple independent regions, so if you're inclined (like Netflix does with AWS) you can develop a way to fail-over to another region.

At least then, it won't be "Heroku is down", it will be "Heroku West is down" or some such thing.

cardmagic2• 14 years ago

http://AppFog.com has multiple regions today: Ireland, Singapore and US, addin Rackspace and HP soon. All free and backed by CloudFoundry

kitsune_• 14 years ago

We've recently begun to move our entire infrastructure to external providers (vps and cloud) because we felt that we couldn't guarantee a sufficiently high level of quality and had to devote too much time and money to operations.

In over four months we've probably already had more downtime than in the past 5 years. Despite paying quite a lot for "redundant", clustered offerings. There was one nasty bug in the hypervisor that killed the entirety of one of our providers' vps infrastructure - for over a day.

Our experience with "the cloud" was a little bit better, but we aren't entirely satisfied.

I don't know, maybe we should just order services from three different providers mirror our applications ourselves as a fail-over mechanism.

dholowiski2• 14 years ago

> maybe we should just order services from three different providers mirror our applications ourselves as a fail-over mechanism.

Yes, yes a thousand times yes. And dedicated servers don't cost much more than VPS's.

patrickgzill2• 14 years ago

Yes. I recommend having a look at either ProxMox (proxmox.com) which can simultaneously do OpenVZ and KVM; or, XCP for Xen-based stuff - see http://www.xen.org/products/cloudxen.html

neilmiddleton• 14 years ago

Heroku status is currently showing production uptime as 99.97%. That's not too shabby, and definitely not as bad as you make out.

jsprinkles8• 14 years ago

He implied that Heroku is not his sole provider. I'm skeptical of that figure, too, as it's probably something on the Heroku sales site (99.7% to the 30-byte health check!), or it's manually updated, in which case it's probably rounded.

cardmagic2• 14 years ago

http://AppFog.com has ruby, python and node, multiple regions today: Ireland, Singapore and US, adding Rackspace and HP soon. All free and backed by CloudFoundry

ralphleon2• 14 years ago

It's so scary to think that our entire startup almost died with this glitch. Our enterprise customers use our product in the mornings... this happened at 9-fucking-am right during our peak hours.

Obviously it's our fault for not having a redundant system / for trusting heroku. Though I still can't help but be a little pissed as I email 100 people about how sorry I am for their service disruption. Heroku didn't email me.

joevandyk3• 14 years ago

You can subscribe to notifications on http://status.heroku.com.

ralphleon2• 14 years ago

Thanks so much! Though I can't imagine anyone who wouldn't want these notifications...

Heroku is down

Related Stories