Heroku was Down

https://news.ycombinator.com/item?id=30456688

by ericpauley • 4 years ago

254 141 4 years ago

news.ycombinator.com

Incident: https://status.heroku.com/incidents/2402

Update: our apps appear back up after 23 minutes total downtime. Others are reporting applications still down.

Update: it appears most or all services have been restored.

VWWHFSfQ2• 4 years ago

It is hard-down for me

emilsedgh• 4 years ago

Yes it appears to be down for us as well.

ericpauley3• 4 years ago

Our app is currently 100% down at the router layer, and dashboard requests are mostly failing.

VWWHFSfQ2• 4 years ago

Routing and logs are down for me as well

btown• 4 years ago

Same here, hard down for us

nickrubin• 4 years ago

Yes, all apps are down

pwned12• 4 years ago

It would be nice if they updated their status page. sigh

greset• 4 years ago

Same here

teagee• 4 years ago

HN is truly a market leader in status page technology

duxup• 4 years ago

Except when it is a false alarm.

I've seen a few get to the front page.

But when we're right we're right.

jcuenod• 4 years ago

10/9 times

dec0dedab0de2• 4 years ago

Even when it's a false alarm it's usually something else is having a problem that is affecting many people and manifesting itself as a particular service being down.

rileytg2• 4 years ago

*was

jonpon• 4 years ago

I came to HN to make sure it was actually down.

js4ever• 4 years ago

HN was also down for a moment because too much traffic

Tainnor• 4 years ago

I experienced that too

chasd00• 4 years ago

A couple of outages that have affected my client i learned about first on HN. We were able to mobilize a team and get on top of it faster than any monitoring team at the client ( a state government ). I feel like HN should invoice us haha

castaway3000• 4 years ago

You might benefit from some better monitoring!

odiroot• 4 years ago

Such a lightweight tool at that! Barely any JS loaded.

thecleaner• 4 years ago

I'll make my employer pay for it if HN starts a paid service. Honestly they should write a blog post about when down times were first reported on Hn.

typeofhuman• 4 years ago

Yes all of my apps are down.

iamricks• 4 years ago

So... what are the chances this is related to a cyber attack from Russia? Also our apps are down

ezekg6• 4 years ago

I wasn't gonna say it...

brimble• 4 years ago

Now is a good time for everyone to double-check their backups and go over their disaster recovery plans. I know I am.

I mean, yesterday was a good day. But today's the best you've got if you didn't do it then.

driverdan2• 4 years ago

Practically zero. Why would Russia care about Heroku?

adamnemecek• 4 years ago

Maybe not heroku per se but something hosted on heroku?

dc-programmer• 4 years ago

Collateral damage (but probably DNS lol)

Trasmatta• 4 years ago

Heroku runs on AWS, so it would be an AWS attack of some sort. But AWS in general seems to be up from what I can tell, so I'm feeling like it's not a cyberattack, and just a Heroku specific problem.

briandear• 4 years ago

https://www.cnn.com/2022/02/24/tech/russia-ukraine-us-sancti...

adamrezich4• 4 years ago

what would lead you to this conclusion?

2OEH8eoCRo03• 4 years ago

https://www.cisa.gov/shields-up

seannui• 4 years ago

No update to their own status or their Twitter (@herokustatus) after 10+ minutes. Pro.

canadianwriter• 4 years ago

Gotta love asking in slack "is uh... literally everything down right now?"

ageitgey• 4 years ago

Same here

ezekg6• 4 years ago

I've been with Heroku for 7 years and this is the first time in recent memory that I've seen Heroku completely go down. Nothing at all works.

digerata3• 4 years ago

Ahh... were you sleeping in 2021? They went down in September, November, and December. Probably some other times earlier in the year as well. I stopped keeping track at how bad their service is because of how bad AWS's service is.

ezekg6• 4 years ago

I'm well aware of their other outages. But I don't remember another time when they were completely down like this, even down to their status page.

jrochkind19• 4 years ago

My app didn't go down during any of those times.

I recall at least one of them I could not deploy new versions of my app though, which is pretty bad. I don't believe I had any downtime at all due to heroku platform outages in 2021. i believe you if you say you did though!

This particular outage definitely seems to be of a rare level of severity.

driverdan2• 4 years ago

You haven't been paying attention. We're very heavy Heroku users. Their dashboard/API has gone down a few times in the past year. Overall they've been reasonably good.

It's much better than it was 5+ years ago. Back then they had almost weekly downtime.

ezekg6• 4 years ago

I am paying attention. My business runs on Heroku. I never said they don't go down... they go down quite a lot (much to my dismay), but never completely down like this one. Even their status page is down, which is a first for me. Must be bad.

bin_bash2• 4 years ago

they had their big layoff in 2020 or 2021 though so I wouldn't expect it to improve

moeadham• 4 years ago

US down for us, EU is up

jlangenauer• 4 years ago

All our apps are down as well. And it seems (and I really have to almost laugh here) that https://status.heroku.com no longer loads.

dspillett• 4 years ago

As always, a note to infrastructure providers: HOST YOUR LIVE STATUS DETAILS ON OTHER INFRASTRUCTURE.

Of course, they won't. If they host is on someone else's then that might look bad (tacitly saying that a competitor is reliable and might be up when they are down) and if they hive off an extra copy of some of their infrastructure there will still be single points of failure either accidentally, by human error (someone somehow messing up both segments at once), or by design (possibly through management trying to save pennies when they noticed this extra bit of infrastructure on the balance sheet).

bin_bash2• 4 years ago

I am pretty sure Heroku has a Linode box or something for their status page. I may be wrong though. They may also have moved it but that would seem like an incredibly dumb idea.

I worked there and I vaguely remember something like this but it's been a long time.

If I had to guess this is probably a DNS issue (it's always DNS).

wuputah• 4 years ago

I agree with your lead statement and argued as such, but was overruled. Last I knew and understood, Heroku Status is static pages pushed out to Fastly, with the internal admin site (that does that work) running in a Heroku Private Space. If you look at the DNS, it still appears to be served by Fastly, and Heroku Private Spaces are generally pretty isolated infra, so I would be curious what the failure mode was here. But ultimately this is the fire you play with when you self-host your status site...

amotinga• 4 years ago

yes, site is down

reubano8• 4 years ago

Yep, all 6 of my Heroku apps are down.

reubano8• 4 years ago

Interestingly enough, my 2 Squarespace sites weren't affected. Anyone know who they host with?

andygcook• 4 years ago

Our app is currently down as well and we host on Heroku. We received a down alert from our monitoring service at 11:32AM EST.

tempnow987• 4 years ago

Status Page was running a bit slowly for me connecting to developers.salesforce.com.

Link here to latest snapshot:

https://postimg.cc/McZHpCWg

No issues noted there.

perryraskin• 4 years ago

My Heroku app is completely down as well

reubano8• 4 years ago

AWS is down too, this is likely the cause since Heroku runs on it. https://downdetector.com/status/aws-amazon-web-services/

jermaustin14• 4 years ago

AWS status page is up, but going VERY slow, and says no issues.

XzAeRosho• 4 years ago

As usual whenever there's an AWS outage

reubano8• 4 years ago

I still don't get why they use images

    <td class="bb pad04 top center" style="width: 32px">
      <img src="/images/status0.gif">
    </td>

Instead of unicode:https://www.htmlsymbols.xyz/unicode/U%2b2705

mattwad• 4 years ago

AWS must be only partially down, all of it is running fine for me in us-east-1. Elastic beanstalk ftw! :)

throwthere• 4 years ago

If us-east-1 is up I have a heard time believing ANYTHING aws is down

jmartens4• 4 years ago

What is the proof that AWS is down? Functional monitoring of AWS by metrist.io (I'm a co-founder) shows no AWS problems. Downdetector is not a reliable source.

reubano8• 4 years ago

What makes Downdetector unreliable? It's showing a huge spike right now.

djbusby2• 4 years ago

Neat! Wish you had a clickable demo rather than just screenshots.

nuggien• 4 years ago

maybe your service isn't as reliable as you think it is ;)

nickphx• 4 years ago

No, AWS is not down.

jrochkind19• 4 years ago

AWS status page is all green.... Just kidding! I know the information content of the AWS status page is literally zero, it's always green!

somenewaccount1• 4 years ago

Nah, it's just on a 12 hour delay between updates.

mabsboza• 4 years ago

reporting from Nicaragua, our apps are down!

rlopezc• 4 years ago

Down for us too

yekurtal• 4 years ago

Mine is still up.

mhale• 4 years ago

Heroku apps are down, but API access is available.

Might be a good time to run: heroku pg:backups:download

jpmoyn• 4 years ago

Down along with the status page right now

mmarcant• 4 years ago

2/3 of our apps have been timing out of synthetic checks since 8:30a PT

pwned12• 4 years ago

Back up just now.

pedroborges• 4 years ago

Our app is down too :(

noodle• 4 years ago

Things just came back up for me

nordec• 4 years ago

Same here

vendiddy• 4 years ago

Down for us as well.

digerata3• 4 years ago

Ours are up. US East. Heroku Shield.

kennysmoothx• 4 years ago

Our Heroku applications are all down, it seems to be that it is a Heroku Router/DNS issue.

We can access our logs and the application is running, just no incoming requests.

jpmw2• 4 years ago

Let me bet: it's DNS.

It's always DNS.

ne0n• 4 years ago

It was, but I don't know why. I'm curious to hear if Heroku releases any information about how this happened. Heroku's DNS was returning a single 100.64.x.x address which is in a reserved range.

jpmw2• 4 years ago

They typically publish post-mortems, I think? Not 100% sure. They definitely do in-house.

We'll have to wait and see I guess.

rileytg2• 4 years ago

status, dashboard and landing page are down for me. (socal.)

sharksauce• 4 years ago

% heroku status

› Warning: Our terms of service have changed:

https://dashboard.heroku.com/terms-of-service Apps: Yellow Data: No known issues at this time. Tools: No known issues at this time.

=== Availability of Common Runtime apps 2022-02-24T16:50:47.249Z https://status.heroku.com/incidents/2402 investigating 2022-02-24T16:50:47.249Z (1 minute ago) Engineers are looking into reports of connectivity issues to Common Runtime apps in the US and EU regions.

jrochkind19• 4 years ago

oh I didn't know that was a CLI command, nice!

Justsignedup• 4 years ago

is it related to an AWS outage? https://downdetector.com/status/aws-amazon-web-services/

jmartens4• 4 years ago

Functional monitoring of AWS by metrist.io (I'm a co-founder) shows no AWS problems. Downdetector is not a reliable source.

steeef• 4 years ago

"User reports indicate problems at Amazon Web Services" User reported outage sounds like people confusing the Heroku outage with AWS.

Gaussian• 4 years ago

We're down

andycloke• 4 years ago

All 6 of my apps & dashboard are down too.

ferajay• 4 years ago

Same here

ericpauley3• 4 years ago

Anyone still seeing outages? Our services are all back up.

ezekg6• 4 years ago

All our applications are still down.

ezekg6• 4 years ago

We're back up. Total 47 mins of downtime.

cbonser• 4 years ago

We are still down as well.

leetrout• 4 years ago

Heroku just sent an email to us confirming issues. Links to a status page but that isn't loading.

https://status.heroku.com/incidents/2402

neilobremski• 4 years ago

https://status.heroku.com/incidents/2402

(It took a few minutes to get this link to work for me)

smoldesu• 4 years ago

Rust's crate repository (crates.io) seems to be down too. I wonder if they're connected...

jrochkind19• 4 years ago

At first the heroku status page was still all green for me... although took 30 seconds to load. I guess when the status page takes 30 seconds to load, that's an indicator!

I did figure, okay, a 30-second-to-load status page probably means my app outage is a heroku platform problem.

(Also an indicator the status page is sharing too much platform with the platform it's supposed to be reporting on? Also in this case an indication that the platform problems are pretty deep?)

Interestingly, my app logs (via papertrail, which is still up) show that some traffic is getting through continually through this current outage, although I can't (and my monitoring app can't either, which pinged me).

escot• 4 years ago

Mines back up, at least 20min down time

bisRepetita2• 4 years ago

Some of my apps are back up. Others are not.

forgingahead• 4 years ago

Our apps are all down as well, Heroku Status not loading either (or just incredibly sluggish).

drstewart• 4 years ago

Cypress is also down

jamil7• 4 years ago

And we're back? I think.

jrochkind19• 4 years ago

My heroku app was back up again as of about 100 seconds before this comment timestamp.

My app was down about 35 minutes total, according to my monitoring.

craigkerstiens2• 4 years ago

It appears to only be affecting apps that have multiple dynos.

While Heroku is busy fixing the issue if you scale your app to a single dyno (assuming it can handle the traffic) it should restore availability.

bisRepetita2• 4 years ago

I had single dynos affected.

ericpauley3• 4 years ago

We also had single-dyno apps affected.

craigkerstiens2• 4 years ago

It may have been during an earlier part of the instance, but we just scaled multiple affected apps to a single dyno and all recovered. ps restart with multiple dynos and no effect.

jeremyjh• 4 years ago

I did too. I'm curious if you also have pre-boot enabled? I think that may set the routing configuration the same as if multiple dynos are up.

reubano8• 4 years ago

My 6 apps are all single dyno. Not sure if I have pre-boot enabled or not though....

jrochkind19• 4 years ago

My single-dyno app was affected.

However, while I could not connect to my app, nobody I know could connect to my app, and my monitoring service could not connect to my app for a ping... my app logs showed that some traffic continued to connect throughout the outage. So it was not entirely universal. And was clearly a routing problem of some kind.

reubano8• 4 years ago

Back up now!

dec0dedab0de2• 4 years ago

I've only ever used heroku for free tier level personal projects, but as I understand it they use AWS to do the actual hosting. I can understand an outage affecting their deployment process, but what could cause running servers to go down?

As I typed that out I remembered that they handle DNS, load balancing, and databases, so I guess any one of them.

alpb• 4 years ago

All it takes is a load balancer misconfiguration on a core service to take a large scale service down.

jrochkind19• 4 years ago

it indeed behaved like a routing issue of some kind (my app was still UP, and was still logging, just no traffic could get to it), and a heroku incident status line said "Engineers are recovering affected routing components," so, yup.

henryaj• 4 years ago

EU outage towards the end of last year was similarly bad, but lasted much longer. Asked Heroku for at least a refund for the dyno time when our apps were unavailable and was flatly told no.

bovage• 4 years ago

my apps are back up now.

leros• 4 years ago

I was able to get my apps up faster by restarting the dynos

ab-dm• 4 years ago

Well that was a fun one to wake up to. We were down for ~33 minutes overnight.

Heroku was Down

Related Stories