Heroku’s log-runtime-metrics Can Crash Your Rails App

http://siliconisland.ca/2013/04/25/herokus-log-runtime-metrics-can-crash-your-rails-app/

by ttronicm • 13 years ago

11 8 13 years ago

Do not run log-runtime-metrics for extended periods of time, as it can crash your Heroku dynos (and cause them to reboot) multiple times daily.

dschwartz88• 13 years ago

No. Freaking. Way. I noticed this same exact issue pop up on my New Relic interface. Multiple times a day our app would get H13's, crash, and then restart. Have been trying to get to the bottom of this ever since. Awesome detective work on this. Our app (and tech team) thank you very much!

fn5• 13 years ago

You're welcome! Our community runs on paying it forward, afterall!

fn5• 13 years ago

Hey, this is my blog post! Thanks for submitting it to HN. :)

ivarv• 13 years ago

In addition to the actual topic of the post (issues with a Heroku beta feature), I find this post pretty interesting for a number of reasons. It seems that for blog posts are the preferred method of reporting deeper issues. I guess it's a way of publicizing the work of tracking down the problem and earning developer cred. (Do people have blog posts for pull requests too?) Also, it's impressive to see how fast Heroku's Ryan Daigle replied. Hopefully this issue will be short-lived as the runtime metrics are pretty handy.

fn5• 13 years ago

Yeah, I actually sent the post to Heroku and NewRelic so they were the first to see it. I think someone from NewRelic submitted it to HN (just a guess).

Curious: What other method would you suggest for reporting deeper issues? I can't think of anything else that would allow for free-form writing like blogs.

mark_ellul• 13 years ago

Thanks for blogging this. Today I had my Account manager and another support person from Heroku pushing me to add that experimental feature, so I could see the memory usage of our dynos. I had reported on twitter and in a support ticket that the memory reported by newRelic didn't match the values that our Account Manager sent me. He sent me this because we were trying to add 4 Unicorn processes to a 1 Gig Dyno, and newRelic was reporting our app never went over 256MB. Our Account manager sent a different trace of memory usage.

I have lost all faith in the values that newRelic reports that come from Heroku.

fn5• 13 years ago

log-runtime-metrics does work -- just enable it for a half hour or so, observe the stats, then turn it off. Just can't keep it running long-term.

fn5• 13 years ago

It turns out I was wrong about log-runtime-metrics! Please see the followup post: http://siliconisland.ca/2013/04/26/beware-of-the-zombie-proc...

Heroku’s log-runtime-metrics Can Crash Your Rails App

Related Stories