Rails 3.2.13 - Performance regressions and major bugs

http://blog.bugsnag.com/2013/03/20/rails-3-2-13-performance-regressions-major-bugs/

by sephine • 13 years ago

125 72 13 years ago

blog.bugsnag.com

scottshea• 13 years ago

You know when you're already having performance issues on Heroku having a performance regression is not fun.

nifoc2• 13 years ago

How is this

> […] handing that task to Sprockets instead of resolving internally […]

even remotely related to performance issues because of random routing? You're mixing two issues that have nothing in common.

josephlord2• 13 years ago

1) Grandparent has performance problems on Heroku.

2) The Rails release has performance regressions (i.e. in at least some scenarios is slower).

In situation (1) you don't want to add further slowdown so (2) is a bad thing.

Grandparent doesn't even blame his Heroku performance problems on the routing issue (although it is a plausible guess). Actually effect of the routing problem can be significantly worsened if even a subset of requests are slowed down.

kawsper• 13 years ago

The asset path and sprocket performance hit doesn't seem to affect most Rails apps, they precompile their assets, so that shouldn't be a problem.

calinet64• 13 years ago

But it does slow down dev, which doesn't not suck.

sergiotapia5• 13 years ago

At this point I'm seriously considering dropping Rails as my framework of choice and reverting back to ASP.Net MVC.

Why?

1. Security fixes are released very quickly (good thing), but more often then not they break existing code (bad thing) - and while you _can_ wait for the next patch to fix those break points, you're left wide open since everybody can see what was broken and how to exploit it.

2. I'm not nearly smart enough with Ruby to apply 'band-aids' I read people write in Rails source code. I just don't have the time and energy to fix code in the actual framework when I need to be fixing _my_ code.

======

So there you have it; I'm bailing.

It was a fantastic ride, Ruby is a beautiful language, but Rails is just a clusterfuck (_for me_).

I need stability and predictable behavior. If that means releases every 6 month to 1 year, so be it.

So long and thanks for all the gems.

masklinn2• 13 years ago

You do know there are other Ruby frameworks than Rails, and other development environments than ASP and Ruby right?

sergiotapia5• 13 years ago

Yes, I used to use ASP.Net MVC for pet projects, then switched to Rails - I also use CakePHP and Django for actual paying work/contracting.

But thanks for the passive aggresive comment.

eduardordm• 13 years ago

I've been running Java and Rails since 2006 in my company (large), we use C# for a small number of desktop applications.

This will come out harsh, but I'm being honest: If you are not willing to fix/adapt/improve the framework you are using, you better not be using ASP.NET also (try a google search on asp.net issues), or any framework at all. In fact, developing larger more complex software might not even be possible without adapting frameworks for your specific problems.

That said, our run so far with Rails and Java (we are still using 2.3!!!):

- We have 38 patches applied to Spring

- 4 patches in apache santuario.

- One security patch on Grizzly (glassfish)

- Rails: One single pull request https://github.com/rails/arel/pull/174

cooldeal• 13 years ago

>If you are not willing to fix/adapt/improve the framework you are using, you better not be using ASP.NET also (try a google search on asp.net issues), of any framework at all. In fact, developing larger more complex software might not even be possible without adapting frameworks for your specific problems.

Huh what? Stack Overflow with it's extreme load works quite well with ASP.NET.

>try a google search on asp.net issues

Okay, I did, what I am supposed to be looking at?

narrator• 13 years ago

It's really nice being able to trace into the guts of hibernate, spring, etc to see what the heck is actually going on and fix it if it's broken. When I used to do Microsoft stuff, I'd run into bugs in the stack that I couldn't fix. This was enormously frustrating and time wasting. The typical advice given out by support was re-install or buy the latest upgrade. There were also a lot of deliberately imposed architectural limitations to prevent you from working around lack of "enterprise" features in the basic versions which were incredibly frustrating and the enterprise version pricing was astronomical.

Axsuul• 13 years ago

Would you really be willing to give up the productivity benefits and developer happiness over a few mishaps? Also, it's generally a good idea not to upgrade to the next software version until seeing the repercussions its had.

sergiotapia5• 13 years ago

That's the thing! I find myself ping-ponging in my mind between "But Ruby and ActiveRecord are so beautiful and easy to use." and "But what happens if the 'magic' breaks?"

Trust me, I have been thinking about what you say every day for the past 5 days.

static_typed• 13 years ago

You are now in the place I was recently -- the decision we tool was to migrate onto Python and their web ecosystem.

We haven't looked back since -- well, we have given an glance or two back since, and viewing the slow-motion car crash that is Rails today, we just feel sorry for those left behind that are discovering 'The Rails Way'.

Ironically, given DHH's comment on Rails about it being 'Omakase', I should point out that it translates as "I'll leave it to you" -- security, I'll leave it to you, software engineering, I'll leave it to you.

Food for thought indeed.

Remember -- Ruby to pose, Python for Pros.

lparry• 13 years ago

Wow, you're either one of those incredibly rare ruby devs who actually run windows, or you're going to switch your entire environment in order to switch frameworks. Something doesn't smell right

sergiotapia5• 13 years ago

I'm a polyglot. I don't encase myself in any 1-thing. I use Mac at work, Windows at home for media and games, and Linux Mint 13 on my laptop for Ruby/Rails. I only used Linux because of Rails, but since I'm switching back (I have Visual Studio 2012 downloading as I type this), I'll have no need for Linux on my end-user machines anymore.

calinet64• 13 years ago

You're on the cutting edge of an extremely popular framework.

If you want, you're welcome to use Rails 2.3 or older and you'll encounter many fewer changes. I imagine the 3.0 branch will have much the same stability once major development moves to 4.0.

rst2• 13 years ago

FWIW, the Rails security team says they're about to drop suport for 2.3, and this time they really mean it. (They say.) The current announced security policy is that once 4.0 gets released, they'll only be doing security patches for 4.0 and 3.2; 2.3 will be left out in the cold.

It's quite possible that someone will pick up the baton, as there still are a number of production 2.3 apps out there, and porting to 3.0+ --- more a "port" than an "upgrade" --- is a real pain. But I'm not sure anyone has stepped forward yet, and until someone does, you're taking your chances.

jjoos• 13 years ago

You should seriously consider running mono>=2.10 with asp.net mvc. It completely fixes your rails security issues, while still letting you use ruby as your main languages.

Also you gain a _lot_ in security by obscurity and hipster points. I don't know about stability, haven't tried this myself.

Check out the mono project for details: http://www.mono-project.com/Mono_Project_Roadmap

threedaymonk• 13 years ago

I've not heard of anyone using IronRuby in earnest, and it doesn't seem to have been updated in a couple of years. Is it a realistic option?

ryanSrich• 13 years ago

As someone who is relatively new to Rails can someone explain to me why it seems to update more than anything else I've used?

davidw• 13 years ago

Mostly to give people on forums something to bitch about.

Serious answer: a few months ago, some serious vulnerabilities were found in Rails. This apparently had a "blood in the water" effect on security people, who started dissecting the entire thing. Predictably, more vulnerabilities have emerged and rapidly been fixed.

All in all, it means that things are improving because it's getting a ton of attention.

philipDS• 13 years ago

This. I think the rapid release cycle of Rails is a good thing, rather than a bad thing. The past few months, some serious vulnerabilities were found in Rails, and I guess this made the Rails core team a little anxious.

One might argue that these performance regressions are unacceptable, and they are. I agree. Still, Rails is relatively young framework and these things happen. I expect any framework that is widely used will see some serious vulnerabilities in the next few months/years.

By the way, a good explanation of what these past Rails security issues mean is explained on patio11's blog (http://www.kalzumeus.com/2013/01/31/what-the-rails-security-...).

OT: I'm also writing a book on upgrading your Rails 3 app to Rails 4. With these rapid releases, a lot of things change and it's not always clear what has changed or what is new. Feel free to check it out at http://upgradetorails4.com/.

evilduck• 13 years ago

Bad timing. The JVM has had an abnormal increase in security patches over the same time frame too.

bluedino• 13 years ago

You don't have to stay on the bleeding edge. We're stuck with Ruby 1.8.7, and we've got a mix of Rails 1, 2, and 3 apps. For better or for worse.

thinkbohemian2• 13 years ago

Not happy? Do something about it: try release candidates, report bugs, and help triage and fix reported bugs.

foobar2k• 13 years ago

I completely agree with this. We (bugsnag) are running on rails 4 beta right now to help iron out any production issues before that is released.

We didn't, however, realize that "patch" releases also have release candidates. I'd recommend following the rails blog (http://weblog.rubyonrails.org/) which announces release candidate builds too.

thibaut_barrere2• 13 years ago

Thanks for the blog post - and I didn't realize that patch releases also had RC.

It looks like this release was quite fat, for a patch release.

I added "check rails diff" to my deploy todo list.

EDIT: I just noticed that http://weblog.rubyonrails.org/ doesn't have a way to subscribe by email. I signed-up by email with this site instead http://blogtrottr.com, in case it helps other here.

thinkbohemian2• 13 years ago

You can also subscribe to the rails-core mailing list, all betas, RC and full releases get announced there.

calinet64• 13 years ago

Ok, yes, that's an extremely great and open-source way to handle this.

But seriously, can we at least expect moderately stable releases? I do not think that's too much to ask, especially for a framework which has, in the past, achieved that.

jrochkind1• 13 years ago

Okay, some of these problems are (unintended) direct side effects of the security fix bugs.

Like I the one about scopes that effected Github -- github quoted [this commit](https://github.com/rails/rails/commit/f980289fd2c1b9073a94b5...) as introducing the bug, which is the commit meant to address CVE-2013-1854.

So, okay, bugs happen, even with security fixes, I can forgive bugs.

But others of those performance regressions... " handing that task to Sprockets instead of resolving internally, which seems to be the cause of the performance issues."

This doesn't sound like the unanticipated side effect of a security fix. This _sounds_ like an independent change they threw into the same patch release with four (four!) security patches.

Why is Rails putting security fixes and other non-security-related enhancements together in the same patch, so you can't get the security fixes without also getting the possibly buggy new features/refactors? Does Rails actually do this by policy? Was it a mistake?

Bugs happen, including bugs in security fixes, sure. But it's BECAUSE bugs will happen that you don't throw unrelated new features or refactors into the same patch release as a security patch, so the only way to get the security patch is to risk MORE problems from the other stuff too! Come on!

badgar• 13 years ago

AFAICT Rails does point releases by just taking whatever's at head on the 3.x master branch, building a release candidate there, and iterating until nobody is reporting bugs.

This isn't a security patch - a stable branch with deliberately cherrypicked fixes - it's just a plain old "point release." And as is to be expected with Rails at this point, point releases break apps.

obviouslygreen• 13 years ago

And as is to be expected with Rails at this point, point releases break apps.

As a primarily Django guy with an interest in Rails... is this hyperbole/overstatement or is this a common perception?

timdorr• 13 years ago

This is a weird comparison, but they should adopt the same release policies as Drupal. Security releases are done separate from maintenance releases, even if they are released at the same time.

That allows you to provide the security releases as a separate patch file so you can apply individually. You can skip maintenance releases that cause problems but still keep things secure.

rst2• 13 years ago

Here's a pointer to the security fixes in the release as a monkeypatch to 3.2.12, without any of the other cruft:

https://github.com/discourse/discourse/blob/master/lib/freed...

The discourse folks who threw this together have an app (Rails-based forum software) that was hit very hard by the performance regressions.

grey-area2• 13 years ago

The rails team already does this. Here's an example patch for one change:

https://groups.google.com/forum/#!msg/ruby-security-ann/o0Ds...

Unfortunately that wouldn't have avoided this issue, as the problem was actually with the small security patch.

FuzzyDunlop• 13 years ago

I think the only mistake made with this patch release was merging in 250+ commits, ranging from last year to the present day. Maybe that's standard for such a large project, but it feels like a hell of a lot of tweaks for a minor release.

As for the rest - I think the issues in January has put Rails in the spotlight, and while scrutiny is certainly warranted, I think the extra pressure on the project over the past months to clear this stuff up has had its consequences.

yxhuvud• 13 years ago

I think the problem is that they do too many patch releases and not enough minor releases. This could easily have been a minor.

renownedmedia• 13 years ago

Since when do Rails developers worry about performance?

1qaz2wsx3edc• 13 years ago

296 commits for a security patch is silly; What the shit tenderlove.

calinet64• 13 years ago

Seriously. Usually Rails feels like riding in an Uber, but this release felt like Crazy Taxi.

ChrisInEdmonton2• 13 years ago

There's been a few people suggesting that, if you are bitten by a bug, you fix it and submit a pull request. That's a good idea. I did exactly this to a regression introduced in 3.2.9, complete with a unit test and a pull request against 3-2-stable and one against master. It took two months to get anyone to look at it, and while one of the two pull requests has a thumbs-up, it's still not been merged. Admittedly, this isn't a security issue as far as I can see, but still.

steveklabnik3• 13 years ago

I personally read every single issue and comment filed against rails/rails. I often don't comment if it's not a part of the codebase I'm not familiar with, and it's the same with merging.

Can you point me to the specific ones so I can look at them again?

tvongaza• 13 years ago

Pretty sure Chris is talking about: https://github.com/rails/rails/issues/8856

ChrisInEdmonton2• 13 years ago

Yeah, tvongaza is correct. Admittedly, it's not a _big_ issue, and perhaps I'm going about things the wrong way. If you could tell me a better way of getting this issue fixed, I'd love to hear it. The only other time I tried, it took more than two years. Luckily, it was pretty easy to monkey-patch our own code to deal with this, once it broke our live site.

bratsche• 13 years ago

I disagree with the final statement of this, that a "security only" release would have somehow helped the situation. I feel like part of the problem is that things are being merged into 3-2-stable and nobody is using them because they're never put into a release.

3.2.11 and 3.2.12 were both security-only releases, and so fixes and patches that were not security-related have just been piling up for the longest time.

Seems like a "release often" mindset might be better and might ensure that things are actually being used. Sure, I know someone is going to blast me saying that bugs like this shouldn't make it into 3-2-stable at all, and since this is Hacker News so whoever says that knows they're ten times the developer that everyone on the Rails core team is. But the reality is that bugs make it into stable branches, and almost nobody is running against the stable branch (Rails core people are probably mostly running against the 4.0 branch, everyone else is running against the latest release).

mnarayan01• 13 years ago

Actually 3.2.12 also added some sort of silly (and broken) patch related to quoting numeric values in ActiveRecord. The primary thing I want out of 3.2.13 is reverting this.

To your more general point, while I don't disagree with your diagnosis of things piling up in stable, I'm also not a big fan of your proposed solution. It looks to me like 3-2-stable is getting filled with a lot of stuff that just should not there; increasing the release frequency is not going to solve this problem.

martinced2• 13 years ago

I'm sure that Rails is great and powers many websites and that many devs loves it.

I'm also sure that people are working hard to fix these recent security issues and perfs issues.

But as a non-Rails dev I can tell you one thing: with all the attention that Rails got lately I'm sure I'll never be learning Ruby / Rails.

I'm into Clojure right now. Next target is Go.

Sadly all these Rails exploits do have a negative effect for Rails: I'm not saying this to criticize Rails. I'm saying this because I'm honestly 100% sure I'll never ever be doing anything serious with Rails and I know I'm not the only one in this case.

So, Rails devs, fix this and fix this rather sooner than later because every exploit is missed opportunities.

erydo• 13 years ago

When you bring up Clojure, I think it's fair to point out that, beyond even the default-unsafe reader thing brought up a couple weeks ago, the JVM itself has had several fatal security issues recently. No platform is completely without bugs.

That said, I don't really know if RoR has had more than normal or whether they're just more widely talked about. It does have a huge and vocal user-base.

ldh• 13 years ago

I don't write Ruby either right now, but I would suggest you not discount it as a nice language because of your perceptions of Rails.

Rails 3.2.13 - Performance regressions and major bugs

Related Stories