I left Heroku for OpsWorks and shaved 40% off response times

http://www.stefanwrobel.com/heroku-to-opsworks

by swrobel • 12 years ago

73 49 12 years ago

andrewvc• 12 years ago

Opsworks is nice, and it's a good packaged solution. A large number of smaller apps would benefit from a switch. It's also more fairly priced.

That being said, opsworks is still based on chef, which is still cumbersome. For most people Ansible is far easier to deal with if they don't fit into the opsworks box. The overhead of starting from scratch with ansible can be made up for with the faster overall development time.

swrobel10• 12 years ago

Good to know! I've been meaning to look into Ansible & SaltStack, and you gave me the kick in the butt I needed.

EDIT: ohai Andrew! Thanks for the comment.

wc-2• 12 years ago

Big thumbs up for ansible, I have had great results with it it in a number of different production environments, I think the docs are great, and the devs in the irc channel on freenode are very helpful.

If you are about to embark on the journey of provisioning for the first time, or if you are looking for something a little more straightforward than chef/puppet (in my opinion at least...), I highly recommend checking out Ansible!

phowat• 12 years ago

Another thumbs up. I'm currently in the process of migrating away from CFEngine to Ansible and it's been a breeze.

sync• 12 years ago

Seems like Opsworks does a lot more than just the provisioning, no?

"AWS OpsWorks can scale your application using automatic load-based or time-based scaling and maintain the health of your application by detecting failed instances and replacing them. You have full control of deployments and automation of each component."

How would you suggest using Ansible to cover that side of things?

wc-2• 12 years ago

Autoscaling groups with ansible-pull + cloud init to configure instances as they are spun up by AWS is what I've had the most success with. So your autoscaling group spins up a new instance using a prebaked AMI, and on first boot cloud-init will run ansible-pull locally to grab the latest configs and start up services as needed. Alternatively you can just use ansible to bake a new AMI as needed and skip the ansible-pull + cloud-init part altogether.

imperialWicket• 12 years ago

This just introduces additional tools/products to the stack, but you can use Ansible with the Netflix OSS stack, such that Ansible provisions your AMIs and Asgard manages auto-scaling and deployment (with very fine-grained control).

darklajid• 12 years ago

Ansible: I was in the "Don't go there" camp a while ago - they documentation was horrible, the whole website a mess and lots of things weren't discoverable at all (people using ansible told me that they used google site search to find stuff in the manual).

I returned a couple of days ago to look at the current state of things and I have to admit that I was impressed. Technically I really liked ansible for quite some while. Now it even comes with decent documentation. I'd back the suggestion to check it out.

ghayes2• 12 years ago

I was really excited by ansible, but I quickly found out that using `register: result` and `when: result|success` throughout so many playbooks meant I was back to the mess that is understanding the flow of a chef script that you didn't personally write.

Maybe that's not a problem since I rarely found an open source ansible playbook for anything I needed and had to always write my own.

mercurial• 12 years ago

The doc is indeed pretty good, and available offline (ansible-doc). I quite like that it uses a very declarative syntax, and it's relatively easy to get idempotent operations. And the fact that you don't need to install anything on the machine you administer is definitely a strong point.

One downside is that I can see how tracking down role dependencies would be tricky if you had many different roles. I also haven't tried at scale.

tgeek• 12 years ago

"That being said, opsworks is still based on chef, which is still cumbersome."

re:chef

I don't think this is necessarily a fair assessment and true for all. There are a ton of community cookbooks out there, and a very large community now online to help people with Chef. I've been using Chef for years and like anything else once you get over the learning period, it becomes second nature. Any tool that you use will have some learning curve to it, and so the trade offs that make it cumbersome for some, make it easy for others.

Also, you could technically Chef in Ansible in OpsWorks(if OpsWorks met other needs for you), but thats just crazy talk haha

iagooar• 12 years ago

While I can understand the advantages the author of the article mentions, I must say that one should think about making the move to OpsWorks, because you can't compare their prices without considering other costs / tradeoffs.

A friend of mine and I have been working on a startup for the last 8 months. It has been tough as we have been working full time for a company and hacking like crazy once we got home, or on weekends.

We chose to use Heroku because of the dead simple workflow that would allow us to spend time developing features instead of maintaining the infrastructure. This may sound like a cliché, but boy was it a good decision. The platform is rock solid, deployments are a bless and the add-on system is just one of the best time savers you can get.

We know OpsWorks (or earlier Scalarium) from our full-time job, and we know how cool it is. But we also know that it requires a lot of time and additional effort to keep things running.

This may pay off if you run a company which can afford it, but you should think about it twice if you have limited resources (and other, more important things to do!).

Even now, after we have launched, we still see no reason why we should move from Heroku to OpsWorks (or another platform). We have made some optimizations and clever (to us) decisions to avoid running into scalability problems. Maybe we will think about migrating once we get a ton more clients and even more traffic. It is cheaper to just add some dynos than spending time on server administration.

In the end, it's all about making the right decision, so don't hurry and if you think Heroku is too expensive for your business, maybe you are not making enough revenue from your customers. On the other hand, if you have the money and time, than go for it.

swrobel10• 12 years ago

Agree with most of what you say, but adding dynos increases concurrency but doesn't do anything to reduce response times. Sometimes you just need a faster box.

thinkbohemian4• 12 years ago

Or a bigger (more RAM) box, I posted this to the Ruby Rouges mailing list on tail latencies: https://gist.github.com/schneems/8849116

Concurrency is only half the battle with tail latencies. Of course dedicated CPU helps as well :)

nthj• 12 years ago

I'd be really interested to know more about how the switch shaved 40% off of the response times. What was the major bottleneck? RAM? CPU? Network IO to PostgreSQL? Network IO to memcache?

Heroku is offering 2X and soon, 4X dynos; did he experiment with these?

The author also sounds rather excited about nginx serving static assets. The Rack::Cache problem on Heroku is easily solved with a CDN layer (and Heroku even recommends you do so.) (Cult Cosmetics /is/ set up with CloudFront, so the author obviously knows this.)

There are a lot of Rails architecture smells in this article, such as worrying about the 60-second boot timeout and the 30-second request timeout. [1] I have to wonder if the time invested in learning Chef and managing the architecture may have been spent making the app more performant.

Developer time is really, really expensive. Not only can you bill $100-$150 hour instead of learning how to use Chef, you could be shipping features for your own products, which can bring you passive income forever. A lot of companies are successfully running on Heroku, and it seems strange to throw everything out for the explained reasons.

[1] Specifically, adding 'require: false' to your Gemfile and only loading heavy libraries when you need them greatly helps with boot time. Ideally, you won't run any image processing libraries (like RMagick) except in background jobs, and your main app should not even need these libraries available. Same goes for PDF generation, scraping technology, or any other heavy processing.

The 30-second timeouts either need to be handled via background jobs (the author mentioned they use sidekiq) or uploading directly to S3, side-stepping the web process entirely.

swrobel10• 12 years ago

I was using 2X dynos on Heroku. I did have memory issues and had to continually reduce the number of unicorns that I was running. I think CPU and disk were the major bottlenecks, but again, Heroku makes it hard to track down exactly what the cause is.

Also, we're an ecommerce site so we rely on a lot of external services (specifically for payments) where we have to run the call in the web process ... hence the issue with 30s timeouts.

thinkbohemian4• 12 years ago

I agree with the CDN comment. Please use a CDN to serve your assets no mater what you host on. It's so easy, and the performance gains to the end user are good, it should be an obvious choice.

Aqueous2• 12 years ago

If companies think Heroku is more expensive they are failing to factor in how expensive it would be to have an employee to do all the work that Heroku takes off their plate. I work 45 - 50 hours a week developing. If I have to do devops - provisioning, deployment and system administration work - to maintain a server infrastructure there goes 15 - 20 hours! Surely my company is paying less to Heroku to manage that redundant, and at this point formulaic infrastructure than to me to do the same things that they've automated away. I can then work on our products. That seems less expensive, not more.

grey-area2• 12 years ago

Once you have done the setup once and have it all automated, it's more like 1 hour per week keeping on top of upgrades, backups etc. So for many people heroku is an expensive option, even when starting out. Nothing wrong with using it, but outsourcing your infrastructure does not come cheap, compared to other options like a VPS.

incision• 12 years ago

One limitation with OpsWorks is that as far as I know, there's no option to use spot instances whereas it's not a problem with EC2 launch configurations for AutoScaling Groups. Also, ASGs and OpsWorks both have options for time and load-based scaling with different interfaces.

It's easy enough to understand how this happened as OpsWorks came about via an acquisition, but consistency would certainly be nice.

Presenting overlapping yet incompatible features under the blanket of AWS is confusing and potentially pretty frustrating.

Ryel2• 12 years ago

How's Science-inc working out for you?

swrobel10• 12 years ago

That's a whole other conversation ;)

Ryel2• 12 years ago

Hopefully good, lol. I've been watching them recently and thinking about reaching out with a couple prototypes.

juandazapata• 12 years ago

At WeHostels we moved from Heroku to Rackspace last year and we shaved ~30% off response times.

thinkbohemian4• 12 years ago

Curious from a user experience perspective would you consider going back to try out the PX dyno service? Do you use chef or puppet or any other software to manage your Rackspace instances?

ericd4• 12 years ago

Based on his mention of SSD-backed instances (c3.large) being only $100/mo, I looked at EC2 pricing again to see if it had become more competitive. Turns out it only has 2x16 gigs of storage and 1 physical core.

Looking at the c3.8xlarge instances, they're specced pretty much exactly like the servers I just built for $9k/each, except they have half the ram (60 vs. 128) and half the storage (2x320 vs 4x320). And they cost the same to rent for 6 months as it does to buy one, including coloing.

So, if you need more than a fraction of a physical server, EC2 still loses pretty hard on the pricing. It seems as though it has become competitive with Softlayer, though, at least.

EDIT: Ah, but if you pay $11.6k upfront to reserve it for three years, it reduces the monthly bill from $1728 to ~$370, which is a much better deal than Softlayer offers. That's actually a pretty reasonable premium compared to racking them yourself, though you lose flexibility and the hardware is worse than what you'd get yourself. Well played, Amazon.

ericd4• 12 years ago

Your servers are also likely 2-5 times as fast in the CPU department too.

FYI, you've been hellbanned, Sssnake.

They use slightly better processors for the C3s than I use (2x8 core 2.8 vs. 2x8 core 2.6). I'd assume that if you get the largest instance, you probably have the whole machine to yourself and don't have issues with bad neighbors, so it's probably actually slightly faster than mine.

vinceguidry3• 12 years ago

Possible he's been hellbanned, but his latest post is the only dead one.

joeblau• 12 years ago

Nice writeup. I recently switched from Nodejitsu over to Heroku for my open source project gitignore.io and I noticed there was a slight slowdown in response time. The app score index in New Relic went from around 92 to 100 though so I guess Heroku's servers on AWS were a bit more reliable. I also added in a CDN to increase the speed of serving the static assets.

All in all, I can say that I'm happy with the transition, but I'm also running one Dyno which I'm far from maxing out. I feel like for a larger product, Heroku would be more problems down the line than it's worth.

swrobel10• 12 years ago

Exactly how I felt, great at small scale, but increasingly painful to use as you reach scale.

By the way, great tool, thanks for building it!

banachtarski• 12 years ago

What do Opswork, Chef, and the Heroku platform have to do with response time?

Did the author investigate why the response time were that slow to begin with?

swrobel10• 12 years ago

I wouldn't say they were "slow to begin with" but improved speed is always nice. I think the major contributing factor was access to newer, more powerful instances. Heroku Dynos run on very old EC2 instances.

DrJ• 12 years ago

FYI OpsWorks runs their Chef on Ruby 1.8, make sure your cookbooks passes on 1.8, and it doesn't depend on +1.9 :)

This is nothing on AWS OpsWorks, as they greatly simplified interface into managing clusters (for deployments, organization).

I found about this recently as I updated a community cookbook(python), as the Chef community is moving forward (with Chef 11+) that 1.8 has reach EOL and 1.9 is stable (2015) and 2.0 current, and 2.1 is new-hot-ness.

I left Heroku for OpsWorks and shaved 40% off response times

Related Stories