The Switch from Heroku to Hardware

http://justcramer.com/2012/08/30/how-noops-works-for-sentry/

by zeeg • 14 years ago

100 48 14 years ago

timr5• 14 years ago

"On the limited hardware I run for getsentry.com, that is, two servers that actually service requests (one database, one app), we’ve serviced around 25 million requests since August 1st, doing anywhere from 500k to 2 million in a single day. That isn’t that much traffic, but what’s important is it services those requests very quickly, and is using very little of the resources that are dedicated to it. In the end, this means that Sentry’s revenue will grow much more quickly than it’s monthly bill will."

There's nothing in this justification that doesn't also apply to Heroku. The cost structures just aren't significantly different at a two machine scale. However, as people keep pointing out, the roll-your-own-cloud approach requires that you build and maintain a bunch of infrastructure that Heroku has already built for you, or you forego redundancy and fault tolerance that Heroku has already built for you.

The best lines of code are the ones you don't have to write.

zeeg8• 14 years ago

I didn't note in this post (but in others), but before I switched, my Heroku bill was almost $700 (and I couldnt get it to perform well), the current bill is far less even with growth.

timr5• 14 years ago

Yeah, I read your original post. I've worked pretty extensively with Heroku and with large, custom, in-house infrastructure, and I don't share your experience.

There's an I/O penalty for working on AWS, but it's on the order of tens of percent, not hundreds. I suspect that your original problems were related to working set size relative to cache (since Ronin => Fugu bumps cache by over 2GB, and you said that Fugu was working well).

Heroku's largest database has a 68GB cache at an (admittedly expensive) $6,400 a month. But even so, $6,400 is a small expense for a growing web application. A mediocre developer costs more than that. Trading off server cost for developer cost is an asymptotically bad bet.

count• 14 years ago

"DNS being slow (fuck it, use IPs)"

So, yeah, ops isn't hard at all, if you don't fucking take the time to do it right.

zeeg8• 14 years ago

It takes just as much time for DNS to propagate as it does for me to deploy a config change. It literally doesn't matter at this stage.

koko775• 14 years ago

Run DNSMasq locally (as in, same datacenter as the computers that will be using it) and tell it to cache. It's dead-simple to set up. Then point your computers to resolve using it.

You can even add to /etc/hosts and the computers using it as their DNS will resolve it. Depending on how much control you have, DNSMasq will also function as a DHCP server and TFTP server from which you can netboot other servers and do such nifty thing as automatic reinstalls. Useful if you have a separate, internal network and want to set internal IPs, too.

donavanm2• 14 years ago

"DNS being slow (fuck it, use IPs)" GAAAHH!! <table flip>. The 1980s called, they said youre an f'n idiot. Seriously, this is not "operations". This is "phoning it in" and/or "not understanding the problem".

Ps: to the op, not parent.

zeeg8• 14 years ago

The problem is DNS resolution isnt cached and potentially has to do a lookup everytime it does a connection. When I was doing that every request (potentially several times), it became expensive. Sounds like I understand it?

That DNS mapped to one node, which will rarely change. If it does, I can spend 30 seconds and deploy a new config, rather than worry about using virtual IPs or anything more complex, let alone with having to configure a cache which would have the exact same problems (delay to change).

Guess what, it's 2012 and the same shit that worked back then works just fine now.

There's a fine line between doing things right, and doing things just to do them.

papercruncher• 14 years ago

Not directly relevant to the blog post, but Sentry is an amazing piece of software. We're running it on an m1.small AWS instance along with a bunch of other stuff and it is rock solid.

jacques_chester2• 14 years ago

So basically nothing has gone wrong yet, therefore it's a slam dunk case for Doing It Yourself.

I'm sorry, but no.

You don't pay an ops guy or Heroku buckets of $$ for when things are going well, just as you don't pay $$ for software that only handles the happy case.

You pay $$ for someone who has fixed shit that went horribly wrong and has the scars to prove it. "That deep purple welt on my lower ego ... is where I only had a backup script and never tested the backups. This interesting zigzag is where I learnt about things that can wrong with heartbeat protocols ..."

Edit: though see below for a more nuanced discussion of reasons from OP.

zeeg8• 14 years ago

You're saying before Heroku (or the cloud) everyone paid an operations guy to do anything?

Nope.

I don't need to suffer the same acts of terror operations people have gone through to be able to avoid, prevent, or recover from them. I'm paying myself to be the operations guy, as well as the engineer.

jacques_chester2• 14 years ago

It's about value delivered and opportunity cost paid.

You're betting foregone engineering time against Heroku's value-as-delivered.

I don't think that the hours you will inevitably spend fixing the ops infrastructure will turn out to be very profitable.

For a large business, moving off Heroku to inhouse operations is probably justifiable, because they can capture sufficient value from a smart ops team to offset the potentially very high monthly bill Heroku would levy for their business.

But for a smaller firm, Heroku so abundantly oversupplies value compared to the bill that you will simply never be able to capture that value from internal work at anything like the same price.

It's like saying "I should stop buying food from the supermarket, it would be cheaper to grow my own vegetables!"

In a naive dollar-cost analysis this is true. In an actual consideration of whether raising a vege patch, bearing the risks of starvation, doing it poorly because you're new at it, spending the hundreds of hours of labour it will require -- means that letting a professional farmer do it is much smarter.

Gains from trade are gains.

I have a pretty strong opinion on this given how much time I've already sunk on similar work: http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign...

And for my secondary startup I am thinking I will just let Heroku handle it.

fdr• 14 years ago

Speaking of backups, you might want to run wal-e:

https://github.com/heroku/wal-e

It is not the best program I ever principally authored (since I know you are very discriminating Python programmer), however, it does work very well and has an excellent reliability record so far, seemingly for both Heroku and users of wal-e per most reports. I also tried quite hard -- and in this I'm pretty satisfied given both the feedback I've gotten and my personal experiences -- to make the program easy to use and administer, for one server and operator to many servers and one operator.

If someone is not doing continuous archiving and can't use our service for any reason, we do try to urge them to use something like wal-e or any of the other continuous archiving options. And to that end, I tried to make a pretty credible wal-e set-up only take a few minutes for someone who already knows how to install Python programs (I would love for someone to contribute more credible packaging).

eckyptang• 14 years ago

Actually before you embark on such things, you should probably be "the scarred person who knows how to fix shit that goes wrong". Then you know which things are likely to hurt you and you can avoid them.

To be in control of your venture means knowing all the corner cases.

I'd trust dedicated hardware more than Heroku as when it does go wrong, you're at the mercy of yourself and not others (other than the colo facility).

dangrossman3• 14 years ago

A bit off topic but this is a good thread to ask in:

Those of you running startups that don't colocate your own hardware, and don't run in a cloud, where do you rent servers from these days?

Most of my stuff is at Softlayer, but their RAM pricing is killer ($25/mo/GB).

zeeg8• 14 years ago

SL has become very expensive in the last couple of years (we host primarily there at Disqus).

For Sentry I'm using Incero, but will likely be switching to Hivelocity (or something similar), as I currently don't have internal network and that's a big, annoying deal for me.

Also, if you're loooking for deals or more information on hosts: webhostingtalk.com

benologist2• 14 years ago

Funny that we'd run into each other here again - Hivelocity is super cheap, I still have a massive, single server there and have used them for ~3 years now and actually all those servers I replaced were with them! They usually have really good specials, incredible value. But their live support is useless, the best thing they can do is log tickets on your behalf for the real support. Most of the time that doesn't matter.

SoftLayer have much better support, geographically diverse locations and a wicked suite of offerings. They're also giving startups $1000/month credit for 12 months, I got almost instantly accepted... I was still asking questions and then the rep was all "I made you an account here's the details".

https://www.softlayer.com/partners/catalyst

corford3• 14 years ago

It might not work for you if you're US based (and want your servers in a datacentre you can easily fly/drive to) but I hear a lot of good things about www.hetzner.de and the prices are fantastic. I plan to use them for the venture I'm currently working on.

boundlessdreamz• 14 years ago

Hetzner.de is awesome. They are cheap and reliable which is a combination you don't see often

Negitivefrags• 14 years ago

Softlayer will drop their RAM price to $12.50/mo/GB just for asking. Seriously, just ring up and they will offer it to you in a heartbeat.

After a bit more chatting with a sales guy I managed to get the price per gig of RAM well below that.

Softlayer is a bit expensive out of the box, but I've found them very open to negotiation.

hk_kh• 14 years ago

I started with two Linode basic vps. I keep them just because their customer support is great, and even let me stay one month without paying when I had problems with money. Also, all inbound traffic is free (basically it's what these two machines do).

Now, from UP2VPS[1] you can get a pretty good deal (1GB memory, 1TB transfer) for just $6/month.

[1] http://up2vps.com/

pooriaazimi• 14 years ago

Their customer service is "dreadful". The support guys are as unhelpful as it can be.

But if you don't need a support, their service is very affordable and good.

JDavo• 14 years ago

My company is self hosting sentry for about 30 Python & Java in-house services and it was the easiest deployment of anything that's taken me more than apt-get install to deploy.

kuanghe• 14 years ago

test,hahahaha

kingrolo• 14 years ago

I'm okay with the concept of trading cost/time for convenience, and why that might work for some folks, but even before we get to that argument, my experience with Heroku is that it just isn't reliable enough for client sites. Since reviewing it (over the last few months) we've had a couple of instances of real downtime (ie, greater than 30 mins), and a few spots of smaller amounts (a few mins each). We don't get that from our Linode+Puppet sites (taught myself Puppet as we went along, I'm a dev rather than an ops guy really).

taligent• 14 years ago

Well I guess you are lucky then because Linode has had massive downtimes in their London and Dallas data centres in the last few days and when I was a customer their Fremont data centre went down all the time.

http://status.linode.com

Heroku at least uses EC2 which will be far more reliable over time.

citricsquid• 14 years ago

Not sure if the London one was really "massive", I have 12 Linodes in London all on different machines and I had one become unavailable during the downtime a few days ago and it was resolved very quickly.

slurgfest• 14 years ago

Heroku goes down every time US-East goes down, so it isn't true that its reliability is the same as EC2's in general

davestheraves• 14 years ago

Sounds to me that the main issue was running out of RAM on workers. Would this not be soved by moving to another cloud provider (such as AWS) where you are not limited to the tiny RAM provided on Heroku?

Is this not similar in strategy to choosing hardware and spinning up your own stack?

EDIT: I getting at the rather sweeping statement against all cloud providers based on a specific Heroku problem

mark_l_watson• 14 years ago

I had the same thought: why not run the workers on a few large memory EC2s and use Heroku for everything else. Run on east coast region for free bandwidth between Heroku and EC2s.

Xylakant• 14 years ago

The moment you do this, you forego one of the biggest advantages of using heroku. As long as you're on heroku only, you don't need to take care of securing the underlying stack - that is firewall rules, OS-updates, general maintenance. The moment you spin up a single AWS instance besides it, it's your problem. Depending on your use-case it could be a better choice to just go all the way to dedicated hardware: The primary advantage AWS has over dedicated hardware is flexibility. You can spin up instances depending on your current need. If your load behavior is a flat, predictable curve you might just not need that - and then real hardware is cheaper in most cases.

josephlord• 14 years ago

This stuff is really useful even if you don't want to move off Heroku. Just knowing that you can is really reassuring in case they stop offering what you need. Simple how to guides are even better.

It was the fact that Heroku offer an environment to run fairly standard Rails + Postgres that made me pick them over the more unique and harder to move from Google platform. Even though I was starting from scratch.

It's always good to have an exit route.

The Switch from Heroku to Hardware

Related Stories