Building Servers for Fun and Prof... OK, Maybe Just for Fun

http://www.codinghorror.com/blog/2012/10/building-servers-for-fun-and-prof-ok-maybe-just-for-fun.html

by bussetta • 14 years ago

86 83 14 years ago

randomchars2• 14 years ago

    > But not the kind that go in your home. No, that'd be as nuts as the now-discontinued Windows Home Server product.

What's wrong with home servers? I've been wanting to build one for quite some time.

ios84dev• 14 years ago

I think he means rack mount home servers, those are pretty nuts because of the noise that most rack mount cases make. Figure in a 1U case you normally have at least 4 15K RPM fans going at all times, even if you put it into "quiet" mode its enough to hear from the next room normally.

timc34• 14 years ago

My rack mount case is silent enough to record with a high quality condenser microphone in the same room. But it is 4U with 120mm slow fans.

Its all about how you build them. Obviously a 1U with 40mm fans running fast could make a lot of noise.

zdw• 14 years ago

HP makes a "Microserver" that's designed for home/office use: http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-42...

Quiet, ECC RAM, and decent speed for most home-related tasks (backup, etc.)

stordoff• 14 years ago

I can highly recommend these. I recently got one of these for £100 (after cashback), and it is a brilliant piece of kit. I've got 4x1.5TB HDDs, and a 250GB OS drive, in mine, running Windows Server 2008 R2, and the performance is pretty good. 50MBps+ over a gigabit network is what I'm getting, which is more than enough for me.

burningion2• 14 years ago

In lieu of building a home server, I highly recommend the Synology line of NAS machines. They've got everything you could possibly want out of a home server, with a much faster configuration. Automatic backups, a built in Torrent server, Plex media server, and remote file access all in one container.

Seriously a great product and value if you're looking for a home server.

randomchars2• 14 years ago

That looks really nice! Torrent server is one of the primary uses I would need this for so it's tempting.

evandena• 14 years ago

I don't want to come across as a comment crapper, but I've been recommending FreeNAS for years now. ZFS is perfect for home NAS purposes.

hkarthik• 14 years ago

+1 I love my Synology NAS. Truly a great product.

mdgrech23• 14 years ago

Hardware is cheap, programmers are expensive. System admins even more so. For this reason I feel like cloud hosting providers are the way to go for bootstrappers or startups. Once you get big you can do things like FB and create your own datacenter. However the enjoyment and power that comes from having full control over your entire environment should never be over looked.

Detrus• 14 years ago

But it also costs programmer time to tweak software for weak hardware. Sometimes it makes sense to spend that time on setting up a few big iron servers yourself.

Muzza• 14 years ago

> System admins even more so.

Really? Honest question.

timc34• 14 years ago

Depends on your business. If your small business is writing code, having a sysadmin around and not 100% busy they are could be costly compared to hiring another programer.

But looking at general rates, I would say no.

thaumaturgy3• 14 years ago

Good sysadmins, yes. Keep in mind that the ideal sysadmin is someone who knows the guts of BSD or Linux (typically, not both) inside and out; can troubleshoot the strangest of problems very quickly; is constantly measuring and improving performance; and can hack together shell scripts or Python or even C code as needed.

Linux and BSD are hellaciously complex and prone to very strange behavior once you start taking them into high-performance land.

If you're OK settling with, "can read syslog and look stuff up on the web", those guys are cheap.

donavanm3• 14 years ago

Not more expensive, more difficult to source. IME - mid level sys admin/eng job code will get 10-20% less compensation than an SDE. A principle level SEs comp will also top out before an SDE.

For junior or mid level SE I've seen about 10% of candidates meet the hiring bar. SDEs I've had less involvement, but I'd hazard 20% get an offer where I've been.

The other problem is the pipeline. Colleges keep Pumping out SDE candidates from comp sci and maths. These junior SDEs will also have a reasonably consistent skill set and experience. There are (effectively) no colleges or curriculum creating new SEs. The incoming candidates also vary widely in their depth and breadth of relevant knowledge. Pretty sure that contributes to the higher negative signal.

tobias3• 14 years ago

I myself like to do this. But only for home servers. I would never do that for a "serious" application. There are several reasons, the most important one being that this is a perfect example where outsourcing the work is both cheaper (economies of scale), more reliable (parts are tested better) and less risky (companies which do that at a larger scale have better risk management). If one the SSDs dies he has to drive there and switch it out, instead of one person simply doing that for the whole data center. This is simply inefficient.

And of course Amazon hosting is more expensive. It is more flexible; you can spin up instances at your whim. You pay for that. It would be better to compare it with standard dedicated server hosting.

peterwwillis3• 14 years ago

Have you worked with "professional support" in a colo before?

When a hard drive dies under vendor support, one guy from the vendor drives out there to replace it. Half the time they don't test it after it's replaced unless you request it. Sometimes the parts are duds. Sometimes they bring the wrong part, or it doesn't fix whatever was broken on the server.

If it's the datacenter's remote hands, it can be hours, and on rare occasions days, before somebody starts working on your issue or even answers the phone or e-mail. Same issues come, and you have to have your own spare hardware for the remote hands to use.

It's highly variant based on the datacenter and vendor(s) and other issues. There's no guarantee that outsourcing will be reliable. You have to find the best one possible and build a good relationship.

codinghorror12• 14 years ago

Having done this twice before, I strongly, STRONGLY recommend picking a colocation center that's less than an hour drive from someone who works for the company. Or you.

Just in case, but that does come up from time to time and if it's many hours away (or worst of all an airplane ride) you will get hours/days to fix things as you noted.

vmind2• 14 years ago

I much prefer the middle ground of dedicated servers over the hassle of colocation and hardware management. (16GB RAM quad core with 2TB raid1 from OVH for £65 a month are a good affordable level, and very quick and easy to spin up new ones).

girasquid2• 14 years ago

If bang for your buck is a concern, you might be interested in this offering from Hetzner: http://www.hetzner.de/en/hosting/produkte_rootserver/ex4s - you get double the RAM and a bit more disk space for a little bit less. I only have a month's worth of experience to talk about it, but other than the fact that it's in Germany (timezones) it's been great.

vmind2• 14 years ago

Yep, have used Hetzner servers too for some things and would use again, but for our customers being 90% in the US, and OVH offer a canadian location (and we may need some EU servers in the future).

henrikschroder• 14 years ago

I was also gonna come in here and mention the middle ground. Amazon is great either if you don't want to commit to anything, or if you have very spiky load and sometimes need to spin up a lot of servers.

Building your own servers is dirt cheap, but there's always the risk of hardware breaking, and you can't exactly scale up very quickly.

But the middle ground, rented dedicated servers, can be a perfect compromise for a lot of companies. Scaling up and adding servers might take a day instead of instantly, and it costs more than owning servers, but you get all the benefits of managed hosting, broken hardware simply isn't something you need to worry about anymore. If a server goes wonky, just scrap it and rent a new one.

stephengillie• 14 years ago

What happens when a HDD starts to go out? Do you have to pay & trust colo personnel to replace it, or do you have to pay to have your server shipped back to you?

I could see putting my own server into a colo if it were like a storage locker - carry/cart your server into the building, mount it in the rack you've rented, plug into the provided power and network connections, and lock the security door.

patio11• 14 years ago

Typically you'd pay for the first option. The term of art is "remote hands" service.

There are options where you can install servers in cages. (Some industries/companies require very strict physical access control.) This is more for show than because it meaningfully increases your security against the attack "your colocation provider is secretly The Adversary."

peterwwillis3• 14 years ago

If you buy the hardware from a vendor and get their 24/7 support they'll send a tech to your cage to replace the drive. It's a little bit more expensive than the machine he built, but with the knowledge that someone will go fix it when it's broken. Worth the money, still ridiculously cheaper than AWS.

jaf656s• 14 years ago

>This is more for show than because it meaningfully increases your security against the attack "your colocation provider is secretly The Adversary."

It does help protect you against the "another colo customer" attack, whether an actual attack or accident.

spindritf• 14 years ago

There's always a third option: go to the collocation facility and change the disk yourself. You won't usually be charged for access during business hours. Attractiveness of that solution depends greatly on the distance to the DC.

codinghorror12• 14 years ago

Good to think about since drive failure is easily the #1 most likely failure case for a server. (#2 is power supply failure, but anecdotally that seems much rarer in practice. Drives can and do fail regularly in my experience.)

That's why I used a RAID mirror, so we'd have to lose two drives in one server before it mattered, e.g. the server is down.

Ideally the RAID array would run degraded (unmirrored) until you can get remote hands or yourself down there to drop in a new drive.

They are front-accessible drives in caddies so changing them is relatively easy, no need to re-rack the server.

gprasanth• 14 years ago

Network connectivity is one very big problem. What good is a blazing fast server with 2Mbps (disruptive!) link? This is the very reason a lot of people would hate their own servers. Cloud = connectivity + (computing power) Yay! for connectivity.

codinghorror12• 14 years ago

Of course it depends on what you're doing.

Running a porn site or image/video sharing service? You're going to go through a ton of bandwidth and that might end up being your bottleneck way more than performance.

dholowiski3• 14 years ago

> Of course it depends on what you're doing.

This sums up every comment here. Every web app is a unique snowflake and what works for you (EC2/Dedicated, SSD, RAM bandwidth) is just as unique.

Much more important than building your own server, is fully understanding your requirements, and deploying the appropriate solution. But it is still wicked fun to build a server.

jaequery2• 14 years ago

i've known this secret for a while. i went from paying $4k+/month spanning over 10+ servers in amazon ec2, to just 2 dedicated servers < $1k/month in an unlimited 1gig colo. the performance difference i see is huge. i never realized how actually slow EC2 is. their issue is definitely in slow IO, which i think only their SSD instances can fix (which will cost me in the $8k+).

i now have a wicked setup, Xenserver Cloud, SAN, all Highly Available, don't have to worry about bandwidth overcharges, and much much faster .

codinghorror12• 14 years ago

Particularly now that we have 6 GBps SATA and cheap(ish) 512 GB SSD drives, the I/O differential can be enormous.

I thought about putting four 512 GB SSDs drives in a Raid 10, which would give me striping performance levels without losing the mirror, but that seemed like a bit too much overkill. These servers have 4 front drive bays, only 2 are filled with the SSD mirror, so we could decide to drop in 2 more drives and rebuild the array if we need even more I/O perf.

btgeekboy• 14 years ago

I like the part where he builds new servers on a rug. What's the over/under for how long until parts start failing?

codinghorror12• 14 years ago

I have built many dozens of computers over the last 20 years and I've never ever seen any static-related problems. I don't wear a static wrist strap, but I do always touch something metal on the case before touching the internals.

ersii• 14 years ago

I've been saying this as well, but has realised I don't know what problems static electricity could or would cause - long-term nor short-term.

I've stopped saying that I've never ever seen any static-related problems and I'm using the silly bracelet connected to something metal these days.

timc34• 14 years ago

I had a summer job building many computers, and have been ever since. Apart from making sure that you are not building up lots of static and then discharging it into components, its never been a problem for me either.

darkarmani2• 14 years ago

If you live on the East coast, there is usually enough humidity 9 months of the year to not cause static problems.

dholowiski3• 14 years ago

Same here. First PC repair skill I learned was grounding myself (wrist to grounded computer case, or other grounded metal). Never zapped a single component and I live somewhere with almost no humidity.

JoeCortopassi3• 14 years ago

"...you don't need the redundancy, geographical backup, and flexibility that comes with cloud virtualization"

This is perhaps the single most glossed over topic in the entire article. If I am a 1-5 person shop, maintaining a web app, virtualized hosting pays huge dividends in that I don't even notice if a hard drive or motherboard takes a dump. There are additional costs that come with the benefit of being abstracted away from hardware failure or geographic problems (building fire, power out, etc), and that's something that every business has to evaluate for itself.

codinghorror12• 14 years ago

Well, you'd always colocate enough servers so that you can lose at least one machine without caring. E.g. HAProxy to 2 web tier machines on the back end. HAProxy will fail over to just one server no problem.

(And yes, you can heartbeat so you have two cheap physical HAProxy machines, too. This gets into sub-blade territory, where the 1U server is internally two or more complete low-ish power servers with independent power supplies, etc. )

That's the whole premise behind FOSS, you don't need to worry about all the licenses, and the hardware is so cheap it is effectively free and getting freer by the second, so you throw a lot of cheap hardware at the problem.

But agreed, "lot" in this case means at least two so one can fail and you don't need to care.

JoeCortopassi3• 14 years ago

I guess my bigger point though, would be that when using something like AWS, I don't have to think/spend time on a lot of implementation details. When using colocation services/hardware/failover, I'm just adding a bunch of little things to my daily tasks and responsibilities. Sometimes this is a big deal (like in a bootstrapped two man team) and other times it's not.

Ultimately, I think it comes down to priority instead of possibility. If your company lives and dies on having reliable servers, you should probably roll your own. But if servers are 'just' a technical detail to your overall business model, then a cloud solution can be well worth the additional cost

abtinf• 14 years ago

That article is not just comparing apples to oranges; with that default aws setup, he is comparing apples to a herd of donkeys.

The correct comparison is his server vs a single EC2 High-Memory Double Extra Large instance with a 3 year heavy-utilization reservation. This instance costs $3100 upfront plus $0.14/hour. The total 3 year cost for this server on AWS would be 3100 + (.14 * 24 * 365 * 3) = 6779.2, or about $188.31 per month.

Sure, its more expensive. But AWS provides an insane of value on top of the server. Like instantly being able to provision additional capacity. I wouldn't be at all surprised if, on a full-loaded cost basis, it is extremely competitive with building his server. Heck, the employee salary expense of building your own server will easily drive the cost of the server well beyond the $3100 up front amazon fee.

I love building hardware too (never had a computer I didn't build except for laptops). But my mind boggles at AWS value proposition

codinghorror12• 14 years ago

Wait, so you can basically buy a server at AWS EC2 for a term of 3 years? That's what this heavy utilization reservation sounds an awful lot like.

recuter• 14 years ago

Not only that, now that they have that marketplace going you can pawn off your contract unto somebody else if your needs change.

I'm not sure how liquid it is yet. You could also equally go dumpster diving for unused virtual instances, say, if some startup goes under and liquidates.

twelve45• 14 years ago

Yeah for the closest possible comparison in your scenario, you want to do a 3 year reserved with heavy utilization.

When you reserve an instance, you are committing to a higher upfront in order to get a lower hourly for the reservation period. The low/medium/high utilization is sort of a knob that allows you to further control this upfront vs. hourly cost. With high utilization you have the highest upfront, but also the lowest hourly cost. If you are planning to run the server 24x7, this will also give you the lowest total cost.

rdw3• 14 years ago

Where's he colocating those servers? Last time I dabbled with colocation, the bandwidth costs per-server were by far the dominant cost. I found it difficult to get anything at all for small quantities of servers.

Looking at he.net at the moment, I see they have a deal for $1/Mbps. Presumably someone like Jeff Atwood can get twice as good a deal as that, so he'd pay around $500/month for bandwidth for those servers. Going by the cheat sheet (https://blog.cloudvertical.com/2012/10/aws-cost-cheat-sheet-...), that is within a factor of 2 of a yearly-reserved h1.4xlarge ($2263/month * 0.47 savings = $1199). It's almost equal to the three-year reserved machine ($2263 * 0.30 = $678).

Edit: He probably only needs 1 Gbps for all 4 machines, driving his bandwidth costs down by a factor of 4, but we could start to take power/space/cooling costs into account at that point.

So is he getting a better deal than that? I'd love to know where.

aes256• 14 years ago

Maybe I've been out of the game too long, but $1/Mbps sounds very low. Last time I looked into it (one or two years ago) you could only get bargain bucket bandwidth approaching those prices, with fairly crumby providers.

I'm assuming Amazon are a cut above that.

rdw3• 14 years ago

Yeah, $1/Mbps sounds like a steal to me as well, which may be why it's a limited-time special offer.

thatamurgy's Pinboard links lead to this hosting provider: http://prgmr.com/san-jose-co-location.html The machines in the original post being ~250W, they'd need the $362.50/month plan. But it's only at 10Mbps, which would only work if the application has low network requirements. Maciej appears to have gotten a much better deal, "For $206, I got 10U, 4 amps, and 100 Mbps connectivity", perhaps in exchange for promoting it through his blog. Everything's easier when you're a celebrity!

codinghorror12• 14 years ago

A lot of colocation centers offer "unlimited" bandwidth, though I strongly suspect that may depend on whether you're running a porn site / megaupload clone or not. :)

Fortunately bandwidth is one thing that has gotten substantially cheaper over time:

http://www.codinghorror.com/blog/2007/02/the-economics-of-ba...

rdw3• 14 years ago

Dangit, Jeff, you didn't answer my question. Share your secret colocation deals with me! :)

Power is also a big cost as well. To quote the prgmr colo: "Please note; power, generally speaking, is a bigger deal than rack units. I'm more likely to let you slide on an extra rack unit than on extra watts; watts cost me real money."

trin_• 14 years ago

hetzner will colo a 1HE 100mbit for 39€. 5tb traffic.

donavanm3• 14 years ago

Rough cost for transit from top providers: US $1-5 EU $5-20 AP $20-40 Africa/china/india/mid east/etc $$$$-go eff yourself

Prices are from someone like L3, qwest, ntt, telephonica. Assuming youre buying 10s or 100s of gb. Don't forget to factor in a couple routers, a bunch of 10gig optics, line cards, maybe some local loops, PMs legal & accounting staff, a bunch of NEs to run it.

dholowiski3• 14 years ago

I love building servers, it is amazingly geeky fun. And no doubt colocation with your own servers will give you the best bang for your buck, with renting a dedicated server coming in second. But you do have to be careful about redundancy. If you can't do without a day or two down-time, you'll need 2 or more servers, because when it's a dedicated server it's all up to you (or to someone you hire/pay for) when something fails.

Sadly I live a 5 hour drive from the nearest co-location facility, so I'm forced to rent a dedicated server.

codinghorror12• 14 years ago

Yeah I would never advise colocating just one server. Two at minimum. We'll have five when all is said and done (still building the other four here).

thaumaturgy3• 14 years ago

maciej from pinboard has written some useful stuff on this too:

- The five stages of hosting: http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/

- Building servers: http://blog.pinboard.in/2012/05/a_cloud_of_my_own/

- Going colo: http://blog.pinboard.in/2012/06/going_colo/

A bunch of people are probably going to respond to Jeff's article by saying things like, "But VPS hosting means you get other people to deal with problems for you", but in reality all that means is that you're at maciej's "monastery" or "dorm room" stage of hosting, and your needs haven't yet driven you to get "the apartment".

codinghorror12• 14 years ago

Maciej is 105% awesome, I'm a huge fan, and I linked several of his articles in the actual text of my post already. Go read them all!

mmariani• 14 years ago

The five stages of hosting is priceless! I'd really like to go with #5, but I'll stick to #3 for now. :)

wiredfool• 14 years ago

I love maciej's footnote:

"""* What is it with these aggro facility names? Rather than Hurricane Electric or Raging Wire, I would much prefer to host with "Calm Multihomed Oasis" or "Granite Mountain" or " Cooling Breezes Pure Sine Wave Mount Bitmore". """

timc34• 14 years ago

Happy rackmount home server user here, running ESXi with a load of dev instances, staging servers, OpenIndiana for ZFS running decent amount of storage connected to my editing workstations, databases, plus looks after our family video and photos.

I still use our company servers that are co-located but having things at home on the same network where you are developing is very compelling for me.

Its very quiet and cool (latest i7 processor).

But if you are hosting your own production boxes, I usually buy from HP or Dell and know that I rarely need to worry about those machines. Would hardly ever build a box to put into production unless it was something that was rather more specialized.

darkarmani2• 14 years ago

The core of this article is that you get more performance for your money by building your own servers and racking them. I think we can all agree there.

The problem with "hardware is cheap and programmers are expensive" is that your hardware will fail when you least expect it and have programmers sitting idle. Hardware is cheap, so have someone else assemble it and rack it.

If you don't need it up all of the time, this is a great way to get a lot of performance -- IO and memory in particular.

halayli• 14 years ago

I always say that there got to be an ec2-like solution but with real hardware.

Provisioning physical machines at the same convenience level as ec2 would be awesome.

I ended up doing what Jeff did. Bought my own server and hosted it at HE for $75/month. It's xeon 5650 with 48GB Ram + 1TB disk for $2k. Assuming the machine will last 3 years, it's a $131/month. That's way cheaper than the closest that softlayer offers (https://www.softlayer.com/Sales/orderServer/41/2087/)

Most of the times, EC2 is really about convenience and not cost.

BryantD4• 14 years ago

Dell's Crowbar project: https://github.com/dellcloudedge/crowbar

Ubuntu Orchestra: https://launchpad.net/orchestra

Or build your own around cobbler & puppet/chef, but you're sort of reinventing the wheel at that point. Still, sometimes that's fun.

donavanm3• 14 years ago

Also razor from emc/puppet. IMO a large scale cobbler config takes a bunch of work to get right. the provisioning and host management space is just starting to catch on to APIs and SOA. You're going to spend lots of time rolling your own apis, libs, services, and integration.

papsosouid3• 14 years ago

Did nobody notice that his AWS figures are totally wrong? AWS is expensive as hell, but even still $1400/month for 3 instances immediately looked incredibly wrong to me.

>The instance types included in the Web Application customer sample are 2 small (for the front end), and 1 large (for the database).

Nope, the instance types are 2 small for web, 2 small for app, and 2 large for DB. That's fully double what he's claiming it is. And he's ignoring the 4 300GB EBS volumes that are in that $1400/month as well as the load balancer and 120GB of bandwidth. And that is entirely on-demand instances, if you are comparing to a colo setup, you should be using the much cheaper reserved instances.

codinghorror12• 14 years ago

We're just comparing the Amazon Web Application template as provided in their calculator, with no changes.

You're right that it is x2 for each EC2 server which I didn't notice until later, but that doesn't change the economics or performance story very much.

papsosouid3• 14 years ago

Yes, we're comparing the web application template as provided. Except you compared less than half of it instead of comparing it. I think "I claimed that AWS is twice as expensive as it actually is" certainly changes the economics much. Your 3 servers is not 6 servers and a load balancer. Reliability counts too, not just performance. Especially since most web apps are going to be 90% idle on the AWS hardware.

duggan• 14 years ago

Here is an incredibly lengthly argument which tries to be more balanced: http://rossduggan.ie/blog/infrastructure/cloud-vs-metal-infr...

Jeff is glossing over a lot.

If you're already massively invested in hardware, in terms of people, processes and hardware, then you could argue that cloud architecture is less valuable in the general case, otherwise, it's usually no contest.

alberth• 14 years ago

Why is Jeff building a server when he can get the EXACT same server at Hetzner [1] for just 79 euro per month.

That translates to literally the exact same 3 year cost as building the server.

[1] http://www.hetzner.de/en/hosting/produktmatrix/rootserver-pr...

Building Servers for Fun and Prof... OK, Maybe Just for Fun

Related Stories