160 Mac Minis, One Rack

http://hackaday.com/2012/12/09/160-mac-minis-one-rack/

by barredo • 14 years ago

214 117 14 years ago

It's very cool, but is anyone actually tied to OS X as a server platform? Couldn't they move to FreeBSD and save a ton of money in an application like this? I'm wondering if there's a real business case for this, or it's just a fun hack.

edit: I guess lumped into this is the small market that seems to exist for colocated Mac Minis. Is there something about them that is better than renting commodity x64 hardware?

nbpoole• 14 years ago

First idea that comes to mind: http://stackoverflow.com/questions/3837043/best-practice-for...

chrismealy• 14 years ago

In comments the builder said it's for testing.

taligent9• 14 years ago

I am currently using a cluster of Mac Minis as a server platform. Some benefits:

* Launchd is a massive improvement over the equivalent mess on Linux. This can't be understated if you are managing your own hardware. * You can develop on the same machine you are deploying to. * You have exactly the same toolchain as on Linux. * Lots of remote monitoring options that are unique to OSX e.g. OSX Server * The OS is stable and upgrades are safe enough to enable auto update. I could never do that on CentOS.

But really it comes down to hardware and resale value for me. 2 Mac Minis in 1RU is great value.

cheald• 14 years ago

I don't have extensive experience with Launchd, but what makes it better?

OS X doesn't have the same toolchain as Linux - notably, Apple doesn't ship anything licensed under GPLv3. This leads to OS X versions of common GNU tools being extremely outdated.

I do agree with the point about developing and deploying in the same environment. I do both in Linux for that exact reason; far fewer surprises when your deploy environment matches what you've already debugged.

astrodust• 14 years ago

How is remote management for these systems? I'm a big Mac fan, but for server infrastructure I've often wondered if Software Update and related things could be more trouble than it's worth.

alanctgardner23• 14 years ago

Most modern Linux distros use systemd now, which is a pretty good unified platform for starting services.

I don't quite understand the other points; you can certainly develop on Linux and there are lots of mature monitoring options.

The trick with auto-updates is, you're getting everything from the package manager. This forces you to keep everything up to date, not just some notion of the 'core system', which is what you're getting from Mac update. Of course, you have to have some discipline and stage major updates in a testing area before you actually update production servers, but I think you'll find key packages like OpenSSL should probably be kept up-to-date.

Personally, I maintain a system which consists of about 30 CentOS servers; not large by any means. But things like upgrades and monitoring are a non-issue; we test updates before we apply them, and we use Nagios for monitoring.

As someone downthread pointed out, you can get a SuperMicro 1U server with a hell of a lot more horsepower than a Mac Mini for an equivalent price; the motivation for Mac servers seems to be almost exclusively to test and build Mac-specific software. I'd be curious to know what your specific application is that motivates this?

mminer• 14 years ago

It's a niche use case, but Apple's Qmaster software facilitates distributed video rendering and exporting for Final Cut Pro and some of their other pro applications. A server cluster like this would probably be overkill, but when you're working with massive video files, extra (OS X) machines to do the heavy lifting make a huge difference.

hugs• 14 years ago

> anyone actually tied to OS X as a server platform?

Yes, if you make software that runs on OS X (or iOS and you want to test on iOS Simulators) you need OS X machines for your build and test process. You need lots of machines so you and your fellow developers can run speed things up and run tests in parallel.

ahi• 14 years ago

One of my old employers has legacy systems built with WebObjects. In theory, WebObjects is Java and should run on anything, but typically it is on OS X Server. I left just as one of my colleagues was trying to figure out what to do about the death of the XServe.

madao2• 14 years ago

Considering you only really have to pay around the 60 dollar mark for the OS now, I dont think its much of a big deal, I use one of these at home as a mini fileserver/wiki it draws sweet FA makes little to no noise and has HDMI connector direct into my tv. I would happily deploy one for our company marketing team or small scale offices.

dfc• 14 years ago

"Draws sweet FA"?

alanctgardner23• 14 years ago

FA = "Fuck All"

It draws precious little electricity, in other words.

dreadsword• 14 years ago

Seems like an expensive way to do data center stuff. Why not create a rack full of alienware laptops or something?

taligent9• 14 years ago

Actually it's far cheaper than even an equivalent Supermicro solution let alone HP/Dell etc. You are getting at minimum 2 Mac Minis in 1RU which as of today could be a 8 core Core i7 / Dual SSDs / 16GB RAM.

Plus if you want to upgrade them then you can put them on eBay and get 75% of the original cost back. Try doing that with a server.

gonzo4• 14 years ago

The machine you quote (Mac mini: 8 core i7 / Dual SSD / 16GB ram) does not exist.

The closest thing to it (Mac mini: 4 core i7 / single SSD / 16GB ram) costs $1499 from Apple.

Are you proposing that I can't get a dual i7 @ 2.6GHz, dual SSD, 16GB ram from Supermicro for less than $3,000?

I have a mini on my desk, but even as an Apple fan, I think you're off the mark here.

simbimbo8• 14 years ago

We actually use SuperMicro machines for our Windows and Linux racks. They work great.. This rack required Apple hardware.

ghshephard• 14 years ago

The company requires a legitimate test bed for their OS X testing. It wasn't designed for generic data center work.

gonzo4• 14 years ago

it was when he started talking about using solder on a 220 VAC connection that I lost the faith in him knowing how to do it right.

wlesieutre• 14 years ago

Out of curiosity, what's the problem with that? I assume he'd have the copper twisted together to make a good connection, with the solder just holding it in place instead of acting as the conductor.

zalzane3• 14 years ago

depending on the resistivity of the copper wire versus the lead-tin mix in the solder, there may end up being more current going through the solder than the actual wire.

Even if the resistivity values of the copper and solder are similar (I'm pretty sure they're at least in the same order of magnitude), you'll still end up with the same amount of current running through the copper wire and the solder.

gonzo4• 14 years ago

Read the (national electric) code, Dude.

Household wiring must always use screw fittings, wire nuts, etc. Solder is not "generally approved" on dc, either, unless inside an approved housing, and then only when done in certain ways.* Household wiring is all the construction codes usually bother to cover, and they apply to 120/240Vac up to the switch, socket or outlet. Lamps, stereos, etc. are covered by code in some areas, like LA county, but by "Underwriter Labs Approval" in most others, in the US. A "UL" label (o/e) on the power cord is required for sale in most larger US communities.

120VAC can be attached by solder to an approved type of glass or phenolic circuit board inside the correct kind of enclosure. UL generally will not approve its use just about anywhere else for power-line connections.

This is something to keep in mind when doing any DIY project. Get a copy of the UL construction-code book. You'll find that all connections should be inside a non-meltable (fire-retardant) enclosure, via screw terminals, wire nuts, or compression fittings of some kind. Heat and humidity can quickly cause electrolytic corrosion on any soldered joint, eventually causing enough resistance for it to get hot and melt down.

Solder is great for some low-voltage dc purposes. When a connection might get subjected to heat (as in a house fire, resistive connection, etc.) the mischief that running solder can cause is just too great for it to be a good idea, so codes do not approve it in most areas. [Sorta like teflon tape in high-pressure gas lines. :-)] Over time, electrolysis slowly destroys soldered copper connections that aren't hermetically sealed, too. Some fungi accelerate that.

simbimbo8• 14 years ago

Sorry about the wording of my sentence. It was supposed to imply that I built a prototype cable at home to show to the cable vendors who would have to build them. Also, there was plenty of discussion about the potential of these cables to be misused, the vendor manufactured the cables to handle 15A. I have corrected my Wordpress page to reflect this discussion.

gonzo4• 14 years ago

cool

tzaman• 14 years ago

I'm a proud owner of a Mini Server (slightly customised - replaced memory and primary disk with SSD) for over a year. I use it as my main workstation and I love it; So small (and relatively cheap including the upgrade), yet so powerful.

jurre2• 14 years ago

Before the thread becomes cluttered with people suggesting alternatives or questioning why you wouldn't just run <insert manufacturer, OS, etc>, the person that did this replied the following:

simbimbo says: December 9, 2012 at 11:03 am Thanks for the great write up Hack A Day. I would like to answer some of the questions posted. @Geebles these machines all run SSD’s and I ordered them with AppleCare, so I hope to never have to change a drive ;-)

As for the reason I built this.. Well, I guess I’m just like a challenge ;-), but seriously, the company I work for has a need to have large numbers of machines to build and test the software we make.

There were plenty of discussions of Virtual environments and other “Bare Motherboard”/Google Datacenter-type solutions, but the fact is, the Apple EULA requires that Mac OS X run on Apple Hardware, since we are a software company we adhere to these rules without exception. These Mac Machines all run OS X in a NetBooted environment. We require Mac OS X because the products we make support Windows, Linux and Mac so we have data centers with thousands of machines configured with all 3 OS’s running constant build and test operations 24 hours a day 365 days a year.

As for device failure, we treat these machines like pixels in a very large display, if a few fail, it’s ok, the management software disables them until we can switch them out. This approach allows us to continue our operations regardless of machine failures.

@bitbass I tried the vertical approach, but manufacturing the required plenum to keep the air clean to the rear machines cost too much for this project, but it’s not off the table for the next rack

@Kris Lee When I open the door I can literally watch the machine temps go up, but I can keep it open for 15-20 minutes before the core temps reach 180F

@Adam Ahhh.. Nope, you can’t have my job ;-)

shardling• 14 years ago

I believe Mozilla has large racks of mac Minis for testing Firefox on. (Same issue, I believe -- to support multiple platforms on the same hardware, you need something that's legal to run OSX on.)

e: Ah, according to [this article](http://www.wired.com/wiredenterprise/2012/05/mozillas-new-da...) they have 500 Minis in their data center.

monochromatic4• 14 years ago

> these machines all run SSD’s . . . so I hope to never have to change a drive

lol

sliverstorm3• 14 years ago

Try actually contributing.

lukeholder• 14 years ago

You misinterpreted the sentence. They were saying that since the drives were bought with applecare, if one of them dies, he does not need to change the drive himself. Applecare to the rescue.

hapless• 14 years ago

Someone didn't finish his homework. The legal, supported method of virtualizing OSX is:

Guest: Leopard Server

Host: ESX 5.1

Hardware: Mac Pro

Mac Pro is supported on the VMware HCL. Leopard Server is legally virtualizable on Mac hardware. VMware supports OSX Leopard as a guest on ESX 5.1

Mac Pro towers are going to be less dense, but given his cooling situation, lower density is probably a win. What datacenter wants 8 kW of laptop CPUs stuffed into a rack? Virtualization would also overcome the lack of redundant PSUs.

frozenport2• 14 years ago

Why not use 2 racks?

This would solve his thermal dissipation problem and probably be easier when compared to getting custom hardware.

liquidise• 14 years ago

i would guess floor real estate constraints. Why do in 2 racks what you can do cost effectively in 1?

keypusher• 14 years ago

Because it's going to be extremely hot.

simbimbo8• 14 years ago

Rack "foorprints" at the datacenter are expensive, and you pay for 15kw per footprint whether you use it or not. it just made sense to fit it all into one. Having had this rack running at full power for a couple of weeks now, I can say the temps stay lower than our SuperMicro racks in the same row!! and the Supermicro racks can only hold 20 machines before they run out of power in the footprint.

meaty2• 14 years ago

I'm actually surprised any DC would take that equipment. They, in my experience at least, are very fussy about what what you put in the racks and power draw etc.

Oh and we get 640 cores in 20U (8x4 core xeon machines each 1u) and that leaves enough room for a 32Tb SAN, FC switches and a pair of redundant LAN switches.

REgarding splitting the power using the hack described, 160 melted minis and a halon cloud coming up.

Looks pretty though.

jws• 14 years ago

REgarding splitting the power using the hack described, 160 melted minis and a halon cloud coming up.

You should have a talk with your power cord provider. You should be using cables that can handle at least 4 amps in anything with a 110v plug on it. I don't think you can buy one smaller than 18ga and those are good for 10 amps. Remember, you have to handle enough current to blow the breaker if something goes wrong (unless you are British and have your own fuse in the plug).

simbimbo8• 14 years ago

We went through several designs with the vendor. This cable is designed to operate at the circuit rating. The wire from the PDU plug to the split is 12GA 600V and the cable from the split to the mini is 14GA 300V.

michaelt• 14 years ago

A 1U xserve maxes out at 334 Watts [1]. Each Mac mini takes 85 Watts [2], so 4 is 340 Watts. At 220v that's 1.54 Amps.

Is splitting the power really so dangerous?

[1] http://support.apple.com/kb/HT3513 [2] http://support.apple.com/kb/HT3468

alexkus• 14 years ago

It must be their own DC.

Most rackspace rented (in UK DC's at least) tend to be a maximum of 16A (at 240V) per 42U cabinet, so just under 4kW. By my estimation those Mac Minis will be drawing ~13kW at peak.

EwanToo• 14 years ago

You can get higher power density in a rack in pretty much any UK DC, you just need to pay for it.

The problem is you often end up paying for the rack space that would be allocated for the total power you're using, without actually getting the rack space!

In exchange, the DC's own electrician turns up during your install and makes sure things are working correctly and safely, you might end up with different sockets if required, 3-phase power, etc.

w3pm• 14 years ago

I understand the idea of treating them like pixels, so if a fan dies or a NIC card dies, no problem, just stop using that Mini. But what about memory corruption or other issues that are more difficult to detect? Normally server hardware has things like ECC memory to prevent these issues, but in this case a Mini with bad RAM could intermittently corrupt data for some time before it's noticed (if ever).

lallysingh2• 14 years ago

The machines are for testing. They'll detect those through secondary means. If a machine's faulty, it'll cause two cases: (1) faulty software will register as faulty; (2) good software will register as faulty. The third case (faulty software marked as good), is really unlikely, and any time it does happen, a later bug report will give a hint.

A test failure will probably bring up an engineer that will track down the issue, and a re-test will inevitably occur. The faulty machine will eventually (hopefully) get labeled flaky and will get repaired.

Of course, nobody may care and just use a double-test to verify that an executable is good.

rdtsc6• 14 years ago

> A test failure will probably bring up an engineer that will track down the issue, and a re-test will inevitably occur. The faulty machine will eventually (hopefully) get labeled flaky and will get repaired.

Depending on how valuable the engineers' time is. I have seen this played out like this: hardware gets blamed last after hours and days of testing have been wasted. So tests are run and re-run, blame goes all around until finally after hours and hours of testing it is determined that maybe it is hardware after all.

In the end an engineers' time is worth a lot more than savings obtained by running flaky but cheaper hardware.

rdtsc6• 14 years ago

Mac Minis are horrible server hardware. We've had a couple running as servers. They fail randomly. Their hard drives fail. They don't rack mount easily. The only reason to have them is if you inherit some old ones, don't want to throw them away, and then don't mind replacing and throwing failed units away pretty often.

sliverstorm3• 14 years ago

Well, they do provide good machine density when you rack them like he did.

rdtsc6• 14 years ago

Yap definitely true.

taligent9• 14 years ago

All servers fail randomly and so do hard drives so not sure why Mac Mini would be any different.

Also they rack mount very easily: http://www.sonnettech.com/product/rackmacmini.html

rdtsc6• 14 years ago

They do but we found that mac minis do so more often. I have a sample of about 15 out of say 100+ other regular 1U rack mountable machines. Mac minis failed at a much higher rate.

The worst part is that some failures were not a 'stop dead failure' where it was easy to detect and replace. Some would just freeze intermittently, some just got really slow.

One could tell that there is a practical difference between server grade and consumer grade components. In retrospect time spent debugging and messing with this was probably not worth the gain in density and the cost.

simbimbo8• 14 years ago

I agree, Mac Minis are not good servers, but these are simply for testing software.

rdtsc6• 14 years ago

Yap. I really depends on the type of loading. In other words I didn't mean "servers" as in necessarily web servers just machines that handle server loads. In this case it just means constants high CPU and disk churn. So if this is a high contention test rack, running tests non-stop its hardware will experience a server type level of stress.

datums• 14 years ago

Definitely a fun challenge. If you're going to invest in the hardware and custom build. Forget the y cable. Figure out a better solution. Rent 1/2 rack next to it to hold the pdus. +1 on the massive door fans.

patangay2• 14 years ago

We did something similar at Facebook for iOS and OSX automated testing and a few of them doing iOS app builds.

Here is a post that Jay Parikh (VP of Infrastructure) made about it. http://tinyurl.com/cnvss4v

Our density isn't as high (we have 64 minis) because of cooling and cabling that we designed according to our datacenter cooling standards.

@jurre - If you want to chat about our design, message me and I can put you in touch with our hardware designer.

simbimbo8• 14 years ago

I would like to chat about it.

i386• 14 years ago

Would love to hear more about your automated testing!

jurre2• 14 years ago

I just copied that reply from the person that built the actual rack, but I would love to hear more about your iOS testing infrastructure/process!

jevinskie• 14 years ago

Me too! Before we used fruitstrap[0] we used DeviceAnywhere. This was a 2U rack unit that contained a franken-iPhone. The hardware buttons were soldered up to GPIO on the DeviceAnywhere.

[0] https://github.com/ghughes/fruitstrap

patangay2• 14 years ago

For those who showed interest in learning more about our infrastructure/automated testing process, if you could drop me an email? gp at our corp domain fb.com.

I don't work on the team anymore, but I can probably start off a thread with the right people involved from Facebook's side.

j452• 14 years ago

I would love to get the specs on the tray that he put the 4 mac mini's on.

simbimbo8• 14 years ago

They're in production now.. H-Squared. great vendor. http://h-sq.com/products/minirack/index.html

j452• 14 years ago

Great, thanks for the pointer!

georgebarnett• 14 years ago

Interestingly, it looks like the front fans blow _into_ the rack. This means that if the door isn't securely closed it'll blow open - being on hinges and having massive fans attached.

It would be better to have the fans on the back and suck air through the rack rather.

That said, DC floor space is cheap compared to power and cooling. I'm surprised they didn't lower the density so as not to have a massive fire risk.

simbimbo8• 14 years ago

I looked into some rack cooling options, but was unable to find a solution that would provide the amount and wide coverage of airflow I needed to move air slowly and uniformly through the rack to provide effective cooling. ( I was designing this to be used with the active cooling rear doors, so I couldn't overwhelm the door with too much air or it wouldn't cool the air effectively, and would raise the ambient temp of the room). So the fans move a high volume of air through the entire cabinet (including the corners) at low velocity resulting in very effective cooling of each of the 40 shelves.

The fans are large so they move a high volume of air at a low speed, the door doesn't move is left unlatched.

Also, can you please explain your "Massive Fire risk" comment? All of the hardware installed in this rack is UL certified and all of the machines will simply shut down if they get too hot.

lallysingh2• 14 years ago

I'm surprised both that the high density worked for consumer devices, and that the rack wasn't prettier.

mewmoo• 14 years ago

Yes.. but why?

nsxwolf• 14 years ago

Curious - what do we call a computer like this? It's obviously not going to make the TOP500, but is it a "supercomputer"? I thought perhaps "minisupercomputer" might be fitting, but according to Wikipedia that is a term for a class of computers that became obsolete in the early 90s.

sproketboy• 14 years ago

Yeah but can it run Crysis? /s

160 Mac Minis, One Rack

Related Stories