Home Lab Hardware Guide

https://haydenjames.io/home-lab-beginners

by ashitlerferad • 5 years ago

196 102 5 years ago

sgarland2• 5 years ago

This is a good, if not opinionated, guide. And to be fair, r/homelab and its ilk are so full of options that it's easy to become overwhelmed.

Personally, I settled on Supermicro because they're modular, and don't care what brand of stuff you throw in them (HP is notorious for spinning fans to turbo if you put non-HP disks into them), although I may be picking up two Dells to complement the one I have for a Proxmox HA/Ceph cluster.

whalesalad• 5 years ago

I love Dell. IDRAC is killer.

sgarland2• 5 years ago

The HTML5 interface is definitely very nice. I haven't had a chance to use Supermicro's HTML5 IPMI, as I have an X9 board, and to my knowledge the minimum support for it is X10.

croutonwagon2• 5 years ago

I dont run server anything.

I have dell 7050's running a VMware/virtualization lab. They can have 64GB of ram and can handle anything i throw at them. Not iKVM though. But i just walk across the room and hook up a monitor the one or two times a year i need to.

Honestly Synology has been the best godsend. I was a SAN admin in a previous life. I have run freenas, openfiler, openfiler in HA, linux+NFS+iSCSI etc over the years. Synology generally makes it simple and totally integrated and allows me to play with other things rather than getting storage working.

walterbell3• 5 years ago

Dell 7050 may have vPro, which enables Intel AMT remote KVM access via Management Engine.

willis936• 5 years ago

I'm on board with most of this except the suggestion that rack mounted hardware should be kept where people live.

No.

I can't afford a house but like to tinker with networking. I pay more for weaker equipment so it will be low power/low noise. I'd love to just buy half a dozen used Dell PowerEdges but rack-mounted hardware is insanely loud.

A basement is the ideal spot. Water isn't an issue as long as your rack is not directly under anything that could leak (including on higher floors) and has a palette underneath it. If there is the possibility of your basement flooding more than 2 inches then you have bigger problems you need to address first. Keeping a rack with electronic equipment there will motivate you to do what you should be doing to the place anyway: dehumidifying and managing pests.

croutonwagon2• 5 years ago

Yeah i tend to agree..

I had a small rack in our old house. Mostly just to house the router/switch because the little cabinet wouldnt fit.

My wife dubbed it the EyeRack, short for eyesore.

For the most part you can easily hide this stuff in a cabinet or a bookshelf, no one will be the wiser.

Ill never really understand people using pizza box style servers, especially 1U units, in a home. With the sole exception of the one time i saw one stood veritcally behind an entertainment center. I think i was the only person at the party to notice it.

bluedino2• 5 years ago

IF you live near a co-lo, you might be able to get a half rack and go in on it with a buddy or two.

Saris• 5 years ago

Agreed, I started with rack mount stuff and quickly moved away from it. Very loud and for the budget stuff frequently used by homelabs, very power hungry for not that much performance.

snuxoll• 5 years ago

I'd love to just buy half a dozen used Dell PowerEdges but rack-mounted hardware is insanely loud.

This is a common belief that isn’t quite correct. My 2U Dell R520 is quite quiet after the initial boot (once the BMC takes over fan control), albeit I had to do some ipmitool magic to get it to not ramp them up with non-OEM PCIe cards installed.

My 1U R420 and R320 boxes? Yeah, they’re a little loud, 40mm fans have to run at higher speeds to get air flowing.

Ultimately my lab lives in my home office and the noise doesn’t really bother me, I wouldn’t put it in the bedroom or living room though.

theandrewbailey• 5 years ago

I've kept my server[0] in my basement since I moved into a place with one. I keep it elevated off the floor (not just for water concerns, but airflow, too) and under a table. Although it doesn't, it can make a lot of noise since no one's near it most of the time.

[0] is really an old desktop.

deeblering43• 5 years ago

I’ve done this and it wasn’t good. The servers heated up the basement, and chewed through power like crazy. Plus it was heavy as hell and a pain to recycle.

Rack servers are that form factor to maximize expensive datacenter rack space. Once you are not in a datacenter regular commodity hardware is a better bet.

For home use really laptops are ideal. They have an inbuilt UPS and KVM.

jjav3• 5 years ago

but rack-mounted hardware is insanely loud

A 1U chassis stuffed with high power dissipation components will be insanely loud like a jet engine. But it's easy to avoid that. The key for quiet is to use 4U chassis for anything that requires significant cooling so it has large fans.

On my home rack I have three 1U computers, two are Atom-based (fanless with SSD, so zero noise) and one is a Celeron-based with slow fans (about as loud as my MacBook). The larger machine is a 4U with larger fans, so same sound level as any tower case (a 4U is literally a tower case sideways, with rack mounting tabs).

KaiserPro• 5 years ago

For UK people https://www.bargainhardware.co.uk/ is an _excellent_ source of kit.

Personally I steer away from cisco. Yes, some people in enterprise swear by it, but I _personally_ hate it with a passion. However there is a fucktonne of it on ebay.

I use ubiquiti for APs, but I've not tried their switching. Currently I have some dlink "smart" stuff. Its PoE and has vlans, which is good enough for my purposes. Can do 10gig, not bad for < £120 (second hand)

Firewall, I'm all for pfsense. I've never liked hardware firewall/router appliances. They've always sucked.

rektide• 5 years ago

AMD has really raised the bar. I’m most impressed with the CPU performance of the M715q. They both run quiet and cool, with Ubuntu Server and Windows 10.

The M715q was offered with a fantastic 4750G chip, a Ryzen 7 Pro chip with 8 core. Today all one can buy in terms of small form factor business PCs is an M75n with low end low power Ryzen 3300U, a multiple-generations old Ryzen 3 with 4 cores.

Small business PCs are great, and for a while, there was serious excitement that AMD was going to make this segment much more interesting. Those dreams seem to have all been cancelled. I'm glad to see that affordable, competent AMD laptops are about, because in many ways it feels like AMD has succeeded so greatly that they have vanished from the market. They don't seem to be allocating production capacity to consumer GPUs, they seem to have withdrawn from this price-conscious market segment,... AMD keeps vanishing.

the_third_wave• 5 years ago

I made a rack out of some dumpster-dived supermarket shelves, lumber, a truck air filter and a forced draft fan. The thing doubles as drying cabinet for produce (mint, mushrooms, fruit etc.) by having the equipment in the top half of the rack followed by an air flow divider and 8 rack-sized metal-mesh-covered drying frames. From top to bottom the thing contains:

* D-Link DGS-3324SR (managed switch, €35)

* HP DL380G7 with 2xX5675 @3.07GHz, 128GB (ECC) RAM and 8x147GB SAS drives (€450)

* NetApp DS4243 (24x3.5” SAS array, currently populated with 24x650GB 15K SAS drives, €400)

* the mentioned airflow divider

* 8 drying frames

It is managed through Proxmox on Debian and runs a host of services including a virtual router (OpenWRT), serving us here on the farm and the extended family spread over 2 countries. The server-mounted array is used as a boot drive and to host some container and VM images, the DS4243 array is configured as a JBOD running a mixture of LVM/mdadm managed arrays and stripe sets used as VM/container image and data storage. I chose mdadm over ZFS because of the greater flexibility it offers. The array in the DL380 is managed by the P410i array controller (i.e. hardware raid), I have 4 spare drives in storage to be used as replacements for failed drives.

The rack is about 1.65m high, it looks like this (here minus the DS4243 array which now sits just above the air flow divider):

https://imgur.com/a/M4Lbf1K

In the not-too-distant future I’ll replace the 15K SAS drives with larger albeit slower (7.2K) SAS or SATA drives to get more space and (especially) less heat - those 15K drives run hot. After a warm summer I added an extra air intake + filter on the front side (not visible on the photos), facing the equipment. This is made possible by the fact that cooling air is pulled through the contraption from the underside instead of being blown in through the filter(s).

I chose this specific hardware - a fairly loaded DL380G7, the DS4243 - because these offered the best price/performance ratio when I got them (in 2018). Spare parts for these devices are cheap and easily available, I made sure to get a full complement of power supplies for both devices (2 for the DL380G7, 4 for the DS4243) although I’m only using half of these. I recently had to replace a power supply in the DL380 (€20) and two drives in the DS4243 (€20/piece), for the rest everything has been working fine for close to 4 years now.

On the question whether this much hardware is needed, well, that depends on what you want to do. If you just want to serve media files and have a shell host to log in to the answer is probably ‘no’, depending on the size of the library. Instead of using ‘enterprise class’ equipment you could try to build a system tailored to the home environment which prioritizes a reduction in power consumption and noise levels over redundancy and performance. You’ll probably end up spending about the same amount of money for hardware, a bit more in time and get a substantially lower performing system but you’d be rewarded by the lower noise levels and reduced power consumption. The latter can be offset by adding a few solar panels, the former by moving the rack to a less noise-sensitive location - the basement, the barn, etc.

As to having 19" rack equipment in the home I'd say this is feasible as long as you don't have to sit right next to the things. Even with the totally enclosed, forced-draft rack I made the thing does produce enough noise to make it hard to forget it is there.

walterbell3• 5 years ago

Enterprise features: Ubiquiti EdgeRouter ER-10X, 10 Port Gigabit Router with PoE Flexibility – $110 (specs) – (10) Gigabit RJ45 Ports, PoE Passthrough on Port 10, Dual-Core, 880 MHz, MIPS1004Kc Processor, 512 MB DDR3 RAM, 512 MB NAND Flash Storage, Internal Switch, Serial Console Port

Has anyone been able to purchase a small Ubiquiti EdgeRouter in the last six months? They've been out of stock at Amazon, Newegg, B&H. Beginning to wonder if they have deprioritized the consumer market, since other vendors are shipping routers.

tbyehl2• 5 years ago

Ubiquiti appears to have been prioritizing their own online store since the supply chain disruptions began. The ER-10X in particular may also be suffering from a general lack of popularity -- they've only had stock twice this year in fairly low quantities. They sell through fairly quick, but not nearly as fast as the ER-X which has been stocked regularly and in much larger quantities.

There's an inventory tracker for the Ubiquiti store on the Discord.

https://discord.gg/ui

mbreese• 5 years ago

Are they still making them? I thought they had switched to pushing their Dream Machine as opposed to the Edge Router series.

nullwarp• 5 years ago

I _hate_ the Dream Machines. We've been switching to them at work and the whole cloud UI is just an absolute mess. It's so hard to find anything.

I will be sad when the last of our Mikrotik stuff gets swapped out.

p_j_w2• 5 years ago

EdgeRouter 4 is available on their own store right now. Most of it's lighter weight siblings aren't, but their 3 most expensive models are also out of stock. None of their consumer wifi gear seems out of stock, though. I'd guess Si shortages before assuming they deprioritized the consumer market.

walterbell3• 5 years ago

Yeah, at that $250 price point there are x86 coreboot alternatives.

rafaelm• 5 years ago

I was hunting the 8 port switch for a while. I had to setup my own stock monitoring with alerts and managed to catch one that way.

psim12• 5 years ago

Yes, I bought an ER-4 last month from NewEgg. In fact, it is in stock and has a promo code for $10 off currently. https://www.newegg.com/p/0XM-0013-00087

youngtaff• 5 years ago

Serve the Home did a fab set of reviews on the different SFF machines and how useful they are for a homelab

https://www.servethehome.com/?s=tinyminimicro

tbyehl2• 5 years ago

I've been re-establishing my home lab and decided to get away from rackmount gear. I found ServeTheHome's TinyMiniMicro[1] series invaluable for choosing some mini PCs that would be right for me.

I went with three HP Prodesk 600 G4 that averaged $250/ea with the i5-8500T/i5-8600T, 256GB NVMe, and a total of 40GB RAM. They can go to 64GB RAM, the dual M.2 M-key plus potentially an SFF SATA drive offer plenty of storage potential, they're effectively silent, and power consumption is much lower than a big server full of fans. vPro potentially offers out-of-band remote management but I haven't tried digging into that yet.

I have two dedicated to Frigate with M.2 Coral TPUs. On the third I've been consolidating the sprawl of Linux VMs and Docker containers running home automation and network management stuff. Could probably make do with just two but why buy only two when you can have three?

[1] https://www.servethehome.com/tag/tinyminimicro/

KingMachiavelli6• 5 years ago

Rack mount hardware is almost always expensive, loud, and power hungry. I just have never seen the point of building a home lab like this.

A single ATX desktop can do almost [1] everything a homelab at a fraction of the cost & power consuption. I think a lot of the reason for homelabs/server hardware was to get access to more CPU cores, now that 8+ cores are very cheap it is actually cheaper to buy a new consumer desktop than it is to run an old server.

What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU. It's like using an excavator to dig a fire pit. Server hardware & software makes sense when it's not your home because then you do need a remote access tool like iDRACK. (There are DIY options if you just need something for personal use).

That said if you enjoy it as a hobby (or your homelab is actually a business thing) then go for it.

jrm4• 5 years ago

Right? Obviously, everyone should do their thing. But I suppose the tiny tiny "issue" I might have would be - I kind of feel like this reinforces the idea that "having server things in the house is big and complex."

So I encourage everyone who does this also tells the newbs, "I mean, you could also just slap linux on that old computer in the corner and do 95% of what I"m doing here, BUT MINE WILL LOOK COOLER."

p_j_w2• 5 years ago

What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU.

I always just assumed it was as a learning experience. When I was in my 20s, I didn't try to get Apache, qmail, and bind running because it was practical for me, I wanted a marketable skill. There are lucrative jobs out there for people who know these technologies.

awat• 5 years ago

That was my experience as well. Home labs seem to be cyclical for a lot people including myself. Started with a big overkill rack to learn technologies, and now I’m down to a 10 inch desk rack that is ATX case size with just a few things to run my network and small VMs.

KingMachiavelli6• 5 years ago

The application side is a more understandable. There are lots of reasons to know and use apache, postfix, etc.

As far as learning goes, if you really want to work in enterprise IT then vSphere is good to know but if you are willing to learn things on your own then you might as well learn kubernetes, docker, etc.

Knowing things like vSphere is cool & perhaps useful but it also hides the how things work. If you want to know and understand things it is better to stick to the open source & interact directly with KVM and Xen. Like you wouldn't use cPanel to learn how a LAMP stack works.

R0b0t1• 5 years ago

Best to avoid vSphere/ESXi in that case. I learned using qemu and was able to step into many roles immediately, including some VMware ones. The Linux/qemu/Xen ones pay better.

PaulWaldman• 5 years ago

What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU.

I disagree. Assuming your example of a single ATX desktop, ESXi really is easy to setup and modern versions provide a graphical web client. This assumes you're staying away from vSAN, vMotion, and iSCSI storage.

kazen443• 5 years ago

but, assuming you are learning the vmware stack because of employment. Not knowing things like vsan, ISCSI and vmotion makes your lab nearly worthless.

Learning vsphere and esxi properly requires atleast a decently sized cluster, especially if you start throwing NSX in the mix.

deeblering43• 5 years ago

The lenovo “tiny” hardware recommended in the article is really ideal. Its essentially laptop components in a micro case (no hid/battery/screen), they even are powered by a laptop style external DC adapter.

They are affordable, quiet, powerful (modern x86_64 with basic gpu) and light on power usage.

I run a pair of m92p myself.

KingMachiavelli6• 5 years ago

The downside to these small form factor kit PCs is that then you are very connectivity limited. You can't use it to build a NAS directly or GPU connected VMs, etc.

They are quite good as a cheap thin client that you use to access your more powerful hardware. As hardware ages it tends to lack some of the nice feature's like dual 4K@60Hz output, thunderbolt, etc. so having a new but cheap/lowpower machine helps.

Steltek• 5 years ago

Yes! Companies refresh Dells pretty often and you can find an i7 for not much money on eBay. Buy two to double up the RAM. If you know the right people at a company, you could get them for free even.

Home labs aren't about millions of hits. It's a playground.

kaladin-jasnah2• 5 years ago

A single ATX desktop can do almost [1] everything a homelab at a fraction of the cost & power consuption.

So this is true, but the iDRAC/iLO on my big, loud server has a virtual KVM feature that lets me lazily sit at my desk and click buttons and install my server operating system of choice. It saves a lot of space and effort compared to flashing a USB, going and plugging in a keyboard, (mouse), and monitor, and going through the whole deal. I'd wager that's one of the big things that would compel me to buy a rack server. I recently built a nice ATX desktop, fitted with a 5950X and everything, and I found that the PiKVM project [0] does a pretty good job at replacing that "integral" part of the server for me (you can also look into an ASRock Rack PAUL [1], but good luck finding one for sale right now.

What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU

A lot of people (not me, I end up using libvirt/QEMU as it suits my needs) buy homelabs to work towards having hands-on experience for their system administration job, which uses ESXi/vSphere. It might also be for working on getting certifications from VMware, in which case they really don't have any choice but to use ESXi on their servers.

Server hardware & software makes sense when it's not your home because then you do need a remote access tool like iDRACK

Now, I addressed this earlier (laziness), but these BMC things are very useful—you can monitor the health of various components of your server, and I believe even update the BIOS without stepping out of your chair. It makes administering a homelab much easier, and even the Pi-KVM, a DIY option, I'm pretty sure, doesn't have monitoring features. Plus, those DIY solutions require wiring stuff into your ATX motherboard, which can get janky and might put off people who want a turnkey solution.

[0] https://pi-kvm.org/

[1] https://www.asrockrack.com/general/productdetail.asp?Model=P...

PragmaticPulp2• 5 years ago

So this is true, but the iDRAC/iLO on my big, loud server has a virtual KVM feature

This is available on AMD Ryzen motherboards, too.

Just get one of ASRock's motherboards with the BMC controller built-in: https://www.asrockrack.com/general/productdetail.asp?Model=X...

No need to mess with PiKVM or add-in cards. It's a server board with KVM management that works out of the box with Ryzen processors. It might need a BIOS update to support your 5950X, but it will work.

KingMachiavelli6• 5 years ago

Yea that's true. ESXi/vSphere can be very relavent although I found that it was easy enough to learn on the job. The real complicated stuff of vSphere probably isn't going to come up in a homelab but if experience is necessary to get the job then it's worth it.

pi-kvm looks very nice. I would really like to not have to use iDRACK or pay the license fee.

Normal ATX motherboards do lack a lot of features. I'm not sure why none of the normal mobos don't just use LVFS [1] to update the BIOS but luckily they can read the update file directly off the vfat EFI partition so pi-kvm would solve that. I think every modern ATX motherboard also supports UEFI network boot so you could setup a simple DHCP+iPXE server for onboarding machines.

[1] https://fwupd.org/

PragmaticPulp2• 5 years ago

Rack mount hardware is almost always expensive, loud, and power hungry.

Buying old rack-mount server hardware for home use is almost always a mistake. Old server hardware may feel cheap when you see an old dual-socket rack mount server on eBay with hardware that was fast 8 years ago, but you can probably meet or exceed the performance with something like a cheap 8-core Ryzen.

Rack mount servers are also exceptionally loud. Unless you love the noise of small, high-RPM server fans, you don't want rack mount server hardware in your house.

And don't forget the power bill. Some old servers idle at hundreds of watts, which will add up over the several years you leave it running. 24/7 server hardware is a good example of where it makes sense to be mindful of power consumption.

What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU.

I disagree. ESXi is actually extremely easy to use, as long as you pick compatible hardware up front. The GUI isn't perfect, but it's intuitive enough that I feel confident clicking around to accomplish what I need instead of looking up a tutorial first.

beerandt• 5 years ago

I've always ignored advice even people say somethings too hard or not worth it. And I pretty much never regret it.

I absolutely regret trying to get a used rack mount server running.

The combination of steep learning curve from workstations to server hardware, plus parts that were failing but tested ok, made for an extremely difficult path to troubleshooting and getting it running right.

And that's before you get to the quirk's of getting it to boot and installing an OS and drivers and software.

I love it now that it works, but it easily took 100x the time (yes 100x) and probably 2-2.5x the total expected cost getting it to that point.

Not counting the additional AC unit I installed to keep it (somewhat) quieter.

I usually expect one or two aspects of my projects to have unexpected roadblocks, but for this it was issues with what seemed like every single step.

It's eventual replacement will be factory new.

kube-system• 5 years ago

I have used consumer hardware and enterprise hardware in my rack. There’s pros and cons either way.

Consumer hardware is cheaper, more power efficient, and quiet.

But, there are a few reasons I have switched to enterprise hardware:

* remote management and redundancy features make it more reliable if I’m traveling and want to remote in to do something

* some software works better with enterprise hardware features (i.e. special disk controller modes)

* I feel like my development experience is closer to what I can expect in production

* 100+ gigs of RAM on one system without a big upfront expense

* the occasional PITA of working with enterprise hardware helps me to understand what I might be expecting out of infrastructure team in production, or design ways to make their life easier

fomine35• 5 years ago

Latest Ryzen's CPU perf and max 128GB ECC UDIMM should be enough for most users, but it still lacks homelabby features and flexibilities. I need more than 16x PCIe lanes that supports ACS and IPMI, so I go for EPYC build. It's great except idle power consumption. I with they improve it to Xeon level, but maybe chiplet architecture (8CCD and massive CCD) isn't good about it.

For hypervisor, I recommend Proxmox VE.

thomaskcr• 5 years ago

And don't forget the power bill. Some old servers idle at hundreds of watts, which will add up over the several years you leave it running. 24/7 server hardware is a good example of where it makes sense to be mindful of power consumption.

My rack has an always on laptop for applications that always "need" to be running. I then have an arduino in my office that's sole purpose is to wake on lan or power on via UPS until the servers are online when I turn my switch to the "on" position, and put them to sleep when in the off position. Any servers still on after 15 mins in the "off" position just get halted.

Once I did that and the friction of going on/off was so low my power consumption went way down.

Lammy• 5 years ago

I just have never seen the point of building a home lab like this.

https://www.youtube.com/watch?v=38ApYaywLzs

I have to pay PG&E rates for power ($$$) so I'm a big fan of lower-power hardware for a system I'm going to leave on 24/7, e.g. my 25W-TDP 1U racked 8-core ECC-equipped Atom server built on this board: https://www.supermicro.com/en/products/motherboard/A2SDi-8C+...

c_o_n_v_e_x• 5 years ago

I like the Atom C-series... interesting option for edge deployed equipment.

bradstewart• 5 years ago

I had more space in my network rack than near my desk, so I bought a cheap Rosewill rackmountable ATX case, and rebuilt my old desktop into it (since I pretty much only use my laptop these days).

But I agree that buying old Dell servers for home use is rather silly at this point.

aftbit• 5 years ago

Rack mount hardware is almost always expensive, loud, and power hungry.

It really depends on the particular hardware. I recently picked up an R720 with 128GB of memory and dual E5-2670v1 for $450 on eBay. It idles at 120W (about the same as my brand new Ryzen 5950X desktop). It is not much louder than my old air-cooled 8700K consumer PC. Of course, it's not much faster either, and it's definitely slower than my Ryzen.

I bought it to learn how to use iDRAC and practice ZFS with 10x $20 1GB 10k RPM SAS drives. Also maybe to give Proxmox a try. All of my practical home-prod stuff runs on an old i5 desktop, not my rack servers.

fomine35• 5 years ago

120W at idle on desktop computer looks not good.

dheera• 5 years ago

expensive

This isn't always true. I got a 24-port Aruba 802.3at PoE switch with FOUR 10-gigabit ports, for a grand total of $120.

loud, and power hungry

Again, not always true. Enterprises do care about power a lot of the time.

I highly recommend the Ubiquiti stuff for home lab use -- most of it is pretty quiet.

If you buy other rack-mount hardware, try to buy at least 2U hardware, the bigger fans are much quieter than the 40mm fans in 1U equipement.

If you must buy non-Ubiquiti 1U equipment, you can usually change the fans out for Noctua fans.

A single ATX desktop

You can build an ATX desktop into a 4U case. I highly recommend the SilverStone RM42-502 for about $250 on Amazon. It takes standard components including standard ATX power supply, standard ATX, micro-ATX, or mini-ITX motherboard, standard fans (or even a CorsAir liquid cooler), it's basically a standard case. If you use quiet components it will be quiet. My ATX rack mount PC is not even noticeable unless I'm running up my GPU doing machine learning stuff.

There are much cheaper cases available as well, but the SilverStone case is quality, and will last you forever, you can just keep building new PCs into it for as long as ATX/ITX exist.

One of the advantages to building your PC in a rack mount configuration is that it's very easy to stack multiple PCs along with your network routers, switches, NAS, UPS, in one nice rack that's easy to move from apartment to apartment in one piece, and all your cables and connections stay nice and tidy.

It's also ideal if you play with a lot of smaller devices. For example if you want to have a cluster of 10 RPis, a rack mount solution is great for keeping the ethernet and power cables tidy, and it isn't going to be loud or any more power hungry than if you had them spread out across the table.

You can also 3D print rack mounts for non-rack equipment, just to keep them tidy.

My rack: https://i.redd.it/xcss9uassrg71.jpg

actually_a_dog• 5 years ago

I agree. My setup is very much like the setup shown near the end of the article (the one that consists of a couple Synology NAS boxes and what looks like a few Mac Minis or other small form factor PCs). I have a file server with a decently large storage array and a few Raspberry Pis and other small electronic gizmos, some of which connect to my wifi, and some which plug into a PC via various cables (ethernet, USB, etc.)

The only thing out of this that consumes any amount of power worth mentioning is the file server/storage array. I haven't measured how much power it uses (I probably should), but I'm able to minimize it by allowing the disks to go to sleep and the CPU to run slower than when I'm actively using it.

I've never felt limited by this setup at all, but, then again, my home lab isn't really my main hobby.

neilv• 5 years ago

Absolutely, rackmount servers tend to be very loud and often power-hungry. But if you think of rackmount as a form factor, it can make some practical and aesthetic sense.

The IKEA CORRAS Birch Effect "rack cabinet" in my living room currently has a nice virtually silent GPU compute server based on a consumer/gaming PC in a 4U short-depth rackmount case, a mostly passively-cooled Supermicro Atom server in which I've replaced the PSU fan with a Noctua, and a sinewave UPS. (There's an ongoing evolution of gear over time, including more exotic stuff, and most toys I might play with in the future will also fit this form factor.)

Aesthetically, those stack on the bottom of the cabinet, a Birch Effect shelf sits on top, and the plastic OpenWrt WiFi router with all the pokey antennae sits safely in the shelf cubbyhole. (Once I need detached WiFi APs, I'll probably build a 1U or 2U router with opnSense/Linux/*BSD. And I have a discreet slide-out rackmount console for if I ever get a deeper cabinet.) I like to think it looks like understated black home AV that's not out of place in a living room, which is better than a tangle of assorted non-rackmount PCs and UPS.

Practically, besides the tidy organizing and uniform cooling airflow, I have some rack posts I can use if the stack of gear gets too large/tricky for pulling individual boxes without disturbing the others too much. Also, when I lived in a dodgier student apartment, fastening a bunch of rackmount gear together with security-head screws seemed a good way to discourage a burglar from walking off with my data.

luke2m• 5 years ago

Exactly. The point of homelabbing at that scale is to dig a fire pit with an excavator, in order to learn how to use said excavator.

walshemj• 5 years ago

You really need the separate physical hardware a home lab is not just a bunch of VM's

One of the best things about doing my CCNA courses at a CISCI academy was having 20 switches and routers to play with and find those areas where it does not always work out the way the books say they should.

jjav3• 5 years ago

Rack mount hardware is almost always expensive, loud, and power hungry. I just have never seen the point of building a home lab like this.

Rack mount is just a form factor, so that generalization doesn't make sense. You can get pretty much any capacity systems you need in a rack case, including small cheap ones.

When I got tired of having systems in various different case sizes and a mess of wiring and switches etc, I moved everything at home to rack mounted.

Now literally everything is one big box (16U rack), with just power and network cables going in. All the computers, network switches, wiring, UPSs, KVM switch and monitor. Very tidy, very clean.

The computers themselves are cheap, quiet and efficient, since I don't need any monster servers at home. A couple are Atom-based (<20W total), one Celeron and one desktop AMD-based (forgot model). Same level of hardware I had before rack mounting, but now a lot neater and more organized.

KingMachiavelli6• 5 years ago

True, if you build something inside a rackmount case then it can be whatever you want. However, from what I could tell empty rack mount case & rails are still a bit of a premium. Were you able to get rack mount case w/ rails at decent prices? Seems each one would have to be at least a 2U to fit a normal ATX PSU ( I guess you could use an ITX or small PSU but those can be more expensive for the same wattage).

How did you handle the system & case fans? Did you rack cases have the standard fans and were you able to configure those to be slow/quiet?

glangdale• 5 years ago

Agreed. The startup I used to work for (acquired by Intel) both before and after acquisition had a good deal of lab hardware. Between evaluation boards and rack mount hardware our server room was insanely loud - like "it's 25M away and behind a closed door, but you can still tell when the lunchtime QA and benchmarking runs kick off".

There's no way I could live with that in my house, or even a fraction of it.

LargoLasskhyfv4• 5 years ago

But excavators are fun!

userbinator• 5 years ago

Classic cars are also expensive, loud, and power hungry ;-)

Sometimes it's not about being practical, it's about having fun playing with stuff you wouldn't otherwise normally use.

MrFoof• 5 years ago

There's a lot to be said for 1L form factors, which were included in the article, which are basically the size of a Mac Mini.

I have an HP ProDesk 405 Mini, with an AMD Ryzen 7 PRO 4750GE (Zen 2 based). 8 cores, 16 threads, with a base clock of 3.1GHz and boosts to 4.3GHz. Includes 8 Vega 64 GPU cores. Tossed 64GB of memory into it and a 1TB Samsung 980 Pro. Supports AMD Dash for basic LOM.

Idles at like 12-14W. Always silent. When stuff with the 5750GE hit they'll be better still. It's a great (and cheaper!) alternative to something like a Threadripper 5970X (when it hits in Novemberish) desktop if you wanted to cluster a few and have a low-power Apache Spark cluster that can actually rip through things. I think the only downer would be the lack of 10GbE support unless the next round of 1Ls offer 10GbE cards as options.

unethical_ban• 5 years ago

I just did some work on my main server, and my take is a bit different.

I upgraded my desktop and built a server from a Ryzen 1700, put 64GB of RAM in it, and now this one device acts as DNS filter/cache(Pi-Hole), VPN server (Pi-VPN Wireguard), and a 10TB ZFS NAS. This is just the base, I also use it for gaming and labs.

The main recommendations:

Fractal Design Define R5 - this is a large case, and is pretty wide - but it is a dream to work in. The extra width gives plenty of room behind the motherboard for hiding cables. It has quiet fans, it is built to minimize noise, and it can hold 8+ hard drives.

OS: Proxmox. I use this as the host OS, and configure my ZFS on the host. I then expose the ZFS as a NAS via a privileged container running Turnkey Linux.

If you get some multi-port NICs on it, you can put an OPNSense firewall as a VM, and use the machine as your router as well. In the end, you would only need UPS, modem, small switch, and the host.

dragontamer2• 5 years ago

Note that 10Gbe SFP+ switches have come down to $250 or so, and may be worthwhile for homelabs to experiment with. See Mikrotik's CRS309-1G-8S+IN, or servethehome's review (https://www.servethehome.com/mikrotik-crs309-1g-8sin-review-...).

If only because most of us probably already know how to use Cat5/Cat6 Ethernet, but how many of us have experimented with fiber optics?

10Gb Ethernet over Cat6 exists too. But that may be boring for some! Home labs are about experimenting with new things.

bombcar• 5 years ago

Another advantage of fiber is it helps prevent lightning and other power surges from spreading. If your equipment is protected on the power edge, fiber isolates it on the network side.

dragontamer2• 5 years ago

Hmm, maybe "fiber" is the wrong moniker here.

I'm more talking about SFP+ ports, because most of your connections within the rack will probably be DAC (copper cables pretending to be fiber) for lower costs. Fiber is really for longer runs. If you only have a few feet worth of cable, I'm not sure if fiber per se is worth it over DAC.

But learning to work with SFP+ hardware is a skill, just like learning to strip CAT6 cable or run it around. Working with DAC cables, or SFP+ modules and finding what works is the "dumb part" of IT, but the kind of stuff you need to practice a few times to understand.

----

Grabbing a few SFP+ ConnectX-2 cards from Ebay (for $30 or so), a few DAC cables, and a $250 switch... you can be well on your way to a 10Gbit network.

LargoLasskhyfv4• 5 years ago

https://fs.com for all your optics!

Maybe not all, but very usable and informative regarding prices and availabilty of all that 'exotic' stuff.

floatboth• 5 years ago

Aruba 1930 is a great option too

memetomancer• 5 years ago

So many of these posts gainsaying the practice of a home lab are spot on. In my view it's plain foolish to try and cram enterprise gear into a living space. It's almost too hot and loud for my office, why would I take any of that home?

The other posts talking about Tiny Mini Micros are on the right track but I think it goes further yet - there's good reason to have a small rack of crap in the corner:

- ISP hardware.

- pfSense gateway.

- Wifi base station.

- A good gigabit switch for the house.

- Those tiny mini micros or Mac minis for lab stuff.

- a NAS chassis or two.

- Raspberry Pi clusters.

- PiDP-11 or other such hobby stuff that needs a place to sit and blink.

There are plenty of other uses too, like security DVRs, ingest stations for cameras/recorders, optical and tape media devices, etc.

None of that stuff is hot or loud, but you probably wouldn't want it piled up on your desk or spilling out of some bookshelf. And I think the article kinda gets at that point, tbh.

ChuckMcM3• 5 years ago

I strongly endorse this notion of equipping your own laboratory for your experiments. Learning through doing is always more durable than learning through reading only.

While the author is looking at learning about and perfecting their skills as an administrator of networked computer systems there are other "kinds" of laboratories that people set up.

Mine, and one I'm more familiar with, are electronics labs. If you're going to be learning about circuits and such it helps to have the basic kit at hand. Similarly for people doing robotics, having a 3D printer in their home lab is essential these days. Nearly everything you might do in a home laboratory will involve some sort of data processing so the ideas by the author are great for creating the lab's "IT infrastructure."

In California it also makes it easier to defend the "I built this technology my own gear (picture/description of lab) so you don't own it." But that may be unique to California.

mcshicks• 5 years ago

Yeah I think I was expecting a home lab = "home electronics lab", although some networking equipment is still required. The rack mount stuff is nice, but yeah, scopes, bench dmm and power supplies, solder station, cable hangers and many drawers to sort parts was what I was expecting to see. Still the rack mount stuff is pretty cool. I have quite a few raspberry pis these days and was always looking for some "rack mount" style ways to make the power/ethernet cables nice and still have some easy way to temporarily hook up keyboard/mouse/monitor when it's occasionally needed.

jhallenworld• 5 years ago

It's not a lab until you lower the Bridgeport milling machine into the basement. Also, you may need three phase, but you will definitely need 240V for the multi-process welder.

https://www.homemodelenginemachinist.com/threads/getting-the...

flapjaxy• 5 years ago

Exactly, I think this thread is focusing on the utility of an environment from a mature SDec POV. Personally having a cheap R710 allowed me to practice my sysadmin, networking, security and a host of other skills that directly translated to my (then) current and future roles. I think the perception of a virtualized environment being the same is false and diminishes the sys/network/security admin's knowledge set. It's is akin to saying that writing JS is the same as Java, coding is coding. I'll also say the number of times I've been in rooms of IT pros who don't understand underlying systems, OS, or networking is disturbingly high.

Lastly, even if it is frivolous I see tinkering with hardware as part of the hacking spirit. It's fun, I like it and it gives me something to grumble about with the grey beards.

ChuckMcM3• 5 years ago

Here in the Bay Area I know a guy who both wires and de-wires commercial buildings for networking. It is surprising to me how many businesses move out and simply discard any equipment they installed in the POE to support their network. He gave me a Catalyst 9000 series to play with (he usually sells them to the resellers) and it was cool to be playing with "real" hardware but man, that thing was LOUD with the fans.

User23• 5 years ago

UPS plus a generator is a must have if you live someplace with severe weather.

zz8652• 5 years ago

Would be great to have more VGA-IP solutions. My biggest problem is headless servers is when they dont boot its a PITA to move somewhere else to plug in a monitor and keyboard. Its the main reason I gave up and just hire a cloud host - even though its expensive.

LargoLasskhyfv4• 5 years ago

https://www.aten.com/us/en/products/kvm/ ?

Just as an example. There are much more, this was just from the top of my head, because they tend to just work.

zz8652• 5 years ago

KVM still needs a monitor which I'm trying to avoid.

aviatordongle• 5 years ago

When I had a larger homelab, I found it was a large proportion of my electricity usage.

A constant 200W adds up, and 200W of heat is not desirable during Australian summers.

I have settled on a PC Engine APU2 acting as my router, and a Raspberry Pi acting as my IoT brain. 10W.

My other devices sleep.

atlgator• 5 years ago

Highly recommend the Discord on https://www.serverbuilds.net/. It's a great community built around home lab hardware using off-lease or decommissioned enterprise hardware. Guides range from 4U virtualization builds to custom pfSense routers.

psim12• 5 years ago

Ikea Lack Rack for rack mounting on the cheap! https://boingboing.net/2020/08/14/lack-rack-ikeas-cheapest-t...

ncrmro• 5 years ago

6c/12t 64gb NUC with 1tb nvme drive works very nicely. I have offsite NAS but the nuc has a thunderbolt 3 port for a raid array or 10 GB ethernet if needed.

With KVM on Ubuntu 21 zfs root pretty much covers most of these home lab uses. Have a Linux workstation that also does gpu pass through for windows gaming. Use a physical Kvm to switch between hosts.

Really hard to justify a rack these days

Home Lab Hardware Guide

Related Stories