Homelab: Intel NUC with the ESXi Hypervisor

by henvic • 6 years ago

124 106 6 years ago

jedieaston• 6 years ago

KVM and Proxmox also run very well on that platform, if you're more used to those stacks (and the management tools are free).

stock_toaster2• 6 years ago

Some additional options:

* XCP-ng (based on xenserver)

* FreeBSD bhyve.

algo_trader• 6 years ago

Can u run proxmox on a single developer machine ?

(so that you boot up your machine into one of the virtualized OSes)

dsl• 6 years ago

No, you'd want something more like unRAID for that.

j45• 6 years ago

Proxmox boots to a shell as far as I have seen. The interface is all web-based.

Also have seen some folks who have used a Pi or a Chromebit to remote into any machine they choose.

tyingq• 6 years ago

That's almost it's strong point. It's pretty quirky from the outside, but I like Proxmox. You can see, pretty quickly, with a ~$50/month experiment on OVH's discount arm (SoYouStart.com) if it's a good fit.

You can mix and match normal VMs with LXD containers, and it does quite well for single physical hosts or small clusters. Past some scale, I'd go with something else, but it's great at the < 20 physical host range. It's a fairly unsung open source area. OpenVZ used to be a semi for-profit, open source/ but we want money, competitor here. Guessing they are mostly dead now. (Not celebrating that...they did a lot of good).

jlgaddis2• 6 years ago

Can u run proxmox on a single developer machine ?

Yes. Proxmox is built on Debian (or was, last I played with it) and you can actually install it on a Debian host, if desired.

(so that you boot up your machine into one of the virtualized OSes)

No.

fomine3• 6 years ago

The good thing about Proxmox is the flexibility because of It's Debian-based. I can do anything than can do on Debian like custom monitoring, manager filesystem, etc.

anonymouswacker2• 6 years ago

I'm looking for a hypervisor recommendation... I am hoping to virtualize at least two if not all three of my machines: Unraid (pass-thru HDDs); pfSense (pass-thru NIC); and Windows 10 (pass-thru Graphics). I am hoping to have Windows 10 running fast enough to do some light photo editing. Which hypervisor would you think to try first?

mastax• 6 years ago

Have you ruled out unraid? It has built-in VM management and supports pass-through.

smcleod• 6 years ago

I'd usually recommend just using KVM on Fedora/CentOS, but if you're not that Linux savy or don't want to spend time fiddling around I'd probably say give Proxmox a go, or XCP-ng if you want something that's easier to cluster between hosts but don't mind the storage not being quite as quick.

lukevp• 6 years ago

I’m not that into hypervisors, but wouldn’t hyper-v be a good choice here? You could have near native windows graphics performance.

mike_d• 6 years ago

You really don't want to virtualize unRAID.

JoshTriplett• 6 years ago

KVM works well. You can use system-wide libvirt-based management to have the VMs automatically come up and go down with the host machine, and treat them like services.

That said, there's value in having your router/firewall separate from other systems (virtual or otherwise), which will substantially reduce complexity and make it less likely that you accidentally make a system accessible on the wrong side.

tjoff2• 6 years ago

I've done all of that on XenServer, or rather XCP-ng now. (pfsense (nic passthrough), freenas (SAS controller passthrough) and windows 10 with gpu passthrough)

It doesn't have the same hype as the others but far more open. Biggest drawback is that the configuration software is windows-only (you can get it running in wine but it is a bit hacky, also xen-orchestra is a free (for personal use, if you build from source) web-configuration utility that one can use).

Would not want to be at mercy of vmware. Also don't want to invest time into a system with such restrictions on use.

Proxmox might be a decent alternative but I have a few systems and the cost would be prohibitive for hobby use.

whatsmyusername3• 6 years ago

Proxmox. ESXi client is windows only and most of KVM's clients are either linux only, cli only, or dogshit.

Proxmox is great as long as you have the mental fortitude to ignore the tickler that pops up on login if you're on community support only.

LeoPanthera2• 6 years ago

I've always preferred Proxmox because of its container support. I have a number of Proxmox hosts that don't have any VMs on them at all, just containers.

Scarbutt• 6 years ago

One thing I really like about ESXi the last time I used it was the built-in virtual switch, other solutions in KVM world where not up to par. Maybe it's different today.

lostlogin4• 6 years ago

I’m very much a lightweight and I found that ESXi was great when combined with Fusion on a Mac. Being able to build VMs locally then upload them is nice. Maybe other platforms have methods like this too?

henvic5• 6 years ago

I was almost sold to bhyve as I intended to have FreeBSD on this machine, but at the last minute I decided to give ESXi a try first.

gonzo2• 6 years ago

bhyve is cool, but ESXi has some good tools.

stock_toaster2• 6 years ago

If you decide to give bhyve a whirl at some point, I highly recommend giving vm-bhyve[1] a try for managing the VMs. Makes lots of tasks much easier.

[1] https://github.com/churchers/vm-bhyve

znpy• 6 years ago

I've always disliked the fact that proxmox is basically uncapable of hosting virtual machine images to spawn vms at will.

You have to fiddle with template images but it's kind of a tedious task.

jimmcslim• 6 years ago

I've just discovered PhotoPrism [1] through this article, which looks like a great solution for organising my photos in my homelab/home server - which is an HP N54 Microserver (Gen 7), recently reformatted to run FreeNAS; loving it so far (previously was some Frankenstein mix of Windows Server 2012 running an Ubuntu VM via Hyper-V, with the storage being ZFS within the VM... terrible performance!)

There is also PhotoStructure [2] that I like the look of, but PhotoPrism is OSS and already integrates machine learning for identifying objects in photos... yes I know I can't expect great performance out of my N54L on that front!

[1] https://photoprism.org/

[2] https://photostructure.com/

LeoPanthera2• 6 years ago

You'd never guess from the Photoprism website that it actually exists. It's just a huge ad.

But there is actually code, tucked away in the hard to find github link: https://github.com/photoprism/photoprism

Unfortunately, it requires docker, and cannot be installed standalone. That's a deal-breaker for me.

henvic5• 6 years ago

off-topic: Here's a weird thing... I've to admit that one concern I've is to like PhotoPrism (this is what is keeping me a little bit afraid of trying it). The issue is that I'm a Go programmer and it seems to be a good piece of software, but it uses the GPL license, and I've as a motto to avoid reading GPL'ed source code, especially when it affects what I do for a living. It'd be even worse if I think about contributing to it...

I discovered it a few months ago when I was searching the web to see what is out there. Previously to moving to iPhoto (regretted a lot), I used digiKam and I enjoyed it.

* I don't want to risk reading something I like there and ending up copying the idea or writing similar code unwillingly, as this license is viral.

Thanks for showing me PhotoStructure!

HeWhoLurksLate• 6 years ago

*> Previously to moving to iPhoto (regretted a lot)

I used iPhoto for a while and am actively considering buying a used one to run it on again for family photos. What didn't you like about it, and is there anything that can take the metadata from iPhoto and translate it?

yjftsjthsd-h3• 6 years ago

but it uses the GPL license, and I've as a motto to avoid reading GPL'ed source code

So, don't? Just pretend that it's a proprietary application with no source that just drops from the sky as a binary or docker image. Can you not get what you want as just a "normal" user?

mceachen• 6 years ago

Hi, author of PhotoStructure here.

If you've signed up for beta access, know that I'm sending out invites in batches, and hope to send out another this week, with the release of v0.8.0 : https://photostructure.com/about/release-notes/

(sorry if you've been waiting a while!)

Briefly, there are 3 builds of PhotoStructure: a desktop build (packaged with Electron), a Docker image, and a git repo you can pull, `yarn install`, and run directly. More details about why I felt the need to quit my office job and work on it full time are here: https://photostructure.com/about/introducing-photostructure/

If you've got any questions, feel free to ask them here or email hello@photostructure.com.

NickNameNick• 6 years ago

I just got one of those NUCs.

The network device is annoying - I couldn't get it to work on debian stable, but it is supported in Debian testing thanks to kernel 5.5

I also had problems with the HDMI video - It didn't work with the hdmi->dvi cable I got with either of my spare monitors, nor did it work with my TV and the existing hdmi cable I had there. I did get it to work by borrowing the hdmi cable from a friends htc vive.

dragonshed• 6 years ago

I have an older NUC that I've been repaving as needs change, using it as a build agent for some projects, a homelab to experiment with hasura, etc. It's always done well, but it's perf relative to other options would never make me consider using it as a type 1 hypervisor like this. I may need to pick up a NUC10I7FNH.

snorrah• 6 years ago

That was quite a thorough and interesting post, I was pleasantly surprised! :)

henvic5• 6 years ago

Thanks! By the way, I noticed this video https://www.youtube.com/embed/WkA0dMudUG0 showing TempleOS didn't make when building from Markdown to HTML.

* I'm going to fix this later today + some misspells.

asadhaider• 6 years ago

I have an old Intel NUC D54250WYKH which I run ESXi on as my core applications server and where all my inbound homelab traffic gets routed through. There's a single CentOS VM with Docker installed which has all the host resources allocated to it.

Currently running the following containers-

  proxy [1] - Nginx proxy host for all ingress traffic, has a nice web interface and works with Let's Encrypt
  pihole [2] - Primary DNS server running Pi-hole to block ads
  cf-ddns [3] - Client to update Cloudflare records with my IP address (wildcard record *.myhomelab.dev which I run the proxy 
  hosts through)
  unifi-controller [4] - UniFi Controller appliance
  watchtower [5] - Keeps the docker images up-to-date
  portainer [6] - Web interface for managing docker containers
  guacamole [7] - Apache Guacamole remote desktop gateway
  postfix-relay [8] - Open SMTP relay on my internal network which forwards everything to Amazon SES, makes email notifications easier

It's a great piece of kit and love the small form factor, it sits in my network comms cabinet and I've had no issues. I pass through a monitoring cable from a UPS in the cabinet to the CentOS VM which monitors if there's a power cut and shuts the ESXi host down safely.

I initially ran PhotonOS with Docker but had some networking issues so switched to CentOS 7. I have larger, more powerful SuperMicro hypervisor hosts which run the bigger application containers.

[1] https://hub.docker.com/r/jc21/nginx-proxy-manager [2] https://hub.docker.com/r/pihole/pihole [3] https://hub.docker.com/r/joshava/cloudflare-ddns [4] https://hub.docker.com/r/linuxserver/unifi-controller [5] https://hub.docker.com/r/containrrr/watchtower [6] https://hub.docker.com/r/portainer/portainer [7] https://hub.docker.com/r/oznu/guacamole [8] https://hub.docker.com/r/simenduev/postfix-relay

regnerba4• 6 years ago

Why run ESXi at all and not just install CentOS directly on the NUC?

ocdtrekkie2• 6 years ago

The portability of being able to lift a VM off the hardware and onto another piece of hardware is a nice feature when you are using bargain consumer hardware for your "servers". The overhead is pretty minimal.

I'm not currently doing it with my NUC servers, but it's a pretty good choice to do so, and ESXi is free if you don't need to, you know, manage it in any competent way.

sigjuice2• 6 years ago

I have a couple of Raspberry Pis and Onion Omegas on my LAN and a few Linux and BSD VMs on my MacBook (NATed). They can mostly ssh amongst themselves, but the biggest problem I have is that I haven't been able to come up with a satisfactory DNS/mDNS setup. I just don't like fiddling with IP addresses. Any suggestions?

lonelappde2• 6 years ago

I thought the point of mdns is that it just works with zero configuration.

Other alternatives include running pihole on one machine (or all machines for backup), or giving them static IPs in your DHCP server and then copying a hosts file to them all.

sigjuice2• 6 years ago

A couple of my systems have resolvers that can't resolve .local names (OpenBSD, OpenWrt/musl)

yjftsjthsd-h3• 6 years ago

Oh yes, when it works it doesn't need any configuration:) (Well, aside initial setup, but that was years ago on Arch so it might be easier now...) But my personal experience was that it didn't always work, and then it was "fun" to troubleshoot. But seriously, do take this with a grain of salt; I abandoned it years ago, so it might "just work" now.

p1necone• 6 years ago

I've always just given static local IPs to everything in my router settings - usually it lets you map MAC addresses to desired IPs.

Then you pass traffic on certain ports to certain IPs if you want to be able to access anything from outside.

ahyattdev• 6 years ago

You can assign them static IPs in DHCP, then run your own DNS server that returns local IPs for those hosts.

Or bridge those VMs on your macbook so it's a flat network, then use Avahi for mDNS so they all have .locals.

rubatuga2• 6 years ago

Running your own DNS is the best option, I use dnsmasq, which functions as both a DHCP and DNS server. Any DHCP requests from the client also contains a hostname, which dnsmasq then stores in its DNS table. So if your hostname is `macbook-air`, dnsmasq will resolve that to the DHCP IP address assignment.

lonelappde2• 6 years ago

Why is this better than running VietualBox in a host OS?

dade_• 6 years ago

VMs work ok, but much I am much happier with Linux containers. NUCs get bogged down pretty easily and go from quiet purring kittens, to screeching and whining when they get under load. A dozen containers, with different distorts and versions runs nicely.

moondev2• 6 years ago

The ultimate NUC for a vSphere lab is the "hades canyon" NUC8i7HVK. I have three of them in a vSAN cluster.

* dual NVME drives for all NVME vSAN

* dual nics work out of the box for ESXi

* dual thunderbolt 3 ports to enable 10GbE via TB3 adaptor or PCIe expansion

* Linux friendly AMD GPU that has no problems being passed through to a nix or win vm

* low profile and easily stackable for a rack shelf

* Supports 2x32GB SODIMMs

* Power efficient

* Low noise

Love these things!

mtsr• 6 years ago

Aren’t vSphere licenses a bit expensive for the average homelab? I personally went with a larger former workstation with more cores and RAM, because I expect to only ever have a single node, even if I’d love to have a full cluster.

InvaderFizz• 6 years ago

VMUG Advantage membership ($200/year) gets you a 6 socket, non-production license for basically the entire VMWare catalog.

yakk02• 6 years ago

just the ESXi hypervisor is free for non-production use.

ex3ndr2• 6 years ago

I found recently that they got 500 bucks license for 6 CPUs now! I think it is doable since it is lifetime license (without updates though after a year(?)).

auggierose• 6 years ago

Me too, they are so tiny and powerful! Just got the 64GB Frost Canyon one, awesome little thing. Running it with an Nvidia 2080Ti via Razor Core Chroma and Ubuntu 20.04. GPU Compute (CUDA) works, which is really the only thing I need it for, but I cannot get it properly working with a display though.

tpolzer• 6 years ago

If you just want network between them, you should actually be able to connect them directly with a thunderbolt cable, no need for Ethernet in between.

kbumsik• 6 years ago

The newer version is even more interesting and it's just released on the market now: NUC 9 Extreme/Pro Kits. [1]

It's got i9/Xeon options, 3 NVME drives, and even PCIe x4 and PCIe x16 slots. It's bigger but still quite small. The PCIe slots are the killer feature to me.

[1] https://www.intel.com/content/www/us/en/products/boards-kits...

dannyw• 6 years ago

It's incredible. I just wish pricing came down a bit; I am happy to pay a premium for the form factor but this is too much.

woodrowbarlow• 6 years ago

with support for optane memory too, nice.

rwmj2• 6 years ago

Do these have remote manangement? IIRC there are only a few NUC models that have BMCs (or Intel's equivalent - AMT).

sqldba• 6 years ago

6 cores and 64GB max memory? It’s hardly enough to run anything.

notRobot• 6 years ago

It's enough to run a lot of stuff.

regnerba4• 6 years ago

O_o what the heck are you trying to run in a home lab where that isn't enough? I currently run 20+ containers per NUC in my home lab. I have plenty of capacity left and nothing else to really run...

luckman212• 6 years ago

I think sqldba's post was sarcasm. As in, once you open Slack, Gmail and a few Chrome tabs, poof there goes 10GB of RAM.

regnerba4• 6 years ago

I run my homelab on NUCs as well but no hypervisor. Everything I run is in Docker containers. So the NUCs have CentOS installed and then just containers across the Swarm cluster.

Storage is an NFS mount from my Synology NAS.

The only reason I would really care to have a hypervisor is because then I can do more from Terraform.

I use Terraform to configure my DNS, my Ubiquiti gear, and the Swarm cluster but that leaves a gap where I need to do something to manage the actual CentOS machine. There isn't much to manage (users, SSHd config, SSH keys, packages such as vim, docker, and htop, and then NFS mounts) but the less I have to manage myself the better. Just don't think it's worth adding a whole hypervisor just to pull possibly some of the networking into Terraform.

Anyways, NUCs + Docker Swarm = great win.

luzer7• 6 years ago

That's exactly what I want to do. Do you have any guides you used? I'm having a little trouble with nfs and docker permissions (guid/uid I think) trickling down to the nfs share in my Synology.

regnerba4• 6 years ago

With regards to NFS permissions, I set my Synology up as so:

* Client: 192.168.0.8/29 (whatever the smallest subnet is I can use that includes my NUC)

* Privilege: Read/Write

* Squash: No mapping

* Asynchronous: Yes

* Non-privileged port: Denied

* Cross-mount: Denied

This is fairly out of date (I keep my own stuff in a private monorepo, this was a snapshot I posted for a friend), but here is the basics of a script to setup my NUC: https://github.com/regner/homelab-stacks/blob/master/scripts...

Of note, the "Adding NFS share to fstab" part doesn't actually do that...

ocdtrekkie2• 6 years ago

I currently run my Sandstorm.io server on an 7th Gen i5 NUC.

I recently picked up some old budget Gigabyte BRIXs that will take on some other fun roles. (A 6th Gen Celeron and a 4th Gen i5.)

I have wanted to run an AD domain for managing my gaming PCs, but I had some difficulty with ESXi IIRC (probably also NIC issues, like in the article), and Intel has decided to be a complete tool about network drivers for "consumer" chipsets being installable on Windows Server. One of the perks of the old 4th Gen BRIX unit: A Realtek LAN NIC.

One of the things I'm sad about on newer NUCs is the removal of the LED ring. 7th gen NUCs in particular have a neat multicolor LED ring around the front panel that can be programmatically controlled... pretty neat for server status like uses. But I suspect they found it rarely used and so left it out for cost or something.

datlife• 6 years ago

After playing around with ESXi, Proxmox, Hyper-V. I converged my mini-homelab to:

- Any linux-based OS, Ubuntu in this case.

- Ansible / Terraform for task automation.

- K8s / Docker for containerization.

Given I have only 1 desktop + laptop, this option is good enough for me to learn cluster setup / networking / automation.

ct520• 6 years ago

I got a NUC8i7BEH with ESXI. Works great, run dev lab out of it publicly facing vlan'd off on my home fiber.

Only negative I ran into is - I bought the bigger one with NVME and regular SSD. I couldnt fit a SSD in with NVME with heatsink :-/ Besides that this thing is snappy and uptime been perfect.

youngtaff• 6 years ago

For my setup I was running ESXi on a 2012 i7 Mac Mini with 16GB of RAM

At the time I bought it (2013?) it was far better value for money than any of the Intel NUCs.

I've always wanted someone to produce a NUC in a similar form factor, with good design but there doesn't seem much demand

And alas the current Mac Mini's are completely un-upgradable

GekkePrutser2• 6 years ago

You can upgrade the memory on the 2018 mini, but yes that's it.

undebuggable2• 6 years ago

Anyone with similar homelab setup - do you use the free ESXi 5.5 or more recent versions which are quite pricey? If 5.5 how do you orchestrate it? vSphere Client on Windows is awful.

I discovered the Ethernet NIC wasn’t working.
More Ethernet issues

Likewise with 5.5, where enabling the ethernet is often nontrivial (e.g. injecting the drivers for the Realtek NICs).

lostlogin4• 6 years ago

I think I signed up for a trial licence and got 6.7 u3, but I can’t seem to find a link. It’s been running a while so it isn’t a 60 day one.

VMWares website is utterly impenetrable and it’s like a form of hell trying to get out of the loops you end up in with licence agreements and download options.

undebuggable2• 6 years ago

VMWares website is utterly impenetrable and it’s like a form of hell trying to get out of the loops

Oh man, yes. The only way to download installation binary of something specific was a direct link to VMWare servers found somewhere on the internet.

bluedino• 6 years ago

ESX is basically free. You just can't manage your ESX hosts as a cluster using vCenter, which gives you things like vMotion or backup your VMs up using the vCenter API.

Also, the Windows client was dropped a while ago and the Flash version is now deprecated.

GekkePrutser2• 6 years ago

Yes, the Thunderbolt 1 to Ethernet adapter works with ESXi. I use it with my Mac Mini 2011 running ESXi.

aljarry2• 6 years ago

On the cheap side, I got old Lenovo m93p - USFF with 8GB RAM, i5 (2 cores, 4 threads) with 4 USB 3.0 ports, which is just enough for lightweight VMs, some test containers and ZFS mirror for backups / NAS. Whole package with 2x2TB disks was around 350$.

mech4222• 6 years ago

I still swear by my Ameridroid H2's - Celeron quad core, dual channel DDR 4, 2 x 1Gbps nic, 1 x NVME, 2 x Sata, 4K video - less then $120. I have 3 that I got the end of last year - tricked out with 16G DDR4 and 2x480G SATA - $250 each

edit clarify it was $250/each

SparkyMcUnicorn• 6 years ago

A few days ago I picked up a SFF Dell (precision T1700) with a Xeon E3, 32GB DDR3, small SSD + HDD, 11 USB ports, and a thunderbolt pcie card. All for $200.

The CPU is roughly equivalent to a 6th or 7th gen i5, but with 8 threads instead of 4.

Installed proxmox with 1 VM and several containers. Couldn't be happier with everything. My wife can't even hear it under decent load in the living room.

rubatuga2• 6 years ago

Picked up a HP Z230 SFF with a Xeon E3 a month ago. Fits two hard drives, a sata ssd, two nvme, as well as a USB-C Pcie adapter. It runs Ubuntu 19.10 with ZFS, and VMs using libvirt and hand defined XML.

Homelab: Intel NUC with the ESXi Hypervisor

Related Stories