I host this blog from my garage

178 65 5 years ago

rektide• 5 years ago

Very nice set up. Thanks for sharing, thanks for submitting. Very great technology to be able to set up & have running, have at your back. A lot of haters, but this is such a vast leap beyond a personal server, to having real cloud infrastructure, to having one and only one control plane for all your systems.

Recap of tech choices: Cilium networking, MetalLB load balancing, nginx ingress, Rook/Ceph storage, Prometheus/Grafana monitoring/alerting, Loki logs, Harbor container registry, Flux for GitOps.

Also a huge shout out to the Ingress controller comparison spreadsheet[1] the author made! Really nice being able to compare these feature sets. Neat to see a couple folks already working on HTTP3 & QUIC load balancing, who does Maglev routing, who has the best offload capabilities. This definitely brought my interest in Apache APISIX way way up.

The author digs at their own infrastructure some, calling it for learning. But to me, I think it's bad/dangerous having a conservative outlook that personal users should have to learn/use inferior/lesser tools (dozens of hours), it's bad to bifurcate the computing world into high and low computing. It took the author a lot of effort I'm sure to cobble together this system, but my hope is, over time, we start to make works like this much more paved road, and we start to make them much more accessible & documented & supported. We still haven't seen personal-use-focused Kubernetes distributions arrive, but I'm hoping that happens. Here's the author's description:

By trying out "flashy" software in a low-stakes environment, I can have some idea of how it will work in production-critical ones, which helps me make better choices. (As an example, my attempts to run Istio in a homelab setting have convinced me that it should be avoided in most circumstances.)

Again I think this is undershooting the real value, of using real tools. The hill to climb right now seems big. But imo the effort one's going to invest in selfhosting is probably going to be significant, and I like a lot the plan of shooting for good.

[1] https://docs.google.com/spreadsheets/d/191WWNpjJ2za6-nbG4ZoU...

shosti8• 5 years ago

Thanks, glad you liked it! Minor clarification, I didn’t write the ingress comparison, that’s from https://learnk8s.io (who I’ve been working with recently). They have a bunch of awesome resources on their site.

treesknees2• 5 years ago

Heh, I'm actually more interested in the "garage" piece of the blog which is barely touched on. I also run a home lab, but from my climate-controlled relatively clean basement. I've thought about moving it to the garage, but I'm worried about temperature and dust/dirt ingress. Do you have a sealed cabinet or air filters or any other environmental controls around the servers? Or are you using industrial machines rated for dusty locations? Otherwise a garage actually doesn't seem like a great place to put servers.

shosti8• 5 years ago

(Author here) I’m kind of cruel to my hardware lol (they’re all frankenservers hacked together with random parts from eBay). I’m not doing anything fancy to protect from dust and whatnot, but it’s pretty much always chilly in the garage (and I’m using low power parts) so heating hasn’t been an issue.

chomp2• 5 years ago

I live in the Southern US and it's extremely humid and hot in the summers.

Only certain systems are fine with being in the garage. For instance, a Supermicro chassis I put out there began rusting in a few spots after a few months, but my custom system that I built was perfectly fine. I suspect the metals Supermicro uses are treated in some way and then cut, but the edges aren't re-treated, allowing rust to form on all of the edges of the chassis.

I never put spinners out there, only SSDs. My bulk storage lives inside my house.

I have an HP Sandy Bridge Xeon box that lives out there and it's always super quiet, but I don't push it hard either. Raspberry pis are super okay with being outside. I'd say if you can make a small Pi cluster, it's a good fit.

I use an open air rack next to my water heater. I don't do any woodworking or anything, so the dust isn't too bad.

cma• 5 years ago

There was a talk from comma.ai with a part on what they did to prevent corrosion running a datacenter in their garage to avoid the nvidia datacenter tax or something:

https://www.youtube.com/watch?v=D0JO5dr8cPE&t=15m12s

pengaru• 5 years ago

It's not too difficult or costly to frame+finish a corner closet if you have the garage space to spare, especially if the garage interior isn't already finished.

jmnicolas• 5 years ago

I have seen production servers run for ten years upstairs a dirty bus repair shop. The room is never cleaned so there's a few millimeters of black dust on everything.

In fact you know the age of a server depending on the amount of dust on it.

There's a defective A/C unit that sometimes throw some water all around.

There's power cut 2 or 3 times a year, the batteries are old so the servers get their hard reboot.

As far as reliability is concerned, sometimes a disk fail, but that's about it.

Now if you think I'm describing something from the third world, you're wrong. It's in France and I'm not exaggerating about it.

So what I'm saying is that for a homelab, don't fret over cleanliness or optimal conditions.

themadturk• 5 years ago

Yes, I've run an enterprise server stack on an old closet, with a home air conditioner venting into the warehouse. Even when the A/C unit died for three days one summer (we had to keep the door open and run several fans), the servers, ranging from 3-7 years old, kept running. Far from optimal, but the systems stay up and the company stays in business.

Lhiw• 5 years ago

I spent some time with a computer refurber back in the day. He used to buy old computers from mining operations these things were literally filled to the brim with dust and weird shit.

Computers are amazingly resilient if you don't need them for much more than office work.

2Gkashmiri• 5 years ago

yeah. i know people who keep their pcs caked with dust and the stuff just chugs along.

i have to clean my office setup every often and i use a blower to remove the dust. dust bunnies and all. still, everything works.

you are right, people often talk about hygiene for computers but they are more resilient than we are made to believe.

BrandoElFollito• 5 years ago

My "datacenter" is on the top shelf of a closed closet. This is in an apartment and the ceiling that is just above the server is heated (heating of the neighbor above me). There is a fiber ONT, switch, router, a raspberry and tower computer ("the server")

The ambient temperature is 32-35°C

The temperature of the disks is 38°C (ssd) to 49°C (hdd), the CPU cores are 27 to 34°C (not sure why there is a difference, it is one CPU).

When I put the computer and other equipment there, I was concerned that the temperature would be unbearable. I am positively surprised.

superkuh2• 5 years ago

Well, hosting a blog from home is probably not a great idea from a practical perspective.

I'm not a hater, but maybe he perceives it as not practical here because of all the fun unnecessary complexity. Keeping something like this going for more than a couple years would require complex sysadmin maintainence for updates (which is fun till it isn't).

But if you just install nginx from your system repositories, have your hugo generated .html and media files in your www dir, and forward a port 80 on your router to your server LAN IP:80, it's going to work until your distro stops working without input and without security issues. For most people, for most of the time, it works great. And it's okay if it doesn't work some of the time.

Hosting your blog from home, in whatever room, is a great idea and very practical.

shosti8• 5 years ago

(Author here) I mostly worry about security for this. If you have nothing private on your network it's probably fine, but if you have, say, a NAS that isn't using proper authentication (pretty common), an os/nginx vulnerability could end up exposing stuff.

Of course there are much simpler ways to lock things down also :)

scubbo3• 5 years ago

Thank you for a great article! I recently took the plunge of building-and-hosting a blog too - but, due to security concerns, I took the entirely opposite approach of making it fully cloud-based (Git repos for infra and for content -> AWS CodePipeline, Hugo during CodeBuild -> S3 and CloudFront). This was sadly ironic since I'd mostly wanted to blog about my experiences with homelabbing, but I didn't trust myself to open a port to the outside world. Thanks to your blog I might finally learn Kubernetes and use a Cloudflare tunnel to implement a similar truly-selfhosted blog!

iso1210• 5 years ago

Obviously your server would be on a DMZ vlan, probably on its own. Set it to automatically take security updates every night and aside from some zero days I'm not sure what security issues you'd have.

superkuh2• 5 years ago

Then why are you intentionally adding another dozen attack surfaces and bleeding edge stuff constantly full of exploits? nginx remote exploits that matter are a once in a decade thing. Your setup is incomparably more insecure than nginx and a port forward.

I've been running a static webserver from my home for more than 20 years now. By avoiding dynamic languages, databases, and buzzwords, I've never been hacked. Never had any issue.

throwaway8943453• 5 years ago

If all you're ever doing is hosting a static blog, then yeah, you don't need something like Kubernetes; however, if you want to host other services (e.g., database, authentication, comments, etc) then it quickly behooves you to build on some higher-level platform or else you'll end up building your own (poorly) and missing out on all of the experience, documentation, and tooling that are publicly available for Kubernetes, Docker, etc.

btilly3• 5 years ago

Sorry, but we did all of that 20 years ago, and those approaches remain a lot simpler for basic use cases. You know, like serving 100k dynamic pages/hour database backed site. (That's what I was doing 20 years ago, how about you?)

See https://pythonspeed.com/articles/dont-need-kubernetes/ for some of the reasons NOT to use Kubernetes for this.

With, of course, a giant exception for two cases. The first is, as with the author, if the purpose of using Kubernetes is to learn how to use Kubernetes. The second is if you are aware of the tradeoffs and have specific reason to say that Kubernetes really is appropriate for your circumstance.

And if you think that Kubernetes is always the right approach, then you clearly are NOT aware of the tradeoffs.

kayodelycaon• 5 years ago

Running Wordpress on a single server is relatively painless. Document the setup and configure automatic backups of the relevant configs, www folder, and database.

I have another server ssh in every night and drop everything to a folder in my dropbox account. Replicates to computers that have incremental backups and less granular off-site backups.

silverfox17• 5 years ago

Running a server to do database, authentication, comments, etc. is relatively simple. If you're trying to do scaling or whatever, sure. But to do all of that on a single vanilla server is pretty trivial, especially if you're using a CMS or some other software on it.

blueflow• 5 years ago

... and missing out on all of the experience, documentation, and tooling that are publicly available for Kubernetes, Docker, etc.

After working with Docker and Kubernetes for some years, i consider that a huge win. Heck, even a cgi-bin script hacked together in perl is better documented and more reliable!

nonameiguess• 5 years ago

I wouldn't host anything I cared about at home because residential broadband is unreliable as heck with low caps on egress I'm already hitting half of an average month thanks to screen sharing and video conferencing working from home. I don't want Spectrum throttling me and suddenly I can't work any more because someone found my blog and put it on Hacker News.

hinkley2• 5 years ago

If you wanted to 'publish' a more interactive website to the internet while still keeping all of the publishing tools safe inside an enclave/intranet somewhere, then you probably need a read-only copy of a database. But any time I look at read-replica configurations, it looks like they expect the replica to contact the master. You can't set up a proper bastion server if you have to have a secret door back into your intranet to replicate data. This has left me stuck on the starting blocks for a couple of pet projects I might like to work on.

Anybody know how to solve this problem, short of roll-your-own?

int0x2e• 5 years ago

How much latency and inconsistency can you live with? Could you dump full or incremental database backups from your secure enclave, upload them to blobs and ingest on the cloud? Could you publish a change feed of sorts to Kafka and apply changes?

hinkley2• 5 years ago

It would have to be incremental because a full dump does not scale up. Of course if you scale down you could probably get away with shipping snapshots of a sqlite database, and I understand some people do just that.

But it would be nice to have streaming WAL logs with the instigator flipped. Probably I need some sort of middleware to dis-intermediate the pipe.

teraflop• 5 years ago

With Postgres, you can do log-shipping replication using whatever mechanism you like to deliver the WAL files, with no direct network connection between the primary and standby servers.

It's still kind of roll-your-own, because you need to provide shell scripts to handle archiving, fetching and cleaning up the log files on either end, but it can be done.

ivank• 5 years ago

WireGuard

mro_name2• 5 years ago

wouldn't a raspberry pi do it from within the drawer?

Not only https://solar.lowtechmagazine.com does similar, but https://cheapskatesguide.org/articles/raspberry-pi-3-4-web-s... has some numbers.

Seems feasible, unless you are the NYT.

shosti8• 5 years ago

Yeah that’s what I would do if I was just hosting a blog, I already had the k8s lab set up for other reasons and wanted to get more use out of it.

mro_name2• 5 years ago

got it.

Found the solar numbers meanwhile: https://solar.lowtechmagazine.com/2020/01/how-sustainable-is...

probotect0r• 5 years ago

He clearly says:

To stave off the inevitable haters: I know this is over-engineered. That’s kind of the point! By trying out “flashy” software in a low-stakes environment...

mkasberg• 5 years ago

It's unfortunate that there aren't better solutions for securely exposing a part of your home network to the cloud. Imagine if it were easy to set up a secure solution to access your home lab from anywhere - you could use it as your own private cloud. And perhaps then it would also be easy to expose part of that publicly, like a blog.

shosti8• 5 years ago

I feel like Tailscale gets you pretty close to an “easy private cloud”, it’s just the “public” part that’s still a bit tricky.

rubatuga• 5 years ago

You could try our service https://hoppy.network

It tunnels a public IPv4 and IPv6 (/56) over WireGuard. We have a good IP reputation so you can use it as a mail server. We’re using the service ourselves, so overall pretty reliable.

charcircuit• 5 years ago

Imagine if it were easy to set up a secure solution to access your home lab from anywhere

It is. You just need to port forward the ports you want exposed to the "cloud" (aka the internet)

heavyset_go• 5 years ago

I do this with Wireguard.

whoknowswhat11• 5 years ago

I also got a lot of mileage out of a homelab.

I experimented with proxmox and ESXi on the homelab front. Now ESXi is at the business.

Same thing with different server setups (Dell / HP / SuperMicro). Gives you a good feel for updates, KVM options etc.

Same thing with networking - I'm wasting time on Mikrotik and it's pretty fun.

It can be handy as a pool of additional resources. I had a very built out homelab server (with new SSD drives etc). Leadtimes got long for some Dell Server configs, we needed something 1-2 days, just grabbed a box from home. With ESXi once the real machine comes in its not that hard to move stuff over.

The big win is experimenting with low stakes. I find I'm learning 2-3x faster in that setup. I can try something, reinstall worst case, reboot, KVM, and literally plug in very easily (I have a hardware KVM setup in addition to the iDRAC type stuff).

midasuni• 5 years ago

If only he’d hosted it on AWS he too could be offline

lorenzo95• 5 years ago

I am curious why you use an ingress-nginx deployment. It looks like your blog pod is also running a nginx container, which makes sense. You could just point cloudflared at the service of that pod could you not?

Maybe you just wanted an ingress/metallb setup but I guess I was wondering if it adds anything else to the mix. Maybe some security feature?

Nice post. I am interested in the flux piece. I noticed that Azure Arc uses fluxcd as well. Going to have to give that(flux) a try.

shosti8• 5 years ago

The ingress controller is so I can have multiple public apps and configure them with Ingresses (definitely possible to do the overall setup with just cloudflared but I wanted to use built in ingress objects instead of a complicated cloudflared config).

slickdork• 5 years ago

Fun read, thanks. I've always wondered how to securely do such a thing and cloudfare tunnel looks like a good solution.

Of course the next thing you gotta solve is that you're running 3 HA clusters, but on one host. Can you replicate a backup to a pi somewhere else in your house that enables on emergency fail over, in case a rat chews through the power cable of the garage server?

david422• 5 years ago

I used to run my own personal website out of my home. Until I went for a job interview and gave out a link to my website.

But the site was down ... and when I got home my cat was chewing on the ethernet wire.

So found some cheap virtual hosting that was much more stable.

MonaroVXR• 5 years ago

Do you have suggestions for cheap hosting in Europe?

tlamponi• 5 years ago

A OVH or Hetzner VPS can get you a long way for 3.5 to 5€ a month, and they're both quite established and not just a one-man show hoster. Also, you get an actual VM, not just a container, so you can do pretty much everything with it.

For network stability I'd prefer OVH, but they only offer you a single IPv6 address, which is just stupid - but depending on your use case that may not be so relevant to you. Hetzners VPS/cloud offering improved a lot in UX in recent year.

stefandesu• 5 years ago

A cheap one that I had good experience with is https://contabo.com.

nicbou• 5 years ago

My robot vacuum bumped into the power bar and unplugged the server. It was the most clever state-sponsored cyberattack I've ever witnessed.

javajosh• 5 years ago

I self-host... and use Cloudflare Tunnel...

This decision protects your home IP address, but it still makes you reliant on something that is not self-hosted. It kind of feels like cheating, but I can't think of a good way to mitigate. What happens if your home ISP gets DDOS'd? What's that like? How do you (or your ISP, or their ISP or ...someone) fix it?

circularfoyers• 5 years ago

Perhaps having a setup where it has a fallback on Cloudflare Tunnel when you detect you're under a DDOS attack. Could this not be feasible?

watermelon0• 5 years ago

1.) You probably don't want to expose your own IP address.

2.) If your IP is under DDOS attack, you won't be able to connect to CF tunnel.

I host this blog from my garage

Related Stories