adamcharnock

I've been a long-time fan of Mikrotik. I even ran an ISP on it a fair while ago. I have a couple of archived GitHub projects if they are useful to anyone:

- Router OS Diff - Can diff two configs and give you the commands needed to bring the existing config up to date with the desired config. It's certainly not perfect, but a starting point of anyone needs something like this. [1]

- Netbox Routeros – A netbox plugin for updating the config of RouterOS devices directly from the Netbox interface. [2]

It has been many years since I touched these, but perhaps they will be of interest to someone.

Aside from that, I have had excellent experience with Mikroik. Everywhere from in-datacenter to it running in an off-grid hut on a mountainside. I've even heard reports of people finding rain streaming through a CRS and it just happily ticking along.

[1] https://github.com/adamcharnock/routeros-diff

[2] https://github.com/adamcharnock/netbox-routeros

Yeah, fair enough.

And yes, adding a Redis cluster is fine, it is just another moving part to manage. But given that the alternative is made out of unobtainium, I guess that is just the way of it :-)

One, it would be cool to be able to embed it, similar to sqlite, directly into applications.

I've found myself wanting this on several occasions too. I.e. wanting all my rust backend processes (k8s pods) to have some minimal shared state, without having to spin up a Redis cluster. I've talked to Claude about it a couple of times, and it descends into something like, "you gotta use Raft or CRDTs, and pick 2 out of 3 from CAP". Which honestly seems pretty fair, and indicates to me that I'm dreaming for something magical.

Nonetheless, it is nice to hear someone else asking for this. If this is indeed feasible (even if simple/limited), then I'd be interested to try it.

which you can only obtain by having a strong CS base and coding manually for years.

I hope this isn’t the case. It is the route I took, but it also doesn’t seem to be a likely route going forward. Strong CS grounding is feasible for sure, but I have a hard time believing that a meaningful number of people will be spending the requisite years coding manually.

I’m not sure. We’ve always had to pick the level of abstraction we start teaching at. Voltages, transistors, registers, assembly, C, etc. This feels like it could just be a progression of that.

I've been wondering if there would be a benefit to inverting how we teach subjects now. Previously we would teach from the bottom, and build up. Semi-colon goes here, curly brace goes there, and then build up to architecture, systems, etc.

But this doesn't seem to make sense when someone comes to a topic with an LLM in-hand. They need to know high-level techniques, architecture, best practice, etc. As they pursue the topic they start to get down into the details, although probably never learn to do it fully independently.

I quite like this view because it paints a somewhat optimistic way forward from where we are now.

I see a lot of conversation here about provider feature parity, but my hope is that sovereignty doesn't have to be strictly about the region or company you choose. For me, the strongest form of sovereignty can come at a lower level. That is, running on an open-source stack you can pick up and run somewhere else should you need to. If your data and workloads live on standard Kubernetes with Postgres, object storage and the usual Prometheus/Grafana/Loki, then no single provider (EU or otherwise) actually has you over a barrel. As the article points out, the "AWS Europe is a separate subsidiary" argument does nothing if the software still ships from the USA.

Shameless plug: We started our company[0] on this basis, i.e. managed Kubernetes on bare metal in EU DCs. We run everything on open source tooling, provide DevOps engineering time to our customers' engineering teams, take on the migration risk ourselves, and offer response-time SLAs. So yes I'm biased, but I did this because I do actually really believe in this approach.

So I'd flip it around. Perhaps building a sovereign hyperscaler is the wrong approach. I'd say we need workloads that aren't welded to any one provider in the first place. And open source tools have come a very long way since EC2 first came online.

[0] https://lithus.eu/

FWIW - I also really like Wispr Flow, but I moved to running the 'Whisper Large' model locally using Handy (https://github.com/cjpais/Handy), which has been essentially as good, while also having lower latency.

Honestly, I still had to practically stand on the clutch with mine!

I'd teach someone to drive it and say, "now push down on the clutch". They they would heave and struggle, then eventually succeed and look victorious. I'd say, "well done, it is now half way down! But that's all you need for now!"

EDIT: To fully explain: It has a two-stage clutch. You half-press it and it disconnects the wheels from the engine. If you fully depress it all the way to the floor, it additionally disconnects the power-take-off shaft (PTO) from the engine. The PTO shaft is a spindle on the back of the tractor which drives things like your flail mower, wood chipper, etc.

EDIT 2: Edit 1 was for the general audience, not the parent commenter ;-)

Yeah, I was introspecting as I wrote that!

I do, but a friend is taking care of the farm now. I moved back to the big city lights (Munich, as fate would have it).

Mine and a pedal and steering column lever, so I guess I got one of the fancy ones!

Up until a year ago I was regularly using a Massy Fergusson 135 [0] (Perkins Diesel version), made sometime in the 1970s. It was wonderful! So amazing to drive and use. Clunky and heavy, but you really really felt like you were using a machine. In low gears, if you put you foot down on the accelerator the engine would roar, and your speed would barely change!

And there was no fancy technology in it at all. If I was in the forest and had forgotten the key, I'd just reach behind the dashboard and hot-wire it. The air filter was basically a shisha-pipe that bubbled the incoming air through wire wool and engine oil.

Its fuel gauge didn't work either. You just had to take a look in the tank, or quickly react as soon as the revs started dropping. I ran it dry a few times and had to sit there with a spanner in one hand and YouTube into the other, while trying to bleed all the fuel lines. But they were all on the outside of the vehicle, which made it comparatively easy I imagine.

I've never actually driven a modern tractor, so don't know how it compares. I imagine the clutch is easier on the knees these days!

Anyway, this just felt like the place to share this.

[0] https://en.wikipedia.org/wiki/Massey_Ferguson_135

This is something we've[0] done a number of times for customers coming from various cloud providers. In our case we move customers onto a multi-server (sometimes multi-AZ) deployment in Hetzner, using Kubernetes to distribute workloads across servers and provide HA. Kubernetes is likely a lot for a single node deployment such as the OP, but it makes a lot more sense as soon as multiple nodes are involved.

For backups we use both Velero and application-level backup for critical workloads (i.e. Postgres WAL backups for PITR). We also ensure all state is on at least two nodes for HA.

We also find bare metal to be a lot more performant in general. Compared to AWS we typically see service response times halve. It is not that virtualisation inherently has that much overhead, rather it is everything else. Eg, bare metal offers:

- Reduced disk latency (NVMe vs network block storage)

- Reduced network latency (we run dedicated fibre, so inter-az is about 1/10th the latency)

- Less cache contention, etc [1]

Anyway, if you want to chat about this sometime just ping me an email: adam@ company domain.

[0] https://lithus.eu

[1] I wrote more on this 6 months ago: https://news.ycombinator.com/item?id=45615867

FYI, PR created: https://code.forgejo.org/forgejo/runner/pulls/1382

We have two entities. An EU one and a UK one. I'm based in Munich, servers are in Germany.

Ping me an email, adam@ domain.

If you're interested I'll see about getting the PR created sooner rather than later.

We can run a Forgejo instance for you with Firecracker VM runners on bare metal. We can also support it and provide an SLA. We're running it internally and it is very solid. We're running the runners on bare metal, with a whole lot of large CI/CD jobs (mostly Rust compilation).

The down side is that the starting price is kinda high, so the math probably only works out if you also have a number of other workloads to run on the same cluster. Or if you need to run a really huge Forgejo server!

I suspect my comment history will provide the best details and overview of what we do. We'll be offering the Firecracker runner back to the Forgejo community very soon in any case.

https://lithus.eu

We've migrated to Forgejo over the last couple of weeks. We position ourselves[0] as an alternative to the big cloud providers, so it seemed very silly that a critical piece of our own infrastructure could be taken out by a GitHub or Azure outage.

It has been a pretty smooth process. Although we have done a couple of pieces of custom development:

1) We've created a Firecracker-based runner, which will run CI jobs in Firecracker VMs. This brings the Foregjo Actions running experience much more closely into line with GitHub's environment (VM, rather than container). We hope to contribute this back shortly, but also drop me a message if this is of interest.

2) We're working up a proposal[1] to add environments and variable groups to Forgejo Actions. This is something we expect to need for some upcoming compliance requirements.

I really like Forgejo as a project, and I've found the community to be very welcoming. I'm really hoping to see it grow and flourish :D

[0] https://lithus.eu, adam@

[1] https://codeberg.org/forgejo/discussions/issues/440

PS. We are also looking at offering this as a managed service to our clients.

Hello! I think this is a fair question, and improving the communication on the website is something that is steadily climbing up our priority list.

We're not really that kind of product company; we're more of a services company. What we do is deploy Kubernetes clusters onto bare metal servers. That's the core technical offering. However, everything beyond that is somewhat per-client. Some clients need a lot of compute. Some clients need a custom object storage cluster. Some clients need a lot of high-speed internal networking. Which is why we prefer to have a call to figure out specifically what your needs are. But I can also see how this isn't necessarily satisfying if you're used to just grabbing the API docs and having a look around.

What we will do is take your company's software stack and migrate it off AWS/Azure/Google and deploy it onto our new infrastructure. We will then become (or work with) your DevOps team to supporting you. This can be anything from containerising workloads to diagnosing performance issues to deploying a new multi-region Postgres cluster. Whatever you need done on your hardware that we feel we can reasonably support. We are the ones on-call should NATS fall over at 4am.

Your team also has full access to the Kubernetes cluster to deploy to as you wish.

I think the pricing page is the most concrete thing on our website, and it is entirely accurate. If you were to phone us and say, "I want that exact hardware," we would do it for you. But the real value we also offer is in the DevOps support we provide, actually doing the migration up-front (at our own cost), and being there working with your team every week.

This is essentially how it is. Additionally, the reality is that our customers don't often even need to think about using root access, but they have it if they want it. They are putting a lot of trust in us, so we also put trust in them.

FYI, AWS offers free Egress when leaving them (because they were forced to be EU regulation, but they chose to offer it globally):

https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i...

But. Don't leave it until the last minute to talk to them about this. They don't make it easy, and require some warning (think months, IIRC)

To put it plainly: We deploy a Kubernetes cluster on Hetzner dedicated servers and become your DevOps team (or a part thereof).

It works because bare metal is about 10% the cost of cloud, and our value-add is in 1) creating a resilient platform on top of that, 2) supporting it, 3) being on-call, and 4) being or supporting your DevOps team.

This starts with us providing a Kubernetes cluster which we manage, but we also take responsibility for the services run on it. If you want Postgres, Redis, Clickhouse, NATS, etc, we'll deploy it and be SLA-on-call for any issues.

If you don't want to deal with Kubernetes then you don't have to. Just have your software engineers hand us the software and we'll handle deployment.

Everything is deployed on open source tooling, you have access to all the configuration for the services we deploy. You have server root access. If you want to leave you can do.

Our customers have full root access, and our engineers (myself included) are in a Slack channel with you engineers.

And, FWIW, it doesn't have to be Hetzner. We can colocate or use other providers, but Hetzner offer excellent bang-per-buck.

Edit: And all this is included in the cluster price, which comes out cheaper than the same hardware on the major cloud providers

An interesting question, so time for some 100% speculation.

It sounds like they probably have revenue in the €500mm range today. And given that the bare metal cost of AWS-equivalent bills tends to be a 90% reduction, we'll say a €10mm+ bare metal cost.

So I would say a cautious and qualified "yes". But I know even for smaller deployments of tens or hundreds of servers, they'll ask you what the purpose is. If you say something like "blockchain," they're going to say, "Actually, we prefer not to have your business."

I get the strong impression that while they naturally do want business, they also aren't going to take a huge amount of risk on board themselves. Their specialism is optimising on cost, which naturally has to involve avoiding or mitigating risk. I'm sure there'd be business terms to discuss, put it that way.

You sum it up very neatly. We've heard this from quite a few companies, and that's kind of why we started our ours.

We figured, "Okay, if we can do this well, reliably, and de-risk it; then we can offer that as a service and just split the difference on the cost savings"

(plus we include engineering time proportional to cluster size, and also do the migration on our own dime as part of the de-risking)

Indeed! We've yet to go down this route, but it's something we're thinking on. A friend and I have been talking about how to bring Nix-like constructs to Kubernetes as well, which has been interesting. (https://github.com/clotodex/kix, very much in the "this is fun to think about" phase)

Fair point!

5 - Datacenter (DC) - Like 4, except also take control of the space/power/HVAC/transit/security side of the equation. Makes sense either at scale, or if you have specific needs. Specific needs could be: specific location, reliability (higher or lower than a DC), resilience (conflict planning).

There are actually some really interesting use cases here. For example, reliability: If your company is in a physical office, how strong is the need to run your internal systems in a data centre? If you run your servers in your office, then there's no connectivity reliability concerns. If the power goes out, then the power is out to your staff's computers anyway (still get a UPS though).

Or perhaps you don't need as high reliability if you're doing only batch workloads? Do you need to pay the premium for redundant network connections and power supplies?

If you want your company to still function in the event of some kind of military conflict, do you really want to rely on fibre optic lines between your office and the data center? Do you want to keep all your infrastructure in such a high-value target?

I think this is one of the more interesting areas to think about, at least for me!

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between:

1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based).

2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50% cheaper than AWS et al). We rent or colocate bare metal, manage it for you, handle software deployments, deploy only open-source, etc. Only really makes sense above €$5k/month spend.

3 - Rented Bare Metal – Let someone else handle the hardware financing for you. Still minimal cap-ex, but with greater hiring/skilling and risk. Around 90% cheaper than AWS et al (plus time).

4 - Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills, scale, cap-ex, and if you plan to run the servers for at least 3-5 years.

A good provider for option 3 is someone like Hetzner. Their internal ROI on server hardware seems to be around the 3 year mark. After which I assume it is either still running with a client, or goes into their server auction system.

Options 3 & 4 generally become more appealing either at scale, or when infrastructure is part of the core business. Option 1 is great for startups who want to spend very little initially, but then grow very quickly. Option 2 is pretty good for SMEs with baseline load, regular-sized business growth, and maybe an overworked DevOps team!

[0] https://lithus.eu, adam@

Agreed. The one bastion of sanity in all this is (/was) the UK. I formed my first company there, 18 years ago, online, in 30 minutes, for around £20.

I then moved to Portugal and started not one but two companies there. The whole process is so clearly setup to discourage people from actually forming companies. Everything from the attitude of all involved ("are you sure?!"), to the practical bureaucracy and costs involved.

I thought perhaps Portugal was just an outlier. But then I moved to Germany. And just wow. Definitely worse. Rounds of paperwork, notary offices, and fees. A process taking weeks. And for a GmbH a minimum investment of €25k. Sure you can form a UG. company for €1, but that effectively just announces "don't trust us, we're tiny" (IMHO).

It is something that really saddens and frustrates me with the EU.

Edit: And sure, you can form an Estonian company. But then you have to try and fly under the radar with regards to the 'permanent establishment' rules.

Well yes and no.

Yes you can form a company in Ireland while living in France. But you cannot get an Irish VAT number without a physical presence in Ireland.

And if – for example – France learns that you are running an Irish company from France (i.e. you have a 'permanent establishment' in France), they'll want you to file and pay French corporation tax. Which is likely sufficiently annoying that you may as well have formed a French company in the first place.