There are only 12 binaries in Talos Linux

https://www.siderolabs.com/blog/there-are-only-12-binaries-in-talos-linux/

by JustinGarrison • 2 years ago

156 70 2 years ago

siderolabs.com

cedws• 2 years ago

As opposed to systemd which is over 3000 lines of C code I’ll never comprehend.

Well, technically true, but systemd is a whole lot more than 3000 lines...

I can see another binary in the demo video called apid, does that one not count?

Any comparison with Bottlerocket OS?

JustinGarrison11• 2 years ago

Many of the binaries you see in the demo video are showing processes running from inside containers. There will be a lot more processes once you start pulling containers and starting containerized services.

Notably the kubelet is also missing from the list because it's not built into the OS but pulled as needed from the correct version of Kubernetes requested.

Bottlerocket runs systemd and also runs 2 versions of containerd. One for the system and one for workloads. This (in theory) hardens the OS more, but in practice makes things extremely annoying to manage because you have to get a shell on the host to access the API.

disclaimer, I used to work at AWS on EKS and closely with the Bottlerocket team.

bantunes• 2 years ago

The systemd hate is getting long in the tooth now. It's not like it doesn't do anything with its line count, or that the code is obfuscated.

JustinGarrison11• 2 years ago

I’m actually a big fan of systemd. It’s an awesome, general purpose, and flexible init system.

I don’t think the complexity it brings is required for Kubernetes.

LAC-Tech• 2 years ago

I'm not reading that as hate, I read that as criticising systemd in the context of a stripped down system designed to do one thing.

I have systemd on the laptop I am typing on right now. Do I want it on some tiny embedded linux device? probably not.

shrubble• 2 years ago

So there is a quantum of criticism/observations about systemd that can be made but after that, no more is acceptable?

jchw2• 2 years ago

In retrospect, it would've saved a lot of trouble and misunderstandings if systemd had called the init daemon "systemd-init" to make it clear that not literally everything that is under the umbrella is part of the init daemon.

yjftsjthsd-h• 2 years ago

Eh... Most of the other components have a hard dependency on the init part; I'm not convinced that they're all that separate.

Jumziey• 2 years ago

SystemD bashing aside :p Talos is pretty awesome for setting up clusters. At home I just run talos with matchbox for PXE bootstrapping it works like a charm. Been really easy to maintain too. I normally just update matchbox and then reset a machine at a time with talos ctl for a clean install. It's something very reassuring with completely reset your machines so you know you could reinstall or replace them easily.

Granted just used in a home setting running smaller workloads for backups, private projects, git etc.

wmf2• 2 years ago

Where is networking configured? I assume the system has to have an IP address before containerd can fetch images.

JustinGarrison11• 2 years ago

Everything is API driven and static networking can be configured via kernel args

https://www.talos.dev/latest/reference/configuration/v1alpha...

burnte• 2 years ago

DHCP I would assume.

wmf2• 2 years ago

Yeah but where is the DHCP client? In the kernel?

birdiesanders4• 2 years ago

You use machineConfigs that are used to provision the base OS and configure it, and the clusterConfig is used to bootstrap k8s on those machines. You can make subtypes and super types, you can have different networking setups, whatever you like, just apply and the OS is driven to state, then k8s is brought up from there. You are presented a kubeconfig after. Changes are done via application of updated machineConfig. Works great in practices and if you write an operator you can manage the config generation via k8s manifests and get wild with it.

E39M5S62• 2 years ago

It's disingenuous to say that /sbin/init (machined/main.go) is less than 400 lines of code. Sure, that file is. What about all of the in-tree modules that are being imported? A super lazy summing of Go lines in the master branch of the repo:

$ find . -name *.go | xargs wc -l | tail -1

  354085 total

Heck, there are almost 100k lines under internal/app!

$ find internal/app -name *.go | xargs wc -l | tail -1

  96885 total

I'm curious what argument you are making here with regards to the number of lines in a single file.

JustinGarrison11• 2 years ago

I know there are a lot more lines and I didn’t count any of the imports from systemd either. 300 loc (machined) vs 3000 loc (systemd) was the closest comparison I could think of without crawling all imports and deps.

Would be happy to update with a different comparison you think is more fair.

raziel2p• 2 years ago

How about we just don't compare lines of code at all, as if it's a useful metric of anything?

coldtea2• 2 years ago

Kind of like TextAdept claiming that text editor is "just 2K lines of C/C++ and 4K lines of Lua", when it's in fact wrapping a well known third party text editor widget.

At least here it makes more sense, since it's not like "our init system is just 200 lines" when those lines are wrapping a third party init system library.

It's more like "our init system logic is just 200 lines, not including third party dependencies". That's legit, provided that those deps are stuff like parsers, some library for dealing with strings or running processes, and things like that.

abound• 2 years ago

Big fan of Talos, have used it in some homelab + cloud clusters over the years, currently powers all my self-hosting. The `talosctl` command is great, and any time you need to do node-level debugging, there's always something like node-shell [1].

[1] https://github.com/kvaps/kubectl-node-shell

cperciva• 2 years ago

The /sbin/init binary is hard linked to /sbin/dashboard, /sbin/poweroff, /sbin/shutdown, and /sbin/wrapperd. While this technically is 5 files, it’s a single file hard linked 4 times to provide convenience commands.

Err, that's definitely 1 file with 5 directory entries, not 5 files.

titanomachy• 2 years ago

Wonder why they would use hard links instead of symlinks.

Edit: interesting, seems like there's a mild performance benefit.

https://unix.stackexchange.com/a/20716

0xbadcafebee3• 2 years ago

There's a lot of benefits actually, as symbolic links often need to be handled as a completely different type of file with different semantics which often leads to bugs. Hard links are always better when you know you are dealing with a single static host & filesystem. Symbolic links in such a case are only better for indicating quickly to the user which files are linked to which others.

vundercind• 2 years ago

Reverse question: why use symlinks when you can get away with hard links?

cpach2• 2 years ago

Not sure it really matters that much in this case. But why not? Hard links are a quite clean and simple concept. One “object”, multiple names.

rwiggins• 2 years ago

Super cool. I always enjoy reading about systems that challenge, well, "ossified" assumptions. An OS not providing a shell, for example? Madness! ... or is it genius, if the OS has a specific purpose...? It's thought-provoking, if nothing else.

I'm a bit skeptical of parts. For instance, the "init" binary being less than 400 lines of golang - wow! And sure, main.go [1] is less than 400 lines and very readable. Then you squint at the list of imported packages, or look to the left at the directory list and realize main.go isn't nearly the entire init binary.

That `talosctl list` invocation [2] didn't escape my notice either. Sure, the base OS may have only a handful of binaries - how many of those traditional utilities have been stuffed into the API server? Not that I disagree with the approach! I think every company eventually replaces direct shell access with a daemon like this. It's just that "binary footprint" can get a bit funny if you have a really sophisticated API server sitting somewhere.

[1] https://github.com/siderolabs/talos/blob/main/internal/app/m...

[2] https://www.talos.dev/v1.6/reference/cli/#talosctl-list

sspiff• 2 years ago

Exactly this. I was thinking of making a similar comment but you made it far better than I could.

Number of binaries is kind of a meaningless metric, especially for a system that historically follows the UNIX philosphy of each program doing one thing.

Sure, a shell is complicated and a potential risk, and perhaps it's a good idea to exclude from the base system in this context.

But I'd rather have ls, tr and wc on my system than some bespoke, all-encompassing API service that has been far less battle tested providing similar functionality.

And like you rightly pointed out, these new binaries all contain their own list of dependencies which are pulled in at build time and need to be taken into scope as well.

That's not to say Talos or its approach doesn't hold merit, but I think it's a little disengenious to simply point at the number of binaries.

JustinGarrison11• 2 years ago

I agree number of binaries is an arbitrary metric but also an indicator that things work differently with Talos. You have to use the declarative API for management which some people could see as a bad thing.

I’d also like to point out that the system API is designed to be extendable and adaptable to different operating systems. We’d love for more vendors to create adapters/shims to get the benefits of API managed Linux

https://github.com/cosi-project/community

0xbadcafebee3• 2 years ago

20 years ago, I used to make custom Linux distros for fun. Floppy distros, CDROM distros, RAM-resident distros, network-boot distros. In a few of them, I custom-made my own binary that was both the init system, and a few applications, stripped it down, and shipped just that as the distro (basically just a few files and my static binary).

A lot of people downloaded them, and it was great fun - to start. Problem is when you want to do more things. You have to start finding workarounds to bolt-on additional tools, or maybe you just throw one or two extra tools in there by default. Over time you find more and more missing things or incompatibilities with other systems, which make it harder to cover more use cases. And finally you realize that "the tiniest system" is a lot more effort than it's worth, and what you really want is "a slim yet compatible system". The system you end up with is a lot fatter, but a lot less headache.

(The security benefits of fewer files are overblown, too. If you audit and harden the system, it doesn't matter how many binaries you have, because the attack vectors they use will be mitigated)

andrewrynhard3• 2 years ago

In the case of Talos, Kubernetes can provide the flexibility you want from a more traditional Linux distribution.

AtlasBarfed2• 2 years ago

That sounds like Kool aid. Can you expound more on this?

JustinGarrison11• 2 years ago

We use “system extensions” to give you flexibility while keeping the base small.

Want GPU drivers? Add the extension. Need Tailscale? Extension.

https://www.talos.dev/v1.6/talos-guides/configuration/system...

Brian_K_White4• 2 years ago

How is that different than adding binaries, in the context of this comment where the point was "In the end you end up just adding everything back in that you originally took out, because managing all the little weird different subsets is not worth the benefit."

In the context of that assertion, adding or subtracting "extensions" and adding or subtracting binaries are equivalent. Both are adding or subtracting "piece of code that provides fuction".

Dwedit• 2 years ago

MX Linux would be that "slim-yet-compatible" system. It can load itself completely into RAM, and has many distro-specific components such as Frugal Install and Remastering, yet also lets you install standard Debian packages.

miki123211• 2 years ago

I wonder how a really slimmed down distro like Alpine would compare here, particularly in terms of image size.

It offers most of the standard Linux utilities we know and love, but most of them are actually just symlinks to Busybox, which is ~900K on my (ARM64) system. That's less than a hello world in Go, for a program that can replace most common Linux utilities in daily usage.

JustinGarrison11• 2 years ago

Busybox is only 1 binary symlinked hundreds of times

Brian_K_White4• 2 years ago

Yes, and?

(They and we all know that. They said as much theirself right in the comment.)

birdiesanders4• 2 years ago

We operate talos and alpine based nodes, many thousands of them. The build chain for alpine is many orders of magnitude more complex than the build pipe for our talos image modifications. Alpine is really not made for doing a lot of “host” tasks and needs much coercion to get it to be capable of running something like k3s, and much more complex to get kubeadm clusters running on it. In the end the complexity is required for flexibility, alpine nodes can be modified on a whim, talos is R/O and ephemeral, but more secure.

suralind• 2 years ago

Big fan of Talos, I use it on Hetzner and it's a joy!

ofrzeta• 2 years ago

How do you install it?

xcdzvyn• 2 years ago

I see Talos only supports XFS, what potential reasons could they have to prefer XFS to competitors?

I've always struggled to compare filesystems fairly. My justification for ext4 is just that everybody else uses it :)

cchance• 2 years ago

xfs is technically more robust and performant than ext4, ext4's just the most widely supported, but most use xfs when available

rickette• 2 years ago

Mongo and Elasticsearch also recommend using XFS. So Talos isn't unique in this. XFS is somewhat more often preferred in data intensive systems.

There are only 12 binaries in Talos Linux

Related Stories