Packaging Kubernetes for Debian

https://lwn.net/SubscriberLink/835599/f3b49b022582e03e/

by pabs3 • 6 years ago

112 109 6 years ago

gitweb6• 6 years ago

The packaging nightmare is why I left Debian and Ubuntu for Arch. I now maintain multiple packages in the AUR with relative ease, and see how easily other maintainers manage multiple packages like Kubernetes without issue, but if it was using Deb packages it would be a full-time job.

viraptor3• 6 years ago

I'm not sure this is a fair comparison. AUR has next to no rules. You could just as easily maintain a PPA. As described in the article, the package is available via Debian-testing.

_y5hn3• 6 years ago

Arch/Manjaro is often praised as simpler, more powerful etc., but there are also complaints about instability. Mint has several QA links: upstream, debian, Ubuntu, Mint. With Mint sanitizing the corporate influence, ie. snap being off by default. I don't need bleeding edge, just stable desktop. So I've found my optimum in Mint, though nothing is perfect.

Vendoring is shifting focus back to individual projects though. K8s could almost replace an OS in many regards. It is made to be modular, but only within its own framework. So it is changing the way software is run, similarly to initd and systemd. Such systems as we've seen, tend to "take over" and not be made to be part of and integrate well from inside an OS. Vendoring is really old practice, but also makes testing and support simpler for upstream. We all know the dynamic approach tend to be more unchanging, or associated with several circles of dependency hells.

It's just different end goals for software development and deployment, that affect how software is distributed. Kind of similar to touted goals of JEE of java fame, which also is too monolithic to integrate well within complex systems.

gitweb6• 6 years ago

I encourage everyone to go evaluate building and submitting a package using the AUR vs Launchpad (PPA). I worked at Canonical and would never go back to Dpkg. I recommend users try it out themselves and see how much easier and cleaner the PKGBUILD approach is.

AUR does have a process to rate, flag and comment on packages. It is also much easier to review the installation script using PKGBUILD. There are a lot of high quality packages in the AUR but you should review everything you install unofficially, just like installing packages from a PPA.

konjin3• 6 years ago

The packaging nightmare also means that when you stand up a system it won't need upgrades for a decade

Leave an arch system without an upgrade for a month and you're playing Russian roulette.

gravypod• 6 years ago

It will need upgrades for those decades you're just hoping someone remembers how to apply them 9years after they're I'm wide spread adoption.

jchw2• 6 years ago

People say this all the time but it isn’t true. Occasionally a package will require manual intervention to update properly. The manual intervention steps are generally pretty easy and documented on the Arch homepage, so if you haven’t updated in a while and are concerned, all you really have to do is check the homepage. But I don’t even do that. I just pacman -Syu and check if it failed. For the past few years this has worked out just fine. Most of the packages, I don’t even have installed, so I hardly ever hit a manual intervention step.

Of course, Debian doesn’t require manual intervention for most updates, but it also isn’t rolling release. If you want packages that are actually recent you have to sit on Sid, which is a lot less stable than Arch. And if you have a Debian machine that’s a couple years old and you want to upgrade it to the latest Debian version? Good luck. Some of that isn’t Debian’s fault, but whereas on Arch major, breaking package changes are rare but irregular, on Debian they tend to hit you simultaneously in one major upgrade. For home machines, I definitely prefer an occasional manual intervention and continuous fixes over major breakages every couple years where I often just give up and reinstall from scratch.

That said I use Arch a lot less nowadays as I’ve moved onto NixOS. Nix is clearly headed somewhere new, though it remains to be seen if the complexity of the approach is actually maintainable.

If I want to make an idiomatic Arch package, it’s usually easy: PKGBUILDs are intuitive and simple. Nix is much the same for most stuff, especially since most of the boilerplate for various build systems has been automated; though for a program as complex as Kubernetes its a nightmare, to be sure. For Dpkg though, it seems like so much complexity, and I don’t think I ever really had a good experience.

And honestly, I have no idea how to make an RPM package anymore.

barumi2• 6 years ago

The packaging nightmare is why I left Debian and Ubuntu for Arch.

I find this comment odd and highly dubious. There is absolutely nothing wrong with Debian's packaging system. At most, some individual packages could have been packaged differently, which is arguably a matter of personal taste. Normally, everything just works, and works very well.

Either you shed some additional light on your personal struggles with Debian's packaging, which you did not do at all, or I have to scratch off your baseless assertion as a kin of someone on the internet shouting at clouds.

jchw2• 6 years ago

It is a recurring theme that whenever Arch proponents mention that they don't like Debian packaging, there is a frustrated reaction to it. But I don't understand what people want to hear; there's clearly a reason why people would choose Debian over Arch, Arch users say packaging is a big part of it, and it seems like nobody can accept that this is right.

I can at least tell you that people don't choose Arch over Debian for the installation experience. :)

raziel2p3• 6 years ago

In my experience, once you have a .deb package, it does indeed work perfectly. However, getting to that stage, unless you're packaging a collection of plain files with very simple dependencies, is the hard part.

Ignoring the weird scaffolding you need just to package a static "hello world" binary, there's also all the dh_ scripts which you should use for your package to be "well-made".

I remember looking up how to properly package a Python application, found like 5 different ways documented on the debian wiki, couldn't get any of them to work, gave up and just shipped a whole virtualenv.

rstuart41332• 6 years ago

I'm a DD, and I can see his point of view. The debian build system started form the simplest thing that would work, and grew from organically from there. Building in the same area as the source for example, and then having to clean the mess up as opposed to just building in a clean temporary directory like rpm did from day 1. We only got clean separation of debian patches from the upstream sources a while ago. Kinda. Sometimes debian has to repackage the upstream sourced due top copyright problems. uscan is hardly a nice solution syntax wise, nor is is policy so there doesn't have to be a fully scripted way of going from upstream source to debian package. And there is the dh sequencer, which splits the configuration into a zillion tiny little files. It goes on.

Like any organic solution it works, and somewhat impressively works for all the use cases created by debian's 60k odd packages. However, pretty or simple it isn't. Learning the interactions of the various dh_ helpers with the build systems they try to automate, the "guesses" (term they use in the manual) they make and when they are applicable is a huge job. The learning hill is consequently steep, and I would say unpleasant.

However, I don't want to undersell the end result. The QA checks done FTP masters, lintian, the discipline imposed by testing and the long transition from testing to stable mean the end result is very, very good. Better than I've seen in any other distro, including the .rpm ones I used to use. From a sysadmin point of view, that trumps everything. From what I can tell debian could be described as a bunch of programmers beavering away for free to produce something that none of them could produce on their own - a Linux distribution so good the base their day jobs on it.

csande17• 6 years ago

Say what you will about Gentoo, but I've actually really liked making some small packages for personal use. It's the closest system I've seen to just writing down the steps you'd normally take to compile the software manually.

1_player• 6 years ago

That how Arch Linux packages work. Just a shell file with some metadata

cbmuser2• 6 years ago

Arch doesn’t have to support multiple architectures and releases.

Also, Arch usually puts all files into a single binary package while Debian splits arch-dependent and arch-independent files into separate packages. Also, Debian separates packages into runtime and development libraries, another thing Arch doesn’t do either.

gitweb6• 6 years ago

This is incorrect

pushedx• 6 years ago

Kubernetes binaries are meant to be statically linked. I can understand the licensing concern, but if each of those libraries was packaged separately the experience of using Kubernetes on Debian would be markedly worse.

tupputuppu• 6 years ago

In which way would it be worse?

captn3m0• 6 years ago

Does anyone know of any packaging efforts to run microk8s without snap? I love microk8s, but running snap just for that one package feels like an overkill.

dharmab4• 6 years ago

When we implemented Kubernetes at my employer, one of the things we made clear from the outset was that there is no LTS release of Kubernetes. Keeping k8s and friends up to date is at least one full time job (ideally a half-pizza team).

This also heavily influenced our choice of OS away from ones that emphasized LTS package management and towards more recent kernels with better BPF and kTLS support.

(We also intentionally used short-lived TLS certificates to prevent anyone from running a cluster longer than several months without updates. Worked great. There are workarounds of course but anyone aware of them is the kind of person who updates anyway.)

polskibus2• 6 years ago

What distro did you end up with?

dharmab4• 6 years ago

CoreOS, then Flatcar.

zimbatm5• 6 years ago

That's why k8s is not a good fit for early startups. It will consume too much of your time that is better spent developing a product that the public wants.

AlphaSite• 6 years ago

In that case use a managed cluster and it’s less work.

znpy2• 6 years ago

imho if you're an early stage startup and you don't have an operations or devops person, you're better off either running everything off a single machine or doing devops the old way (as in baking machine images, updating launch configurations and doing service rollouts with via autoscaling groups).

Once you get more revenue, hiring an operations/devops person might be a good idea. A good operations/devops person might make you save his own salary in infrastructure optimization (depending on the scale of course).

another good pattern that i've seen in action is contracting the operations to an external company that's specialized in doing this. the approach would be to let the external company design and build the cluster for you, define responsibility boundaries for your company vs that company, and let such company take care of daily operations and cluster upgrades, possibly with on-call availability. A small support/consulting package (say, 15-20 hours/month) can get you very far.

We've been doing something similar with SigHup (sighup.io) and their Kubernetes Fury distribution, it's been working very well so far.

(disclaimer: i'm not employed by sighup, just pleased to work with them)

amq• 6 years ago

Running several production clusters on GKE, I can say that it is definitely not a full-time job to keep it up too date and healthy in that setting. I would imagine it's a completely different story on-prem.

d33• 6 years ago

half-pizza team

Could you define this term? I tried Googling, but it looks like there are just two references and neither is clear to me.

snarfy• 6 years ago

When you buy food for the meeting, how many pizzas do you need? If it's one pizza that's a one pizza team.

pseudalopex• 6 years ago

2 or 3 people. It's a reference to Amazon's 2 pizza teams. The idea is your team is too big if 2 large pizzas isn't enough to feed everyone.

dimitar• 6 years ago

Do you use managed Kubernetes or kubeadm?

I think if one uses a managed Kubernetes keeping it up-to-date is pretty easy. I did an AKS update in 10 minutes.

dharmab4• 6 years ago

Neither. Both the managed offerings and kubeadm don’t meet our requirements. (Largely related to security, compliance, scale and multi-tenancy.)

viraptor3• 6 years ago

What's the driver for people who actually want an OS-provided k8s? What holds them back from using upstream releases instead? (Or given large enough team - building/patching themselves) I guess it also applies to Ubuntu/openstack to a lesser level.

If you're planning to run those as a platform, it sounds like more pain then necessary.

_y5hn3• 6 years ago

For many it could just be a convenience, assuming the distro way just works. With conflicting goals and approaches, you might end up worse off though. Ideally, you mostly want to spend time and effort on higher pursuits than managing or building your own platform.

viraptor3• 6 years ago

Ideally, you mostly want to spend time and effort on higher pursuits than managing or building your own platform.

I've run openstack for a while and... realistically if you run a platform, you need to have people/time to manage it. I understand from conversations about k8s that is not that much different.

Fumtumi• 6 years ago

If you use microk8s from ubuntu, you know that there is a team behind it who takes care of updating it, upgrading it and making it simple for you to do so.

You also know, that your issue with that setup might get noticed by someone else using the same and will fix it for you.

You get sane defaults and you get k8s optimized for the os.

I run microk8s at home. Its simple, fast, easy and works.

Not seeing any issue with it.

thespoonbends• 6 years ago

I prefer to use Debian packages because it conforms to conventions that make managing systems easier: predictable naming, integration with service management, logging, docs, manages.

(I use NixOS for personal use.)

throwaway54235• 6 years ago

I worked on one of the large commercial projects that offers OS-packaged k8s.

The customers were screaming for it. They want a simpla and reliable way to deploy k8s without a team of specialist, and the guarantee to receive security updates for years without having to upgrade to a new version.

jancsika5• 6 years ago

There are other packages in Debian with vendored dependencies, though none, he acknowledged, have anywhere near the 200 found in his Kubernetes package.

How many are vendored in Chromium/Firefox?

kubanczyk• 6 years ago

The decision to share libraries between many apps has some benefit, especially on desktops. It also has costs. It's a decision you can make.

The article uses so many convoluted phrasings and arguments to avoid saying that. A bureaucracy of *.deb.

inshadows• 6 years ago

Avoid saying what? That it has pros and cons? What a shocking revelation you made there!

It's implied in few paragraphs. It's not that hard to infer if you take off your hat. Thou who read Corbet often know a thing having pros and cons is implied in his reasoning.

rstuart41332• 6 years ago

Thou who read Corbet often know a thing having pros and cons is implied in his reasoning.

His skill is the warts are laid bare, but not unkindly and often with humour.

dsr_2• 6 years ago

There's another interpretation available: kubernetes is not a mature system, and that's why it's too much work to package properly.

200 library dependencies is not too many for Debian to handle, as long as they don't change very often and the work is useful for other packages. But Kubernetes' dependencies do change often, and there are not enough other Go systems around to spread the work around. This is not a stable system, therefore Debian can't ship it in a stable release.

orf• 6 years ago

I think there’s a difference between “stable” and “evolving”. Software with a much lower release cadence than k8s works well with Debian’s distribution model, but it falls down with frequent releases and a large dependency tree. I guess you’d see that with NodeJS applications as well - I don’t think it’s practical for Debian to re-invent NPM and try to distribute and ship the large and changing dependency trees you find in those projects. However that doesn’t mean the software is not _stable_, for the typical definition of stable.

justinclift• 6 years ago

It kind of feels like the same problem OpenStack had, and failed due to.

That being said, OpenStack is technically still around, so it's not completely "dead". ;)

loxs• 6 years ago

Also I don't get it, what's the problem with shipping a statically linked binary? Yeah, it goes a bit against the debian way, but it will facilitate things like out-of-the-box init scripts etc.

tremon3• 6 years ago

I think there’s a difference between “stable” and “evolving”.

No, that's pretty much the Debian definition of "stable": software that doesn't need changing for multiple years. The Debian term "stable" has nothing to do with whether or not a program crashes regularly, but how often it requires maintenance. In that definition, "evolving" software indeed isn't "stable" (yet).

jancsika5• 6 years ago

There's another interpretation available: kubernetes is not a mature system, and that's why it's too much work to package properly.

If that were true, one would expect the packagers at Arch to be leveling similar complaints about the "maturity" of the software or the "awful" build system used by Kubernetes.

I'm going to rankly speculate that:

1. That is not the case.

2. Kubernetes was packaged in Arch in less time than it's taken for Debian to discuss the policy disagreement between the previous and current package maintainers, neither of which have been able to deliver a maintainable Kubernetes package in Debian yet.

3. This says more about the efficacy of Debian's packaging policies and package dev UX than it does about Kubernetes.

jancsika5• 6 years ago

Also-- I'm willing to bet there are similar charges somewhere in a Debian discussion list wrt Virtualbox and its packaging. And I'm willing to bet that there's currently a virtualbox package in Arch and little to no discussion about how "unstable" or whatever virtualbox is.

Foxboron3• 6 years ago

It's funny. I was discussing looking into packaging kubernetes with another Arch developer yesterday, and noticed this discussion today.

I'm the package maintainer for go and maintain a number of go packages in Arch Linux, along with writing the current guidelines and looking into packaging strategies for golang.

I'll offer a more cynic view then the one presented: golang is not a mature packaging ecosystem.

So 1) is partially correct; go just sucks from a packaging perspective. Not really the issue at kubernetes.

We can probably package up kubernetes within the next hour and drop it into `[community]` as Arch has less structure and QA around the packaging process. However our largest hurdle is that we package the latest version. Is kubernetes going to work properly on go version `1.15.3`. By experience container software brings out the worst of the go runtime and any changes to the syscall, goroutine or memory management is a cause of concern.

The other hurdle is the cadence of when kubernetes decides to support which container runtime. Docker is wonkey at best, but I haven't seen a lot of details on cri-o nor containerd.

So frankly our problems are not really related to packaging, but the challenges of providing recent versions of packages and making sure they work.

effie2• 6 years ago

Lots of people present their observations and opinions, but this problem is not hard, and the technically best solution is obvious - Debian curbs their ambitions and just distribute the damn thing, with all the vendored stuff, as upstream intended.

Some Debian people don't like it. Users don't care, as long as they can apt install it.

debianUserNew• 6 years ago

Debian Uses care. Otherwise they would use any other distro.

The good the about the distro fragmentation is precisely that, If debian curbs their ambitions then it is no longer debian, and those that prefer the debian way over ubuntu or fedora or arch would get screwed in the process.

It’s fine to not like it and not use it, but I do like projects that keep their ways.

effie2• 6 years ago

Well some may care, I agree with that. But Debian already changed its ways in controversial shift to systemd that leave many users disappointed - pragmatism and modernisation efforts prevailed over traditional ways of composability, stability and predictability. Why shouldn't this happen for the policy of packages as well? The Debian way is hitting its limits and is ripe for some updates. The amount of work needed to maintain foreign hard to comprehend software has to be curbed to make it manageable in the future. If enough people say "yeah lets package it as is and be done with it" why shouldn't Debian go that way? It is natural to change and adapt. Big software with fast release cadency is going to get even more complex and harder to package each release.

throwaway7281• 6 years ago

Naive question: I package small and midsize Go projects as deb packages all the time and find it relatively easy. All you need is the binaries and some config. Cross-compilation is easy with Go, so what exactly is a problem with k8s? Isn't it just a bunch of executables for kubectl, kubelet and api-server?

ibotty• 6 years ago

The problem is building and packaging (and versioning) all dependencies. It's a reasonable policy to not allow bundling which conflicts with usual Go developing.

moondev2• 6 years ago

kubeadm kubelet kubectl + socat and a few more standard utils. Where are the 200+ dependencies they mention?

cuillevel3• 6 years ago

Build time dependencies. Normally distributions would put every library from the dependency tree in a package and build it. Then install all the dependencies on the build host and produce the Kubernetes package.

moondev2• 6 years ago

Good to know, thanks for the clarification. I can see now I oversimplified the packaging process.

Packaging Kubernetes for Debian

Related Stories