Show HN: distri: a Linux distribution to research fast package management

https://michael.stapelberg.ch/posts/2019-08-17-introducing-distri/

by secure • 7 years ago

92 66 7 years ago

michael.stapelberg.ch

viraptor2• 7 years ago

I'm curious about the no-triggers claim. There's stuff that happens after installing packages sometimes, which you can't put into an image. For example refreshing the font cache. How does that work here?

secure17• 7 years ago

An ideal cache implementation would transparently recognize the need to update itself when required, and do so efficiently and transparently.

In the particular case of the font cache, I’d say that the library which uses the cache should recognize that an update is needed. I.e., the update happens at next use, not at package installation time. On server systems, where fonts might never be loaded, this saves some compute :)

viraptor2• 7 years ago

But in this case they're building an OS from existing components and not redesigning GUI libraries. The current reality is that you run fc-cache to update the cache.

Hooks in this case don't impact servers because fonts are not installed in the first place so caches don't need updating.

aasasd2• 7 years ago

You might find this a problematic approach:

- You make packages track more state―“stuff has changed”―and if there are no triggers, you can't even set a flag for the package's runtime: instead, you have to paranoically check for changes at the run time and compute the difference. Meanwhile, an installer/updater knows exactly that something was changed, and what precisely it was―because the installer just done that.

- You're shifting work from install time to run time, which e.g. in backend web programming is exactly the opposite of the right thing to do. Because modifications, in most cases, occur much more rarely than usage, and because making the user wait is a no-no. So, an admin who wants to prepare installed packages for the use, would have to patch the ‘trigger’ stage back in, via their scripts or something―with the caveat that they can't pause the installation in the meantime, so that all changes are processed before the updated programs are run.

waddlesplash5• 7 years ago

So ... this is more or less exactly how Haiku's package management system works. (It was designed in ~2008-2009, merged into the nightlies in 2013, and of course has been used by default since then.)

We use different terminology for a number of these things, but all the concepts are almost identical. Is this really the case of two completely separate creations of the same idea? Or is this another instance of the Linux world NIH'ing something a different project had for a while without even mentioning it?

secure17• 7 years ago

I learnt about the Haiku package management system when I had already developed distri for several months.

I do actually reference Haiku, but in the referenced post https://michael.stapelberg.ch/posts/2019-08-17-linux-package..., not in the distri introduction post itself.

I think it’s very telling that the two approaches look so similar, so I was really happy to learn about the similarities with Haiku!

waddlesplash5• 7 years ago

"HaikuDepot" is just the name for the GUI interface. The package manager itself has no real "name." The number of similarities being so high, I am still skeptical you really did not come across Haiku before, but well...

On a different note, some key things we have discovered in working with this system for almost a decade now are:

(1) One really needs a dedicated kernel module for this. Managing the abstraction in userland only gets too tedious after a while, especially around updates and the like. Moving mount management to a kernel "packagefs" makes things so much simpler (and of course more performant). Not to mention that you can then write a dedicated file format which supports random access better.

(2) Perhaps the most powerful feature of this system, the ability to boot into a previous state, is virtually impossible without kernel support.

(3) You will be surprised at how many users want to change things inside packages; and how many Linux users refuse to switch to a system that does not allow this. On Haiku we have mechanisms for overriding packaged files and "blacklisting" files; but on Linux you may find people are far too wired into the "old way" to get a radical change like this off the ground.

(4) The second most powerful feature, the ability to install packages in ~ (or, eventually, anywhere else for that matter) sounds great, but it requires a massive amount of software patching, and for applications to use APIs (that Linux does not have) to iterate through paths rather than hard-coding them.

So if you are going to make such a big, systemic change, why not just ditch Linux for Haiku instead of re-creating what we have spent so long making work already?

siscia2• 7 years ago

At work we avoid package manager altogether following the mantra that the best package manager is no package manager at all.

All the software is installed in a global directory /cvmfs and distribute to all the clients using FUSE and simple http servers.

Then different departments have ownership of different subpaths. General software that can be useful to everybody is installed in /cvmfs/sft.cern.ch software that is useful to only a specific collaboration is installed in, as an example, /cvmfs/lhcb.cern.ch

secure17• 7 years ago

Thanks for sharing, this sounds cool! I hadn’t heard of cvmfs, but we use a similar system at work, and it works really well.

siscia2• 7 years ago

Indeed, it is one of the, I would say, key technology developed at CERN that we were unable to push enough in the industry.

What system do you use at your place?

marmaduke• 7 years ago

I think this sort of setup is frequently found in HPC systems with lots of compute nodes. Tons of software installed into a NFS share, so the nodes can be reimaged easily with no breakage of user software (ideally).

Our /soft mount is at 200 GB for a small group after 8 years operating.

return_0e• 7 years ago

This is very interesting as this is almost identical to how the Haiku Operating System does package management using their own packaging format (hpkg) which uses packagefs. [0] [1]

This format is used more than just to package applications, but to update the whole OS in a consistent manner [2] as it is also versioned in with shared-libraries and this was implemented in 2013.

[0] https://www.haiku-os.org/blog/zooey/2011-01-08_package_manag...

[1] https://www.haiku-os.org/guides/daily-tasks/install-applicat...

[2] https://www.haiku-os.org/blog/bonefish/2011-06-20_package_ma...

secure17• 7 years ago

Thanks for sharing! Repeating my comment from the other thread for visibility:

I learnt about the Haiku package management system when I had already developed distri for several months.

I think it’s very telling that the two approaches look so similar, so I was really happy to learn about the similarities with Haiku!

Boulth• 7 years ago

That's super interesting! I've been pondering better package management tools for a while as the current landscape seems stagnated.

I hope some ideas from distri will make their way into mainstream distros. Maybe the more agile ones (Gentoo, Arch) would be interested?

secure17• 7 years ago

Thanks! Yeah, I certainly hope many distros can pick it up. I have heard from someone at SUSE: https://twitter.com/fleming_matt/status/1162819502050070528

jchw3• 7 years ago

There’s been some related work lately, too.

- Nix packages are still archives, but they do have separate roots, and use runpath to hardcode shared library paths into the Nix store.

- rpm-ostree provides something of a hybrid between images and packaging, allowing for more seamless OS upgrades while still supporting more or less traditional packages on top.

I hope that in the near future, Linux package management can jump into the next phase. Package management on UNIX likes already offered some neat advantages with their complications, but it’s interesting to watch this occur while Windows and macOS largely maintain the same model they’ve had basically forever, for better or worse. (I guess on Windows and macOS there’s an increased focus on an app-store-based distribution model, but it’s not really full system package management. The most interesting bit is probably the sandboxing.)

techntoke3• 7 years ago

The advantage of Linux packaging for distros like Arch become apparent when you actually want to create a package, and realize how easy it is.

jchw3• 7 years ago

Arch is definitely nice, with PKGBUILDs that are basically exactly what you want.

NixOS has its Nix language, and the Nixpkgs system, which definitely takes some time and I'd even say is more cumbersome than PKGBUILDs, but it has its advantages.

But going back, building packages for RPM or Dpkg based distros does not really feel that simple or easy. I think easy packaging is something we have today by virtue of simplifying things.

lone_haxx0r• 7 years ago

The only thing I want is a distro that lets me download a package from the web and install it locally without connecting to any central repo.

I recently learned that Slackware does that, so as soon as I have a little free time, I'll switch to it.

oneplane4• 7 years ago

Pretty much every distro ever already does this. At the same time, practically every repository is already a web page with packages.

Take Ubuntu and Debian for example; if you download a .deb file you can install it from the GUI and CLI as-is.

An example with pictures from a random Google result: https://itsfoss.com/install-deb-files-ubuntu/

stonogo• 7 years ago

There is also checkinstall, which replaces the 'make install' step in building from source and outputs a native package.

lmz• 7 years ago

And the problem with this approach is of course the dependency management. Just because you have the file for package X doesn't mean it will work if you're missing some of its dependencies. And those dependencies have dependencies etc. Which is why now we use e.g. yum and apt to install stuff, and not rpm and dpkg directly.

CameronNemo4• 7 years ago

Endless OS is specifically geared to "commonly offline" distribution models. Packages are shipped as mostly self contained images.

oneplane4• 7 years ago

While I see the advantages and really enjoy this type of work (and the ones like Nix and OS-as-an-image efforts), I don't think it would be a one-true-solution in a one-size-fits-all construction.

Sometimes you want a single filesystem, sometimes you want multiple filesystems, sometimes you want an image copied into RAM and nothing more, sometimes you don't have the resources to even load something like FUSE.

I believe that if we are to make anything better we have to be able to either switch or combine the many ways one can make a modular or composable system.

We can already swap bootloaders, kernels, desktop environments, transplant storage from one system to another and continue working. It should be just a feasible to switch filesystem/package models.

secure17• 7 years ago

Certainly distri isn’t suitable for all environments, but I think the general concept can be very resource-efficient.

A few thoughts inline:

Sometimes you want a single filesystem, sometimes you want multiple filesystems

distri presents packages in a single file system (mounted at /ro). The fact that there are many images underneath remains an implementation detail to the user.

sometimes you don't have the resources to even load something like FUSE.

I suppose in those environments you just can’t have package management, then, given how light-weight FUSE is.

We can already swap bootloaders, kernels, desktop environments, transplant storage from one system to another and continue working. It should be just a feasible to switch filesystem/package models.

Supporting more than one packaging model seems like an enormous task to me. I think distributions are better off not blowing up their test matrix like that :)

heavenlyhash• 7 years ago

Super agree.

What would it take to make that a reality, though -- without, like secure's sibling comment says, blowing up the text matrix?

Which things would you make standardized (or conventional) to make switching easier?

What things would you strip out of consideration to make switching easier?

One of the things I like about Distri is that by questioning whether a bunch of inter-package interactions are necessary (and finding out that, often, the answer is "no"), it gets closer to a model of packaging where things are less prone to create either conflicts or sprawling dependency trees. That feels like a step in a good direction, at least, to me.

secure17• 7 years ago

The most significant difference in this context are the separate hierarchies—programs need to be recompiled to work when made available under the /ro mount point.

I don’t see a way to avoid having two versions of a package when you want to support distri-style package management and traditional-style package management.

solarkraft• 7 years ago

This is very nice and it comes as rather surprising to me that it doesn't exist yet in a "serious" Linux distribution.

While reading about the file system structure I was reminded a of Gobo Linux concepts - have you heard of it?

I would really like to use something like this on my computer. Are there disadvantages serious enough to make pursuing this not worth while?

secure17• 7 years ago

While reading about the file system structure I was reminded a of Gobo Linux concepts - have you heard of it?

Thanks for the pointer! Yeah, on twitter a few people have pointed to GoboLinux, too. I had read about it a few years ago, and indeed there are a number of similarities.

I would really like to use something like this on my computer. Are there disadvantages serious enough to make pursuing this not worth while?

I don’t think there are inherent disadvantages, other than that the project goals I want to spend time on are experimental/exploratory in nature, rather than building a community around it. I’m not looking to start a new Linux distribution with a user base, I’m trying to show established distributions how much room they have for optimizations, so that we all profit :)

rurban• 7 years ago

A package-specific /ro prefix is annoying and will not persuade many.

void linux uses standard paths, and still has a blazingly package manager, without the need for images being mounted ro. https://wiki.voidlinux.org/XBPS

pacman is also pretty fast. apt, nix and rpm/yum are by far the slowest package managers. suse's has the fastest undo, via btrfs snapshots.

secure17• 7 years ago

What’s annoying about the prefix?

Thanks for mentioning void linux. I gave it a shot in Docker just now, but it seems rather traditional in that it does dependency resolution, does not allow for co-installability, uses transactions, unpacks archives, and runs hooks/triggers.

CameronNemo4• 7 years ago

I think his point was that xbps is fast enough for normal operation, while still offering a traditional package management workflow. Alpine's APK is even faster according to one of the XBPS developers and another void maintainer that uses Alpine at work.

(paragraph 2) https://www.reddit.com/r/voidlinux/comments/ccppu0/opkg_vs_a...

CameronNemo4• 7 years ago

By the way, all of the common void xbps triggers are found here: https://github.com/void-linux/void-packages/tree/master/srcp...

These are one of the largest time sinks for the package manager.

codedokode2• 7 years ago

I don't think that possibly slow package manager is main Debian problem. There are more serious issues:

- bug tracker is email-based, not web-based, very old, and without keyword search.

- old versions of software

- unnesessary duplication: repositories contain Python packages that can be installed with Python package managers.

- Debian is not friendly to third-party software, to closed-source software. For example, to install Slack or VS code you have to add third-party repository and give permanent root access to your system to Slack Inc or Microsoft. You have a choice between backdooring your system or not using third-party apps. Compare this to Android that is also linux-based but provides a reliable and comfortable jail for every application except Google's.

- Debian doesn't protect your information from malicious software. For example, if you install third-party app, it will be able to read your browser's history and cookies in your home directory. Also, any unprivileged program can read hardware identifiers like MAC address, hardware list, disk serial number, BIOS data etc. This is perfect for tracking users even if they reinstall or change their operating system. Again, compare this to Android.

- package manager doesn't allow to install several versions of PHP or Node.JS if you are working on several projects. But for language like Python they allow to install two different versions, why Python gets such a privileged treatment and PHP or Node doesn't I don't understand.

There are projects aiming to solve some of these problems like Snapcraft, but as far as I am aware, they are not integrated into Debian yet.

Also, it is a bad idea to hardcode paths (like /use/share) within application, so that you have to recompile it to change the path. All applications should be portable by default to make installation into home directory easier.

johnmarcus• 7 years ago

how does slack have permanent root access to my system again?

CameronNemo4• 7 years ago

The repo can theoretically hijack a package like util-linux, unless you creatively set up apt pinning.

codedokode2• 7 years ago

If you download and unpack Slack's .deb package [1] , you'll find a script "/etc/cron.daily/slack" (contained within "data.tar.xz") that is a bash script run by cron daily under root account.

This cron script adds a line "deb https://packagecloud.io/slacktechnologies/slack/debian/ jessie main" into apt sources list. Also it adds Slack Inc. public key into list of trusted keys. This allows Slack Inc. to replace any package on your system including bash, firefox, libreoffice or sshd by publishing a package with the same name but higher version number. Note that they can provide malicious updates only to specific users, identifying them by country or IP address, so that other users won't see anything suspicious. For example, law enforcement agencies can ask them to backdoor a suspect's computer to gather evidence or important data for intelligence.

This is the problem with apt and with Linux package managers in general: they assume you only add package sources you completely trust and do not support installing third-party packags without this trust (unlike Windows that supports portable applications or Android that doesn't give root privileges to installed packages). Linux package managers are not suited for installing untrusted third-party applications.

[1] https://downloads.slack-edge.com/linux_releases/slack-deskto...

antpls• 7 years ago

I agree. Furthermore, it feels like creating and maintaining a Debian package becomes redundant with various git/docker/pip/maven repositories and tools.

Half of the package management problems are social issues such as security, audit, openness to newcomers, collaboration, and trust.

We could probably solves those problems for all package managers (pip, various image registries, npm, etc) at once. We would maybe need a new package manager protocol that everyone agrees on rather than new filesystem tricks. Something like a federated package manager (sounds a bit buzzword).

drudru112• 7 years ago

So is there any performance hit or gain when using SquashFS? I wonder why all the container folks aren’t using this.

secure17• 7 years ago

I haven’t noticed a gain, but also no substantial hit for day-to-day use.

I can only speculate as to why containers are not using SquashFS, but my first guess would be that they wanted to use something more broadly supported. Manipulating SquashFS images is not as commonly available in different programming languages as dealing with e.g. tar archives.

drudru112• 7 years ago

If there isn’t a performance hit, and you get all the benefits (especially atomicity), this seems like a nice, easy win. I will tinker with SquashFS today.

Thanks for writing this up.

elFarto• 7 years ago

Interesting, this mirrors a lot of thoughts I've had on package management on Linux.

I'm not sure I like the idea of using file system images as packages, wouldn't that have overhead in terms of the sheer number of mount points you'd need, and additionally the permissions needed to actually mount it.

Keeping each package separate is a good idea, I certainly think the current 'soup' method of package management is not ideal. By 'soup' I mean adding files all over the place, and then giving it a good stir by running some scripts; it's very difficult to get back to the state the system was in before the package was installed without doing extra work to keep track of those changes. I was looking through the Alpine package management system, a lot of packages post-install script would add a user, but the remove script rarely removed that user.

Ideally packages shouldn't care where they're installed to. If a binary in the package needs to know where it's installed to, it's possible to work that out at run time (even for libraries, although it's not that straight forward).

In my idea for a package manager, each package would have different types of files in different directories, e.g. all command line binaries would be in /bin, all libraries in /lib, all fonts in /share/fonts, etc. In addition, it would be possible to have scripts run when another package was installed that provided a set of files you could use, for instance: when a package that contains fonts is installed, the package that is responsible for fonts on the system would add the relevant directory to the config and refresh the cache.

If a package had a library in /lib, a package could update /etc/ld.so.conf and refresh the cache. Linking files to shared directories would be avoided as much as possible (you probably need to do it for /bin). Using an exchange directory is somewhat of a crutch, patching the system to work as applications expect a traditional Linux system to work.

Hmm, this post seems to have just turned into a bit of a brain dump.

compsciphd• 7 years ago

see https://www.usenix.org/legacy/event/atc10/tech/full_papers/P...

and

https://www.usenix.org/legacy/events/lisa11/tech/full_papers...

Show HN: distri: a Linux distribution to research fast package management

Related Stories