Debian discusses vendoring again (2021)

https://lwn.net/Articles/842319/

by goranmoomin • 3 years ago

115 123 3 years ago

lwn.net

brycewray• 3 years ago

(2021)

avgcorrection• 3 years ago

This is Debian so default oldness should be expected.

teddyh• 3 years ago

Discussion at the time: <https://news.ycombinator.com/item?id=25758863>

dang• 3 years ago

Thanks! Macroexpanded:

Debian discusses vendoring again - https://news.ycombinator.com/item?id=25758863 - Jan 2021 (159 comments)

worksonmine4• 3 years ago

I use Debian because it is boring and stable. My core system is small and never changes by design. For anything that needs bells and whistles I either spin up a VM, use another package manager or build from source.

Don't try to change Debian, not every package needs to be on every system.

jampekka2• 3 years ago

Even unstable is nowadays boring and stable, and it's hard to get into trouble even with experimental!

Decade or two ago it was more exciting when apt upgrade on unstable (or testing after a recent stable release) quite regularly got your computer into an unbootable state, the DE fubared or packages into some interlocking dependency mess that could be more or less impossible to solve. Haven't had that excitement many many years now.

PedroBatista• 3 years ago

Yeah, but I don't know if that's true throughout all the ecosystem.

One of the main selling points ( if not THE selling point of Debian ) is the stability and lack of "surprises".

If that fails and Debian starts to be perceived as "once was the gold standard, now they are slacking", trust erodes. Even if you're right about 80% of it.

lmm10• 3 years ago

The Debian model of an artisanal handcrafted build for every library is just not sustainable. Integrating deeply with the language's packaging ecosystem might be one answer, but they did that with Perl to (IMO) not great results, and I don't think there's much appetite for doing that with Javascript (from either side).

edhelas• 3 years ago

Yet its there today and still one of the more reliable packaging system and distro you can find.

lmm10• 3 years ago

I don't know anyone who uses Debian and actually sticks with their packaged apps. The most important programs they're using on their computers will be installed either from third-party repositories, or some other tool (whether that's snap/flatpak/nix/etc., or a language-specific package manager like NPM), or built from source themselves, because Debian's "native" packaged versions of everything are chronically out of date.

Barrin92• 3 years ago

I wish Opensuse's Build Service gained more traction. Automated, reproducible builds for most distributions, itself open source etc. One of the many underrated tools in the ecosystem.

mrweasel3• 3 years ago

The Debian model of an artisanal handcrafted build for every library is just not sustainable.

Yet it has functioned for 25 years (apt turned 25 this year). There has been mistakes, like the OpenSSL "fix", but not many. One issue that I, now that you mention Javascript, is stuff like npm, but to me that more a sign of npm being badly broken by design, or at least designed for a different environment. Build Python projects on Debian, using apt as you dependency manager works pretty well for systems that you expect to be stable over time.

One think so seems to forget is that you can often pull in newer versions of libraries and other packages from Backports. That does mitigate some of the issues with packages being to old.

lmm10• 3 years ago

Yet it that functioned for 25 years (apt turned 25 this year).

Up to a point. Even 15 years ago the issues were clear and people were having to work around them.

There has been mistakes, like the OpenSSL "fix", but not many.

That was the single worst bug in a general-purpose computer system IMO, and a predictable result of the Debian way of building packages. But they never felt the need to change their policies.

Build Python projects on Debian, using apt as you dependency manager works pretty well

Python has notoriously terrible dependency management. It's a pretty low bar.

One think so seems to forget is that you can often pull in newer versions of libraries and other packages from Backports. That does mitigate some of the issues with packages being to old.

The fact that there is such a big ecosystem of workarounds for linux distro packaging should tell you there's something fundamentally wrong with the basic idea.

paldepind2• 3 years ago

One issue that I, now that you mention Javascript, is stuff like npm, but to me that more a sign of npm being badly broken by design, or at least designed for a different environment.

npm being "badly broken by design" is one opinion. But the fact is that almost all modern language tooling is moving in the direction of becoming more npm-like or has been created from the beginning with heavy npm inspiration. See for instance Rust's Cargo, Dart's Pub, Go's modules, Python's virtual env, Esy for Ocaml, and more. Developers love being able to easily manage and add dependencies, and this problem is only going to grow for Linux distributions across language ecosystems.

duped• 3 years ago

If it's functioned for 25 years why does no one use it for their ecosystem?

There's a wide range of package managers in the wild that exist to help build software using features that apt fundamentally does not support because of its broken model of the universe.

hedora3• 3 years ago

Back when I wrote a lot of Perl code, I limited myself to the debian packages, since they were tested and API stability was enforced by the debian team. It was missing a few packages, but they were either niche or low quality.

CPAN was untested garbage by comparison. (In the same way that pip is today.)

rollcat• 3 years ago

It's what OpenBSD ports have been doing for npm packages for a while, until they eventually gave up and just told people to install using npm.

There are very strong arguments for both approaches. Some people will tell you to use a version of a library in your distro's repo, because it's been vetted by a third party. It's nice to be able to choose an OS/distro that fits your needs.

curt152• 3 years ago

Besides the obvious people scaling problem, trying to create a deb package per pip package or rust crate will also explode the size of the apt metadata to download.

nerdponx5• 3 years ago

This is one of the places that the often-maligned Homebrew package manager does a good job.

App dependencies in languages like Python and Lua are fetched manually, with exact filenames and hashes, and installed into an isolated location just for that app. If they turn out to be used by several apps, then they get packaged normally.

I wish other package managers did it this way. It would make packaging complicated applications much easier and result in many fewer dependency conflicts, and allows devs to produce more stable software.

It's basically static linking but for Python packages. Sometimes you want shared libraries, but for one-off dependencies it's possibly more trouble than it's worth.

lloydatkinson2• 3 years ago

Nix/NixOS too

gcr• 3 years ago

I didn't know this about homebrew. Is this only true for installing homebrew python packages?

To be honest, I would `brew install python3` or whatever, then manage my own per-project virtual environments with direnv, so I'd bypass all of homebrew's efforts to encapsulate and package python packages...

nerdponx5• 3 years ago

It's at the discretion of the package maintainer, but it's pretty common. Just look at the `Formula.rb` for any standalone app in Python or Lua. Example: https://github.com/Homebrew/homebrew-core/blob/66c42fbd814b3...

I think using per-project venv is still the sensible thing to do, not just in Python but using the equivalent in Lua (hererocks), Ruby (Bundler), etc.

This is for system-wide installations of standalone CLI tools and other "apps".

However if you do find something on PyPI that is a standalone app, but isn't in Homebrew, you can create an isolated env for it automatically using Pipx https://pypa.github.io/pipx/.

cogman10• 3 years ago

You are basically describing appimage, flatpak, and to a lesser extent, snap and docker.

AppImage is the closest analog as it will reuse files across images.

nerdponx5• 3 years ago

I'm not sure about that. I think it's more akin to vendoring libraries in a C application by unpacking someone else's tarball into your own source tree.

The overall philosophy is that of AppImage I suppose. Flatpak doesn't really count because you still have "runtimes" that other apps depend on.

hedora3• 3 years ago

The article suggests using debian’s contrib package category in this way. (Packages with such dependencies live in contrib, since they don’t live up to Debian’s quality standards, which is fine.)

Fnoord• 3 years ago

Any reproducible package manager is suffice. Debian aims for that, too, AFAIK.

With Homebrew you end up having to link the binary, and you cannot link different binaries with the same name.

I just use venvs for Python, and pipx takes care of that, together with topgrade which ensures all software repos remain updated, Homebrew or Nix or not. For Rust I use Cargo, etc etc.

I'd agree Nix does it more elegantly but it is a very different beast to handle. Anything I want to do with Nix(OS) I have to look up how they do it.

nerdponx5• 3 years ago

The whole point of the thread is that Debian doesn't currently vendor dependencies, so you need to package every single dependency. Reproducibly, but fully packaged and maintained nonetheless, and necessitating careful testing to avoid conflicts between versions.

My point is that Homebrew takes the vendoring approach instead.

mikemcquaid• 3 years ago

As the person who has probably influenced Homebrew's policy on this the most: I'm very pleased to read this, thanks for saying it!

I was also reading this link today to think more about how Homebrew can handle this stuff better.

To me the main differentiation is "is the package for an application or a library?". If it's a library: it's a trickier balance. If it's an application: vendoring the e.g. NPM/RubyGems modules is most likely to create the best, most useful experience for the user, upstream creator and Homebrew maintainers.

nerdponx5• 3 years ago

As a very very occasional and light-duty packaging contributor, IMO the best thing Homebrew can do at this point is start writing much better docs for packagers. The Ruby API doc is very difficult to navigate if you aren't already a Ruby programmer. And it's very hard to know what the "right way" is for doing anything, other than browsing other formulas and hoping you land on a useful example.

This is another area where Homebrew has an opportunity to do better than the alternatives.

Making it faster wouldn't hurt either, but that's another story ;)

dgrabla• 3 years ago

My solution: Debian + Flatpak for the few applications you need at the latest version

RedShift12• 3 years ago

I think this is partially caused by the frankly anemic standard library some languages come with. Take node for example, it tooks years (almost a decade?) for "fetch" to be available and you still can't connect to a database without external libraries, so you need to install those which then bring in their own dependencies because the standard library is so limited...

masklinn• 3 years ago

“Go” is cited as one of the problematic ecosystems and it’s generally considered a “batteries included” language.

The problem is not anemic standard libraries, it’s ecosystems where vendoring is common, because will unvendor packages so you use whatever debian ships.

Which on one hand is understandable, if they ship a security fix in one of their packages they want every dependent on the system to get that fix, on the other hand there’s no guarantees whatsoever the package works once unvendored.

mananaysiempre• 3 years ago

there’s no guarantees whatsoever the package works once unvendored.

And it’s precisely the job of the maintainer to ensure that it does, and more generally that all the various things that they ship together not conflict with each other. Assembling a coherent system despite each individual developer’s understandably narrow view—in explicit opposition to that view, if need be,—is and always was a distro maintainer’s job description.

Unfortunately, the bug reporting story has become worse over time. GNU packages included a configuration-time override for the bug reporting email to accomodate distributions and other modifications, but in most software today you’ll more often see a hardcoded link to the upstream homepage. And of course people will often just ask Google for the bug tracker address instead. This leads to understandable annoyance on the part of upstream developers. (It still doesn’t make acting against their intent improper in any way, though.)

loloquwowndueo• 3 years ago

Do database drivers belong in a languages standard library? I think not, to be honest.

RedShift12• 3 years ago

php has PDO which is a nice abstraction over all the database drivers it has.

alexvitkov• 3 years ago

Not sure there's a corelation here, Node has a much more featureful library than say C, and there's still a dependency hell there and not in C.

bkallus• 3 years ago

C makes it too inconvenient to pull in 300 dependencies. I think the recipe for dependency hell is insufficient stdlib + decent included package manager.

michaelt2• 3 years ago

C developers basically invented vendoring - loads of C libraries are distributed as "the whole library is in a single .h file, just copy it into your repo"

zozbot2342• 3 years ago

TBH it's mostly caused by the lack of consistent semver use in the npm ecosystem, which in turn is driven by JavaScript being a highly dynamic language and not being designed for programming 'in the large'. The whole point of semantic versioning is to make it possible to auto-upgrade dependencies to the latest compatible version, at which point devendoring a dependency and packaging it separately starts to make a lot of sense. If every dependency upgrade requires a complex review of the code, there's no point in devendoring since it will just result in lots of bespoke package versions adding pointless clutter to the archive.

andrewaylett3• 3 years ago

One difficulty with NPM packaging is that it's traditionally quite hard to work out whether your dependency spec is correct or not. You can either keep bumping the version, or leave it be and hope no-one checks becuase it's hard to test.

Shameless self-promotion: https://www.npmjs.com/package/downgrade-build

This duplicates your project in a temporary directory and locks all the direct dependencies to their lowest semver-compatible version. It's not infallible, but it helps.

The only problem is that some projects rely on their direct dependencies bumping versions in order to force update transitive dependencies. Which is horrible, and also relatively easy to avoid needing to do by using a tool like Renovate.

lloydatkinson2• 3 years ago

Pretty cool! Does it work with PNPM? I’ve mostly fully switched to that because NPM has for many years stood out as one of the worst package managers across ecosystems.

andrewaylett3• 3 years ago

I'm afraid I've not tried it with PNPM, but it's unlikely to work without at least a small tweak as it executes an "npm run <script>" internally. You get to pick the script though.

philipwhiuk2• 3 years ago

But the package.json isn't the important bit. The lock file is.

andrewaylett3• 3 years ago

If you're shipping a library, it's the consumers' lock files that are important. You probably want your lockfile to be as up-to-date as it can reasonably be, for testing, but you also (at least in my opinion) should keep your dependency bounds as wide as is correct so you're not forcing dependency upgrades on your consumers.

What Renovate recommend (and I concur) is specifying exact versions of development dependencies in your package.json, and broad version specifiers for peer and production dependencies. Then you use Renovate to test and bump the lock file versions, keeping them up-to-date.

Downgrade build then helps you ensure that the lower bound on your peer and production dependencies isn't set too low.

rurban• 3 years ago

Not much happened since: https://wiki.debian.org/Javascript/Nodejs/Tasks/gsa (last modified 2020-12-17) still hundreds of npm packages missing.

gsa itself still at 0.0.0 (npm view gsa)

skywhopper• 3 years ago

This problem in various forms has been around for decades now. In 2007 it was clear to me that Debian style packaging was insufficient for handling anything beyond trivial deployments of any tools or services from the Python, Ruby, and Java ecosystems. Things have only gotten faster-churning and more complex since then. The old model of packages for shared libraries that are updated across the board just isn’t workable. Both from the packagers point of view, because of the busywork, but also from the users’ point of view, since the low package coverage means many things just haven’t been packaged anyway.

In my opinion it’s probably fine for Debian to stick to the old model still for the core OS features, but operators need to know when to use something different for their own needs.

tremon6• 3 years ago

The old model of packages for shared libraries that are updated across the board just isn’t workable

Why is it not workable, other than "developers no longer care about API stability"? Should we uncritically accept the state of the developing world as it is today, or should we strive for something better?

gettodachoppa• 3 years ago

"I'm talking about packaging xyz 1.3.1 and 2.0.1, as separate xyz-1 and xyz-2 packages, and allowing the use of both in build dependencies. Then, a package using xyz-1 can work with upstream to migrate to xyz-2, and when we have no more packages in the archive using xyz-1 we can drop it. "

What a naive soul, thinking anyone in the JS world cares even remotely about API stability. xyz-1 will become obsolete within 2.5 weeks and the dev will tell the users to deal with it. You can't bring accountability into that culture.

loughnane2• 3 years ago

I’ve only been using Debian b as a daily driver and a home server for the past 4 months, but I love it.

I had been on Ubuntu for years but ultimately turned away because of them pushing snap packages. Also I had been curious about Debian and wanted to give it a shot to see if it was too bare bones.

Most of the work I do is scientific python with a dash of web dev. The standard packages “just work” for most of my needs, and when I need to venture out for something like Signal or Slack, flatpak has me taken care of.

For projects I either use venv or docker, which feels cleaner to me anyway.

kwk1• 3 years ago

Debian b

Just FYI, although the Ubuntu version naming scheme lends itself to being abbreviated to a single letter, this isn't true for Debian: the last three releases were Bookworm, Bullseye, and Buster.

checkyoursudo• 3 years ago

And yet, saying you use Debian b could plausibly mean any of those anyway. Several of my servers are stuck on Debian b, and I am not 100% sure which b it is.

giancarlostoro• 3 years ago

The article is from 2021 btw, the title should be updated.

sumanthvepa• 3 years ago

This is a huge problem for Linux distributions. I've been thinking of trying to open source a project (it's a web application simulator) that we have internally, that is built using Javascript/NodeJS and Python/Flask. While the mechanics of slapping a GPLv3 license on the code we wanted to publish were fairly straightforward, severe problems arose when we wanted to package it for distribution with an OS like Debian. The application is a web app that uses both Node and Python, and the complexities packaging it were overwhelming. For now we planning on releasing it as a container image. I'm not fan of doing this, but it seemed like a solution we conceivably implement with the time and resources we had available to us.

zbentley• 3 years ago

If you ever revisit that decision, check out FPM. It can shave off a few of the rough edges related to packaging: https://github.com/jordansissel/fpm

johnny22• 3 years ago

maybe im reading it wrong, but it sounds like they wanted it to be able to be included in the distribution and not just make a package for it. If that's the case, fpm wouln't help. They'd still have to split out all their deps and get them packaged and aso hope the deps don't collide with existing deps version wise.

tupolef• 3 years ago

Please just provide a list of dependencies with min/max versions, one for Build and one for Usage. Then provide a detailed example of the config files, a systemd unit file, and maybe some config examples for the 3 main webservers.

People will then deal with it. The target for such a project is not a noob user and it's more ressource friendly and it help people to understand how to debug the app and how it works.

I'm testing around 10 new tools every week and I hate when the project page only gives me a docker/flatpack/snap solution, I always find a way to install it raw without any indications, and it's painful.

vbezhenar• 3 years ago

I'd prefer vendoring approach to all packages. I don't care about disk space or traffic, but I care about pinning precise dependency versions. I'd even say, I'd prefer to virtualize every application in a docker manner. That's what I mostly do with server applications using docker or k8s and it works for me. There's no reason it wouldn't work with desktop applications.

worksonmine4• 3 years ago

I agree for projects I maintain, where I can bump a package and the dependencies myself.

But on a server with 10 services, depending on 10 versions of a badly maintained project that requires an urgent security patch? Not so much, I rather just update a shared lib. It's a matter of scale and resources.

The great thing is that there are options, Flatpaks do what you want, no need to demand that Debian does the same.

tremon6• 3 years ago

If you don't care about disk space or traffic, does that mean that other people who do care about it are automatically wrong? What about people who care about software security and want to know which vulnerable versions of which software exist on their servers or desktops?

here's no reason it wouldn't work with desktop applications

Have you tried it with desktop applications?

moomin• 3 years ago

I had a think about how you could best deal with this and then realised I was probably just designing a worse Nix.

I think, unfortunately, the Debian model just can’t scale to the sheer volume of what it’s being asked to do. And, as pointed out by one of the original posters, it’s not clear what benefit creating a common standardised version of lodash buys anyone.

yxhuvud• 3 years ago

Distro specific packagers are broken because the surface area and workload is too high as it is essentially n packages * m distros to maintain. Language specific packagers are broken because they can't express dependencies across the language barrier in a good and automated way.

The latter seems easier to solve as long as there is someone that actually wants to solve it. Currently the language packager maintainers doesn't seem to be interested in a common solution (understandably, as it is a very complex problem).

smilliken• 3 years ago

Language specific packagers are broken because they can't express dependencies across the language barrier in a good and automated way. The latter seems easier to solve as long as there is someone that actually wants to solve it.

This is solved by Nix. Packages of any type can depend on eachother.

pmontra3• 3 years ago

They should just drop gsa. It's a vulnerability scanner. People interested in that and in its web fronted are supposed to have the skill to follow the instructions on GitHub and install those two pieces of software. One has a dockerfile, the other one must be built from source. It can probably run in a socket container with the correct dependencies. Problem solved.

I use Debian on my laptop but in no way I'm using their vendored languages or packages except what's needed to run the OS. Everything else moves too fast. It's either language managers or docker containers.

What I want from a distro is a solid foundation to run my desktop, editor and browser, then I care about everything else.

hedora3• 3 years ago

The article suggests moving it to contrib, and having it download its stuff post-install.

That seems completely reasonable. If it uses common npm libraries that don’t break api’s every few months, then those could be moved to stable.

I’d rather libraries packaged by debian releases continue to work for more than 30 days than to have them package unsupportable dev packages that bitrots my dev environment every “apt get update”.

It’s kind of amusing to me that there are so many comments saying this is unsustainable, given the fact that multiple BSDs maintain a quality bar similar to Debian’s.

NegativeK• 3 years ago

There are a number of benefits that come from letting the distro maintain a piece of software, such as unattended, or even just a unified UI for updates.

pmontra3• 3 years ago

I know, but they are usually months or years behind. If a customer tells me that they are working with Language x.y.z I need to install that version, no matter what my distro gives me. And if a project is N versions ahead of the version in my distro, I usually install from either their own PPA or docker image.

If I had one euro for every time I read something along the lines of "the version of our software available in Debian is terribly out of date, do not install it, it's full of bugs that we fixed on the last years plus new features. Follow our installation procedure instead."

phkahler2• 3 years ago

> If we continue to push back on bundling, we need to improve our tools and processes and policies to make it feasible to maintain unbundled packages. Otherwise, we need to build tools and processes and policies around bundled dependencies.

Or neither. The dependency hell created by that approach could be considered a bug with the software created that way - even a blocker. Like hey, your project isn't going to get included until it's not a giant spaghetti pile of dependencies.

jancsika• 3 years ago

But they've already caved for Firefox and Chromium.

rcxdude2• 3 years ago

Given users are generally much more attached to applications than distributions, that's unlikely to work out well for the distros.

phkahler2• 3 years ago

It was just another possible way to look at it. I prefer distro-provided packages myself, and won't jump through many hoops beyond that simplicity.

raverbashing• 3 years ago

Keeping local versions of every dependency and trying to get around the mess is a 90's/2000's model of dependencies

This does not scale

Linux distros are too big. They need to determine what to ship and be part of the system and what to have the bigger vendors bundle

ChrisArchitect• 3 years ago

(2021)

Previous discussion: https://news.ycombinator.com/item?id=25758863

josephcsible• 3 years ago

One of my favorite things about Debian's approach is that it provides immunity to the left-pad problem.

Debian discusses vendoring again (2021)

Related Stories