Traditional packaging is not suitable for modern applications?

https://theevilskeleton.gitlab.io/2022/08/29/traditional-packaging-is-not-suitable-for-modern-applications.html

by pabs3 • 4 years ago

59 128 4 years ago

theevilskeleton.gitlab.io

auggierose3• 4 years ago

The text suggests to just containerise any app, so I am wondering, what is the current status for containers (on Linux):

Can you access files outside of a container from inside a container?

Can you access devices like an eGPU from inside a container?

korijn• 4 years ago

This is addressed in the article. Search for permissions and e.g. XDG portals.

auggierose3• 4 years ago

I have seen the link to XDG portals, but it doesn't say what is possible with it.

IceWreck2• 4 years ago

Yes to both.

auggierose3• 4 years ago

Is there a good book about all of that?

forrestthewoods5• 4 years ago

The solution is simple: ship your fucking dependencies.

Every application should ship every dependency it requires. Attempting to use random shit that may or may not be installed on the system is a disaster.

Containers that include all dependencies achieves this. Infact I like to argue that the only reason containers exist is because reliably launching software is so outrageously complex the only way to do is create a full system imagine. It’s a modern day tragedy.

Disk space is cheap. If every application shipped duplicate dependencies it’d be… totally fine. The downside is inability to update security vulnerabilities. But if everyone is shipping containers that seems kinda moot.

iOS and Android don’t have this issue. When you install an app it just works. Windows has the issue somewhat, but it’s mostly a problem with dev tools that are trying to pretend their in Linux. If you ship your fucking dependencies it’s a non-issue.

mab122• 4 years ago

What is the difference between shipping dependencies and shipping container? Thin layer of metadata? Or am I missing something?

mjevans• 4 years ago

Every security update means updating every big bundle of packages when it's a small change in some commonly used core library.

Even IF my bandwidth and time were free (neither are) that is still a ton of duplicated work and wear on non-volatile memory life.

forrestthewoods5• 4 years ago

Every security update means updating every big bundle of packages when it's a small change in some commonly used core library.

Replacing all instances of libFFMoeg.so on a drive is not particularly harder than replacing one.

that is still a ton of duplicated work and wear on non-volatile memory life.

Nonsense. We’re talking about replacing libraries on the order tens to hundreds of megabytes during a monthly update. It’s peanuts.

Time I’ll give you. But the current system of system-wide libraries is objectively broken. As evidenced by the increasing usage of containers to work around the brokenness. I’d wager far more time is wasted on docker orchestration.

It’s probably worth differentiating between servers and personal machines. Lord knows how many vulnerabilities are in your Steam library. It’s a lot.

compsciphd5• 4 years ago

It's only due to the way docker containers and the like are constructed. Predating docker by a few years I built a docker like mechanism that viewed what docker calls layers (i also used the term) as packages in a traditional distribution.

Instead of unpackaging a package into a file systme, the package is just a layer that can be unioned in on demand (for live updates, though in the age of containizerization, this hasn't aged as well, so in practice it be running a new container).

link to paper is somewhere in the threads here

FooBarWidget• 4 years ago

The solution is simple: ship your fucking dependencies.

And then you get Electron, and people start complaining about the fact that you ship all your dependencies instead of using the dependencies that are already there.

forrestthewoods5• 4 years ago

Who is complaining about the install size of Electron? The problem with Electron is it’s laggy and slow. Using a different source for dependencies wouldn’t solve performance. It’d just slightly reduce storage footprint.

I mean there’s always someone who complains about something. But if Electron apps had a large folder of binaries but it was super fast and responsive that would eliminate the vast majority of complaints.

bhedgeoser• 4 years ago

Flutter is as good if not better than electron and requires a minimal set of dependencies (gtk, libc). Everything else can be statically linked using plugins.

alkonaut4• 4 years ago

Yes. For a desktop system where you want to update complex multi-gig apps perhaps once per week this isn't just the better option, this is the ONLY option.

For a smartphone, the same.

For a server or security sensitive system? Perhaps a different scenario. You wouldn't update a dozen large complex apps every week on such a system.

I think the issue is that the distros with their application trees are trying to solve a handful of completely different problems. Me updating a 5GB IDE and a Game once a week is not the same problem to solve as running a web server.

pasc1878• 4 years ago

And this is what NeXTStep/macOS has done for 30 years. Well you could also ship common Frameworks but they all have a version number so should not cause a problem if different apps need different versions of a library.

The AppStore or using a library like Sparkle will update when the developer finds an issue or security issue with a dependency.

The advantage here is that the OS knows about bundles and so gives an organisation above the file. Anything in that bundle should not be used by anything else. I suspect this is what using containers gives Linux

Alternative do as work did 30 years ago - explicitly link to a versioned number of a library. OK then you need to identify which apps use a particular library that needs to be updated for security reasons.

Karellen2• 4 years ago

Modern applications are inherently fragile.

Well, there's your problem.

zokier• 4 years ago

I feel the article is missing the point of distros. They are supposed to be fully self-contained integrated operating systems, not just container platforms.

Additionally, essentially vendoring full dependency trees for each application separately would massively increase the maintenance burden because instead of just maintaining one version of a library the distro will need to maintain n versions

Modern applications are inherently fragile.

There is nothing inherent in the fragility, it is self-inflicted problem of some applications.

compsciphd5• 4 years ago

I'm not sure distro needs to "maintain" N versions. The distro maintain 1, but all N remain available and able to fulfill the dependencies of the underlying application

It's not the distro's job to ensure the application is secure and up to date, its up to the application packager (i.e. docker container maintainer) to do that. i.e. to rev new releases of the application image even if nothing has changed in the core app, just to take care of security updates in distro provided packages). In pr

cuu5083• 4 years ago

It's not the distro's job to ensure the application is secure and up to date

I would prefer a distro that ensures applications in its repository are up to date.

Consider a log4j-type issue. In one case, the distro quickly fixes the vulnerability in a library package, the user installs the update and that is that.

In the other case, the user has to track down which installed applications use the affected library, update them if there is an update available, and figure out what to do if there is no update available.

nine_k4• 4 years ago

A distro often has to maintain more than one version of a,library: consider Debian stable, oldstable, and backports.

goodpoint5• 4 years ago

It's not the distro's job to ensure the application is secure and up to date

On the contrary, it's the most important feature of distributions.

forrestthewoods5• 4 years ago

self-inflicted problem of some applications

It sure seems distro inflicted to me. The architecture induces fragility. Apps needing to very carefully step to avoid breaking things is the definition of fragile.

throw_a_grenade2• 4 years ago

This is written from the point of view that implicitly assumes only a single application installed (= a container), where distro is reduced to API provider and some common utils the app author doesn't care about. It would be correct only if I was building an appliance. But as the user I have a general purpose computer which gets the work done over multiple apps.

I think there lies deeper, cultural division between developers, who think the world (at least the computer) revolves around their app and their downstream, who desperately try to avoid more and more code shipped unto them for the upstream's convenience. It's not really a difference for upstream if they bundle a dependency or not (or 10k+, in the case of npm), but it quickly becomes a big support burden if you have multiple such "apps" installed.

microtonal2• 4 years ago

It's not really a difference for upstream if they bundle a dependency or not (or 10k+, in the case of npm), but it quickly becomes a big support burden if you have multiple such "apps" installed.

But only because distributions want to take control of everything. If a distribution just provided the necessary infrastructure to empower developers to maintain their own dependencies (like eg. Flatpak does), we wouldn’t have this issue.

mmis1000• 4 years ago

I think you are going to have 20 gcc available at same time if you do want this model. Windows literally showed you what would happen if everyone just require their own dependency version.

It is choice between a system with bleeding edge / ancient(or sometimes insecure) program mixed together or a system with well audited recent enough programs.

Windows/Snaps go for the first, and apt/yum..etc chose the later. It's just all about it.

kkfx4• 4 years ago

There is nothing inherent in the fragility, it is self-inflicted problem of some applications.

The architecture is fragile due to too many layers. Anything is "fragile" in the sense that you can't really design a really complex, generic system with all possible state pre-computed and handled up front, the point is how to handle the fragility.

Classic systems was "the OS is a single application where anything is ready available source code", this was utterly fragile in potential faults BUT far easier to correct any error so in the end far more robust. Think about some CL systems: you'll easily get far more issues in normal usage, but essentially any issue is easy to handle.

The real underneath issue is business: software can't be a commercial product, must be a common open knowledge, like a kitchen recipe, where any party involved can see and act and anything is designed ONLY for the human. We need desktops in a network not dummy terminals of a central service designed to serve their owner.

nine_k4• 4 years ago

software can't be a commercial product, must be a common open knowledge, like a kitchen recipe

Kitchen recipes rarely need features added or even bugs fixed. But when this happens, it's usually done by a cook on the spot.

This is only possible due to utter simplicity of a typical recipe, compared to any substantial piece of software. Also, small variations in cooking usually result in pretty edible dishes, while small variations in a program can have drastic results.

So no, sadly, software already cannot be like a cookbook.

alkonaut4• 4 years ago

massively increase the maintenance burden

The maintenance burden for who? Using isolated apps (Assuming that no-shared-libraries is what we want, which is probably true for desktop systems like and less true for security critical systems or "traditional" linux systems) just places the maintenance burden on the apps where it belongs. Yes, unmaintained apps should run out-of-date libraries.

JohnFen4• 4 years ago

Assuming that no-shared-libraries is what we want

That's certainly not what I want on any system, desktop or otherwise.

compsciphd5• 4 years ago

I literally wrote part of my phd dissertation on this (can call it proto docker as it predated it and they seemed to use much of the same nomenclature I used, also was sort of a toss in an the very in, though arguably my most important contribution).

see - https://www.usenix.org/legacy/event/lisa11/tech/full_papers/...

teddyh2• 4 years ago

People are generally opposed to forking, right? If you fork a project, you effectively take over all the responsibilities to fix bugs and security issues etc. in that project, which is, most of the time, more work that the fork is worth. Now, a popular recent trend is “vendoring” your dependencies; i.e. bundling the dependencies with your software. However, vendoring does not differ much from forking. The responsibility may be slightly less, since you only have to fix bugs and issues affecting your application’s usage of the software. But, just like a fork, the longer you go without merging from the upstream version, the more work it is to maintain your fork.

I suggest that vendoring, just like forking, should be avoided, for largely the same reasons.

doubleunplussed2• 4 years ago

vscode example notwithstanding, I do not encounter the 'conflicting packages' problem very often on Arch. In fact, I don't remember the last time I encountered it, other than when it's an explicit choice - like, Python can only have one version of a package installed for the system Python interpreter, so if you try to install both python-numpy and python-numpy-mkl-bin, well, you have to choose. That's how Python works, though you can of course make additional Python environments (venv/conda/whatever) with different packages.

I notice the vscode example involves the AUR, which is not the official repositories. I consider that a broken package, and it should be fixed. One of the packages should rename its main executable or whatever.

Chrome, for example, has various versions, but they ship binaries with different names, e.g. /usr/bin/google-chrome-stable.

The great thing about Arch is that there is one canonical user repository, the AUR. So the official repos should not have conflicts unless absolutely necessary, and the AUR should endeavour not to have conflicts, though it's more of a wild-west so they're somewhat more likely.

However, you generally don't have to think about how to not conflict with whatever random packages may be out there - you only need to consider the repos and the AUR. Anyone making their own packages that are not in the official repos or the AUR, and yet conflict with them, well, that's their problem.

Arch being super up to date and the existence of the AUR really does obviate the need for containerising apps to a large degree, at least from the perspective of ensuring dependencies are satisfied (there are still security reasons to want containerisation). I don't believe I have any containerised apps, and yet I'm running loads of modern common programs like zoom, chrome, discord, spotify, the list goes on.

tempay• 4 years ago

if you try to install both python-numpy and python-numpy-mkl-bin, well, you have to choose. That's how Python works

It's possible to have as many Python versions as you like installed at the same time. All of the libraries and headers include the major/minor version, packages are installed in `lib/pythonX.Y` and the Python binary itself is available as `pythonX.Y`. Few distributions support this well for anything other than a single 2.x and 3.x but it's nothing to do with how Python itself works.

doubleunplussed2• 4 years ago

That's for different Python interpreters altogether, you still can't have multiple numpy packages in any one of them.

Arch actually supports multiple Pythons just fine, I currently have a bunch of 3.x Python interpreters I installed from the AUR. But installing other Python packages with these interpreters indeed needs to be done outside the system's package manager, e.g. with pip -U or a venv (usually a venv for me).

ddevault2• 4 years ago

Nonsense. There are essentially two conflicting ideals here: what developers want and what distros want. The problem is that distros are generally right and what distros want is also what users want. So it's devs against the world.

Devs want to ship updates whenever they want and force them into their users' machines as soon as possible, and forget about updating dependencies and pin them to ten versions ago so they never have to eat the cost of upgrading them. Devs don't want to make sure their application plays well with others, they just want to package it up in its own universe, the user's hard drive space and bandwidth be damned. Devs never want to finish their software or declare it stable, they just want to ship their machine on Monday morning so users have it installed by Monday afternoon, just to be updated again on Wednesday. Devs want to ship spyware and adware and malware without any oversight. All of this is cheap and profitable for devs but highly antisocial.

Everything is better about the distro model: security, efficiency, and stability, and they represent the users interests even when they conflict with the dev's interests.

"Modern applications" are developed by children and distros are the teacher slapping them on the wrist for misbehaving. Don't bitch about it -- just grow up.

alkonaut4• 4 years ago

As a user I just want the app to be exactly like you describe: a) isolated from all other apps b) self-updating and have a release cycle decoupled from all other apps on my system.

In this scenario (for my personal desktop OS) I don't care about bandwidth, disk space or whether an app uses an out of date version of a library. Having a release per week isn't uncommon and isn't somehow a sign that an app has poor quality.

I'm not going to say a generalization is dead because of one counterexample, but It's perhaps not as clear cut as you make it out to be.

ddevault2• 4 years ago

As a user you are asking for a measurably worse experience. And given context, I suspect you are a developer, not just a user.

A release per week is a sign that an app has poor quality.

akerl_2• 4 years ago

For what it’s worth, I’m in the same boat as you.

I have different thoughts in different use cases (for example, I have servers farms and other specific systems where I more strictly batch updates and value a slower cadence), but for my desktop, my mobile devices, and essentially all of my personal systems, I want constant rolling forward progress.

goodpoint5• 4 years ago

As a user I just want the app to be exactly like you describe

...and with zero security. No 3rd party reviewing it for vulnerable dependencies, homecalling and telemetries or worse, or breach of licenses.

No guarantee of receiving security updates. Especially given that most upstream do not maintain stable release train and do not backport security fixes.

And keep in mind that flatpak sandboxing is only partially effective and it also does not protect the data you put in the sandboxed app in any way.

(Not to mention that more than 4 billion people on this planet have very slow, expensive and unreliable Internet access, and old devices with limited storage space.)

No thanks, I'll stick with distributions.

NoGravitas• 4 years ago

Thanks for the voice of sanity, Drew.

forgotpwd162• 4 years ago

Nix solves the mentioned issues without using containers. Not sure why they keep referring to containers as only solution when Nix is included. Also not sure why they talk specifically about graphical applications. The issues mentioned apply to every program running on the system. That said, agree on those issues.

nine_k4• 4 years ago

I bet it has something to do with Dockerfile having far simpler syntax than Nix config files.

qwery3• 4 years ago

However, Electron based applications vary, as they are built on top of different versions of Electron.

I thought one of the selling points of Electron et al. was building apps on web standards. What is changing so fundamentally between versions of Electron that prevents apps from running on the latest release? I wouldn't expect Element to not work on Firefox 432[0] after using it on Firefox 431...

[0] I lost track of Firefox version numbers somewhere between 3 and 50

kkfx4• 4 years ago

I think many, too many still fail to see two thing:

- packages/package management is intimately tied to filesystems and so installers. Actual package models are based on '80s Unix model that do not scale anymore since decades. Such model was designed decoupling "system code" to "user code" (script, mostly) while all along the IT history, both before and after unix we see a better model (see below);

- packages in FLOSS are NOT meant do be made by upstream developers, that's a need of commercial only software. In FLOSS systems packagers of any distro package upstream code.

Now a step at a time: classic systems where designed as a single application, indefinitively extensible by many, and users included. At a certain point in time AT&T/IBM and others say "ah, no that's bad" IMVHO just to deprive users of much power and gain the ability to sell limited, limiting and poorly designed crap. For more than a decade such decoupling take shape. After decades of big mess frameworks spring to life as a kind of middle-ground and they prove to be no better than the decoupled crap they try to solve without coming back to the classic OS-as-a-single-application model. Now essentially ALL non trivial applications being unable to effectively integrate each others try to be "the only one application" and finding no way to do so some pull out of the hat an even worse cage: hiding the OS on someone else computer, used as a bootloader of a modern more limited and limiting Java VM: the WebVM improperly named browser for legacy reasons. Pushing "the network is the computer" to "the cloud is the system".

To solve this mess what we need is the original solution: the OS as a single application where source code is in the runtime, no "packaging" just SCM. Oh, surely that does not solve version conflict between source, these actually are ALREADY solved in architectural terms by the FLOSS distribution model: packagers assemble code. But working directly on source makes that far easier with simple sharable patches who happen to be also a bit distro-neutral. Nix/Guix offer the actual middle ground.

To solve the storage mess it's far more complex. Today most solutions simply use a backend storage service, a kind of DB, no matter if SQL-alike or key-value or something else. SUN with zfs realize a very first limited evolution in that sense, but far from current needs. BeOS with if raw "query-able file system" have done another very small part before. https://twizzler.io seems to offer something more modern, but that's still an open issue.

edent2• 4 years ago

There is, I think, a bit of a paradigm shift in action.

In the old world, disk space was limited and download speeds were slow. So it made sense for every app to use the system-provided version of Library X.

Additionally, stability and backwards compatibility was assumed. So if the distro updated to Library X v1.2, and your app relied on v1.1, everything would still work.

Today's world is very different (for many users). Disk space is cheap and plentiful and downloading a couple of GB takes an instant. So why not bundle everything together?

Additionally, developers can't be sure of stability in their dependencies. And the app developer is the person who receives the complaints from angry users, not the library developer.

So it makes perfect sense to use a self-contained set of libraries and dependencies.

The pendulum will swing the other way in time. We'll all get annoyed at having a dozen different Electron libraries taking up space, and we'll go back to one-library-to-rule-them all.

But - as someone who used to hate battling dependency hell - I love Flatpack.

okasaki• 4 years ago

Disk space is cheap and plentiful and downloading a couple of GB takes an instant.

True in a data center. Not true for many people.

So why not bundle everything together?

Security updates, RAM usage.

edent2• 4 years ago

True in a data center. Not true for many people.

Even the cheapest laptop you can buy today has literally 1,000x the disk space of a 1990's tower PC. The packaging format APT was developed in the late 1990s when the average user's modem speed was unlikely to hit 56kbps.

Times have changed. Time to update the way we package software.

Security updates, RAM usage.

True in a data centre. Not true for many people.

ginko• 4 years ago

Additionally, developers can't be sure of stability in their dependencies. And the app developer is the person who receives the complaints from angry users, not the library developer.

That seems like the main issue to me. Why has it become acceptable for libraries to break backwards compatibility left and right?

goodpoint5• 4 years ago

In the old world, disk space was limited and download speeds were slow

No, the problem is that production systems need to receive fine-grained and timely security updates.

You can't rebuild, test and deploy a whole OS every time a vulnerability in OpenSSL is fixed. It would be extremely risky for any non-trivial production workload.

Disk space is not the issue. Distributions do the crucial and painstaking work of backporting security fixes.

This cannot be done when applications are shipped with huge blobs of dependencies, like with flatpack or containers, it's simply too much work for any organization to do that.

Karellen2• 4 years ago

Additionally, stability and backwards compatibility was assumed. So if the distro updated to Library X v1.2, and your app relied on v1.1, everything would still work.

I thought all the cool kids had adopted `semver` these days, so this shouldn't be a problem? Except for crappy libraries/dependencies, in which case surely the answer is: Don't use crappy dependencies; find an alternative that does the same thing, without sucking.

Additionally, developers can't be sure of stability in their dependencies.

Again, be more judicious in which dependencies you use. Stay away from the crap ones, find ones that do the right thing.

Disk space is cheap and plentiful and downloading a couple of GB takes an instant.

Maybe for you. But not for everyone on the planet. Or, maybe people who live in places with intermittent internet connectivity don't deserve new software, or for their needs to be considered?

So why not bundle everything together?

Security. If a serious vulnerability is found in Library X v1.1, I want to be able to upgrade it to Library X v1.1.1 and know that every single app that uses it is safe, without even needing to know which apps use it. I don't want to have to figure out which apps use it (which, OK, might not be too hard if the tools are written to find that out), and then have to download an update for every single one. Even if I do have unlimited gigabit broadband, that could still be awkward if the library is sufficiently popular and widely used.

choeger• 4 years ago

The problem is that containers are opaque and authors simply don't care about quality. If say, fedora would allow containers by having deviating Versions of rpms per app, it would be quite cool. The OS could inform me about apps using deprecated Versions, dependencies could be shared, etc. But people don't want to put up much effort when shipping their app. They want to build on some version of Ubuntu, add some manual hacks and expect everyone to use that container for the next decade or so. No one cares about dependencies, security or other nonfunctional aspects, sadly.

kasabali3• 4 years ago

Yet another article arguing against distribution package managers without understanding them a bit.

Traditional packaging is perfectly suitable for modern applications.

What author is arguing against is the distributions' packaging policies. Distribution policies are what prevents shipping multiple versions of the same library or program, not package managers. These policies are in place because they're necessary for releasing and maintaining a 50000 package distribution.

But for shipping your own software you're free to ignore distribution policies. Package managers have always provided the necessary mechanism. Nothing is preventing you from shoving in all library, binary or data dependencies your program needs into a deb package. You're also free to embed version number in package name and installation paths so different versions could be installed in parallel. The package manager will obey dutifully. Build systems have always been capable of installing libraries into arbitrary paths, and deb/rpm packages are just dumb archive files with metadata slapped on top. If you shove everything needed into it and declare no dependencies, it'll be no different than a dumb tar file and it'll work just fine if you did it right.

You can even imitate flatpak runtimes if you want. Put same things a flatpak runtime has into /usr/lib/freedesktop-sdk-21.08 into a package like freedesktop-sdk-2108.deb, and make your application deb depend on freedesktop-sdk-2108 and you're good to go.

I'm really tired hearing of these baseless arguments over and over over the years. I just wish everyone could be honest and just say "We don't like deb/rpm, they're not cool, they're not hip, we just want to NIH them" so we could be spared of these made up justifications.

P.S. I have nothing against nix. Nix is legit innovation.

hnaccountme• 4 years ago

Better title would be "Traditional Packaging Is Not Suitable For Low Skilled Developers"

eqvinox• 4 years ago

Traditional packaging introduces several issues, such as dependency and package conflicts, which usually require workarounds that differ from one distribution to another.

I feel that there was no thought here at all regarding the reverse perspective:

Current application development and deployment practices introduce several issues, such as dependency and package conflicts, which usually require workarounds that differ from one application to another.

friendzis4• 4 years ago

The fundamental problem is not that distribution maintainers do not leverage containers, but rather we do not understand dependency models/trees of our programs.

Containers are golden images. You find combination of stuff that works, carefully package them together and ship this whole package. Gold images run on bare metal, containers run on container runtime enabled kernel.

Sure, containerizing an application is /easier/ than working out the dependency trees and solving dependency conflicts between them. It is by no means fundamentally better approach.

If anything, this argument dismisses most work done by distros - building and solving those dependency trees. Maintainers of free OSes sometimes do not bother to solve all possible conflicts, however instead of breaking the system/packages, they do inform you of possible conflicts that are not solved.

Inability to cherry-pick dependencies is deceptively wrong. Traditional distributions resort to use of so called meta-packages and alternatives to keep the system up to date. Deb/rpm packages are simple archives with additional metadata slapped on. The packaging system is transparent to contents: nothing forbids you to package versions 1.0 and 2.0 and have them happily coexist as long as they do not share common paths. In fact, a lot of packages are distributed in exactly the same way.

It seems that the author has a solution - containers - and tries to find a problem for said solution.

jasode• 4 years ago

Sure, containerizing an application is /easier/ than working out the dependency trees and solving dependency conflicts between them. It is by no means fundamentally better approach. [...] The [distro] packaging system is transparent to contents: nothing forbids you to package versions 1.0 and 2.0 and have them happily coexist as long as they do not share common paths.

I think your comment is analyzing the problem from a different abstraction level than the author's examples.

In other words, your alternative approach of "solving dependency conflicts" is abstract advice unless you try to make it concrete by focusing who solves those dependency conflicts.

Using the author's example of a VSCode conflict (collision of "/usr/bin/code") with Actor in italics:

- Microsoft owns commercial the build of /packages/visual-studio-code-bin

- Filipe Laíns & Massimiliano Torromeo owns the open-source-license build of /packages/community/x86_64/code/

- Levente Polyak, et al owns the Arch Linux distribution

- end user owns the computer that needs to run conflicting software

Generalize the above matrix to every software package and consider which of those 4 actors can practically solve the conflicts in a scalable way?

Filipe Laíns & Massimiliano Torromeo can't control what Microsoft does and vice-versa. (Ironically, the freedom to not be controlled by the other actor is one reason there's a package conflict!)

And Levente Polyak could "solve" it by blessing Filipe Laíns's package and rejecting Microsoft's but this creates a new "problem" for users who specifically want Microsoft's commercial build.

The fundamental issue is that you have independent actors that can't control what others do. Result is that sometimes software packages clash (namespace collisions, dependent libraries not in sync, incompatible licenses, etc).

The approach of using containers -- however flawed it is -- gets past the stalemate of independent actors in the software landscape being incompatible with each other.

humanrebar• 4 years ago

This is why we need packaging standards, especially around metadata, instead of further decamping into language-specific ecosystems. Because even with containerized apps, you still want a few launch scripts and a compiled program (for instance) that might contain multiple dependencies from different languages (C, Go, and Fortran, maybe).

kasabali3• 4 years ago

The approach of using containers -- however flawed it is -- gets past the stalemate of independent actors in the software landscape being incompatible with each other

What does that is bundling your dependencies, not containers. You could always bundle everything an app needs into a single tarball or a package. Containers didn't make it possible, they made it easy and popular.

friendzis4• 4 years ago

The fundamental issue is that you have independent actors that can't control what others do. Result is that sometimes software packages clash (namespace collisions, dependent libraries not in sync, incompatible licenses, etc).

Yes, I agree. Where we are in disagreement is what the role of a distro is here. Distros are there to solve that stalemate in one way or another. In Code's case distro maintainers chose to mark these packages as conflicting. They could have changed namespace of one (or both) package[s].

As I have mentioned in reply to a sibling comment this is exactly the work distros are doing - solving the stalemate between upstreams.

The approach of using containers -- however flawed it is -- gets past the stalemate of independent actors in the software landscape being incompatible with each other.

Sure, containers do try to get past the stalemate, but they try to solve a different problem - how to make conflicting dependency trees coexist. Containers can also fail to solve the stalemate. Sure, containers can isolate `/usr/bin/code` between containers, but contention on `/etc/code`, `$HOME/.code` or other static user data still has to be solved one way or another.

redkoala• 4 years ago

There are design choices that can be made to guarantee namespace isolation - having different software packages installed within the same /usr/bin/ directory is a guarantee for namespace collisions.

pid-1• 4 years ago

If anything, this argument dismisses most work done by distros
It seems that the author has a solution - containers - and tries to find a problem for said solution.

I don't really follow this. Distro maintainers are working in a problem and some people are happy with the results.

But the fact Nix, Guix flatpack and others exist shows other folks are not and are coming up with their solutions.

It seems you're painting the author as some container weirdo just for not liking your favorite distribution method.

friendzis4• 4 years ago

Distro maintainers are working on a problem (conflict resolution in dependency trees) and solving it. Use of containers is sidestepping of the problem.

Containers solve entirely different problem - how do you make conflicting dependency trees coexist in isolation. Layered containers solve this particular problem quite well. But it's a different problem than distros try to solve.

camgunz4• 4 years ago

I generally think these arguments (containers, flatpak) boil down to: I don't want to figure out how a distro works. I'm unsympathetic to that; it's actually pretty easy to figure that out: they almost all use systemd and making a deb/rpm is also straightforward (there are great guides). Distros are also great at updates, to the degree devs can largely outsource a lot of it to them (compared with Windows where you're writing update functionality into your app, or whatever).

I'm not even sympathetic to weird cases like OBS needing a forked version of ffmpeg. 1 this sounds like a wacky thing to do, 2 it sounds like they created a new project that is a different dependency. Call it something else (I suggest obsmpeg), package it for distros, pull in updates from ffmpeg whenever you want, reap the benefits of FOSS. Inventing an entirely new software delivery system instead seems like an overreaction.

I am sympathetic to working this out across multiple distros though. Finding the intersection of packages across multiple versions of Ubuntu, Debian and Fedora isn't super easy, and it can really hamstring you when building an app (you want to use libcool-1.8.x, Ubuntu LTS has libcool-1.2.25, :sad:). The ecosystem isn't set up for this and it's a pain to deal with in your app (you more or less #ifdef functionality out, which product managers will freak out about).

iudqnolq3• 4 years ago

As a regular Linux user who doesn't publish apps but has been reading this argument for years I've never understood the value distros provide.

They're clearly doing a huge amount of work editing everything, but it also doesn't seem like they have the resources to properly test their edits. And while their edits are necessary to make everything work with their system, from a high level perspective they don't seem to be making the software better at what I want it to do. Given that, their model looks like dangerous busywork that at best delays updates and at worst introduces new bugs.

Distros use the word "maintain" to describe what they're doing, but they don't seem to be doing that. To maintain a piece of software I'd expect the maintainer to need a deep understanding of the source code and to continually monitor changes. The complaints maintainers post that software changed seem to prove they aren't doing that.

When I first saw a maintainer complain a library only posted about a breaking build change in their release notes and mailing list I was shocked. I had been assuming they were reading diffs, when they weren't even reading the changelog.

openmapsguy2• 4 years ago

Generally, distros supply security updates. For example, Debian picks a version for a release and makes sure it has security updates for the entire release. Sometimes this requires backporting patches because the version is no longer supported upstream. This means an app can be shipped that will work for the life of the release without changes, while staying secure.

camgunz4• 4 years ago

Yeah, quality varies. But the vast majority of this is volunteer work. If I were developer of some app and I wanted it to work on Fedora, I'd probably be the maintainer of its package. But if I didn't want to do that, I'd at least be in contact with the person who was. "Hey heads up, we broke compat. Let me know if we need to huddle or something." Also seems a lot easier than inventing Flatpak and getting distros and users to use/like it.

So much of this stuff is like a default unwillingness to work with humans over the internet, but that's fundamentally how FOSS works. A couple years ago there was a kerfluffle between the scientific community and SciPy--the community wanted SciPy to maintain compatibility literally forever and also continue adding new features. SciPy was like, "that's impossible, and even if we wanted to it would take the resources of Microsoft to do". But they never talked about it! Instead there were open letters and forum threads.

The quickest route to fixing social problems like this--and they are social--is to deal with people. I'm pretty misanthropic, but even I have to admit this is true.

Distros use the word "maintain" to describe what they're doing, but they don't seem to be doing that. To maintain a piece of software I'd expect the maintainer to need a deep understanding of the source code and to continually monitor changes. The complaints maintainers post that software changed seem to prove they aren't doing that.

I just want to point out that this is asking a lot. You're essentially asking Debian to find volunteers to become as knowledgeable about, say, Libreoffice, as the developers themselves. That's probably not gonna happen. I think a more reasonable definition of "maintainer" is "ensures the package continues to work on ${DISTRO}, reaches out to the developers if a new update breaks that". I'm not defending the case you brought up, that sounds pretty lazy to me, but in the vast majority of cases it works great.

Roark663• 4 years ago

How can one write an article like this and fail to mention guix? It solves _a_lot_ of the problems mentioned. Guix (and all its channels) is a great piece of software, but unfortunately its potential for populartity is severely diminished by the insistence of the core user/dev group that its goals are not purely technological, but ideological(perhaps even primarily ideological).

It seems that for many devs guix is less of a technical achievemt,and more a way of preventing people from being able to run what they call "non free software". BTW, "non free software" also includes open source software that was released under certain non-compliant licenses.

The biggest issue I have with guix is that any effort of packaging "non-free" software (such as cuda, intel MKL, nvidia drivers etc). Is heavily discouraged, and if it it does happen its discussion is banned from official channels and all authors which make such software available in their personal repos seem to meekly include various pledges in the READMEs "to not advertise this horrible evil thing, or otherwise let people know of its existence". To me this behaviour is very far away from encouraging "free software", but my and their definition of free differ a lot.

My second issue with guix is that the language used to write everything(guile) still lacks many crucial debugging and ease-of-use features such as: being able to display more meanigful error messages, being able to show on which line of compiled code the error/exception happens at, being able to easily interrogate variable value types (there seem to be lots of is_this_value_a_string?, or is_this_value_an_integer? functions, but no what_type_is_this?), no standard way to convert nested objects to string, or to pretty-print deeply nested objects. To this one has to add pretty substantial, but really poorely laid out documentation of the language (from the very first chapter the authors in their examples use all language features that may not be explained until 300 pages in - so the reader basically has to read the whole thing multiple times to "get it"). And so on... All those things matter a lot to beginners and they matter less to advanced users. If this language and guix in general is to catch on, it has to be more beginner friendly.

However as a package manager it really is pretty good. It delivers everything nix does, but uses a more standardised and powerfull general purpose language.

I can't wait until someone forks guix without the whole ideology thing.

bayindirh2• 4 years ago

I can't wait until someone forks guix without the whole ideology thing.

What's wrong with the ideology behind it?

coldacid• 4 years ago

Strict ideology itself is wrong if for no other reason than that reality is not black and white.

the_duke• 4 years ago

Guix is basically an open source focused variant of Nix, with a few different design decisions.

It's also considerably less popular than Nix, which is already niche.

Roark663• 4 years ago

The main difference between guix and nix(other than ideology) is the language used to define the packages. Nix uses its own language, guix uses guile(compatible with scheme). Many people will find the nix language a lot more readable and easy to learn, but one may argue scheme/guie is a more general purpose language. It has been in existence for a long time. As such it has a large number of libraries, editing tools and it is easier to do stuff without having to resort to third party tools and scripts. Therefore in the long run guile (for one that learns it well) will prove more useful.

I have to admit I'm not 100% convinced by the argument. I've been learning guile for a couple of weeks and I just start to "get it". Compared with other languages I learned the difficulty with guile was many many times more (for various reasons). I now see it as a pretty powerfull language, but not without many faults.

The above notwithstanding I think something like Guix/Nix will become really popular when a truly easy for beginners to learn and powerfull language is used (such as python for example). Perhaps it would be an interesting project to attempt to auto-translate guix from guile into python. I can definitely see it takeoff then.

forgotpwd162• 4 years ago

I can't wait until someone forks guix without the whole ideology thing.

There's no reason to fork Guix since ideology doesn't affect the software itself at all. Rather what you want is make an alternative definitions repo. Pretty sure I've seen such one that includes software not accepted on official channels.

Roark663• 4 years ago

Indeed alternative "channels" exist. I mentioned them in my post. However, their authors don't question the ideological side of things. As I mentioned they even include statements how this software is "evil" etc.

What does it matter, you may say, the software is available! Yes, it is, but because of the whole "please don't advertise non-guix non-free(by their definition) software" it is hard to find to new users. Additionally pretty much all support there is for guix is provided by maybe 5 people on the irc channel. If you go to nonguix that number shrinks to perhaps 3 or 2. It is not easy to get support when the whole community is few people.

By "forking without the ideology" I would like to see guix under a new name, the name new users would search and find an installer that used normal Linux Kernel that supports a lot of software,after installing they would find a (hopefully) large community that is helpful in how to package any kind of software. One where a user doesn't have to hide the fact their trying top package software with proprietary components to get help etc. The "pure" guix has a different mission and it can continue with its libre-kernel . As of now, the software things I mention (guix with normal Kernel) exist, but they are not popular, as a result they are not as up to date as they could be. Also the whole project popularity suffers as a result.

al2o3cr• 4 years ago

I dunno about "modern" - I can recall distro packaging making a total disaster of Ruby gems over a decade ago.

noobermin• 4 years ago

Just spit balling ideas here, how much of the problems containerizing is meant to solve would be solved by merely static linking? Because it feels like that's what containers are mostly used for. Static linking + a sandbox to get most of everything.

Using shared libraries but then shipping the .so's in a container defeats the point of a .so in the first place. If the reason containers are okay now since disk space is cheap, why not return to static linking of executables?

tannhaeuser• 4 years ago

The question is why now? Distros have for the longest time worked as curated sets of dependencies without "nuclear" options such as containers. At the same time, there have been exactly zero new end-user apps on Linux for over a decade. Incidentally just today I bought a MBP after 12 years on Linux. Won't miss the stagnation, bike shedding, nerding around with new unproven languages, projecting everything into the "next big thing" such as rust, systemd, containers while delivering very little, the DE regressions, comical lack of apps, qa, taste, professionalism, and common sense.

qwery3• 4 years ago

I don't want to start a war about static/dynamic linking / shared libraries, so: some people want some shared libraries for some reasons.

One problem with having different applications use different versions of libraries is that you can have multiple versions of a library installed.

Having multiple versions of a library installed undermines some of the reasons for using shared libraries in the first place.

A lot of (most?) software was never meant to be used this way. It's uncommon to see an application that separates its data/config with unique filenames per version, for example. And there's not a global answer to whether or not that data should be shared between versions anyway. [0]

Container-style software distribution largely avoids these issues by not having shared libraries[1] for the most part. This has some benefits like the sandboxing and security features and making consistent behaviour across environments more achievable[2]. But you miss out on benefits from using shared libraries. Surprise -- a software development decision is a trade-off!

[0] Blender stands out for separating config files per version and also letting you easily import an existing config from an earlier version after an update.

[1] or not sharing shared libraries

[2] consistent behaviour across environments is more achievable as the dependencies are shipped with the application (you can ship dependencies without containers as well, e.g. static linking)

Traditional packaging is not suitable for modern applications?

Related Stories