The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

https://github.com/quaadgras/graphics.gd/discussions/242

by Splizard • 6 months ago

231 206 6 months ago

einpoklum2• 6 months ago

This seems interesting even regardless of go. Is it realistic to create an executable which would work on very different kinds of Linux distros? e.g. 32-bit and 64-bit? Or maybe some general framework/library for building an arbitrary program at least for "any libc"?

quesomaster9000• 6 months ago

Cosmopolitan goes one further: [binaries] that runs natively on Linux + Mac + Windows + FreeBSD + OpenBSD + NetBSD + BIOS on AMD64 and ARM64

https://justine.lol/cosmopolitan/

oguz-ismail23• 6 months ago

Linux

if you configure binfmt_misc

Windows

if you disable Windows Defender

OpenBSD

only older versions

dontdoxxme• 6 months ago

Clearly a joke if it uses the .lol tld.

sambuccid• 6 months ago

Appimage exists that packs linux applications into a single executable file that you just download and open. It works on most linux distros

greyw• 6 months ago

I vaguely remember that Appimage-based programs would fail for me because of fuse and glibc symbol version incompatibilties.

Gave up them afterwards. If I need to tweak dependencies might as well deal with the packet manager of my distro.

iberator• 6 months ago

Yup. Just compile it as static executable. Static binaries are very undervalued imo.

flohofwoe5• 6 months ago

The "just" is doing a lot of heavylifting here (as detailed in the article), especially for anything that's not a trivial cmdline tool.

account428• 6 months ago

As TFA points out at the beginning, it's not so simple if you want to use the GPU.

pjmlp4• 6 months ago

We had a time when static binaries where pretty much the only thing we had available.

Here is an idea, lets go back to pure UNIX distros using static binaries with OS IPC for any kind of application dynamism, I bet it will work out great, after all it did for several years.

Got to put that RAM to use.

amelius3• 6 months ago

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

mdavid6265• 6 months ago

You can "package" all .so files you need into one file, there are many tools which do this (like a zip file).

But you can't take .so files and make one "static" binary out of them.

fc417fc8023• 6 months ago

Well not a static binary in the sense that's commonly meant when speaking about static linking. But you can pack .so files into the executable as binary data and then dlopen the relevant memory ranges.

geocar2• 6 months ago

But you can't take .so files and make one "static" binary out of them.

Yes you can!

This is more-or-less what unexec does

- https://news.ycombinator.com/item?id=21394916

For some reason nobody seems to like this sorcery, probably because it combines the worst of all worlds.

But there's almost[1] nothing special about what the dynamic linker is doing to get those .so files into memory that it can't arrange them in one big file ahead of time!

[1]: ASLR would be one of those things...

TheDong• 6 months ago

There are things like this.

The things I know of and can think of off the top of my head are:

1. appimage https://appimage.org/

2. nix-bundle https://github.com/nix-community/nix-bundle

3. guix via guix pack

4. A small collection of random small projects hardly anyone uses for docker to do this (i.e. https://github.com/NilsIrl/dockerc )

5. A docker image (a package that runs everywhere, assuming a docker runtime is available)

6. https://flatpak.org/

7. https://en.wikipedia.org/wiki/Snap_(software)

AppImage is the closest to what you want I think.

amelius3• 6 months ago

AppImage looks like what I need, thanks.

I wonder though, if I package say a .so file from nVidia, is that allowed by the license?

gilli• 6 months ago

I wish AppImage was slightly more user friendly and did not require the user to specifically make it executable.

a0223112• 6 months ago

It should be noted that AppImages tend to be noticeably slower at runtime than other packaging methods and also very big for typical systems which include most libraries. They're good as a "compile once, run everywhere" approach but you're really accommodating edge cases here.

A "works in most cases" build should also be available for that that it would benefit. And if you can, why not provide specialized packages for the edge cases?

Of course, don't take my advice as-is, you should always thoroughly benchmark your software on real systems and choose the tradeoffs you're willing to make.

aa-jv• 6 months ago

AppImage comes close to fulfilling this need:

https://appimage.github.io/appimagetool/

Myself, I've committed to using Lua for all my cross-platform development needs, and in that regard I find luastatic very, very useful ..

formerly_proven• 6 months ago

I don't think you can link shared objects into a static binary because you'd have to patch all instances where the code reads the PLT/GOT, but this can be arbitrarily mangled by the optimizer, and turn them back into relocations for the linker to then resolve them.

You can change the rpath though, which is sort of like an LD_LIBRARY_PATH baked into the object, which makes it relatively easy to bundle everything but libc with your binary.

edit: Mild correction, there is this: https://sourceforge.net/projects/statifier/ But the way this works is that it has the dynamic linker load everything (without ASLR / in a compact layout, presumably) and then dumps an image of the process. Everything else is just increasingly fancy ways of copying shared objects around and making ld.so prefer the bundled libraries.

fieu• 6 months ago

Ermine: https://www.magicermine.com/

It works surprisingly well but their pricing is hidden and last time I contacted them as a student it was upwards of $350/year

secure• 6 months ago

https://github.com/gokrazy/freeze is a minimal take on this

alas44• 6 months ago

There is this project "actually portable executable"/cosmopolitan libc https://github.com/jart/cosmopolitan that allows a compile once execute anywhere style type of C++ binary

lizknope• 6 months ago

15-30 years ago I managed a lot of commercial chip design EDA software that ran on Solaris and Linux. We had wrapper shell scripts for so many programs that used LD_LIBRARY_PATH and LD_PRELOAD to point to the specific versions of various libraries that each program needed. I used "ldd" which prints out the shared libraries a program uses.

mkoubaa• 6 months ago

Sounds painful. Better to distrib a separate bundle per platform and use RPATH

ValdikSS4• 6 months ago

It you know what you're doing and scrupulous enough, you can package the software in a way that it works 25 years later.

https://xcancel.com/ValdikSS/status/1843044963443253678

ryan-c• 6 months ago

(not an endorsement, I do not use it, but I know of it)

https://www.magicermine.com/

saghm3• 6 months ago

I don't think it's as simple as "run this one thing to package it", so if the process rather than the format is what you're looking for, this won't work, but that sounds a lot like how AppImages work from the user perspective. My understanding is that an AppImage is basically a static binary paired with a small filesystem image containing the "root" for the application (including the expected libraries under /usr/lib or wherever they belong). I don't line everything about the format, but overall it feels a lot less prescriptive than other "universal" packages like flatpak or snap, and the fact that you can easily extract it and pick out the pieces you want to repackage without needing any external tools (there are built-in flags on the binary like --appimage-extract) in helps a lot.

jcalvinowens2• 6 months ago

  mkdir chroot
  cd chroot
  for lib in $(ldd ${executable} | grep -oE '/\S+'); do
    tgt="$(dirname ${lib})"
    mkdir -p .${tgt}
    cp ${lib} .${tgt}
  done
  mkdir -p .$(dirname ${executable})
  cp ${executable} .${executable}
  tar cf ../chroot-run-anywhere.tgz .

saidinesh53• 6 months ago

You're supposed to do this recursively for all the libs no?

Eg. Your App might just depend on libqt5gui.so but that libqt5gui.so might depend on some libxml etc...

Not to mention all the files from /usr/share etc... That your application might indirectly depend on.

Conscat2• 6 months ago

Someone already mentioned AppImage, but I'd like to draw attention to this alternate implementation that executes as a POSIX shell script, making it possible to dynamic dispatch different programs on different architectures. e.g. a fat binary for ARM and x64.

https://github.com/mgord9518/shappimage

sekh60• 6 months ago

So autotools but for execution instead of compilation?

fdgdd• 6 months ago

Exodus (https://github.com/intoli/exodus) used to be good for this but is giving me a Python error these days.

MrDrMcCoy• 6 months ago

A couple similar projects:

https://github.com/sigurd-dev/mkblob https://github.com/tweag/clodl

Meneth• 6 months ago

That seems mostly useful for proprietary programs. I don't like it.

seba_dos13• 6 months ago

Yeah, in my 20 years of using and developing on GNU/Linux the only binary compatibility issues I experienced that I can think of now were related to either Adobe Flash, Adobe Reader or games.

Adobe stuff is of the kind that you'd prefer to not exist at all rather than have it fixed (and today you largely can pretend that it never existed already), and the situation for games has been pretty much fixed by Steam runtimes.

It's fine that some people care about it and some solutions are really clever, but it just doesn't seem to be an actual issue you stumble on in practice much.

whizzter• 6 months ago

The solution to games is to load Windows games instead of Linux binaries.

Basically the way for the year of the Linux desktop is to become Windows.

paddim83• 6 months ago

Probably because your distro purposefully keeps software out of date because it is too fragile otherwise. I don't think that is reasonable at all for desktop use.

juliangmp• 6 months ago

Why? Foss software also benefits from less dependency hell.

breezykoi2• 6 months ago

For distro-packaged FOSS, binary compatibility isn't really a problem. Distributions like Debian already resolve dependencies by building from source and keeping a coherent set of libraries. Security fixes and updates propagate naturally.

Binary compatibility solutions mostly target cases where rebuilding isn't possible, typically closed source software. Freezing and bundling software dependencies ultimately creates dependency hell rather than avoiding it.

Faelian2• 6 months ago

I did wrote a small open-source tool in Rust. And I too did encounter that kind of issue when I did start to build a .deb.

Honestly, it was the kind of bug that is not fun to fix, because it's really about dependency, and not some fun code issue. There is no point in making our life harder with this to gatekeep proprietary software to run on our platform.

athrowaway3z• 6 months ago

I'd never heard of detour. That's a pretty cool hack.

ckbkr10• 6 months ago

they were prominent in game hacking 2005ish windows

made hooking into game code much easier than before

sidewndr464• 6 months ago

Aren't all DLLs on the Windows platform compiled with an unusual instruction at the start of each function? This makes it possible to somehow hot patch the DLL after it is already in memory

mgaunard2• 6 months ago

It's funny how people insist on wanting to link everything statically when shared libraries were specifically designed to have a better alternative.

Even worse is containers, which has the disadvantage of both.

vv_2• 6 months ago

It's easier to distribute software fully self-contained, if you ignore the pain of statically linking everything together :)

diego_sandoval• 6 months ago

What's the pain?

fc417fc8023• 6 months ago

Dynamic linking exists to make a specific set of tradeoffs. Neither better nor worse than static linking in the general sense.

arghwhat• 6 months ago

Dynamic libraries have been frowned upon since their inception as being a terrible solution to a non-existent problem, generally amplifying binary sizes and harming performance. Some fun quotes of quite notable characters on the matter here: https://harmful.cat-v.org/software/dynamic-linking/

In practice, a statically linked system is often smaller than a meticulously dynamically linked one - while there are many copies of common routines, programs only contain tightly packed, specifically optimized and sometimes inlined versions of the symbols they use. The space and performance gain per program is quite significant.

Modern apps and containers are another issue entirely - linking doesn't help if your issue is gigabytes of graphical assets or using a container base image that includes the entire world.

holowoodman6• 6 months ago

Statically linked binaries are a huge security problem, as are containers, for the same reason. Vendors are too slow to patch.

When dynamically linking against shared OS libraries, Updates are far quicker and easier.

And as for the size advantage, just look at a typical Golang or Haskell program. Statically linked, two-digit megabytes, larger than my libc...

rlpb5• 6 months ago

Imagine a fully statically linked version of Debian. What happens when there’s a security update in a commonly used library? Am I supposed to redownload a rebuild of basically the entire distro every time this happens, or else what?

RicoElectrico• 6 months ago

That would be a good point if said shared libraries did not break binary backwards compatibility and behaved more like winapi.

flohofwoe5• 6 months ago

Dynamic libraries make a lot of sense as operating system interface when they guarantee a stable API and ABI (see Windows for how to do that) - the other scenarios where DLLs make sense is for plugin systems. But that's pretty much it, for anything else static linking is superior because it doesn't present an optimization barrier (especially for dead code elimination).

No idea why the glibc can't provide API+ABI stability, but on Linux it always comes down to glibc related "DLL hell" problems (e.g. not being able to run an executable that was created on a more recent Linux system on an older Linux system even when the program doesn't access any new glibc entry points - the usually adviced solution is to link with an older glibc version, but that's also not trivial, unless you use the Zig toolchain).

TL;DR: It's not static vs dynamic linking, just glibc being a an exceptionally shitty solution as operating system interface.

uecker3• 6 months ago

I do not think it is difficult compiling against versions by using a container.

AshamedCaptain13• 6 months ago

(e.g. not being able to run an executable that was created on a more recent Linux system on an older Linux system even when the program doesn't access any new glibc entry points - the usually adviced solution is to link with an older glibc version, but that's also not trivial, unless you use the Zig toolchain).

In the era of containers, I do not understand why this is "Not trivial". I could do it with even a chroot.

mgaunard2• 6 months ago

Static linking is also an optimization barrier.

LTO is really a different thing, where you recompile when you link. You could technically do that as part of the dynamic linker too, but I don't think anyone is doing it.

There is a surprisingly high number of software development houses that don't (or can't) use LTO, either because of secrecy, scalability issues or simply not having good enough build processes to ensure they don't breach the ODR.

sebastos2• 6 months ago

Genuine question - are there examples (research? old systems?) of the interface to the operating system being exposed differently than a library? How might that work exactly?

abigail95• 6 months ago

Why would I want to be constantly calling into code I have no control over, that may or may not exist, that may or may not be tampered with.

I lose control of the execution state. I have to follow the calling conventions which let my flags get clobbered.

To forego all of the above including link time optimization for the benefit of what exactly?

Imagine developing a C program where every object file produced during compilation was dynamically linked. It's obvious why that is a stupid idea - why does it become less stupid when dealing with a separate library?

uecker3• 6 months ago

You call into dynamic libraries so that you do not need to recompile and distribute new binaries to all your users whenever there is a security issue or other critical fix in any of the dependencies.

gethly2• 6 months ago

Isn't the sole reason why linux sucks(sucked?) for games and other software exactly that there is a gazillion of different libraries with different versions, so you have zero assumptions about the state of the OS, which makes making sw for it such a pain?

paddim83• 6 months ago

Yes. Premature optimisation when it comes to dynamic linking is the reason for why the year of the Linux desktop is far away in my opinion

Rochus2• 6 months ago

So what we need is essentially a "libc virtualization".

But Musl is only available on Linux, isn't it? Cosmopolitan (https://github.com/jart/cosmopolitan) goes further and is available also on Mac and Windows, and it uses e.g. SIMD and other performance related improvements. Unfortunately, one has to cut through the marketing "magic" to find the main engineering value; stripping away the "polyglot" shell-script hacks and the "Actually Portable Executable" container (which are undoubtedly innovative), the core benefit proposition of Cosmopolitan is indeed a platform-agnostic, statically-linked C standard (plus some Posix) library that performs runtime system call translation, so to say "the Musl we have been waiting for".

sidewndr464• 6 months ago

At the rate things are going we'll need a container virtualization layer as well, a docker for docker if you know what I mean

miduil• 6 months ago

Do you mean something like gVisor?

rafale• 6 months ago

"All problems in computer science can be solved by another level of indirection"

binsquare2• 6 months ago

I'm building in this space, I take a docker inside a microvm (vm-lite) approach.

https://github.com/smol-machines/smolvm

VikingCoder2• 6 months ago

I desperately want to write C/C++ code that has a web server and can talk websockets, and that I can compile with Cosmopolitan.

I don't want Lua. Using Lua is crazy clever, but it's not what I want.

I should just vibe code the dang thing.

tgittos• 6 months ago

You should, it’s fun.

I have a devcontainer running the Cosmopolitan toolchain and stuck the cosmocc README.md in a file referenced from my AGENTS.md.

Claude does a decent job. You have to stay on top of it when it’s writing C, easy to turn to spaghetti.

Also the fat binary concept trips up agents - just have it read the actual cosmocc file itself to figure any issues out.

drowsspa2• 6 months ago

I find it amazing how much the mess that building C/C++ code has been for so many decades seems to have influenced the direction technology, the economy and even politics has been going.

Really, what would the world look like if this problem had been properly solved? Would the centralization and monetization of the Internet have followed the same path? Would Windows be so dominant? Would social media have evolved to the current status? Would we have had a chance to fight against the technofeudalism we're headed for?

joshmarinacci• 6 months ago

How does this technical issue affect the economy and politics? In what way would the world be different just because we used a better linker?

AshamedCaptain13• 6 months ago

What I find amazing is why people continously claim glibc is the problem here. I have a commercial software binary from 1996 that _still works_ to this day. It even links with X11, and works under Xwayland.

The trick? It's not statically linked, but dynamically linked. And it doesn't like with anything other than glibc, X11 ... and bdb.

At this point I think people just do not know how binary compatibility works at all. Or they refer to a different problem that I am not familiar with.

Conscat2• 6 months ago

If the APE concept isn't appealing to you, you may be interested in the work on LLVM libC. My friend recently delivered an under-appreciated lecture on the vision:

https://youtu.be/HtCMCL13Grg

tl;dw Google recognizes the need for a statically-linked modular latency sensitive portable POSIX runtime, and they are building it.

weebull• 6 months ago

If you're using dlopen(), you're just reimplementing the dynamic linker.

112233• 6 months ago

that's cute, but dismissive, sort of like "if you use popen(), you are reimplementing bash". There is so much hair in ld nobody wants to know about — parsing elf, ctors/dtors, ...

pilif• 6 months ago

Isn't this asking for the exact trouble musl wanted so spare you from by disabling dlopen()?

netbioserror• 6 months ago

I've been statically linking Nim binaries with musl. It's fantastic. Relatively easy to set up (just a few compiler flags and the musl toolchain), and I get an optimized binary that is indistinguishable from any other static C Linux binary. It runs on any machine we throw it at. For a newer-generation systems language, that is a massive selling point.

zoobab• 6 months ago

I have an idea for a static linux distribution based on musl, with either an Alpine rebuild or Gentoo-musl:

http://stalinux.wikidot.com

The documentation to make static binary with GLibc is sparce for a reason, they don't like static binaries.

cb321• 6 months ago

Yeah. I've been doing this for almost 10 years now. It's not APE/cosmopolitan (which also "kinda works" with Nim but has many lowest common denominator platform support issues, e.g. posix_fallocate). However, it does let you have very cross-Linux portable binaries. Maybe beyond Linux.

Some might appreciate a concrete instance of this advice inline here. For `foo.nim`, you can just add a `foo.nim.cfg`:

    @if gcc:
      gcc.exe       = "musl-gcc"
      gcc.linkerexe = "musl-gcc"
      passL         = "-static -s" @end

There is also a "NimScript" syntax you could use a `foo.nims`:

    if defined gcc:  # nim.cfg runs faster than NimScript
      switch "gcc.exe"      , "musl-gcc"
      switch "gcc.linkerexe", "musl-gcc"
      switch "passL"        , "-static -s"

ValdikSS4• 6 months ago

`dlopen`'ing system libraries is an "easy" hack to try to maintain compatibility with wide variety of libraries/ABIs. It's barely used (I know only of SDL, Small HTTP Server, and now Godot).

Without dlopen (with regular dynamic linking), it's much harder to compile for older distros, and I doubt you can easily implement glibc/musl cross-compatibility at all in general.

Take a look what Valve does in a Steam Runtime:

    - https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/docs/pressure-vessel.md
    - https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/subprojects/libcapsule/doc/Capsules.txt

surajrmal• 6 months ago

Binary comparability extends beyond the vide that runs in your process. These days a lot of functionality occurs by way of IPC which has a variety of wire protocols depending on the interface. For instance there is dbus, Wayland protocols, varlink, etc. Both the wire protocol, and the APIs built on top need to retain backwards comparability to ensure Binary compatibility. Otherwise you're not going to be able to run on various different Linux based platforms arbitrarily. And unlike the kernel, these userspace surfaces do not take backwards compatibility nearly as important. It's also much more difficult to target a subset of these APIs that are available on systems that are only 5 years old. I would argue API endpoints on the web have less risk here (although those break all the time as well)

leni5362• 6 months ago

Do I get this right that this effectively dlopens glibc (indirectly) into an executable that is statically linked to musl? How can the two runtimes coexist? What about malloc/free? AFAIK both libc's allocators take ownership of brk, that can't be good. What about malloc/free across the dynamic library interface? There are certainly libraries that hand out allocated objects and expect the user to free them, but that's probably uncommon in graphics.

Splizard2• 6 months ago

You have to tell musl to use mmap instead of brk. You're right that it doesn't work in all cases but as long as you switch TLS on calls (and callbacks), at least with a project the size of Godot, you can approach a workable solution.

leni5362• 6 months ago

You have to tell musl to use mmap instead of brk.

How do I do that? Is there a documented configuration of musl's allocator?

dunder_cat• 6 months ago

Related discussion (the actual project is mentioned in the issue): "Detour: Dynamic linking on Linux without Libc" https://news.ycombinator.com/item?id=45740241

aspbee555• 6 months ago

I managed to get this combo going not too long ago with my musl rust app but I found that even after it all was compiled and loading the lib it did not function properly because the library I loaded still depended on libc functions. even with everything compiled into a huge monolithic musl binary it couldn't find something graphics related

I eventually decided to keep the tiny musl app and make a companion app in a secondary process as needed (since the entire point of me compiling musl was cross platform linux compatibility/stability)

tuhgdetzhh• 6 months ago

The best binary compatibily you can get on Linux is via Wine.

Source: https://blog.hiler.eu/win32-the-only-stable-abi/

nektro• 6 months ago

you'll likely be interested in [1] and [2] for further reading

[1] https://github.com/ziglang/zig/issues/7240

[2] https://www.youtube.com/watch?v=pq1XqP4-qOo

The Holy Grail of Linux Binary Compatibility: Musl and Dlopen

Related Stories