Patching GCC to build Actually Portable Executables

https://ahgamut.github.io/2023/07/13/patching-gcc-cosmo/

by latchkey • 3 years ago

319 118 3 years ago

ahgamut.github.io

"But I've spent a good chunk of time removing obvious counterexamples, and a lot of popular software builds seamlessly."

Publishing a list of software that will successfully compile might be helpful. Perhaps this already exists.

ahgamut14• 3 years ago

You can build software like vim, emacs, ninja, bash, git, gcc etc -- here's a list of software I got to build with this technique: https://github.com/ahgamut/superconfigure

The superconfigure script is just a wrapper around the usual configure script used to build your software, supplying flags like --enable-static.

KerrAvon• 3 years ago

The switch-to-if thing is pretty inelegant. It seems like you could fix this by mapping to a platform-neutral set of constants, so you could dereference the runtime symbol with a function -- `switch(CosmoMappErrno(EINVAL))` and `case COSMO_EINVAL:` and so avoid turning the code into goto mush.

outworlder• 3 years ago

Do we even care if the replacement is done automatically?

jsmith45• 3 years ago

But that won't help with compiling unmodified code. Nobody should worry about gotos being created inside the compiler. It is doing that all over the place anyway.

Unless of course you mean that the compiler should be recognizing switches like this, and instead of rewriting them to if trees, it should be rewriting them to switches, changing the labels to use special cosmo specific constants for each of the values, and wraping the input value to the switch with a call to a function that maps the current runtimes platform's values for these over to corresponding cosmo specific constants (and letting other values pass though unchanged).

That... actually might be a simpler compiler transformation to implement. There would be complexity in needing to recognize which of the multiple sets of of constants are being used, and applying the right call or calls to the switch input to map them. It also require complexity on the library side though (creating these ). Not sure if the author would want to implement it, given they have a working implementation of the other transformation.

Lastly such a mapping approach would have risks of inappropriately mapping, or trickiness like having to invert the input before mapping for code that does the negative errno value thing.

ahgamut14• 3 years ago

Dereferencing the runtime symbol with a function sounds interesting! If you can show me an example of where it works, I'd be happy to try it out. I like the if-else-goto arrangement because it fit in perfectly with the other parts of gcc -- if you look at my patch[1], you will find that I had to change very little of gcc's existing code to add this capability.

[1] https://github.com/ahgamut/gcc/tree/portcosmo-11.2

charonn0• 3 years ago

N.B.: "Portable Executable" or "PE" is also the name of the Windows executable format.

epcoa• 3 years ago

Looks like they changed the story headline to Actually Portable Executables (which is the original “correct” name). Other than an audio format relatively few use I suppose APE isn’t the most clashy file format name.

1vuio0pswjnm72• 3 years ago

It seems that the term "portable" in PE refers to portability to different hardware whereas the same term used in APE refers to portability to different operating systems.

throwaway8356• 3 years ago

Portable is a generic term. The PE format is the same for x86 and ARM64 and other architectures.

https://learn.microsoft.com/en-us/windows/win32/debug/pe-for...

The name "Portable Executable" refers to the fact that the format is not architecture specific

https://en.m.wikipedia.org/wiki/Portable_Executable

On Windows NT operating systems, PE currently supports the x86-32, x86-64 (AMD64/Intel 64), IA-64, ARM and ARM64 instruction set architectures (ISAs). Prior to Windows 2000, Windows NT (and thus PE) supported the MIPS, Alpha, and PowerPC ISAs. Because PE is used on Windows CE, it continues to support several variants of the MIPS, ARM (including Thumb), and SuperH ISAs.

In my opinion it qualifies as portable

appleflaxen• 3 years ago

This is incredible cool.

yjftsjthsd-h2• 3 years ago

In case the author sees this:

Of course, my patch isn’t perfect. It can’t handle some anonymous structs, enums, const ints, or amazing things like using SIGILL as an array index [...]

Is "SIGILL" a typo?

inglor_cz2• 3 years ago

The SIGILL signal is raised when an attempt is made to execute an invalid, privileged, or ill-formed instruction. SIGILL is usually caused by a program error that overlays code with data or by a call to a function that is not linked into the program load module.

[0] https://support.sas.com/documentation/onlinedoc/ccompiler/do...

yjftsjthsd-h2• 3 years ago

Thanks, didn't know that was an actual signal!

spacechild1• 3 years ago

Another common cause is trying to execute instructions that the CPU doesn't support, e.g. trying to run AVX2 code on a machine that only supports SSE4.

ahgamut14• 3 years ago

SIGILL is for "illegal instruction", according to https://man7.org/linux/man-pages/man7/signal.7.html

I too thought it was a typo when I caused it to occur while building BLIS, but then I found out it was because I had added some AVX thing to the config that my local computer did not have.

flohofwoe6• 3 years ago

Some compilers (like gcc) emit an 'ud' instruction (https://www.felixcloutier.com/x86/ud) in some situations where the compiler detects undefined behaviour. IIRC this instruction triggers SIGILL on Linux.

cratermoon4• 3 years ago

An array of, say, strings, or functions, indexed according to IPC signal numbers. So array[SIGHUP] would be the same as array[2] so SIGILL would be array[4], or the 5th element of the array.

dataangel2• 3 years ago

Is there any practical purpose for APE? Like who is shipping single executables that need to run on N OSes?

Timon3• 3 years ago

I'm trying to release future side projects that way. I even build a Redbean executable for any client-only SPAs. The idea of this stuff just working on so many systems is amazing.

nicoburns• 3 years ago

Problem with this is that self-modifying code will likely trip security blocks on most platforms, which is probably more annoying for users than having to download separate executables.

jjnoakes2• 3 years ago

Storing executables on file systems shared across operating systems or architectures gets a lot easier if you don't have to have special paths for each os+arch combination.

saagarjha• 3 years ago

Doesn’t the executable get overwritten after first invocation?

dataangel2• 3 years ago

I've seen this solved before by giving each combo a different mount with the same path.

jcelerier2• 3 years ago

i've been asked this hundreds of times in my life, by customers, bosses, etc...

mhh__• 3 years ago

But they don't know or care the difference between the code being in the same binary or not

lmm• 3 years ago

I can imagine it making it slightly easier to distribute executables from e.g. maven repositories (where having multiple artifacts for the same id/version combination is cumbersome). I think PyPI might have a similar issue?

Also cross-compiling is painful; a lot of small OSS projects that don't have a build farm with all of the different BSDs etc. to hand simply don't ship binaries for those. Even e.g. kubectl has a binary for Linux but not for any of the BSDs (unless you count OSX). So this seems like an easy win for those.

corysama• 3 years ago

Java/Closure/Kotlin programmers. .Net programmers. JS, Python, Ruby, Erlang, every VM-based language programmers. Statically-compiled programs are under the hood of everything, but at the application level in the minority.

APE is a big “Who needs a VM if an AOT compiled language as low level as C can compile once and run universally?”

up2isomorphism• 3 years ago

Portability is not the reason you see VMs everywhere.

pjmlp4• 3 years ago

If libc is the only leaf dependency.

tstack• 3 years ago

I use an APE executable as an agent for communicating with remote hosts in the Logfile Navigator (https://lnav.org). While lnav itself is not built as an APE, the agent built into itself is. That agent is transferred to the remote when the user wants to read logs on that host. This way, there is no extra step to determine the type of OS or building in multiple versions of the executable. Here's a short blog post on the subject:

https://lnav.org/2021/05/03/tailing-remote-files.html

electroly2• 3 years ago

My biggest problem with APE: not portable enough, in the sense that when people adopt APE, they lock themselves into less portability than they might have supported otherwise. I live in an ARM world and because of APE you only offer Intel x64 binaries. All new Macs are ARM64, Raspberry Pi models are a mix of ARM32 and ARM64 (mine is ARM64), and I use exclusively Graviton instances in AWS. Running x64 binaries with QEMU/Rosetta emulation on an ARM system is a cute trick but not something I want to do for a piece of infrastructure. Raspberry Pis are barely fast enough to run native binaries. Consider offering APE for x64 but then still producing ARM binaries the old fashioned way. APE is pushing the idea that x64 is a good bytecode in a world that is moving from x64 to ARM. I'd like to see the project accept that ARM exists and produce an ARM version of APE, maybe with some jart flourish--can we make a cross-platform and cross-architecture fat binary?

flohofwoe6• 3 years ago

I have a shader compiler which would benefit from a platform agnostic, ready-to-run executable format.

It's too big to be build ad-hoc on the user's machine (contains various big 3rd party C++ dependencies, and takes between 2 and 20 minutes to build, depending on the hardware).

Needs to run on macOS x86-64+ARM, Linux x86-64 (ARM would be nice too of course), and Windows x86-64.

I'm currently looking into WASM, but that needs an installed WASI runtime.

efrecon• 3 years ago

In vscode extensions that rely on an external binary to do most of the job?

nonameiguess• 3 years ago

It's much more useful for greenfield software. Mad props to whoever wrote the gcc patch just to get something like this work some of the time, but rebuilding existing software is probably a fool's errand. There's a good reason you see things like the rpm package format include not only kernel but distro. Running on "Linux," BSD, Windows, or MacOS is one thing, but Linux is a bunch of different OSes, not one. A whole lot of software out there relies upon assuming you're using a particular DNS provider, filesystem hierarchy that may or may not be FHS-compliant. When I went down a rabbit hole a few years back trying to extend Linux from Scratch to include a bunch more packages, I found that GHC won't build if the file "/etc/os-release" doesn't exist. The file can be empty, but it has to exist.

Someone mentioned busybox elsewhere, which sure, will build with this if you disable a whole bunch of stuff, but busybox is supposed to be the minimal utilities needed to run something close to POSIX-compliant, which obviously assumes your system is a POSIX system. It also includes most of the stuff usually provided by util-linux, which obviously won't run except on Linux. If your software is doing something like trying to read from particular device files to get hardware info from the kernel, sometimes that will work on both Linux and BSD (and even Mac since it's kind of a BSD), but it definitely won't work on Windows.

For software that uses the network, how do you query DNS? If you're calling getaddrinfo, I guess Cosmo libc will do some magic for you to make sure that works everywhere, but a lot of software not written in C is doing something not quite that easily made universal, like just assuming /etc/resolv.conf exists and reading it, which definitely won't work on Windows, or using the native platform APIs, which will only work on one platform.

If you distribute desktop software, I would say good luck. Are you going to comply with the XDG? Well, now I guess it will "work" on Windows, but you're not complying with Windows conventions where you're supposed to put stuff in App\RoamingLow or whatever the hell that I would never remember without looking it up every time. Don't try storing data in the registry, because now it will only work on Windows.

Does your software? It better be present on all systems then. Do you query any environment variables? I don't think LC_TIME and LC_COLLATE are provided anywhere but Linux usually. The XDG_ variables are only Linux. Does Windows have PAGER? I don't even know. I'm pretty sure PATH, USER, and HOME work everywhere.

A truly actually portable executable (TAPE) would require far more than an executable file format that can successfully load into memory and execute its first instruction on any platform. There's a reason vendors just write software that uses a browser's JavaScript engine as a platform instead of the OS's native runtime.

EDIT: I don't know if this is ironic, but I guess consider your choice of programming language, too. Go became popular in part because of the static linking, single executable thing, but it inherently can't be cross-platform since it doesn't use the C library and tries to make all calls into the kernel on its own by keeping track of the syscall table provided by every OS it supports. Well, they're still different on every OS, so an executable that works on Linux will not work anywhere else and vice versa.

tssva2• 3 years ago

I'm pretty sure PATH, USER, and HOME work everywhere.

Out of those 3 only PATH works on Windows.

Windows has a USERNAME environment variable, but depending on your needs you may need to combine it with the USERDOMAIN variable to get the complete username, %USERDOMAIN%\%USERNAME%.

Windows has a HOMEPATH environment variable but it contains only the directory for the home folder. Another variable, HOMEDRIVE, contains the drive the home directory exists on, so the complete home location is %HOMEDRIVE%\%HOMEPATH%. In powershell a read-only HOME variable is set pointing to %HOMEDRIVE%\%HOMEPATH%. The home directory is supposed to store user files. User specific applications, OS components, customization and configuration data should be stored in the USERPROFILE directory. This is very frequently but not always the same as %HOMEDRIVE%\%HOMEPATH%.

regularfry• 3 years ago

A long time ago I wanted an Ansible equivalent that worked by injecting a bootstrap VM, then sending configuration code to run on the remote node, rather than chatting back and forth over an SSH tunnel. A lot of the slowness of applying config was that chattiness, and it would have been a lot faster to avoid it.

I have no idea if there is a tool that actually works like this - I remember an exploit tool that did something similar, but it wasn't quite right - but making that initial bootstrap VM and a useful stdlib an APE executable would also have avoided the "there needs to be a working Python at the other end" issue which persisted in tripping me up for far longer than it should have.

ahgamut14• 3 years ago

I wrote this post: the title should be "Patching GCC to build Actually Portable Executables", because it refers to Cosmopolitan Libc and jart's Actually Portable Executable format.

With my gcc patch, you can now build software like vim, emacs, ninja, bash, git, gcc etc with Cosmopolitan Libc, via their usual autotools/cmake-style build system. The built executables should run on Linux, FreeBSD, MacOS, OpenBSD, NetBSD, and also Windows (although I haven't tested Windows yet.)

Here's a list of software I got to build with this technique: https://github.com/ahgamut/superconfigure

The superconfigure script is just a wrapper around the usual configure script used to build your software, supplying flags like --enable-static.

If you want to build gcc using Cosmopolitan Libc -- try out this repo: https://github.com/ahgamut/musl-cross-make/tree/gccbuild

ChickeNES• 3 years ago

Have you attempted to build busybox?

ahgamut14• 3 years ago

Yes, I have gotten busybox to build successfully, but at present it's not as nice as bash or vim. I expect someone will write a better build script soon enough. Maybe it will be you!

The default build parameters in busybox make a lot of OS-related assumptions, so I'd recommend you use make menuconfig instead of make depconfig. in the TUI, you can disable all console and networking utilities, linux modprobe utilities, some linux system utilities, free, uptime, etc. Start with a minimal config and then slowly add things.

In terms of source code changes: modify `u_signal_names.c` to use a switch statement instead of the amazing "use SIGHUP as an array index" method it follows, use int32_t instead of the "smallint" type that busybox seems to prefer, and IIRC there's an enum in there somewhere that should be rewritten as a #define.

That should get you pretty close to a successful build.

endgame• 3 years ago

Will this patch of yours make it into GCC upstream? APE executables and cosmopolitan libc are incredibly cool tech, and it'd be great if they were easier to use.

ahgamut14• 3 years ago

Reading the GCC source code was fun and educational, and I'd love to find out if this could patch make it into GCC. Haven't discussed it with any of the GCC developers yet.

In the meantime, the Cosmopolitan Libc monorepo contains binaries of gcc-11 and binutils that have been built with my patch, you can use those for now :)

cratermoon4• 3 years ago

Could this be done for Rust? rustup target add cosmo maybe?

ahgamut14• 3 years ago

I did that last year :) https://github.com/ahgamut/rust-ape-example

Also got ripgrep to work: https://github.com/ahgamut/ripgrep/tree/cosmopolitan

Might need to update the build scripts a little to handle the latest updates in Rust and Cosmo (we'll need gcc-11 now), but I expect to get Rust working again soon enough.

foresto• 3 years ago

Nice! Do you foresee any problems with using this behind a transpiled language like Nim?

ahgamut14• 3 years ago

I haven't used Nim much, but I remember a repo on Github had setup a build script for compiling Nim with Cosmopolitan Libc. https://github.com/Yardanico/cosmonim

This gcc patch makes such build scripts simpler, because you will need to change less of your code -- let me know how it works!

mgaunard4• 3 years ago

Building is not such a big achievement, what's more interesting is to know whether it actually works?

As in, was there extensive testing to prove there are no regressions compared to the glibc version?

ahgamut14• 3 years ago

Right now I use the gcc, git, vim, and ninja APE binaries on my local system for daily work, and so far I have seen no visible regressions.

I agree with you though -- an extensive testing setup will reveal more things for improvement, be it in my patch or in the libc itself. At the moment I only have direct access to Debian 11/Fedora 35/FreeBSD 13, but I expect we will soon find a nice way to run tests across the different operating systems automatically. One of my ideas was to setup a separate drive or folder with all the executables, accessible from different operating systems/VMs, and then run the tests from each. What do you think?

blueflow3• 3 years ago

I recently saw that you got banned from lobste.rs for being a sockpuppet from jart and just now i realized that you are an actual, different person.

[unconstructive complaint about lobste.rs moderation skipped]

I hope HN treats you better.

supriyo-biswas3• 3 years ago

I took a peek at the moderation log and the banning of jart and their descendants was a real shitshow[1]. I’m sure a lot of legit folks lost their accounts, and most of the descendants weren’t even posting anything related to APE or jart’s other projects.

[1] https://lobste.rs/moderations

ahgamut14• 3 years ago

Yeah I found out I got banned when attempting to comment about this blog post. I rarely comment on online forums anyway, because I find I don't know enough to comment about most topics.

IIRC over the last year, I've only commented once on lobste.rs, when it came to something about Python's import system, and only upvoted my own blog posts about Cosmopolitan Libc and related comments + maybe one or two other posts. I disagree with the ban, but I can see why my account would get banned as a spammer (one of the accounts banned alongside mine had a double account I think).

lobste.rs is a wonderful community, and I'm happy to see that someone there atleast mentioned this blog post. Hopefully I can get un-banned at some point.

Luckily I can still comment on HN though!

pushcx2• 3 years ago

I'm the Lobsters admin and I enforced this ban. You misread the modlog, only jart's invitee i2 was sockpuppeting. As jart is here claiming that this is all mistaken sockpuppeting, let me explain what happened.

Lobsters has had serious problems with users who try to exploit the site for self-promotion, and it's more serious for us than HN when a few votes is enough to move something to our homepage. As part of addressing it, we have a bright-line rule prohibiting new users from submitting stories from new, unseen domains in their first 70 days on the site.

Five hours after jart invited woodrush, at 2022-01-12 02:26, they tried to violate the rule and got this error message:

woodrush.github.io is an unseen domain from a new user. We restrict this to discourage self-promotion and give you time to learn about topicality. Skirting this with a URL shortener or tweet or something will probably earn a ban.

At 2022-01-12 02:40, jart submitted the exact same link to break the restriction, and woodrush submitted several posts from that domain. After that, jart started submitting stories from all of her invitees to circumvent this rule, which anyone can verify by comparing timestamps on her submitted stories at https://lobste.rs/newest/jart to creation dates for her invitees at https://lobste.rs/u#jart.

On top of repeatedly breaking a rule that has a big red background and explicitly warns of a ban, two of jart's invitees were manually warned and banned for other inappropriate self-promotion. Finally, when another of her invitees started obvious sockpuppeting, I looked at the invite tree, saw this pattern of abuse, and cleaned up after it.

To rebut another claim repeated a couple times elsewhere in the comment tree, all banned users get an email notifying them of a ban with the Reply-To header set to my email: https://github.com/lobsters/lobsters/blob/master/app/mailers... None of the users banned in this action have clicked reply to explain how I've made a mistake enforcing an unambiguous rule of ours.

develatio• 3 years ago

There is a ticket in Go's repo about this: https://github.com/golang/go/issues/51900

cratermoon4• 3 years ago

In case anyone is confused. "Actually Portable Executables" https://justine.lol/ape.html

ahgamut14• 3 years ago

I added a comment mentioning this, but it hasn't changed yet. I expect it will change soon. Perhaps I can ping @dang from here?

cratermoon4• 3 years ago

Justine's post did a great job teaching me the details of leveraging the various executable formats to make it possible.

jwilk• 3 years ago

https://news.ycombinator.com/item?id=36526450

The only way to get reliable message delivery is to email hn@ycombinator.com.

ylyn• 3 years ago

I don't see why you are so surprised by code that indexes using SIGxxx. The signal numbers are well-defined and known to be mostly consecutive and start from 1.

ahgamut14• 3 years ago

I had tried to build a bunch of different codebases (git, bash, gcc, curl etc.) before trying busybox, and I'd never seen code like that before. Plus that error came alongside a compiler error in my patch that took a long time to debug, so it was a memorable surprise :)

Would you consider a code pattern like that to be common? What other codebases apart from busybox have it? If there are many examples, I might spend some time trying to update my patch to handle patterns of that kind.

mgaunard4• 3 years ago

If errno is such a big problem, what's the hope in even running the C++ standard library and its std::error_code...

No way to use magic macros there.

foota• 3 years ago

The issue _was_ the magic macros, I believe. Normally, the macros expand to a constant, but now they don't, which breaks some things. std::error_code doesn't have the same issue, since it's not a constant?

mgaunard4• 3 years ago

Well if they need to replace those integer constants with function calls, clearly that breaks everything.

std::error_code is just a wrapper for several different enums. I should have probably just given the posix one, std::errc.

You can't just make the various enumerations of an enum be function calls.

The right implementation would have been to convert error codes from/to posix, rather than just give native error codes and make looking up the reference value a runtime operation.

Gordonjcp2• 3 years ago

"If EINVAL is not a compile-time constant, four of the initializations above are not valid"

Wait, why isn't EINVAL a compile-time constant, shouldn't it just be an enum?

londons_explore3• 3 years ago

It has a different constant value on different OS's (Linux, Mac, Windows, etc).

Therefore the resulting code, which must run on any OS, cannot treat it as a constant.

mike_hock• 3 years ago

Thus proving that APEs are not viable, the end.

The assumption that constants are constants is a sound assumption, and I highly doubt many would be willing to rewrite the world to make some "clever hack" work.

Gordonjcp2• 3 years ago

Ah, right, so while you and I don't care what numerical value in the enum EINVAL has, if I take that numerical value and stuff the binary into another OS then it means something totally different.

Right, it's not that it *isn't* a constant, it's that its not the *same* constant everywhere?

f33d5173• 3 years ago

I suppose this is all quite neat, but I wonder about the theory behind it. If you're building a new OS target, the logical thing to do is to define new contants. Rather than passing in the OSes constants on every underlying OS, you would translate the OSes constants to be the same everywhere. Then, use of constants as array indexes and case statements would work normally. This might incur more overhead than does the approach described in the article, but it seems a great deal easier, and simpler to use for whoever decides to compile a new program against cosmopolitan libc.

ahgamut14• 3 years ago

New constants do make compilation a lot easier, but my personal opinion is that the overhead to convert to/from the old constants during runtime is too much.

Look at the list of constants here: https://github.com/jart/cosmopolitan/blob/master/libc/sysv/c...

Every time I used any of these constants, I'd have to load a whole bunch of them into my binary as a large lookup table, and go through that table every time I needed a check in my program. It might not be that slow, but I believe it would definitely be noticeable.

My goal was to make porting easier without changing a lot of source code in either the libc or in the software I was trying to port, and still produce binaries that are close or better in performance. Under those constraints, this gcc patch seemed like the best way to simplify the process.

If I run into enough codebases where SIGHUP is used as an array index initializer, I will probably attempt your suggestion just to measure the tradeoffs. Or you could try it out and let me know if a separate set of constants is better.

Patching GCC to build Actually Portable Executables

Related Stories