FatELF is no more. Linux developer community owned Ryan Gordon.

http://icculus.org/cgi-bin/finger/finger.pl?user=icculus&date=2009-11-03&time=19-08-04

by jeff18 • 17 years ago

59 60 17 years ago

icculus.org

chanux• 17 years ago

I don't believe this is the way it should end, leaving one enthusiast all broken.

Anyway I'm not really aware about background details of the matter.

mahmud• 17 years ago

Enthusiasm is not a substitute for pragmatics; all the other contributors are just as enthusiastic, but they know their patches are not entitled to merger with the main tree willy nilly, without there being a very good reason.

orph• 17 years ago

This sort of soul-crush usually means more quality employee fodder for Google & Facebook.

ableal• 17 years ago

This reminds me of P.G.Wodehouse's stories, where you'll occasionally find a victim being dragooned into judging a 'Bonny Baby' contest.

spitfire• 17 years ago

This is sort of an interesting counterpoint to the whole "Open source forsters innovation" meme open source types love to throw out.

Open source can foster innovation, as long as ego's don't get in the way. Sadly there are a lot of ego's involved in open source.

NB: I think the idea of fatelf is a very good one. Mainly for binary applications though. Which OSS guys tend to dislike.

tvon2• 17 years ago

> Mainly for binary applications though. Which OSS guys tend to dislike.

Huh?

dlsspy• 17 years ago

Where binary == closed source.

Imagine a binary-only distribution of some application that runs on all hardware platforms (because we get that a lot in OS X and we used to get it on NeXTSTEP as well).

thingie6• 17 years ago

He could write the patch, he could try it and persuade mainline kernel (and glibc and gcc, and...) developers to include it. If the feature is really that useful, he can even maintain it separately, or some Linux distribution can. All of which would be impossible with closed software. Innovation in the oss way, sadly, means, that there are lots of ideas, and most of them thrown out.

Hostile reaction from kernel developers? Well, maybe it's little sad, they could trash the patch more politely or something, but eventually they have to. Fatelf adds stuff to kernel, glibc and few other parts of system without really solving much. He could try something with binfmt_misc and completely userspace, not add junk into kernel and glibc itself. (With junk I mean something unnecessary that can be done in userspace. (or not at all))

cschwarm• 17 years ago

> without really solving much

On a German Linux news site (prolinux.de), a commercial Linux game developer -- Ankh2, IIRC -- once commented that they could not get their game running on about 2-3% of their customers' Linux computers, despite their best efforts.

I dunno whether FatElf would have solved problems like this in the long run, but there are problems. You may not be affected but this doesn't mean others aren't.

kelnos• 17 years ago

> He could try something with binfmt_misc...

Exactly! And that sounds like it would have been the smartest move from the start. Something like this doesn't need to be in the kernel. It doesn't need to be in glibc, and doesn't need to be in ld. There are probably performance advantages that you miss out on by using binfmt_misc, but that's the perfect way to get your solution working without any external dependencies. You create two userspace tools: 1) an app that stitches together multiple binaries into a FatELF file, and 2) a launcher program that, when given the path to a FatELF file, picks the correct embedded binary and runs it. #2 might need some fancy tricks with the dynamic loader, but it's certainly doable.

Assuming people do like it and use it, only then do you try to get it integrated into the system proper. Why try to push something into three pieces of software very core to Linux and Linux development when there's no evidence that there will be any developer, distro, or user adoption?

davidw• 17 years ago

I think it has more to do with one open source project in particular, the kernel, which has a lot of history. The one time I tried to get a patch in, it was very difficult and I ultimately let it be, despite people regularly saying they thought it was a sensible idea. Indeed, there was never any real technical opposition, just a sort of inertia to ignore stuff.

jdminhbg• 17 years ago

I don't think this is specific to open source; anyone who's worked at a closed-source company can certainly tell the same story about stuff being quashed by egos higher-up. At least in the open source community, if you feel strongly enough, you can fork it and go off without the egos.

shpxnvz• 17 years ago

This is sort of an interesting counterpoint to the whole "Open source forsters innovation" meme open source types love to throw out.

I may be missing something, but it seems to be fostering innovation in this example as well. He has managed to create something that at least a few people consider innovative, thanks in part to the open licenses of the sources this work is built on, and he is free to distribute it however he likes.

Or are you arguing that it's not innovation unless some other group can be forced to take up work on your behalf, by merging and/or maintaining code they don't want or agree with?

wmf• 17 years ago

Look on the bright side: they saved him a lot of time. Usually the Linux developers tell you to redesign and rewrite your patch from scratch. :-/

holdenk• 17 years ago

FatELF seems like a bad idea for Linux. The "benefits" listed on http://icculus.org/fatelf/ fail to persuade me to want to pay any additional cost.

bad_user• 17 years ago

Oh, so having future commercial packages that many people depend on, like Adobe Photoshop or 3D Studio Max ... packaged and distributed as a universal binary for every major distribution and every architecture ... is of no importance whatsoever?

This is a major roadblock for commercial third-parties that would like to distribute ports of their software for Linux.

thingie6• 17 years ago

You can do this without fatelf. Put everything you have into a single package and add "shell scripts and flakey logic to pick the right binary and libraries to load". (He fails to mention, that fatelf is nothing more than this, only hardwired into kernel.)

prodigal_erik2• 17 years ago

If a proprietary vendor is willing to do all the work to cross-compile, package, and test on a variety of architectures and distributions, I can't believe they would be deterred by the need to link a binary for each package. With FatELF they would have to do that anyway, and then glue all those binaries together before building all their packages.

Tuna-Fish2• 17 years ago

It would be of plenty of importance, but fat_elf does not actually help solve the problem in any way.

You can produce all important functionality of the fat_elf patches with a single simple shell script.

lil_cain3• 17 years ago

It doesn't necessarily solve the 'every distribution' thing. And most commercial developers don't support multiple architecture s in the first place.

oddgodd2• 17 years ago

It really is just a bad idea. Or at least one that is working against ideas central to the way Linux is currently used.

Universal/fat binaries made sense on the Macintosh because there is no concept of program installation on these systems. While I think that eschewing installation is generally a better design, one drawback is that if you want to be able to support multiple architectures in one application you have to do the architecture check when the program is loaded.

Central to Linux and Windows is the idea of program installation, either through packages or installer programs. No one is interested in making it so that you can drag and drop items in Program Files or /usr/bin between systems and expect them to run, which is the only thing that using fat binaries really gets you over other solutions.

Nearly all of the commercial binary-only software I have seen in Linux (and other Unixes) uses an installer program, just like windows. There is no technical reason why such an installer couldn’t determine the architecture to install.

FooBarWidget4• 17 years ago

Not quite. Current Linux packaging formats encourage the developer to build one package per architecture. This means presenting several download choices for the user which can be confusing. The user doesn't always know what his architecture is.

The problem can be solved in two ways:

1) Distributing through a repository and have the package manager auto-select the architecture. However this is highly distribution-specific. If you want to build a single tar.gz that works on the majority of the Linux distros then you're out of luck.

2) Compile binaries for multiple architectures and bundle everything into the same package, and have a shell script or whatever select the correct binary.

While (2) is entirely doable and does not confuse the end user, it does make the developer's job harder. He has to compile binaries multiple times and spend a lot of effort on packaging. Having support for universal binaries throughout the entire system, including the developer toolchain, solves not only confusion for the end user but also hassle for the developer. On OS X I can type "gcc -arch i386 -arch ppc" and it'll generate a universal binary with two architectures. I don't have to spend time setting up a PPC virtual machine with a development environment in it, or to setup a cross compiler, just to compile binaries for PPC; everything is right there.

I think the ultimate point is not to make impossible things possible, but to make already possible things easier, for both end users and app developers.

jrockway2• 17 years ago

I don't see what's stopping him from forking Linux and glibc. If this is really better, everyone will just use his fork. (It worked for my fork of cperl-mode, anyway...)

FooBarWidget4• 17 years ago

The point is to get patches accepted without forking. Linux and glibc aren't exactly small. Do you honestly see a single man maintaining millions of lines of forked code as well as handling packaging and stuff?

tamas• 17 years ago

It's not as bad as it sounds, basically great many of the http://git.kernel.org/ repos are actual forks of the kernel, regularly pulling changes from vanilla, resolving the occasional conflicts and maintaining/developing their own patches.

etherealG• 17 years ago

don't deny the fork. the fork is good. use the source luke.

jrockway2• 17 years ago

"git pull --rebase".

coliveira• 17 years ago

I think this guy is trying to solve a problem that doesn't exist in Linux. In Linux, every piece of software is compile for the architecture you want, so there is no need for a fat binary. Mac OS X needed this because they Mac users mostly run commercial software, so it is better to have a single binary for all platforms.

djcapelis3• 17 years ago

As someone who has actually built binaries to run across multiple distros, this doesn't address even half the issues. Very few people are concerned about binaries that run on more than one architecture when distributing binaries that run on more than one distro is a harder problem.

This solution doesn't solve the hard problems, but solves an easy one in an uninspired fashion. Rejecting these patches wasn't a case of holding back innovation, but merely holding back solutions that more experienced developers felt were not appropriate for the platform.

There was a good reason for that and playing armchair critic after reading a sympathetic story from a developer who is understandably hurt after getting his feature rejected just doesn't help anyone.

The kernel is more well managed than most people give it credit for, and it's exactly because of this that hacky incremental solutions get rejected, even when they might legitimately produce a small benefit. Eventually the overall solution to the problem FatElf is trying to solve is going to have to be something different.

I feel sorry for the developer, no one likes getting a feature rejected. But it was the wrong solution and the technical merits must trump other concerns.

antonovka6• 17 years ago

But it was the wrong solution and the technical merits must trump other concerns.

Your post is very long, but I'm curious to hear why universal binaries are the wrong solution?

On the Mac, it's the cornerstone of their transparent multi-architecture support.

Apple (and previously NeXT) inarguably demonstrated its value and ease-of-use over the past 20 years. Producing x86-32, x86-64, armv6, armv7, and PPC binaries all from the same source code -- and having them work the first time -- is incredibly simple on Mac OS X.

NeXTSTEP (the OS) ran on 68k, x86, PA-RISC, and Sparc processors, and it was equally easy to build software that ran on those machines.

djcapelis3• 17 years ago

OS X has a unified set of libraries to work with, Linux does not. This means the problem is much more complex. Creating a binary that simply runs on different distributions all on the same architecture is a fundamentally hard problem. Simply stacking together binaries for multiple architectures together doesn't result in a universal binary. At worst you have to stack together different binaries for different combinations of libraries and different architectures and if you assume even just the latest versions only of the top 5 distributions running on the top 5 supported architectures for Linux you may well end up with 25 different binaries that FatELF would glue together resulting in every program being a cool 2500% larger. This is unacceptable in terms of space and unacceptable in that it still isn't universal after all that as not all arches are supported and you only get the latest versions of the popular distros.

This is why I claim FatELF does not solve the hard problems with universal binaries. The only thing it might solve is someone could distribute a binary for a particular distro that could run on multiple architectures for that distro, but they can already do this with shell scripts and this is an uncommon case and most people would just rather download and store the one appropriate binary.

(This last is easy with a functional package management system. You should keep in mind that OS X does not have a functional complete package management scheme that allows users to discover, acquire and manage 3rd party packages. Not all Linux distros do either, but those that do are less in need of universal binaries. This is really a separate discussion though that gets to the heart of how software ought to be distributed. There are many blind spots in OS X's software strategy some of which Sparkle is making a half-baked (not sparkle's fault) attempt to fix.)[1]

At the end of the day you can't simply assume that the solution for OS X is the right one for Linux and you'd make a much bigger mistake assuming that the developers who rejected these patches weren't both entirely aware of OS X's excellent use of this technique and were doing this for anything but sound technical reasons. (I.E. It's not just NIH, they would happily adopt this if this was a reasonable solution to Linux's unique[2] challenges with binary compatibility across "Linux")

[1] There are many things wrong with this and most package management systems do not incorporate third party repos/packages as well as they should, which IMO is the larger problem here. Not that corporations should have to manage repos for every distro, but that becomes a separate mess.

[2] Perhaps read as special.

Aside #432: As you noticed my replies can often be anything but concise. I apologize if that makes things hard to follow.

djcapelis3• 17 years ago

I re-read your post and wanted to clarify what you meant by:

  Apple (and previously NeXT) inarguably demonstrated its
  value and ease-of-use over the past 20 years. Producing
  x86-32, x86-64, armv6, armv7, and PPC binaries all from the
  same source code -- and having them work the first time --
  is incredibly simple on Mac OS X.

Presumably you aren't under the impression that FatELF has anything to do with making source code easily compile to more targets. FatELF merely allows you to combine the binaries that would result from taking a source code base and compiling a binary for one arch and then another and gluing those together into one file.

The wording on your post wasn't entirely clear, though I'm guessing we're on the same page.

FatELF is no more. Linux developer community owned Ryan Gordon.

Related Stories