Rich Felker of musl libc comments on Google's LLVM libc proposal

http://lists.llvm.org/pipermail/llvm-dev/2019-June/133308.html

by maximilianroos • 7 years ago

387 155 7 years ago

lists.llvm.org

new_realist• 7 years ago

Why not start with musl?

gpm• 7 years ago

This is a common complaint about companies starting new open source projects, and it really annoys me.

Anyone who works with software knows there are tons of very good reasons why you might want to start a new project from scratch instead of contributing to an existing one. You might find the existing architecture limiting. The trade offs taken to be non optimal (for your use case or in general). Consider the existing code to be low quality. The code contribution process to lax (introducing bugs) or strict (slowing work). Find the community toxic. Want to utilize existing internal code bases. Etc.

There are tons of valid reasons to make a new project, and for most of them you don't want to say it out loud. Saying "your code sucks" or "we don't like you (i.e. toxic community)" is rude. Saying "we don't like the processes" is asking for a long political fight, and likely to be taken as rude by the people who invented them. Saying that they want to utilize internal code bases could be problematic internally. Etc.

In this case I get the feeling that quite a bit of it is disagreeing with the tradeoff's musl took (see: Wanting to make libc more like a library), which has to be about the only reason you are allowed to publicly say. Maybe there are other reasons as well, maybe not.

Sorry to jump down your throat, it's just a common type of comment that I wanted to give a general reply to. (Edit: To be clear I'm not involved with this project, or a google employee for that matter)

Pyxl1012• 7 years ago

When a new project is proposing to do something that's been done before, potentially multiple times, I think it is reasonable to want to understand how the new project will compare and contrast to work that's already been done.

Asking "How will this project compare to <existing implementation>?" seems quite reasonable to me. In fact, I would expect the person making the proposal to address this question as part of the proposal. It is not desirable to reinvent the wheel without good reason.

It's totally fine if the reason is something like: we want to use a different license, or a different development process, or a different coding style, or whatever. Whatever the actual motivations are, clarifying them up front will be good for everyone.

nine_k• 7 years ago

This is reasonable.

A simple answer like "we are not starting with musl because ..." would be quite welcome, though.

g82918• 7 years ago

I agree this with as some one working at a company that avoids most open source like the plague. But in this particular case, the code quality in musl is pretty good. I haven't had an issue with it so far, and after trying to upstream some things in glibc I have to say I can't see why they wouldn't use musl.

hinkley2• 7 years ago

Yes, there are reasons to rewrite a library, but people expect you to tell them why you are rewriting it.

So... why are they rewriting it?

MichaelSalib• 7 years ago

Which do you think looks better on your promotion packet?

1. Write a new libc and get it used all over Google, generating lots of users automatically; or

2. Shepard Google's adoption of code some other folks wrote.

chandlerc10245• 7 years ago

Either one looks completely fantastic, I assure you.

I have helped people on my team write promotion packets in both forms. Both worked perfectly well.

DannyBee3• 7 years ago

I don't think this level of cynicism is warranted

However, as the person who actually runs the relevant promo committees for the org, both would be fine, i'd care a lot more about why the choice was made.

netheril96• 7 years ago

Option 1 will take a long time. Writing a libc takes a long time, and transitioning the whole of Google codebase to a new libc takes even longer to iron out all of the incompatibilities.

If one goes with option 2, he/she may have many more achievements to write on his/her promotion packet during the time he/she would have spent implementing a new libc. So I hardly think promotion is the incentive here.

mda• 7 years ago

Instead of resorting to theoretical rhetorics on meta issues on the given proposal,Rich Felker could ask this question as well. But he did not, I am actually quite disappointed with his response, it looks unprofessional.

kbumsik• 7 years ago

Doesn't google use newlib already?

altmind• 7 years ago

Wait, weren't they using Bionic on android?

murderfs5• 7 years ago

Google is too big for the left hand to know what the right hand is doing. Various parts of Google use bionic, newlib, musl, and multiple forks of glibc.

nemothekid• 7 years ago

What does “LLVM implementation of Libc” mean here? Does that just mean an implementation of libc maintained by the LLVM project?

fourier_mode4• 7 years ago

Yeah, I am also confused. Isn't there Clang already? (Please correct me if I am wrong)

wyldfire4• 7 years ago

The C library is generally considered a part of the OS and the overall toolchain. Clang is "merely" a C compiler.

The C library provides implementations for the platform-neutral language functions like "open/read/write/close/ioctl/dup/dup2" and more.

Pyxl1012• 7 years ago

Clang is a C language family frontend for LLVM. That is, it's a compiler for C and C++ and related languages.

Libc is the C standard library, and there are multiple implementations of it, e.g. glibc and musl. Clang does not include an implementation of libc, though it does have an implementation of the C++ standard library called libc++: https://libcxx.llvm.org/

It's not immediately clear to me why an LLVM implementation of the C++ standard library would be desirable while an implementation of the C standard library would be undesirable.

noncoml• 7 years ago

Sounds like an LLVM "recommended" libc?

Loved the third point in Rich's reply:

"there is tremendous value in non-monoculture of libc implementations, or implementations of any important library interfaces or language runtimes. Likewise there's tremendous value in non-monoculture of tooling (compilers, linkers, etc.). Avoiding monoculture preserves the motivation for consensus-based standards processes rather than single-party control (see also: Chrome and what it's done to the web) and the motivation for people writing software to write to the standards rather than to a particular implementation. A big part of making that possible is clear delineation of roles between parts of the toolchain and runtime, with well-defined interface boundaries. Some folks have told me that I should press LLVM to make musl the "LLVM libc" instead of whatever Google wants to do, but that misses the point: there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class"."

wyldfire4• 7 years ago

there is tremendous value in non-monoculture of libc implementations
any one library implementation that's "first class" for use with LLVM while others are only "second class"."

I agree with Zachary here. LLVM's really good at mixing and matching w/target libs IMO (and linkers too, while we're at it). FWIW lld and libc++/abi have been a part of the llvm project for a long time now and BFD ld and libstdc++ are still the defaults for the linux clang driver.

taeric2• 7 years ago

The benefits of not having a monoculture resonate heavily with me. I'm convinced it is in line with antifragility.

Odd that so much of the goals folks make involve building monocultures.

m0zg2• 7 years ago

This sounds like Android NDK development will get quite painful in a few years. It previously included a botched libc++ missing a bunch of features. This is libc, but I imagine subtle feature disparities and incompatibilities there will cause plenty of pain as well.

uep• 7 years ago

There are already some real pain points with bionic. I primarily write Linux code that sometimes has to run on Android (notably not any other UNIX). I've run into tons of problems porting to old versions of bionic, despite the kernel being the same.

mrpippy• 7 years ago

I don’t think the Android team is the interested party here, given the original poster’s interest only in x86_64

m0zg2• 7 years ago

Google doesn't do "only x86_64". Even when I was there (years ago) at a minimum all software had to run on PowerPC as well, and non-portable assembly/intrinsics required explicit approval from "Eng Elders" and a portable fallback. I wouldn't be surprised if ARM compat is also required now.

jdswain• 7 years ago

In my experience NDK development is quite painful now. Most of my NDK work is a little old now but I found the debugging process quite painful. This was for C++ code that was running on other platforms so almost all of the debugging was due to platform incompatibilities or bugs.

But I did notice in the linked article that they are only targeting x86_64, so I don't think this libc is targeted at Android.

pjmlp4• 7 years ago

Android NDK development was been quite painful since ever, to the point sometimes I miss Symbian, and Symbian C++ patterns were not to fall in love with.

Not sure about you, but for me it was felt like a 20% project that someone on the Android team does something when they happen to be free of work.

If it wasn't for JetBrains coming up with CLion we still wouldn't have a replacement for Eclipse CDT, multiple reboots of NDK build tools, no real story on how to share libraries between NDK projects, native APIs are C wrappers to C++ code that we wrap again in C++ for proper type safety, permissions or file access require JNI, no AndroidX/JetPack like libraries and the list goes on.

The IO talk was more of state of nation than anything else.

Apple and Microsoft platforms are definitly more C++ developer friendly regarding the overall tooling, and mixed language development support as well.

catern3• 7 years ago

One of the replies is this email from an LLVM developer working at Google: http://lists.llvm.org/pipermail/llvm-dev/2019-June/133315.ht...

I can't say it looks particularly good for someone at Google to be using their moderator powers on this unrelated-to-Google list to tell Felker not to include parts of his message which reflect badly on Google.

scarface74• 7 years ago

I see nothing wrong with this. The last thing you want in a technical discussion are petty personal attacks. The moderator sounds a lot like something you would see dang say on HN.

catern3• 7 years ago

I don't think they're petty personal attacks. They are important concerns which I share.

Regardless, since this particular member of the project has clear incentives to be biased in favor of Google, they should have recused themselves, and asked another member of the project/moderator of the list to judge whether the message was appropriate.

hyperpape• 7 years ago

Felker's comment was that a big corporation might release botched code that contained hacks, but their influence would mean developers would be stuck coding against it.

If that's insulting at all, it's several levels less insulting than the sort of thing I see dang moderating.

mindslight• 7 years ago

But the criticism had nothing to do with "petty personal attacks", rather it is just tone policing mildly blunt language.

falsedan• 7 years ago

There's a clear conflict of interest: moderating comments critical of the organisation who employs you can easily be considered biased to a disinterested party. It doesn't matter if there is bias or not, the possibility that people will see it as biased is enough for people to recognise it and lost trust. This kind of moderation should be done by a disinterested party, and the Google employee should recuse themselves from addressing it (just refer it to another moderator).

Guess who just finished their Conflict of Interest yearly refresher!

ddevault• 7 years ago

Copy-pasting a comment from "cup" on lobsters, because I think it's on-point:

1000 times yes. If Google wants their own libc that’s their business. But LLVM should not be part of their “manifest destiny”. The corporatization of OSS is a scary prospect, and should be called out loud and clear like this every time it’s attempted.

https://lobste.rs/s/dgmoyi/response_libc_llvm#c_hg8dvg

favorited2• 7 years ago

There has been huge corporate involvement and investment in LLVM from its earliest days as anything beyond a research project.

When Apple open-sourced Clang, or their C++ standard library implementation and contributed them to the LLVM project, was that something to be afraid of as well? It seems to have gone pretty well.

svnpenn• 7 years ago

Fair point, but Google is not Apple. Google has a history of railroading standards to suit their needs, to the point where the standard becomes "what Google is doing". See the Chrome project.

omeid2• 7 years ago

Unfortunately, the good deeds does not protect from bad consequences.

Google has shown little regard to standards and community, specially as of recent, based on their handling of the musl libc in fuchsia, libc++ in Android, Go the language and it's community, and the elephant in the room: Google Chrome. Google has a bad culture of forking things and breaking things for good business reasons which does not align with the community and people outside Google.

The concerns around Google fucking things up with their influence is not unreasonable.

hermitdev• 7 years ago

I'm only a casual observer of LLVM, and have really only used tools such as clang-format.

That said, I haven't observed Apple applying a heavy hand in regards to LLVM/Clang development.

As a C++ developer, clang has been a god send, amd not just for Clang users. For a long while, they had the best error messages around. Actually truly pointing you to precisely where the error is and giving meaningful errors. For a long fucking time, C++ compilers gave really verbose, shitty error messages that took a whole other level of understanding to decipher, which really hindered novices from progressing. Clang put pressure on GCC and Visual Studio to up their game and produce useful errors during compilation.

It was bad enough at one point, I used to present interview candidates with example code that failed to compile and the compiler output and ask the candidatea to fix the code (it was always a trivial mistake in the code involving templates, bit it'd generate a few screen fulls of errors). It wasnt my sole judge of whether I'd recommend them or not, but it was a good indicator of how well they understood the C++ environment and how to decipher compiler errors (sample program was less than 20 lines, but generated at least 3 screen full of errors due to template spewage).

chandlerc10245• 7 years ago

Just so we're completely clear, Google has been an incredibly long-term contributor to basically all parts of LLVM. So have sooooo many other companies. This is a really cool project IMO, but it doesn't represent a significant change in who or how much anyone is contributing to LLVM... It's a drop in the bucket.

And for the record, LLVM is doing fine with all these corporate contributors.

I'm not trying to say that this can't be a concern for OSS communities and projects, but LLVM seems to have a quite successful and sustainable model here.

microtonal• 7 years ago

The problem is that neither the original thread, nor the threads here (which seem to be quite heavy on Google commenters) answers the question 'why?'.

I think this wouldn't have caused nearly as much commotion as it does now, if the original post explained the technical reasons for starting a new libc. What problems do existing implementations have? How could a new libc address these issues?

Instead, this has an air of (regardless whether this is true): we are going to write a new libc for completely opaque company-driven reasons, and either you accept it or you we will make it happen anyway.

fourier_mode4• 7 years ago

I think its primarily because Chris Lattner[0] is at Google now. I wonder even if they pull out this effort how much of that would still be maintained if Lattner changes jobs.

[0] https://en.wikipedia.org/wiki/Chris_Lattner

favorited2• 7 years ago

Chris Lattner said that he has nothing to do with this project

http://lists.llvm.org/pipermail/llvm-dev/2019-June/133274.ht...

theresistor• 7 years ago

No. Chris is working in the AI/ML space, and specifically commented in this thread that he is not involved with this project.

CUViper• 7 years ago

FWIW Roland McGrath, the original glibc author, also works at Google. But of course not every possible person will be involved here, no matter how relevant.

jasonhansel• 7 years ago

Filed under "This is why we need the GPL"

flukus• 7 years ago

This. I definitely sounds like they want a libc that they are free to fork and add proprietary incompatibilities to, which they can do with the GNU libc (under LGPL) as long as they don't want to distribute it, so we can assume they want to distribute it and that this is a further closing of their consumer tech like android. I would've though musl would be a better starting point for them.

taneq• 7 years ago

How does the GPL help with this? Google could build their own libc replacement, and if they push it hard enough to gain widespread community support, it could become the (or at least, a) new standard. At that point Google is free to do whatever they want with it as long as they make it useful enough that the community won't abandon it. The GPL just means that it could be forked later, but if Google is still putting significant resources into developing the main branch then no-one will switch to the forks.

izacus3• 7 years ago

GPL is literally made to allow forks like these, what exactly did you mean with your post?!

ajxs2• 7 years ago

A corporate-led project does not have to answer to the community, and will leave whatever bugs they introduce in place for the sake of bug-compatibility with their own software rather than fixing them.

This speaks volumes very clearly. This highlights an immense hazard. Enterprise scale companies contributing to open-source is a fantastic thing, but enterprise scale companies thrusting their own proprietary libraries onto the open-source world is not. I'm already actively avoiding becoming beholden to Google in my work as it is already, let alone in the world where important software uses a libc written by Google.

If you're not concerned by this, refer to the immense power that Google already wields over the extremely ubiquitous web-standards through the market dominance that Chrome has.

jchw5• 7 years ago

(Disclosure, Googler here, not working on projects related to any of the things being discussed, speaking in a purely personal capacity.)

I hate to disagree with Rich Felker. I think most people have a lot of respect for musl and the amount of effort it took. But, I disagree with the basic conclusions.

For one thing, I don't think it's fair to compare this to Chrome, an end user product. libc is a fundamental piece of architecture, like (edit: as in, fundamental in stacks that they are used in, to clarify) protobufs, gRPC, Kubernetes, etc. I heavily sympathize with folks uneasy about the Chrome situation - like I've mentioned maybe hundreds of times now I use and recommend Firefox personally, but I feel this particular discussion will become more productive when that stops being a focal point.

Their other points are not really related, but I disagree with them also. I don't think there's a good motivation to keep bugs for backwards compatibility: Google runs non-Google software, too; keeping subtle incompatibilities could be a disaster. Also, Google has a pretty good corpus of code to test their libc on, so I think they have a unique opportunity to fast-track a pretty battle-tested libc (I did not know of this project before this Hacker News post, so this is purely a hot take, but for now I stand by it.)

No idea what their plan is, but open governance would probably work wonders here. After all, LLVM has plenty of corporate contributors, and it seems to do fine.

stochastic_monk• 7 years ago

I think trying to work with protobufs by analogy here fails because the scopes of power are so vastly different.

gumby• 7 years ago

The problem was stated in Felker's response: "...corporate development teams are uniquely qualified to utterly botch a libc, yet still push it into widespread use..."

This can come innocently as well. For a simple non-code example: google has a code style guide. For C++ it says "don't use exceptions" and even (or used to?) explains that the writers consider exceptions to be a good thing but they have enough legacy / third party code base that doesn't use them that they can't switch. Lots of people use the google style guide because it exists, rather than write one themselves, or rather than spend time hashing one out. So they end up shackling themselves to a constraint entirely resulting from an environment they are not in.

I see this indirectly in your own comment, where you casually referred to gRPC or protobufs as "fundamental piece[s] of architecture". Well they aren't standards, they are technologies that google likes. Protobufs is a perfect example of a technology in, well not widespread use, but is used by a bunch of projects, and as discussed elsewhere contains a number of decisions that reflect the google environment, probably not resulting in a great fit for a number of places where it is used.

There's nothing malign in that; the whole point of writing code is to meet your needs. But when you're an 800 pound gorilla, don't presume your every step is what everyone else needs. They are what you need and without proactive effort may not help many others....but can crowd out what could.

ajxs2• 7 years ago

Google has a pretty good corpus of code to test their libc on...

That is a good point, but I'm in no way disputing that Google could do a great job of creating their own libc. I would never be foolish enough to challenge the merit of Google's engineers, the proof of this is clear in the tasting of the pudding that is Google's software. My concerns lie in the open-source community becoming further beholden to Google, or even worse with Google dictating the direction of development on what could become a cornerstone of the architecture of many critical pieces of software.

EugeneOZ• 7 years ago

In the beginning of Google's letter they let us understand they are going to create simplified version for their own needs. It does mean they don't care about compatibility and bugs, if it doesn't affect their software. That's not how this kind of libraries should be implemented.

fishnchips• 7 years ago

Also, Google has a pretty good corpus of code to test their libc on

I left Google more than 5 years ago but at the time the subset of the language (C++) one was allowed to use was quite limited. Testing your implementation of libc on that codebase could still lead to poor coverage.

ezoe• 7 years ago

Chrome, an end user product. libc is a fundamental piece of architecture

The web browsers is not the end user products anymore. The web sites(and these days, smartphone apps which is just a wrapper of the browser) are the end user products. The web browser is, just like libc, fundamental piece of architecture.

Sure, the users directly interact with web browsers GUI, but average users can't recognize the difference between the Browser, Internet, Google and Android. They are all same things to most of them.

In that sense, browser and libc are the same, all users use it without noticing it.

sporkland• 7 years ago

I'm not making a comment dressed as a question, I'm genuinely curious. Is this super different from the linux kernel policy that you don't break user space? Both in intent and in effect.

firethief• 7 years ago

In this scenario "userspace" refers to Google's code, and "breaking" it would be necessary to fix it for everyone else.

xvilka• 7 years ago

Quite relevant then: "Why I'm donating $150/month (10% of my income) to the musl libc project"[1].

[1] https://andrewkelley.me/post/why-donating-to-musl-libc-proje...

gok• 7 years ago

This would have been a much better email without the needless stone throwing from one libc maintainer's glass house.

cpeterso• 7 years ago

I'm surprised the proposed initial scope would include x86-64 but not ARMv7 or AArch64. That suggests Google's motivation here is their server software, not mobile devices running Android or Fuchsia.

Also, portability and consistency is an important attribute of a libc. Supporting multiple architectures from the beginning would help shape a portable design. For example, the original target processor for Windows NT development was the Intel i860 specially because it wasn't the standard i386 architecture. Fun fact: the Intel i860 was codenamed "N10" aka "N-Ten", which may have been the origin of the Windows "NT" moniker.

floatboth• 7 years ago

mobile devices running Android

They already have Bionic!

or Fuchsia

They already have a weird fork of musl that Rich described in the email!

typical Google "Not Invented in this Team" approach..

pjmlp4• 7 years ago

Bionic was a very hard child birth.

Check NDK roadmap to see how much years (yes years) has taken them between Android and NDK releases to provide a proper libc.

lone_haxx0r• 7 years ago

Google are going to make it with or without LLVM's cooperation.

If it's done with LLVM, it's highly probable that LLVM won't let them extend,extinguish libc. Google's code will be carefully evaluated by a relatively neutral party: LLVM.

If Google does it alone, they will write whatever code they want. They'll be free to extend, extinguish libc.

saagarjha4• 7 years ago

Why do you think Google has the ability to EEE libc by themselves but somehow be voluntarily reigned in by LLVM?

rurban4• 7 years ago

Rich is of course totally right. There's not only the botched musl in Fuchsia, also Android Bionic, which is a libc written in C++. It's utterly unusable. They had a better, older, safer libc implementation based on some HP code, but that is gone.

Some of Rich' points are moot though. Google already have their own libc's, in Android and Fuchsia with enhanced security measures. Similar to Huawei or Intel or Cisco. Better than Apple's freebsd libc. Now reaching out to the community to outsource their development efforts under llvm is a good thing. There's obviously a need, musl, bsd and glibc cannot address. I'm sympathetic to that.

wbl2• 7 years ago

And the need is? I read the thread and I still don't know.

_pmf_• 7 years ago

Make vs buy decisions work differently for companies with virtually unlimited buying power. The absolute control that doing it yourself provides cannot be bought. And Google has to care only the platforms it wants to support (as some guy at BSDcon said, you have Windows, Linux Intel/AMD/ARM and rounding errors).

Do you think the 100000 cookie cutter game developers for Android will stop developing for it because the platform libc has subtle issues? They rightfully don't give a shit.

eqvinox• 7 years ago

His 3rd argument is absolutely and utterly compelling and a total knockout against Google's approach to this IMHO:

Some folks have told me that I should press LLVM to make musl the "LLVM libc" instead of whatever Google wants to do, but that misses the point: there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class".

If they want to create a great libc - or fuck up another libc - they're much welcome to do that, but they need to go do it outside of LLVM.

Dumping a libc on LLVM is just (1.) appropriating credibility from LLVM, (2.) sticking others with the maintenance bill, and (3.) horseshit.

qalmakka• 7 years ago

This, and I also strongly argue that compiler implementations should be untied from standard libraries, and this also applies to libc++/libstdc++. A compiler shipping its runtime is a bad idea that totally screws up compatibility (just look at how painful it is to use MINGW + C++ on Windows because it can't link with VS's C++ Runtime - I understand that MS headers are crap to parse, but with a bit of effort clang now does it fine), and ties them together and prevents reuse. In an ideal world shipping a runtime should be the OS's role, not the compiler's, and the compiler should make use of whatever the platform ships, instead of having N compilers shipping N different, incompatible libraries. The interface between the library and the compiler would then become a true language standard, instead of "library for compiler version X.Y.Z"

nly2• 7 years ago

Raymond Chen wrote about this in his 2014 blog post "Windows is not a Microsoft Visual C/C++ Run-Time delivery channel"

https://devblogs.microsoft.com/oldnewthing/20140411-00/?p=12...

Stable ABIs are good, but the ability to break it and iterate/improve is even more so. GNU platforms are currently stuck with a shitty std::string implementation (32 bytes but only 15 bytes of SSO capacity) because breaking ABI means recompiling your entire Linux distro.

mehdix• 7 years ago

These are the highlights IMO:

Avoiding monoculture preserves the motivation for consensus-based standards processes rather than single-party control
there shouldn't be a "LLVM libc", or any one library implementation that's "first class" for use with LLVM while others are only "second class".

rjvehn• 7 years ago

Well glibc kind of proved that individual maintainers can be just as bad for stewardship[0].

[0] https://news.ycombinator.com/item?id=7914022

Rich Felker of musl libc comments on Google's LLVM libc proposal

Related Stories