LLD is included in the upcoming LLVM 4.0 release

https://reviews.llvm.org/D29539

by AndyKelley • 9 years ago

226 69 9 years ago

reviews.llvm.org

smlacy• 9 years ago

lld is the LLVM linker. More details here:

https://lld.llvm.org/

AndyKelley4• 9 years ago

Posting this news article is my way to get the attention of Debian, Arch, Gentoo, Homebrew, NixOS, and other package maintainers. As an upstream compiler developer [1], I want to depend on LLD instead of the system linker, and the sooner LLD becomes ubiquitous in the various package managers, the sooner I can depend on it. (Of course I could ask my users to compile it from source, but that increases overhead of people getting started.)

Further evidence that it is time for LLD to be distributed along with LLVM: http://lists.llvm.org/pipermail/llvm-dev/2017-February/11030...

[1] http://ziglang.org/

woodrowbarlow• 9 years ago

can you elaborate on the advantages offered by using LLD over the system linker? is it purely for the sake of consistency, or does this decrease the size of your compiler's frontend component?

MaulingMonkey• 9 years ago

can you elaborate on the advantages offered by using LLD over the system linker?

Performance.

> (usually more than 2x faster than GNU gold).

I switched from bfd to gold to realize a ~6x performance gain on a project (~1 minute linking -> ~10 seconds linking, times 10-20 subprojects, times a few build configurations - really adds up!) I'll keep in mind that I now also have the option to switch to LLD for another 2x (12x total?)

AndyKelley4• 9 years ago

To me, the biggest advantage is cross compiling.

I started working on a proof of concept operating system, and according to the OS Dev Wiki, the first thing you have to do to get started is to build a cross compiler. This is because by default the C compiler and linker are compiled only for the current system. When I started making the OS in Zig, however, I did not have to build a special cross compilation version of Zig, because it supports cross compilation out of the box for every target and OS. For example, unlike in C, compiler_rt/libgcc are shipped with the compiler in source form and built lazily for the desired target.

So not having to build or obtain a special gcc was nice, however, I did have to build a cross compiled GNU binutils so that I could use an ARM linker on an x86_64 host. If I can depend on LDD, then I can support cross compiling for every target with no other dependencies needed.

X86BSD• 9 years ago

The amount of work it took to replace the gnu linker with LLD in freebsd was non trivial. But it's just about ready to be turned on in base.

rleigh• 9 years ago

I'm really looking forward to this. I've hit the 2^16 symbol limit in the ancient GNU ld in base several times over the last few years (the version in ports is fine). It will be great to have a default linker without such 16-bit limits.

kibwen2• 9 years ago

As someone mildly informed of Rust language development, I can confirm that they as well would be over the moon if LLD were usable on all LLVM targets. It's been a long, long anticipated feature.

umanwizard• 9 years ago

Package maintainers aren't like the President. You don't need news media to get their attention. You can just e-mail them.

AndyKelley4• 9 years ago

That sounds like an O(N) solution.

More seriously, being a package maintainer myself I know that the goal is to serve the users and create packages that users expect. So, the best way to get package maintainers to include LLD is to get users to expect it.

pantalaimon• 9 years ago

Well they kind of are, it's up to them to decide what kind of software millions of users are able to install.

ktta• 9 years ago

Link to page explaining LLD

http://lld.llvm.org/NewLLD.html

jey• 9 years ago

EDIT: Disregard; See https://news.ycombinator.com/item?id=13671078

Actually, the page you linked seems to be about an older ELF and COFF specific implementation, despite its URL.

Here's an overview of what makes the new one special: https://lld.llvm.org/design.html

Joky5• 9 years ago

Actually no: `lld` is not a single linker, there has been a split between Elf/COFF folks and the MachO ones. The page you're linking is about the MachO project which is not very actively developed.

haberman3• 9 years ago

I am pretty familiar with object file formats, and I don't get how this "atom" stuff works.

The site says that "atoms" are an improvement over simple section interlacing. But I don't get how you are going to make this leap without changing the object file format. Linkers work on the section level because that is how object files work. Object files have sections, not atoms. Compilers emit sections as their basic, atomic unit of output. Within a section, the code will assume that all offsets referring to other parts of the section will be stable, so you can't chop a section apart without breaking the code.

How does the new linker work in terms of this new "atom" abstraction without changing the underlying object file format?

haberman3• 9 years ago

Wow, this is claiming 1.2x to 2x faster than gold at 10% of the code size. That is a remarkable achievement. Gold was already faster than GNU ld.

I am really surprised that there was this much room for optimization. Ian Lance Taylor who wrote Gold is a really smart guy, and speed was one of the primary goals.

jblow3• 9 years ago

Linkers are mind-bogglingly slow. I don't understand why they are so slow.

lld is still slow, it is just less slow than the other linkers.

This is not to disparage anyone working on linkers or say they are not smart. I think they just don't tend to be performance-oriented programmers, and culturally there has become some kind of ingrained acceptance of how much time it is okay for a linker to take.

stuckagain• 9 years ago

But does it do all of the link-time optimizations that gold can do?

alvarop2• 9 years ago

"The ELF support is in progress and is able to link large programs such as Clang or LLD itself. Unless your program depends on linker scripts, you can expect it to be linkable with LLD."

:( The one feature I need most for embedded development

rui3145• 9 years ago

The document is outdated and needs updating. Linker script support has improved a lot since 3.9, and that's why it can now link the entire FreeBSD base system including kernel. It's still in active development (e.g. it can't link Linux kernel yet) but it's not that bad now.

alvarop2• 9 years ago

Nice! I mostly want to use it on Cortex-M0/3/4 microcontrollers, which are much less complex.

nightcracker• 9 years ago

This can not come too soon. Looking forward to it.

nneonneo• 9 years ago

The LLD webpage touts a big speed improvement over gold, which is extremely commendable.

What I'm curious about, though, is memory usage. On relatively constrained systems, linking large projects can take ages due to swapping - anecdotally, linking is responsible for upwards of 80% of the compilation times I see for a certain very large software project on my developer machine (which, at 16GB RAM, isn't huge, but fairly typical). Worse, memory pressure from linking makes the process rather non-parallelizable, which also hurts throughput. Having a memory-efficient linker could significantly speed up compilation beyond just a 2x improvement in such environments.

rui3145• 9 years ago

I shouldn't be bad, although I didn't actually measure its heap usage. Allocating memory and copying data is slow, so in LLD we mmap input object files and use the files directly as much as possible. This increases process virtual memory size (because large files are kept mapped to memory), but the actual memory usage is much smaller than that.

wyldfire• 9 years ago

Of note is the fact that George Rimar, Dmitry Golovin et al are working on getting Linux to boot linked by LLD. They've making a lot of progress -- when he started it didn't even successfully link.

ThisIs_MyName2• 9 years ago

Woohoo, I've been using lld instead of GNU ld and gold for a while now and I'd love to have everyone who builds my code do the same.

fithisux• 9 years ago

Even though I don't care about usable Linux linker I care about a replacement for Microsoft's linker for windows. LDC is an example where you need the closed link.exe in order to link.

Hopefully all the llvm projects (lldb, libc++, ...) mature for windows in order to have a replacement for msvc's tools. Until then (and beyond then) GNU tools gcc + ld will be my go to tools if msvc compatibility is not a constraint. They work perfectly well with Golang. But it happens people to release msvc link libraries. Even dmd for windows-x64 needs microsoft linker to work). Unilink is an option, but its closed source. And openwatcom-v2 will take time to get there.

AceJohnny2• 9 years ago

This is not the "atom-based" LLD http://lld.llvm.org/AtomLLD.html

While conventional linkers work at the compilation-unit level (one source file, usually), placing that whole source file's functions adjacently in memory [1], an atom-based linker is able to take the smallest linkable units (individual functions, each static/global variable...), and arrange those optimally.

As I recall, the OS X ld is based on this model. However it remains more limited as it doesn't support GNU ld's linker scripts and only limited command-line parameters, so it's not exposing all the power of the flexibility it would provide.

As far as I know, AtomLLD remains an experimental project with only 1 or two people working part-time on it.

[1] although modern linkers also add LTO (Link-Time Optimization) to rearrange things after everything's been integrated/

chisophugis6• 9 years ago

While conventional linkers work at the compilation-unit level (one source file, usually), placing that whole source file's functions adjacently in memory [1], an atom-based linker is able to take the smallest linkable units (individual functions, each static/global variable...), and arrange those optimally.

This isn't quite right. It's just that traditionally (i.e. when not using -ffunction-sections/-fdata-sections) when compilers output an ELF relocatable object, they group all the functions into a single "smallest linkable unit" (a single .text section) and so the linker can't actually do any reordering because the information that the functions are distinct has been lost.

I describe this in more detail here: http://lists.llvm.org/pipermail/llvm-dev/2016-December/10814... This was actually one of the key technical issues that prevented the Atom design from being used for ELF and COFF.

""" the ELF and COFF notion of "section" is a strict superset of its [the Atom LLD's] core "atom" abstraction (an indivisible chunk of data with a 1:1 correspondence with a symbol name). Therefore the entire design was impossible to use for those formats. In ELF and COFF, a section is decoupled from symbols, and there can be arbitrarily many symbols pointing to arbitrary parts of a section (whereas in MachO, "section" means something different; the analogous concept to an ELF/COFF section is basically "the thing that you get when you split a MachO section on symbol boundaries, assuming that the compiler/assembler hasn't relaxed relocations across those boundaries" which is not as powerful as ELF/COFF's notion). This resulted in severe contortions that ultimately made it untenable to keep working in that codebase. """

floatboth3• 9 years ago

Heh, speaking of linking individual functions, GHC Haskell has a flag to emit one object file per function and do that thing with a traditional linker.

It's horribly slow.

But it produces smaller binaries, I've got 2x smaller binaries on my projects with that option.

ThisIs_MyName2• 9 years ago

Doesn't LTO produce even smaller/faster binaries anyway?

mioelnir• 9 years ago

I'm just going to shamelessly quote the FreeBSD Quarterly Status Report, Q4/2016 that came out a few days ago.

   LLD developers made significant progress over the last quarter. With
   changes committed to both LLD and FreeBSD we reached a major milestone:
   it is now possible to link the entire FreeBSD/amd64 base system (kernel
   and userland) with LLD.

   Now that the base system links with LLD, we have started investigating
   linking applications in the ports tree with LLD. Through this process
   we are identifying limitations or bugs in both LLD and a number of
   FreeBSD ports. With a few work-in-progress patches we can link
   approximately 95% of the ports collection with LLD on amd64.

That is 95% of ~27000 ports. Which means for very many things, lld as linker will just work once those patches have been applied at the appropriate places.

[1] https://lists.freebsd.org/pipermail/freebsd-announce/2017-Fe...

gigatexal• 9 years ago

For the uninitiated what is LLD? What does it provide/ are it's benefits?

LLD is included in the upcoming LLVM 4.0 release

Related Stories