60x speed-up of Linux “perf”

https://eighty-twenty.org/2021/09/09/perf-addr2line-speed-improvement

by tonyg • 5 years ago

477 215 5 years ago

junon6• 5 years ago

Yet another case of GPL virality causing more issues than it solves.

E: Downvoters, have you read the article? The program started a new process for each address lookup instead of using a library because that library is GPL.

Shall we have a discussion?

AstralStorm2• 5 years ago

No. That is the case of Linus Torvalds not using the standard "or later" clause. He has his (bad) reasons.

The incompatibility is caused by "lack of restrictions" clause. Without that, GPL becomes essentially BSD with all the corporate thievery that entails.

yakubin3• 5 years ago

The "or later" clause is a backdoor. Not wanting backdoors in your license doesn't seem like a bad reason to me.

kaikki• 5 years ago

What is a good source to get an understanding of all the Licenses and their relation?

kevincox• 5 years ago

"or later" is a hack to get around the fact that the GPL is basically incompatible (I need a better word because this is overloaded) with itself if you change the name. If you want to use GPL and GPLRenamed libraries in the same project the only option is to relicence the project under both licences. This may be possible if you have few dependencies and few authors. But if you want to depend on a new library after some authors have become uncontactable you basically can't.

josteink• 5 years ago

That is the case of Linus Torvalds not using the standard "or later" clause. He has his (bad) reasons.

Linus does not think GPLv3 is a good or fitting license for the Linux-kernel and that it changes too much things to be considered a new version of the same license as GPLv2.

So he refuses to add the “or later” backdoor which would effectively relicense “his” kernel with a license he does not approve of.

Is that not his right?

jraph• 5 years ago

Well, I don't think I agree with him and would licence GPL code with the "or later" clause, but he has a very reasonable and convincing argument against it.

See this video of him explaining his take on GPLv3 and the "or later" clause at Deb Conf if you are interested: https://www.youtube.com/watch?v=PaKIZ7gJlRU

dvdkhlng2• 5 years ago

IMO it's not "GPL virality" that is at the root of the issue here. The authors' decision to license theis works this way causes the problem. You can still ask both libraries' authors for relicensing their work in a less restricted / more compatible way.

If they don't want to do that, then being authors and owning their copyrights, that's their right to do. That's nothing I would blame on the GPL.

I mean we should be grateful that these authors made their works available free of cost in the first place.

for1nner• 5 years ago

The authors' decision to license theis works this way causes the problem.

I'm not sure how you can rationally make an argument for a framework that literally limits the use of improved or more performant code/technology/understanding simply because it does not share a philosophical view of IP/ownership/sharing.

This is the core of the problem with IP in the real world as well, and I find it mind-boggling that we don't look at the SYSTEM and see it as problematic.

nickitolas• 5 years ago

As I understand it relicensing in many open source projects can be a very complicated matter, since in some cases it requires getting permission from all past contributors (Which could number in the hundreds)

guerrilla• 5 years ago

Can someone ELI5 to me where the problem is because both perf and bfd seem to be GPL. Why would the former not be allowed to link to the latter?

Matthias1• 5 years ago

Sure, it's not the license itself. But this does read like a problem. It's difficult to imagine that the authors of libbfd licensed it under GPLv3 because they wanted to ensure that no one would link it against perf. Maybe they do have strong feelings about supporting that license.

But this is an example of the ugly side of open source licensing. A lot of people don't have strong feelings about ensuring the distribution terms of their code, and just don't care how it's used. And in those cases, it can be annoying to have e.g. opensource.org insist that GPLv3 is the best option.

Edit: To be more specific, I mean that this problem could be avoided if both softwares were released under a public-domain-equivelent license. But of course, that will never happen.

Dylan168075• 5 years ago

The difference in license only matters because of a broad interpretation of virality. The easy way to show this is to mentally replace GPL with LGPL and notice that the problem disappears.

Spivak• 5 years ago

It seems odd that the takeaway is "GPL sucks because it got in my way" and not "both authors' explicit wishes were respected." Assuming that this is in error and this isn't what the authors wanted to happen then it's fixable by someone changing their license.

ziml773• 5 years ago

It's fixable with a license change, but in practice that's impossible for large projects. Without a CLA, every individual who's contributed is a copyright owner. If you can't get every single one of them to agree to change it, then the license is stuck. And it doesn't matter if the reason you can't get them to agree is because they've become uncontactable or died.

lnxg33k14• 5 years ago

I guess MIT and Apache would be better, so that company can make billions on the code and contributors would still be poor (see AAPL -> FreeBSD)

NelsonMinar• 5 years ago

You understand there's nothing in the GPL that prevents companies from making billions on the code without compensating the contributors?

junon6• 5 years ago

There's nothing that prevents me from using GPL code commercially...

pathseeker• 5 years ago

Right, and GPL is making all of us open source devs rich. I can't fathom getting into open source and then getting mad when people use your tools to great success.

simion3142• 5 years ago

more issues than it solves.

Can you explain how you reach this conclusion? Assuming the GPL did not exist and the author would have made stuff proprietary, would that be better ?

Some people like me would share code with GPL3 or later, why the f* should we give it to you as BSD? are you running on your devices only BSD code or MIT code ? or are you running proprietary software but for some reason making all stuff BSD will make your job easier since you could mindlesly copy paste shit in your proprietary stuff?

adwn• 5 years ago

Assuming the GPL did not exist and the author would have made stuff proprietary, would that be better ?

That's a false dichotomy – there are many more open source licenses which wouldn't have led to the problem described in the article.

matheusmoreira5• 5 years ago

The root cause is copyright. If it were to be abolished, no one would ever have to think about licensing ever again. None of this pain would exist.

Think about all the time that would be saved when people no longer need to think about all this lawyer bullshit.

junon6• 5 years ago

Not sure if this is a pisstake on my argument, but a world without copyright (or something similar) would be a world of endless exploitation and stifled innovation.

olliej• 5 years ago

In the absence of copyright things like the GPL would have no power - a company could take your work and bundle it into their product and never contribute anything back ever.

bityard• 5 years ago

With the "viral" name-calling you have already demonstrated that you don't want a rational discussion.

Dylan168075• 5 years ago

No, using the normal term does not demonstrate that.

tlamponi• 5 years ago

  > E: Downvoters, have you read the article? The program started a new process for each address lookup instead
  > of using a library because that library is GPL.
  >
  > Shall we have a discussion?

Yeah no, they simply had a bad alternative implementation which could have been just as fast as the licence incompatible library call:

  >  non-bfd, without patch:  7m59s
  >  non-bfd, with patch:       15s
  >  bfd:                       15s

-- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911815#38

The same could have happened due to other reasons for alternative implementation, that range from availability of more than one implementation with different advantages and disadvantages, different OS the code needs to run on, or simply preference.

Blaming the license in this case is just short-sighted and with the wording used it just shows a bias of yours against the GPL, but not actual will to participate in a meaningful discussion on the linked thematic.

faho4• 5 years ago

Michał Sidor suggests building against libbfd, something that the Debian maintainers don’t want to do.

"don't want to" isn't quite correct - they can't.

Perf is licensed under the GPLv2, libbfd under the GPLv3. The licenses are incompatible, which makes the combination unredistributable - which is what Debian would be doing with it.

Debian is legally not allowed to do this.

AstralStorm2• 5 years ago

Unless perf has the standard clause allowing use of later versions of GPL. That would make it GPLv3 if linked against libbfd.

Unfortunately it does not.

xxpor2• 5 years ago

That's pretty typical for things in the kernel ecosystem.

compsciphd• 5 years ago

its arguable when it comes to dynamic shared libraries (i.e. the ZFS argument).

tonyg8• 5 years ago

You're right. I should clarify. Thank you!

varajelle• 5 years ago

Is there not an exception in the license for "system libraries" which libbfd kind of is?

faho4• 5 years ago

Is it? On my system the only things depending on binutils (which libbfd is part of) are gcc, clang and perf (seems archlinux doesn't care).

That makes it arguably not a system library, so it's fair that Debian would want to steer clear of that interpretation.

captainmuon• 5 years ago

If we are nitpicking, they can, there is nothing material stopping them. It is "just" at odds with copyright law.

In a society with different laws, there would be no problem. I wonder what would be greater - the productivity gain when we would not have to make workarounds due to licensing, and when we could just legally access the source code of everything. Or the loss, because people might have less incentive to innovate, or they would just keep the source hidden.

pphysch• 5 years ago

If only we were permitted to view the digital IP rights tyranny as another disruptable industry with latent energy that could be unlocked by a savvy founder...

The economic activity & value directly suppressed by IPR is unbelievable. Hard to even mentally scratch the surface. It's extremely economical (for society) to digitally copy and distribute useful information.

Most Western IP laws are draconic in the purest sense of the word.

Unfortunately their is a very influential sect of "corporate-law fearing" IP fanatics at the heart of many FOSS tech traditions (RMS, et al). Not all of it is pragmatic.

ziml773• 5 years ago

Is it just me or is the incompatibility especially absurd given that perf is currently using libbfd, just 1 step removed via addr2line?

I wonder if a patch to addr2line would be accepted which allows the executable to stay running to accept multiple requests through stdin.

tonyg8• 5 years ago

addr2line already does this, and it's exactly what TFA is making use of to get the speedup.

lathiat• 5 years ago

This is kind of the crux of a long history of these licensing arguments. That dynamic linking is problematic because it potentially creates a derived work. And then you might distribute that derived work.

But shelling out to something or in some cases having an automated script create or download the derived work automatically is sometimes acceptable.

It’s a bit of a silly line in some ways. But it’s a long story.

tverbeure3• 5 years ago

I’ve wondered about similar situations.

Take Yosys, an open source synthesis platform that allows external commands by adding shred libraries. It’s MIT licensed.

There is the Yosys-GHDL plug-in that allows using VHDL instead of Verilog compilation. It’s an independent open source project that can’t be merged into the main Yosys tree because GHDL itself is GPL 2.0.

Is the author of that plug-in violating the GHDL license? The plug-in glue code is GPL2 as well.

Can the GHDL authors ask the plug-in authors to take down their code (which has been forked many times on GitHub, of course.)

fiddlerwoaroof• 5 years ago

My understanding is that there’s only a problem if the distributor combines the two things: if the main application is one license and the plug-in a different one, it’s not a license violation for the end-user to combine the two (unless they turn around and become a distributor in some way).

geofft2• 5 years ago

The plugin is potentially a derivative work of both Yosys and GHDL, but that's fine - it's possible to comply with the requirements of both the MIT license and the GPLv2 at the same time. The resulting work is GPLv2-licensed. (To be precise, you also need to preserve any copyright notices from the MIT-licensed project and the text of the MIT license. The MIT license has only one condition, but you do need to follow it, same as if you were building proprietary software with an MIT-licensed component.) If anyone distributes Yosys with the plugin, the combined work must also be distributed under GPLv2, which is also fine.

In this case, perf is (like the Linux kernel, in whose git tree it lives) GPLv2-only, and modern versions of libbfd are GPLv3-or-later, and it's not possible to comply with both at the same time. The GPLv2 has a "no additional restrictions" clause, and the GPLv3 asks for things the GPLv2 does not. So a distributor of a combined system including perf built against libbfd (like a Linux distro) cannot comply with the licenses.

(I suppose perf was written against the libbfd API back when libbfd was GPLv2, avoiding the question of whether perf is an illegal derivative work of libbfd. Or the backtracing API is de minimis, which means the source is fine but a compiled binary as par of a distro that also includes a modern libbfd is not.)

One possible answer here is to ask the perf authors to relicense under GPLv2 or later. Many years ago, at a startup that no longer exists, I wanted to reuse code from the Linux kernel "dm-verity" module in GRUB, which had moved to GPLv3, and I got a pretty quick answer from Red Hat (who was the only copyright holder for those files) saying that would be fine.

anticristi2• 5 years ago

Kind of ironic that RMS campained against proprietary software to allow reuse of code. Now we have two islands of open source code ... .

josefx2• 5 years ago

Two? Are the AGPL licenses compatible? How about GPL licenses with exceptions for some APIs as used by OpenJDK?

drran2• 5 years ago

RMS advises to use "GPLv2 or any later version", which is compatible with GPLv3, instead of "GPLv2 only" license.

scottlamb3• 5 years ago

At least you can legally build it locally, just not distribute the result. That's still better than what's possible with proprietary software.

As drran mentioned, I'm sure RMS would say the problem here is that Linux is licensed under v2 only. Linus of course would disagree.

LeoPanthera• 5 years ago

Debian could provide a tool which builds it for you, on your system, right? The resulting binary would not be redistributable but you could use it.

I've often wondered why more linux distros don't provide a similar tool to build ZFS into the kernel for you.

mappu• 5 years ago

Debian nicely offers this with the zfs-dkms package.

Ubuntu just bundles ZFS into the kernel statically - not even as a loadable module - and damn the licensing implications(!).

spockz• 5 years ago

So this means the distributor can fix this by building this package locally and use the result to link other binaries to? This isn’t uncommon right?

rossmohax• 5 years ago

DKMS style linking at install time might be an option, but I've never seen it used in practice

roelschroeven• 5 years ago

How do other distributions handle this? Do they also have a slow perf? Or do they just link perf with libbfd regardless of the incompatible licenses?

vlovich1233• 5 years ago

I haven't blogged about this yet, but we saw a 1000x fold speed-up doing several things around symbolication. The more optimal approach we found was to use the gimli crate[1] directly & carefully optimize it to read in the data structures for the executable(s) you are symbolicating upfront & then issuing in-process queries. They also have a drop-in replacement of addr2line that outperforms it (both in symbolication speed & memory usage).

I am curious about how that compares with libbfd since that wasn't under consideration for us as it uses GPLv3.

[1] https://github.com/gimli-rs/gimli

[2] https://github.com/gimli-rs/addr2line

tonyg8• 5 years ago

Steinar Gunderson (who suggested [1] the talk-through-a-pipe approach) just now compared [2] the `libbfd` variant to the pipe-to-`addr2line` version, and found them to take similar amounts of time.

This agrees with what I saw in `top` while testing: with the patch, I see `perf` using ~95% CPU, with `addr2line` using the remaining ~5%.

So speeding up `addr2line` probably wouldn't result in very much of an overall improvement for this workload.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911815#28 [2]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911815#38

vlovich1233• 5 years ago

The piping is a good chunk of the gain but the gimli library is still much faster than addr2line & uses less memory if used correctly. Since libbfd shows similar results, I'm thinking there might be a speed bump from switching to use it (some care has to be taken to preserve 0-copy semantics across FFI).

vient• 5 years ago

Looks interesting. I've encountered some annoying bugs with latest addr2line when trying to get symbols for addresses, wonder how gimli performs in that sense. Also, it seems that addr2line 2.37 has some performance regressions compared to 2.36.1 which I didn't look at yet.

vlovich1233• 5 years ago

In my experience gimli also symbolicated things addr2line couldn't.

tyingq• 5 years ago

Apparently llvm has a drop-in replacement for addr2line: https://llvm.org/docs/CommandGuide/llvm-addr2line.html

I suppose you could make a library out of that.

nonbirithm3• 5 years ago

It is a tough feeling to see someone else's design for a library that would be literally perfect for my needs, but I'm unable to use it because of the license, so I have to spend weeks implementing my own inferior version while carefully avoiding making the code too similar to what I happen to remember.

I'm past believing that I'm smart enough to always be able to come up with/reimplement a competent enough solution for all my needs by myself every time. But copyright still has to be respected.

When I'm forced into that situation from licensing issues, it really makes me feel like my skillset primarily revolves around gluing together other people's well-designed code, and being left to my own devices exposes my weakness in implementing something from scratch.

adsche• 5 years ago

Curious, what license(s) prevented you from using a library?

Or maybe rather, what does "use" mean here?

nonbirithm3• 5 years ago

GPL, and in some cases source-available codebases produced by decompilation.

"Use" in this case means to have some feature that's implemented by those libraries reimplemented in an open-source project (MIT-licensed).

ShrigmaMale• 5 years ago

GPLv2, which is viral.

jeltz3• 5 years ago

GPL by some quite popular interpretations of what derivate work means. I do not personally agree but those interpretations are common enough to be a real concern and I do not think it has been tested in court yet (GPL has been tested in court but I do not think this particular aspect has).

matheusmoreira5• 5 years ago

carefully avoiding making the code too similar to what I happen to remember

Is this really necessary? The GPL allows you to read and study the software. Is it really copyright infringement if you take your understanding and make your own program? Are people really forced to come up with convoluted new ways to solve the same problem just to avoid any similarity to existing work?

Salgat2• 5 years ago

The issue is that the only way to guarantee no liability is a clean room implementation. It's why most open source reimplementations require no mention or examination of the source/machine code. Google got dinged over a trivial method they copied in their java implementation, even though it was extremely trivial. You can't risk any chance of copying, regardless of how trivial the code.

jdavis703• 5 years ago

Think about it this way. If a panel of jurists who only half understood the expert testimony saw your code side by side with other code would it look like blatant copying or not?

toast03• 5 years ago

It depends on what derivative work means, and finding out the specifics may mean spending time in court.

GPLv3 says 'To “modify” a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a “modified version” of the earlier work or a work “based on” the earlier work.'

If you read and study a work, and then write something new that does the same thing, that may be considered adapting or copying from (parts of) the original work.

nonbirithm3• 5 years ago

What I wasn't sure of is, as long as I never actually incorporate the original code, if basing your fundamental design on something counts as infringement. I find it hard to see the difference between taking some class definitions and changing the variable names and going off to use what you learned from other people only to realize you arrived at the same design as they did because doing anything else would mean the implementation would be worse than what you now know it could be.

I don't have much of an understanding of copyright when it comes to cases like those, but my worry is that declaring that your open-source codebase is merely inspired by codebase X with this specific code in the case of these specific data structures would still count as infringement in a court of law. However, I don't intend to infringe on the licenses of the original, and do not intend to directly copy any code.

troutwine• 5 years ago

Dunno! For what it's worth I build a lot of MIT and MPL software professionally and the lawyers at several employers have suggested doing exactly what the GP does. My assumption has been that proving derivation in a court is an expensive process however it turns out, so why risk it.

kmeisthax• 5 years ago

No, what you're describing is not itself copyright infringement. There are actually two legal standards for infringement in the US:

- Striking similarity, or what most of us think of as infringement, where there's literal copies of someone else's code in your own. This is what things like content ID systems try to detect.

- Access plus substantial similarity. This is where you've looked at someone else's code (source or disassembled, doesn't really matter); and then produced something that looks a lot like that code if you squint a little.

Substantial similarity does not have a hard-and-fast rule associated with it, it's usually something that juries or judges decide. It's basically the "yeah just change it a little so the teacher doesn't notice" meme, but in legal form. If you were to read another program's code, and then make another program that looked an awful lot like the first, then you'd be infringing. But this is bounded by other exceptions to and rules of copyright - notably, functionality itself can't be copyrighted, and copyright over interfaces is largely prescribed by various fair use decisions (so emulation and re-implementation is largely still OK even under this standard).

If you read GCC, and then write LLVM, that's not substantial similarity. That's a different, legally distinct compiler design. The FSF could sue the pants off Apple if merely reading GCC source meant any compiler you wrote was infringing.

If you read GCC, and then write a compiler with the same internal representation as GCC; a parser that's structured the same way as GCC; and optimization passes that are organized the same way as GCC's; then you're closer to infringing.

Your responsibility isn't to create an entirely novel program by any means necessary. It's just to avoid doing the software equivalent of tracing over someone else's drawing.

csdreamer7• 5 years ago

This is the comment that points out the issue.

perf is licensed under GPL v2 only and libbfd is a GNU tool that is licensed GPL v3 and higher.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911815

Really wish the Linux kernel would start mandating new patches to be GPL v2 and higher and get buy in from the largest contributors. A decade later new code would like replace the smaller contributors code and we could consider the software GPL v3 compatible.

jabl2• 5 years ago

You might as well blame GNU for updating from GPLv2+ to GPLv3+, creating the problem in the first place.

Since, realistically, neither Linux nor GNU will budge, maybe the practical solution would be to look into integrating with the equivalent library in the LLVM ecosystem (assuming it exists).

whoknowswhat112• 5 years ago

No chance of that.

GPLv3 wasn't just an update, it was a major change from GPLv2. Importantly, it limited developers own rights to use code of projects they contribute to how they want (including in devices that are locked or secured in various ways).

Plenty of people even outside of Linux are not going to be going to GPLv3.

The GPLv3 split also damaged the more copyleft side of things as I think some kernel devs predicted.

I think the momentum is currently more MIT / Apache - not sure if that could be where folks could be encouraged to release under to keep at least the open source part alive even if the copyleft part kind of goes away.

Anyone doing any stats on this? The more true open source players are going MIT / Apache style, the proprietary relicense folks are doing the (A)GPLv3 thing to drive licensing revenue given the risk aversion to GPLv3 that is out there. A lot of the GPLv3 codebases require contributor agreements so they can license outside of GPLv3 so they tend not to be true multi-contributor / multi-copyright holder codebases.

tpxl• 5 years ago

Don't feel down about that. If someone's job is writing database engines or JSON parsers, you can't do a better job than them while also writing business logic. We should be glad some people chose to provide such code in a matter we can use it, and when we can't, a patchjob that does the trick is just as good.

jsolson• 5 years ago

For my personal work, I treat GPLv3 (and similarly licensed) projects as essentially the same as anything behind a commercial license -- they don't exist.

Rather than thinking about how to re-implement a thing that doesn't exist, I think about how to implement the tool I actually need. What I end up with may be less general, but it's actual functionality is often simpler and easier for me to grok/remember.

teleforce• 5 years ago

For those who're wondering what Linux perf can do, please check this excellent examples and descriptions:

https://www.brendangregg.com/perf.html

touisteur4• 5 years ago

Oh no I hoped it was about speeding up perf record that is actually a big thorn on my side, one I wrote a specific tool for... Depending on the number of probes you use, perf record can induce large latency hits or reduced throughput. Batching/buffer disk writes solves the problem for me. But I had to redevelop a perf parser to record/compress smarter. And for network streaming (no touching the disk is even better...).

tonyg8• 5 years ago

Sorry! :-) It was very much just scratching a particular itch I had related to cargo-flamegraph...

touisteur4• 5 years ago

Perf report is indeed slow AF, especially on large files, you're right in wanting to speed it up! Thanks for sharing! This is an interesting tidbit that has thrown me down a rabbit hole of 'profiling the profiler'...

scottlamb3• 5 years ago

Neat! Is your tool open source?

It'd be nice if perf record had a fundamentally faster way of working. I found a nice description of how it works in the README.md for cargo-trace: "perf relies on perf_event_open_sys to sample the stack. Every time a sample is taken, the entire stack is copied into user space. Stack unwinding is performed in user space as a post processing step. This wastes bandwidth and is a security concern as it may dump secrets like private keys."

cargo-trace is apparently dormant now, but I found it really interesting. It does the unwinding via eBPF instead, which should be quicker while recording, not generate as much (sensitive) data, and not require as much post-processing. (Symbolization would still happen in post-processing.)

touisteur4• 5 years ago

I'll ask about opensourcing the tool. But just in case, the recipe is to use pipe mode and pre-parse all frames, stream them as messages, sometimes to several targets (pub/sub) with some streaming-zstd, and also splitting the pmu/probes/Intel-PT streams and treating them separately. Stack-traces are analysed (precomputed cfg optimised structure so unwinding is faster) before storing in adhoc in-house format with all other system traces. Only annoying thing is changing perf-record settings (pid changes, need event X, new probe) means restart and I ran out of interns before we had no-loss switchover...

saurik3• 5 years ago

FINALLY AN ACTUAL ANSWER!! :D

I had done a bunch of research into this a while ago and filed an issue with the Ubuntu bug tracker, but was told it was due to ABI stability (which did not make any sense).

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1894407

Understanding that there is actually an underlying license problem is like finally being told all of the reasons for everything that happened on LOST ;P.

mistrial9• 5 years ago

This bug affects 3 people (!)

fhaldridge7• 5 years ago

I'm sure it affects more people than that. Not everyone who experiences this bug is going to create an account on the ubuntu launchpad platform and write a comment about it

andrepd• 5 years ago

I'm absolutely positively sure that the amount of people using perf on Debian and derivatives (Ubuntu, Mint) is >>3

rtpg• 5 years ago

So `perf` is super slow on Ubuntu as well?

I feel for distro maintainers, real thankless job trying to square the circle on a billion different things like this.

thisisinterest• 5 years ago

Perhaps you should add a link to this article as a comment in that ubuntu bug tracker. That way it's useful for others who have the same question.

halayli• 5 years ago

perf-report's slowness is very frustrating. This slowness started around 2017 or so. However, you can get around it by passing --no-inline and the graphs will be exactly the same.

unique_parrot2• 5 years ago

So all I am understanding is, we could make linux 10-100 times faster, but licensing-quirks don't allow it? Physical electricity, time and whatnot is wasted because of legal foo?

foxfluff• 5 years ago

No, you're not understanding it. Perf is a profiling tool. It's slow on Debian because reasons.

danuker• 5 years ago

The GPL requires that software linking to GPL projects are also licensed in GPL. This issue could be solved easily by releasing GPL software, that respects users' freedom.

Macha4• 5 years ago

Both packages involved here are GPL, but unfortunately have mutually exclusive version ranges (=2, and >=3)

londons_explore• 5 years ago

I would like to see a more general approach to solving the problem of "short lived processes are expensive".

There needs to be a way to dynamically link a binary into your own address space, and call it's logic repeatedly, without incurring all the process startup overhead all the time.

Pretty much, have all command line utilities be linkable like a library.

xuhu• 5 years ago

Afl-fuzz runs a program until main() and then forks it repeatedly. An empty main() can be called at least 10000 times per second this way.

xxpor2• 5 years ago

With AFL++ you can even determine exactly where the fork happens:

https://github.com/AFLplusplus/AFLplusplus/blob/stable/instr...

stabbles• 5 years ago

I wonder how many other packages suffer from this problem... Maybe building everything from sources isn't that bad after all.

a-dub• 5 years ago

is it the fork/exec overhead or the overhead of re-parsing the dwarf data that is responsible for the slowdown? process spawn thrash is obviously bad, but i'm curious how much it contributes to the issue here? forks are pretty cheap these days as i understand, and i think an exec may also be pretty cheap since the program image is already going to be sitting in the buffer cache.

reading/parsing dwarf data, on the other hand, is likely to be slow. not totally sure, but maybe i/o could be sped up by mmap'ing in the dwarf data and maybe part of the parsing could be cached?

fun story, i once solved a similar performance regression in a machine learning context, where calling code would serialize an entire model and pass it to inference code, which would then deserialize it all, make one inference, and then tear it all down. if online/streaming is not required, a huge speedup can come from just batching the work.

tonyg8• 5 years ago

I suspect it's both. But I didn't measure, because the patch does away with both sources of overhead at the same time :-)

glandium• 5 years ago

Parsing the dwarf data has orders of magnitude more overhead than fork/exec. (source: I'm the one who opened the linked Debian bug)

dabreegster• 5 years ago

Does anybody have working instructions to build a patched perf for Ubuntu 20.04? I built from kernel.org following https://michcioperz.com/post/slow-perf-script/, but somehow flamegraph pointed at the new build breaks totally -- after exiting the app, there's no "perf record: Woken up 181 times to write data" message; it just hangs.

tonyg8• 5 years ago

Note that the patch does something other than what Michał's post ends up doing: instead of linking libbfd, it replaces lots of calls to addr2line with a single long-running call to addr2line.

If you want to try the patch on Ubuntu, I recommend using the Debianish technique of "apt source linux-perf-5.10" (or whichever version of the kernel you're running) and applying the patch. Then "make" in the tools/perf directory, and it should work...

gnufx• 5 years ago

Silly rants about the GPL aside, why can't you use (LGPL) libelf, following its version of addr2line? The binutils-static rpm doc even says to use elfutils instead in new code -- no BFD? A quick look suggests RHEL's perf does so, but I'd need to check.

evmar• 5 years ago

In a separate context I discovered the slow performance of addr2line and wrote my own faster one as a learning experiment:

http://neugierig.org/software/blog/2012/03/heap-profiling.ht...

DSingularity• 5 years ago

Why not build from source and link against libbfd?

tonyg8• 5 years ago

That would have solved my immediate problem, but the result is not distributable, so it wouldn't solve it generally, for others suffering the same thing.

tanjor• 5 years ago

I do know Boost.Stacktrace calls addr2line too. From the code(https://github.com/boostorg/stacktrace/blob/develop/include/...), it seems Boost.Stacktrace also shells out to addr2line for every address. But in practice, I found the overhead of boost::stacktrace::stacktrace() is not as horrendous as my crappy implementation, which calls addr2line, too. Is there something I missed in Boost's code?

60x speed-up of Linux “perf”

Related Stories