What is BPF and why is it taking over Linux Performance Analysis?

http://blog.memsql.com/bpf-linux-performance/

by okket • 10 years ago

131 40 10 years ago

blog.memsql.com

Low quality blogs designed to be used as marketing copy for the company. This would be a much better link/resource: http://www.brendangregg.com/blog/2016-03-05/linux-bpf-superp...

binarycrusader• 10 years ago

That so called low quality blog is one Brendan himself linked to recently: https://twitter.com/brendangregg/status/766314203749638145

diegocg• 10 years ago

What is exactly "low quality" about this post?

wcummings• 10 years ago

It's a marketing blog. It exists for leadgen.

brendangregg3• 10 years ago

There is something important about this post (even though it looks like marketing): it's the first application vendor to create BPF scripts. It's a milestone for BPF. It won't be the last.

SFJulie2• 10 years ago

long story short. Average is a poor measure of non linear phenomenon like the measure of a performance for a task .

Histogram are better to see the repartitions of values, and can hint you on a n-modal behaviour. (a «mode» when everything goes well, and a degraded one for instance which can hint on a potential transition of behavior).

I guess the next step is to say this is even more important to know the evolution over time and announce that the equivalent of spectrogram (3D mapped on 2D plus colour) is a good idea to known towards which mode you are tending the more. [EDIT: which can also more significantly graphed as spectrum analysis, colour encoding the amplitude of measure for a given frequency, and frequency being 1/Period thus making sure that slow processes may appear between 0 and average]

A shortcut is to hire of of the many jobless guys with a master in science that actually have experience in labs.

Beware that for a reason that is beyond my reach, people with background in math, management, literature have problems with scientific concepts that has been discovered after 1800 unless it is packaged like magic.

static_noise• 10 years ago

Replace the average (mean) with the median and you will have better results in almost all cases.

When the data is good the result ist virtually identical. When there are outliers, the median just ignores them unless there lots of them. When the data is that bad, that the median doesn't work anymore you need to up your statistics quite a few notches.

SFJulie2• 10 years ago

Well the median of measure that are only aggregated by average (RRD, carbon ...) is ... biased.

Hence the reason I made my own backend for making very high precision measurement using a percentile filter (10%, 50%, 90% and average for the fun).

It was fun to watch linux «monotonic» clock not being monotonic.

omginternets• 10 years ago

In psychophysics we either use the median for reaction times (Gamma distribution) or we normalize with something like the logarithm.

I think some of the techniques might be applicable to system performance analysis, so I submit the following introductory publication, in case any are interested [0].

[0] http://opensiuc.lib.siu.edu/cgi/viewcontent.cgi?article=1077...

gnufx2• 10 years ago

Replacing a single number with a different one hardly helps. The distribution matters, specifically the outliers in performance measurements.

gnufx2• 10 years ago

I'd like to agree about scientific experience, but it's tragic how much trouble many professional scientists, or ex-scientists, seem to have with anything outside their specialism, like quantitative analysis of their computational techniques. When did you last see error bars on a benchmark, even?

vbernat• 10 years ago

SystemTap enables the same kind of analysis for a few years already. And it works well with older kernels (starting from 3.2, but really from 3.13). Of course, an in-kernel solution is better (and I believe that SystemTap will soon use it).

It's unfortunate that SystemTap never had much visbility. It also comes with a huge libraries of tapsets.

dozzie2• 10 years ago

Is SystemTap stable already? As far as I remember from few years ago, it could easily crash your kernel with a valid program that didn't abuse anything, what disqualified it from use in production.

lima• 10 years ago

Yes it's stable. CloudFlare heavily uses it in production.

josefbacik2• 10 years ago

Systemtap was fantastic, but super finnicky. You had to have debuginfo for the kernel installed plus the source. This was fine for RHEL but kind of a pain elsewhere. Then you run into other problems like sometimes the distro compiler wasn't used to build the kernel you are running and suddenly you couldn't load any stap script. Also the only way to get output was through stdout, there was no easy way to programmatically pull results from an stap script.

BPF/BCC solves most of this. You still have to have the sources so you can get the right targets for kprobes, but you don't have the compiler mismatch problem. You can now access the hash maps directly from user space so you can do things like build system monitoring tools that run in production and record really specific information easily. I used systemtap for years, but bcc/bpf is a whole new world.

paxcoder8• 10 years ago

You used past tense in talking about a project last committed to less than 3 days ago. I have no horse in this race but it seems to me like you might.

brendangregg3• 10 years ago

Yes, and I've given SystemTap plenty of chances and hours of my time. But right now let's give BPF a chance -- it's more in-kernel in 2 years than SystemTap did in 10 years.

It's unfortunate that SystemTap never had much visbility.

Right. DTrace was marketed. I don't think SystemTap was.

qwertyuiop9243• 10 years ago

eBPF could use a better interface, though. I REALLY don't want to have to write this stuff in C. Think we could get a scripting language?

okket2• 10 years ago

You only have to write the kernel stuff in C (which is a small part, since you are just collecting data), for the front end you can use BCC (Python/Lua)

https://github.com/iovisor/bcc

Personal favorite: https://github.com/iovisor/bcc/blob/master/tools/sslsniff_ex...

qwertyuiop9243• 10 years ago

Interesting. I'll have to read the docs, but...

This looks like a job for... FUNCTIONAL PROGRAMMING!

brendangregg3• 10 years ago

Yes, we could, and I imagine we will at some point. It'll have become a user-level problem: the kernel capabilities are there (BPF+everything else), and there's the bcc user-level libraries (written in C) that make using the kernel stuff easier. The easiest way to add a front end would be to consume the bcc libraries. There's plenty of examples of how to use them (the Python and lua interfaces).

So if a scripting interface existed, would you use it? For what? Yes, it'd be nice to have, but I think a lot of people will get as far as using the provided bcc tools and be happy with that.

qwertyuiop9243• 10 years ago

If we could write bpf programs in something other than C, it would go a long way towards opening up eBPF to people who wouldn't be able to write their own scripts otherwise.

What is BPF and why is it taking over Linux Performance Analysis?

Related Stories