brancz

We write quite a bit of BPF at work for the Parca-Agent project[1]. And we find that even C is sometimes too high level of a language paired with modern optimization techniques as the code you write often doesn’t come out that way the other and and we have to resort to weird hacks or write assembly directly to appease the verifier.

Apart from that, the usual qualms one might have about C are not really relevant in eBPF land, so I’ve actually found it the nicest experience writing C I’ve ever had, the verifier is just the price we have to pay.

[1] https://github.com/parca-dev/parca-agent

It’s about density in a datacenter. With this you have 1PB in 4 drives, fitting in a 1u rack, which is just incredible. Also these drives don’t use regular SATA or SAS, they use PCIe, so these drives are also quite fast in comparison. Density has a power efficiency aspect as well both in just having fewer drives and requiring fewer servers to put drives into.

I also find this misleading, and could be solved so easily by just explaining that of course varints need resolving and things will just happen lazily (presumably, I didn’t read the code) when they are requested to be read rather than eagerly.

Thank you!

Overhead ultimately depends on the frequency, it defaults to 19hz per core, at which it’s less than 1%, which is tried and tested with all sorts of super heavy python, JVM, rust, etc. workloads. Since it’s per core it tends to be plenty of stacks to build statistical significance quickly. The profiler is essentially a thread-per-core model, which certainly helps for perf.

The offset approach has evolved a bit, it’s mixed with some disassembling today, with that combination it’s rock solid. It is dependent on the engine, and in the case of python only support cpython today.

Already been done:

1) native unwinding: https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...

2) python: https://www.polarsignals.com/blog/posts/2023/10/04/profiling...

Both available as part of the Parca open source project.

https://www.parca.dev/

(Disclaimer I work on Parca and am the founder of Polar Signals)

We’re working hard to bring a lot of Strobelight to everyone through Parca[0] as OSS and Polar Signals[1] as the commercial version. Some parts already exists much to come this year! :)

[0] https://www.parca.dev/

[1] https://www.polarsignals.com/

(Disclaimer: founder of polar signals)

The point of the first one is that you can create snapshots from within the product where profiling data isn't forever. This is so you can use the pprof.me link in a GitHub issue, PR, or elsewhere and trust that the data never goes away even if the original data went out of retention. We actually originally built pprof.me out of frustration that users of Prometheus (several of us are Prometheus maintainers) at best submitted screenshots of profiling data when all we wanted was an easy way to explore it.

I agree that neither of these are terribly complicated features, but as far as I know no other product on the market actually has this combination. (yes, you can export data from most systems and use a different visualization tool but the point of products is to provide a single integrated package)

(disclaimer: Founder of the company that offers the product featured in this case study.)

Keep an eye out on our blog we're working on some interesting things in this area!

I mentioned this on another thread as well, but the point isn't that perf can't catch something like this, but it's that having continuous profiling set up makes it way easier to make profiling data an everyday part of your development process, and ultimately nothing behaves quite like production. This allows things that gradually sneak into the codebase to be easily spotted since you don't have to go through the whole shebang of getting representative production profiling data over time, because you just always have it available. Continuous profiling also allows you to spot intermittent things easier and so on.

(disclaimer: I'm the founder of the product showcased in this case study.)

Nobody is saying that a regular profiling tool can't detect it. However, it's one of those things that if you don't profile it regularly then these are easy things to miss. With a continuous profiler set up you skip everything regarding collection of the right data (and you can see aggregates across time which in the second example of the blog post was important because the memory allocations aren't necessarily seen in a 10s profile). It makes the barrier for including profiling data in your everyday software engineering way easier.

You can capture memory or CPU metrics with top as well, and that's useful, but it's not the same thing as a full-blown metrics system eg. Prometheus.

Prometheus gives you CPU/memory total metrics, the profiler used in the article gets you much higher resolution: down to the line number of your source code.

If you're looking to optimize your project I would recommend using a profiler rather than metrics that just tell you the total.

(disclaimer: I'm the founder of the company that offers the product shown in the blog post, but I also happen to be a Prometheus maintainer)

Thank you for the feedback! Quickly worked with the S2 team to get the screenshot from the change added (it's just enabling the hardware acceleration feature in the sha2 crate)!

I'm familiar with the paper, but it doesn't improve the situation in terms of LBR availability on cloud providers, does it?

If I'm understanding correctly, this is collecting LBR data through hardware support for PGO/AutoFDO, right?

There isn’t, because the indented way to use Parca is to profile production and always-on.

However, we wouldn’t be against adding a mode like this!

FWIW both the server and the agent are single statically linked binaries so while it’s a bit more set up it’s not terribly difficult either[1].

[1] https://www.parca.dev/docs/quickstart/

Glad you liked it! Yeah, we worked with a customer who really needs this badly and has done some unspeakable things to get by until now.

Thanks for submitting! We know HN has a sweet spot for LuaJIT, so we figured it would eventually end up here.

Quick summary: this post dives into the gory details of how we implemented an eBPF based profiler for LuaJIT.

Let us know if you have any questions on this, we’ll keep an eye out on comments!

Thanks for posting!

We missed this being submitted yesterday, so let us know if you have any questions, we'll be sure to watch this thread!

There certainly are, but I’d recommend making the first priority to look for who is actually going to be your most valuable business partner, their location in my opinion doesn’t really play any role in it. And in my experience a VC with conviction in your business will find a way to invest.

Thank you!

Having gone through this a couple of times, I'll say that all of this is true, but in practice, I find the actual founding process not very complicated with neo-banks that understand the founding process well. I just founded a new GmbH last week, and it took 2 days in total.

One more thing I'd recommend doing before going to a notary though is get a "Vorabstellungnahme" from IHK to ensure that they won't reject your company name which would create delays and additional notary cost. It costs some money, but is worth ensuring it doesn't cause chaos afterwards.

1. Get "Vorabstellungnahme" from IHK, takes a few hours.

2. E-mail notary (yes I've worked with them a couple times, that might speed up their response time), get pre-fab founding docs within a few hours, appointment next day. Digital version of the founding docs will be available within a few hours.

3. Create account with Qonto or something equivalent, there's an explicit configuration for your notary email, so they will take care of providing proof of the starting capital.

4. Transfer starting capital.

5. Notary hands in Handelsregisteranmeldung, sends you a copy and upload it to your bank.

In practice, I find the founding process not complicated, but the day-to-day operation, bookkeeping, taxes, etc. way more painful.

Founding a GmbH wholly owned by a Delaware C-Corp, however, is intensely painful, practically no bank wants to work with you, and notaries aren't enough, you need to work with apostilles (international notaries). I highly recommend working with a law firm to set this up correctly but expect easily upwards of $10k between lawyer, apostille, and bank account costs.

This seems interesting for bpf programs that have essentially no user-space interaction which isn’t true for any of the things it lists on the landing page. Pixie, Cilium, Parca, etc have complex interactions between userspace and the bpf program, from loading and configuring to ongoing maintenance of bpf maps. I don’t really see why they would use this. I do like this type of thing for small programs like Inspektor Gadget does, but Inspektor Gadget already exists.

I guess this is feedback to maintainers of the project to clarify these two relationships.

Great comment! Just want to add we are making good progress on the JVM unwinder!

The way perf does it is slow, as the entire stack is copied into user-space and is then asynchronously unwound.

This is solvable as Brendan calls out, we’ve created an eBPF-based profiler at Polar Signals, that essentially does what you said, it optimized the unwind tables, caches them in bpf maps, and then synchronously unwinds as opposed to copying the whole stack into user-space.

Parca's user-space code is apache2 and the eBPF code is GPL.

Just about anyone who works on a profiler for native binaries has to solve this eventually. We had to as well for our eBPF-based profiler: https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...

Right, but judging by the detail of the post, and not mentioning profiling makes it seem like it wasn’t even attempted.

I’m incredibly confused why profiling wasn’t even mentioned in this post.

I would highly recommend anyone troubleshooting something like this to do continuous profiling and the root cause of this type of problem becomes painfully obvious within seconds.

(Disclaimer I founded a company in this space, and we recently open sources a library for rust to do this: https://www.polarsignals.com/blog/posts/2023/12/20/rust-memo...)

We’ll be hanging out in comments, let us know if you have any questions!

We've "fixed"[1] all the tooling that we work on at Polar Signals, and even then I think this is the right thing to do. It enables so much more debuggability aside from profiling.

[1] https://www.polarsignals.com/blog/posts/2022/11/29/dwarf-bas...