felixge

We had reached out to y'all last year to explore taking ideas from your format. It definitely looks interesting! But IIRC nobody from your team ended up making it to one of our SIG meetings?

https://github.com/yandex/perforator/issues/13

OTel Profiling SIG maintainer here: I understand your concern, but we’ve tried our best to make things efficient across the protocol and all involved components.

Please let us know if you find any issues with what we are shipping right now.

Do you think the GC roots alone (goroutine stacks with goroutine id, package globals) would be enough?

I think in many cases you'd want the reference chains.

The GC could certainly keep track of those, but at the expense of making things slower. My colleagues Nick and Daniel prototyped this at some point [1].

Alternatively the tracing of reference chains can be done on heap dumps, but it requires maintaining a partial replica of the GC in user space, see goref [2] for that approach.

So it's not entirely trivial, but rest assured that it's definitely being considered by the Go project. You can see some discussions related to it here [3].

Disclaimer: I contribute to the Go runtime as part of my job at Datadog. I can't speak on behalf of the Go team.

[1] https://go-review.googlesource.com/c/go/+/552736

[2] https://github.com/cloudwego/goref/blob/main/docs/principle....

[3] https://github.com/golang/go/issues/57175

I'd love to hear more! What kind of profiling issues are you running into? I'm assuming the inuse memory profiles are sometimes not good enough to track down leaks since they only show the allocation stack traces? Have you tried goref [1]?. What kind of memory pressure issues are you dealing with?

[1] https://github.com/cloudwego/goref

Disclaimer: I work on continuous profiling for Datadog and contribute to the profiling features in the runtime.

Hacking into the Go runtime with eBPF is definitely fun.

But for a more long term solution in terms of reliability and overhead, it might be worth raising this as a feature request for the Go runtime itself. Type information could be provided via pprof labels on the allocation profiles.

+1. In particular []byte slice allocations are often a significant driver of GC pace while also being relatively easy to optimize (e.g. via sync.Pool reuse).

This is great, thank you.

I'm skeptical that it's worth it myself, this was just a fun research project for me. But once hardware shadow stacks are available, I think this could be great.

To answer your first question: For most Go applications, the average stack trace depth for profiling/execution tracing is below 32 frames. But some applications use heavy middleware layers that can push the average above this limit.

That being said, I think this technique will amortize much earlier when the fixed cost per frame walk is higher, e.g. when using DWARF or gopclntab unwinding. For Go that doesn't really matter while the compiler emits frame pointers. But it's always good to have options when it comes to evolving the compiler and runtime ...

That seems to be windows only? My main target OS is Linux.

Thank you so much, this is very helpful and interesting. I'll try to experiment with this at some point.

Thanks for the reply!

What does the API for accessing the shadow stack from user space look like? I didn't see anything for it in the kernel docs [1].

I agree about the need for switching the shadow stacks in the Go scheduler. But this would probably require an API that is a bit at odds with the security goals of the kernel feature.

I'm not sure I follow your thoughts on CGO and how this would work on older CPUs and kernels.

[1] https://docs.kernel.org/next/x86/shstk.html

I don't think any obvious 10%+ opportunities have been overlooked. Go is optimizing for fast and simple builds, which is a bit at odds with optimal code gen. So I think the biggest opportunity is to use Go implementations that are based on aggressively optimizing compilers such as LLVM and GCC. But those implementations tend to be a few major versions behind and are likely to be less stable than the official compiler.

That being said, I'm sure there are a lot of remaining incremental optimization opportunities that could add up to 10% over time. For example a faster map implementation [1]. I'm sure there is more.

Another recent perf opportunity is using pgo [2] which can get you 10% in some cases. Shameless plug: We recently GA'ed our support for it at Datadog [3].

[1] https://github.com/golang/go/issues/54766 [2] https://go.dev/doc/pgo [3] https://www.datadoghq.com/blog/datadog-pgo-go/

That's what hardware shadow stacks in modern intel/arm CPUs can do! It just needs to be exposed to user space and become widely available.

I know that at least two engineers from the runtime team have seen the post in the #darkarts channel of gopher slack. One of them left a fire emoji :).

I'll probably bring it up in the by-weekly Go runtime diagnostics sync [1] next Thursday, but my guess is that they'll have the same conclusion as me: Neat trick, but not a good idea for the runtime until hardware shadow stacks become widely available and accessible.

[1] https://github.com/golang/go/issues/57175

Thanks! And to answer you question: No, it won't speed up Go programs for now. This was mostly a fun research project for me.

The low hanging fruits to speed up stack unwinding in the Go runtime is to switch to frame pointer unwinding in more places. In go1.21 we contributed patches to do this for the execution tracer. For the upcoming go1.23 release, my colleague Nick contributed patches to upgrade the block and mutex profiler. Once the go1.24 tree opens, we're hoping to tackle the memory profiler as well as copystack. The latter would benefit all Go programs, even those not using profiling. But it's likely going to be relative small win (<= 1%).

Once all of this is done, shadow stacks have the potential to make things even faster. But the problem is that we'll be deeply in diminishing returns territory at that point. Speeding up stack capturing is great when it makes up 80-90% of your overhead (this was the case for the execution tracer before frame pointers). But once we're down to 1-2% (the current situation for the execution tracer), another 8x speedup is not going to buy us much, especially when it has downsides.

The only future in which shadow stacks could speed up real Go programs is one where we decide to drop frame pointer support in the compiler, which could provide 1-2% speedup for all Go programs. Once hardware shadow stacks become widely available and accessible, I think that would be worth considering. But that's likely to be a few years down the road from now.

OP here, happy to answer any question.

Thanks for the reply. I ended up implementing this idea in Go and wrote a blog post about the results: https://blog.felixge.de/blazingly-fast-shadow-stacks-for-go/

I'm curious if you've done any benchmarking for your implementation as well?

Looking at the code [1] it seems like the library is actively trying to handle this problem.

[1] https://github.com/koute/not-perf/blob/master/nwind/src/loca...

Dynamic patching of return addresses is a very cool trick. I don't think I've seen this before. Have you run into any situations where this crashes programs or otherwise interferes with their execution?

This sounds like it was copied from PostreSQL which SQLite cites as a strong source of inspiration.

I wouldn't be so sure. ByteDance (TikTok) implemented a moving GC for Go: https://youtu.be/DpQgJ06ZjGc?si=FviWDGa7HhBRBrO1

I think e-bikes are a good solution for this problem.

Great comments, thanks for sharing. The non-atomic frame setup is indeed problematic for CPU profilers, but it's not an issue for allocation profiling, Off-CPU profiling or other types off non-interrupt driven profiling. But as you mentioned, there might be ways to solve that problem.

I'm not very familiar with Objective C and Swift, so this might not make sense. But JS used to have a similar problem with async/await. The v8 engine solved it by walking the chain of JS promises to recover the "logical stack" developers are interested in [1].

[1] https://v8.dev/blog/fast-async

First of all, I think the .eh_frame unwinding y'all pioneered is great.

But I think you're only thinking about CPU profiling at <= 100 Hz / core. However, Brendan's article is also talking about Off-CPU profiling, and as far as I can tell, all known techniques (scheduler tracing, wall clock sampling) require stack unwinding to occur 1-3 orders of magnitude more often than for CPU profiling.

For those use cases, I don't think .eh_frame unwinding will be good enough, at least not for continuous profiling. E.g. see [1][2] for an example of how frame pointer unwinding allowed the Go runtime to lower execution tracing overhead from 10-20% to 1-2%, even so it was already using a relatively fast lookup table approach.

[1] https://go.dev/blog/execution-traces-2024

[2] https://blog.felixge.de/reducing-gos-execution-tracer-overhe...

AFAIK Go’s spec allows relocating objects. The door towards a moving GC is kept open.

Go doesn't have an instruction (execution) or function call tracer. Go's tracer is primarily tracing scheduler events. So maybe the term scheduler tracer should have been used?

Anyway, using the term "execution tracer" in Go goes back to the initial design doc from 2014: https://docs.google.com/document/u/1/d/1FP5apqzBgr7ahCCgFO-y...

but if it really is 1-2% that'd be totally worth it in many cases

As one of the people who worked on the optimizations mentioned in the article, I'm probably biased, but I think you can expect those claims to hold outside of artificially pathological workloads :).

We're using execution tracing in our very large fleet of Go services at Datadog, and so far I've not seen any of our real workloads exceed 1% overhead from execution tracing.

In fact, we're confident enough that we built our goroutine timeline feature on top of execution traces, and it's enabled for all of our profiling customers by default nowdays as well [1].

[1] https://blog.felixge.de/debug-go-request-latency-with-datado...

I’m skeptical about the numbers reported for Go, they seem too low. Maybe it’s caused by using Fprintf?

Well said. But how much finite time are we talking about (for writes to become visible)? Does it differ between architectures? Can there be extreme edge cases?