More powerful Go execution traces

https://go.dev/blog/execution-traces-2024

by nalgeon • 2 years ago

376 122 2 years ago

Really cool work. I'm still slightly paranoid about the overhead of leaving tracing enabled all the time but if it really is 1-2% that'd be totally worth it in many cases.

More awesome work from the Go team. Thanks!

heyoni• 2 years ago

Go has a race detector that you can leave on for a while in like a dev environment and it’ll flag race conditions. There was documented overhead and at some point even a memory leak but they spent months looking into it and eventually plugged the leak and the saga is now part of their official docs. It’s really interesting to see that kind of stuff laid bare so I would trust that this feature will at least run reasonably well and if it doesn’t, they’ll likely fix it: https://go.dev/doc/articles/race_detector https://go.dev/issue/26813

swatcoder• 2 years ago

The cost of race detection varies by program, but for a typical program, memory usage may increase by 5-10x and execution time by 2-20x.

That's "let's run the race detector and get some coffee" overhead, not "let's leave it on in dev" overhead.

Still cool that they have it available!

klabb32• 2 years ago

The race detector is an invaluable complement to a language like Go that lacks compile time safety. It meaningfully increases the confidence in code and is anecdotally both good at finding races while also giving straightforward error information to plug them. It won’t find everything though, since it’s runtime analysis.

In general, Go tooling makes up for a lot of the intrinsic issues with the language, and even shines as best in class in some cases (like coverage, and perhaps tracing too). All out of the box, and improving with every release.

felixge2• 2 years ago

but if it really is 1-2% that'd be totally worth it in many cases

As one of the people who worked on the optimizations mentioned in the article, I'm probably biased, but I think you can expect those claims to hold outside of artificially pathological workloads :).

We're using execution tracing in our very large fleet of Go services at Datadog, and so far I've not seen any of our real workloads exceed 1% overhead from execution tracing.

In fact, we're confident enough that we built our goroutine timeline feature on top of execution traces, and it's enabled for all of our profiling customers by default nowdays as well [1].

[1] https://blog.felixge.de/debug-go-request-latency-with-datado...

baq• 2 years ago

I love datadog but you guys really give splunk a run for their money in the invoicing department :/

bsaul• 2 years ago

i've sometimes heard that the JVM had best in class tooling for server troubleshooting. How does go compare to it now ?

Xeoncross3• 2 years ago

For good reason, the JVM has a ton more knobs that need adjusting. You can't just run Java code. The JVM has a lot of tricks you have to customize for based on your workload.

For years until 1.19 the Go GC has had only one tuning parameter (GOGC).

rileymichael• 2 years ago

I don't see the connection you're making between knobs that adjust runtime behavior and tooling. As an aside, "you can't just run java code" is a bit hyperbolic, plenty of people "just run" java apps and rely on the default ergonomics. The modern JVM also offers more automated options, such as ZGC which is explicitly self tuning.

tsimionescu3• 2 years ago

Note that Go needs those knobs just as much as the JVM does, at least some of them. They just didn't want to expose them.

Ironlink• 2 years ago

You can't just run Java code. The JVM has a lot of tricks you have to customize for based on your workload.

This sounds like something you would hear 10 years ago in relation to the CMS garbage collector. Since Java 9, G1 has been the default gc for multi core workloads and is self-tuning. The CMS gc was removed 4 years ago. If you do need to tune G1, the primary knob is target pause time. While other knobs exist, you should think carefully before using them.

We run all of our workloads with vanilla settings.

Thaxll4• 2 years ago

Go is one of the best language in term of tooling, it has just one binary that contains everything, I don't know any other language that includes out of the box all of that:

- testing

- benchmark

- profiling

- cross compilation ( that work from any platform, like compiling a windows .exe from your raspberry pi for example )

- some linting

- documentation

- package mgmt

- bug report

- code generation

- etc...

Java is probably more advanced in some fields ( like tracing / profiling ) but it lacks others.

preommr• 2 years ago

Golang also has some of the worst tooling because everything is based off of what comes builtin, and because they're not specialized projects, they're very limited in capability and configuration.

Coming from ts, tooling like tsconfig has a lot of options, but sensible defaults can be set with a single flag like strict mode. If some org has some specialized needs, they can dive into the configuration and get what they need.

With golang, not only would it be a lot for any single team to offer all those features at a decent level of polish, the golang culture in particular is very, very resistant to small bits of comfort because of dogma like "worse is better". It's kind of similar to Haskell's "avoid success at all cost".

pjmlp16• 2 years ago

There is really nothing in Go that Java ecosystem lacks, in their 30 years of existence.

The only thing one could arguably argue that Go does better is value types, but even that requires careful coding so that escape analysis is triggered, and in that sense, it is only a matter to use a JVM implementation like GraalVM, OpenJ9, Android ART, PTC, Aicas, Azul.

Alifatisk• 2 years ago

single binary
cross compilation ( that work from any platform, like compiling a windows .exe from your raspberry pi for example )

This, I think is one of Go's best selling point.

tsimionescu3• 2 years ago

I think Go is catching up, but it's still significantly behind. For example, Go memory profiles are much, much worse than Java's - they don't even have them integrated with the GC to show the ownership graph (they can only show where each object was allocated, instead of which other objects are holding a reference to it). The CPU profiling parts seemed more up to par. This tracing thing is nice, I'm not as familiar with this are of Java. Also, I don't think Java has a built-in race detector (except perhaps for the detection of concurrent write and iterations in collections?).

Also, the OpenJDK JVM supports live debugging and code hotswap, going so far as to de-optimize code you're debugging on the fly to make it readable. Go doesn't support live code reload even in local builds.

tptacek3• 2 years ago

I really like Go's tooling, and while I'm fluent in Java I've never been on a full-time Java team. I've remarked to Java friends in the past that I think Go has best-in-class tooling and had my ass handed to me, in detail and at length. By many accounts, Java is the gold standard for tooling in language ecosystems. It's a compliment to Go that you'd even consider the comparison. :)

lycopodiopsida• 2 years ago

Java's tooling is indeed very good but comes with a huge downside of being bound to an IDE.

atombender• 2 years ago

One of the weakest areas is analyzing heap dumps.

The current format has very limited tooling; "go tool" has some extremely rudimentary visualization. There is gotraceui [1], which is much better, though you need to use Go trace regions to get much useful context.

There's a proposal to support Perfetto [2], but I don't know if anything has come of it.

[1] https://gotraceui.dev/

[2] https://perfetto.dev/

Veserv2• 2 years ago

The usage of the term execution trace really threw me off. I am more familiar with that term meaning a instruction execution trace or maybe just a function trace, so it made no sense why stack unwinding would be a bottleneck. Turns out it is actually a goroutine event (with stack trace at time of event) trace [1]. I guess Go does not have a code execution trace package?

[1] https://pkg.go.dev/runtime/trace

Thaxll4• 2 years ago

Tracing code is a way to see what's happening in your code which Go provides:

https://www.infoq.com/articles/debugging-go-programs-pprof-t...

Many other language use the same technique, see Python with: https://github.com/bloomberg/memray

Veserv2• 2 years ago

I was talking about execution tracing, specifically, not tracing as a general concept. Tracing as a general concept just means a non-sampled time-series event stream and as commonly used these days generally also includes "stacking" to distinguish it from generic event streams or logging.

When you add a additional term like "execution" you are specializing the term to mean a event stream of "execution". In areas I am familiar with, that would normally mean a trace of the complete execution of the program down to the instruction-level so you can trace the precise "execution" of the program through the code.

What is described here would, in the terminology I am familiar with, be more like... a thread status and system event trace just applied to goroutines and the Go runtime, respectively, I guess? It does also include the stack trace at the time of the event, so it does have more data than that, but that is qualitatively different than a instruction execution trace that allows you to trace the exact sequence of execution of your program.

felixge2• 2 years ago

Go doesn't have an instruction (execution) or function call tracer. Go's tracer is primarily tracing scheduler events. So maybe the term scheduler tracer should have been used?

Anyway, using the term "execution tracer" in Go goes back to the initial design doc from 2014: https://docs.google.com/document/u/1/d/1FP5apqzBgr7ahCCgFO-y...

imiric• 2 years ago

This is really amazing! I look forward to giving it a try. Kudos to the Go team and contributors.

Say what you will about the language itself, the Go stdlib is one of the most useful and pleasant standard libraries I've ever worked with. It gets better with every release, and great care is taken to make it featureful, performant and well documented. These days for most general things I rarely have the need to reach for an external package.

traceroute66• 2 years ago

I rarely have the need to reach for an external package

Agree 100 % ! And it gets better as time evolves.

For example the recent `log/slog` introduction to stlib kills the need for 99.9999% of people to use third-party logging, because structured logging is now in stlib.

Same with the new http mux. Many people will now be able to migrate over to stdlib because of the richer functionality, and only the small community of outliers who are doing stuff like regex mux parsing will need the third-party libs, but with time no doubt regex mux will make its way to stdlib too.

throwaway894345• 2 years ago

I remember hearing from people in the early days of Go that it was only nice because it hadn't aged. It didn't have the "barnacles" that Python and Java had acquired, but that it surely would after a decade. At the time, Python and Java were about 15 years old (since their 1.0 releases). Well, it's been 12 years and Go is still clean and minimalist. Maybe all of the cruft comes in years 12-15? :shrug:

arp2422• 2 years ago

Go did acquire some of that though; a few packages are deprecated, some have different ways of doing the same thing (e.g. the addition of context and fs packages added new paradigms, and later netip and new JSON package added "urllib2"-like "v2" packages), and some things weren't really a good idea to start with.

I do think the situation is better than in Python and overall not bad after 12 years, but it's not as clean and minimalist as it could be.

kbolino5• 2 years ago

Go has many barnacles. Off the top of my head:

- context.Context is two different barnacles. The first is the thing itself; do I "need" context here? Probably should add it just to be safe. The second is the myriad of older libraries that don't use it or have tacked it on with a duplicate set of functions/methods. Library authors seem to not understand the point of major versions or else are afraid to pull that trigger.

- http.Client is a barnacle. First, there's the fact that you have to close the response.Body even if you don't use it. Then there's the difficulty of adding middleware or high-level configuration to it. Then there's the fact that they broke their own rules for context.Context and instead of taking it as a parameter in the methods, it's stored as part of a struct. You can work around many of these problems for your own code, but not when using third-party libraries.

- The entire log package is a barnacle. Thankfully we now have log/slog but log existed for so long that lots of third-party libraries use it. Even if the library authors knew to avoid the footguns that are Fatal/Fatalf/Fatalln and Panic/Panicf/Panicln, context and level support are spotty at best.

- The (crypto/rand).Read and (math/rand).Read debacle. Really, the entire math/rand package is a barnacle. Thankfully this has also been addressed with math/rand/v2 but the old one will live on for compatibility.

pjmlp16• 2 years ago

The Flight Recoding pun did not go unnoticed.

https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-g...

Zuiii• 2 years ago

Go's standard library is a shining example of what all standard libraries should strive for. Yet, we still have some languages who's developers refuse to include even a basic http API in their standard libraries in an age where even embedded systems have started to speak http. Imagine if if the same had happened with TCP and UDP...

Here's to the continued success of Go and other sanely-designed languages.

nindalf2• 2 years ago

I think it’s a bit extreme to characterise it as sanity vs insanity. I think you need at least one of a large stdlib or good dependency management.

I love Go, used it since the 1.0 release and have used it at work for years. No complaints about the language. But for the longest time Go didn’t have quality dependency management and pulling in dependencies was annoying. Building your programs with the large, high quality stdlib was the path of least resistance.

Since Go 1.0 (2012) most new languages have coalesced on good dependency management. Rust, for example, copied Ruby’s bundler idioms since before 1.0 (2015). People who needed good quality libraries were able to pull them in with minimal hassle. That’s why they didn’t need a large standard library to succeed. I’ve written more about the tradeoff here - https://blog.nindalf.com/posts/rust-stdlib/

To you, I’d suggest using less charged language like sanity. In this case there’s a good technical case for both ways, and it’s not productive or nice to imply that people who choose a different path from you are insane.

bheadmaster2• 2 years ago

To you, I’d suggest using less charged language like sanity.

To the grandparent, I'd suggest keeping the language as it is. Status quo in the current programming ecosystem is often insane, and when it is, we should call it out.

it’s not productive or nice to imply that people who choose a different path from you are insane.

In a general sense, I agree. However, when it comes to programming, insanity is so widely accepted as normality that I think it would be counterproductive to blunt the language used to describe it.

As an example of insanity for those who may (justifiably) think I'm talking out of my bottom - in NodeJS ecosystem the standard library is so bad that it is considered acceptable to use dependencies for even the simplest tasks, which caused the `leftpad` incident which broke countless programs.

begueradj• 2 years ago

It's funny that your comment here applauds Go but your blog post praises Rust.

norman7843• 2 years ago

I think there's a distinction between Rust and Go in that sense, Go has a clear objective, be a good (if not the best) language for writing web services (that's why Google developed it in the first place), that's why their stdlib is so rich to write this kind of applications, while Rust was developed to be the system language for the next 50 years, that's why the difference in stdlib.

The only downside of Rust is that you have to pull more than 100 dependencies to build a simple hello world server (Tokio + Axum | Actix), then you need a database driver, most likely something that depends on SQLx (not sure here how many, dependencies), but one of my projects got near 500 transitive dependencies with only 20~ish direct dependencies.

trimethylpurine• 2 years ago

sanity - Soundness of judgment or reason.

This doesn't sound extreme. The statement, "designed with sound judgement," doesn't carry the implications that you're defending from.

pjmlp16• 2 years ago

Python (1991), Java (1995), .NET (2001), Smalltalk (1972), Common Lisp (1984), Ruby (1995), Perl (1993).

Go is following not leading.

To this day ISO C and ISO C++ still don't include TCP and UDP on their standard library, that comes from POSIX, the UNIX APIs that didn't make it into neither ISO C nor ISO C++.

jchw3• 2 years ago

Go is not quite the same but definitely on a similar level of abstraction to C, C++, Rust, etc. If that statement makes you raise an eyebrow, I'd suggest that while Go requires more runtime than C or Rust, it does still give you:

- Direct memory access through `unsafe`. Python kinda does through `ctypes`?

- Tightly integrated assembly language; just pop Go-flavored assembly into your package and you can link directly to it.

- Statically-compiled code with AOT; no bytecode, no interpreters, no JITs.

Therefore what really sets Go apart is that it gives you all of these rich standard library capabilities in a relatively lower level language.

Of course I kind of understand why Rust and C++ don't put e.g. a PNG decoder in the standard library, I think this is somewhat an issue of mentality; those are things that are firmly the job of libraries. But honestly, I wish they did. It's not like the existence of things in the standard library prevents anyone from writing their own versions (It doesn't stop people from doing so in Go, after all), but when done well it provides a ton of value. I think nobody would claim that Go's flags library is the best CLI flag parsing library available. In fact, it's basically just... fine. But, on the other hand, it's certainly good enough for the vast majority of small utilities, meaning that you can just use it whenever you need something like that, and that's amazing. I would love to have that in Rust and C++.

And after experiencing OpenSSL yet again just recently, I can say with certainty that I'd love Go's crypto and TLS stack just about everywhere else, too. (Or at least something comparable, and in fairness, the rustls API looks totally fine.)

tptacek3• 2 years ago

I have deep respect for Python and like working in it, but Python's standard library has only one of the attributes listed on the comment you're responding to. The Go standard library is different in a variety of ways from Python's standard library, and, to the original commenter's point, it is far more common to replace portions of the Python standard library with third-party packages than it is with Go's standard library.

jsjohnst2• 2 years ago

Yet, we still have some languages who's developers refuse to include even a basic http API in their standard libraries in an age where even embedded systems have started to speak http.

There’s also the philosophy that the language core should be as minimal as possible. I’m not 100% sold on it, but there’s definitely a valid argument to be made in favor of it.

norman7843• 2 years ago

As I wrote in another comment, it depends on the goal of the language, Go was build for writing web services, that's why they have most, if not everything, you need to write this type of apps.

WuxiFingerHold3• 2 years ago

My take as well. It's not black or white. At times I wish Rust's std lib was greater (by the way, it's not small either), but other times I'm blown away by the quality of some community crates.

WuxiFingerHold3• 2 years ago

I really like that Go has a great std lib, but it comes at the cost of quality or at least feature fullness. Some examples:

- The csv package. Compare Go's csv package with Rust csv crate. The package in Go's std lib is close to useless compared to the Rust community crate.

- Logging. No tracing, limited providers, just recently added structured logging.

- The xml package is very slow and cumbersome to use compared to modern element-tree-like implementation. For larger docs you probably need to resolve to a community package.

- If I'm not mistaken, Java has a build in http server, which usage is probably very low. Instead people are using Jetty / Netty / Tomcat (? my Java times are long ago)

To repeat myself: I personally like the std lib approach of Go. I disagree to any narrow view on this topic.

dang• 2 years ago

Can you please not take HN threads into programming language flamewar? We're trying to avoid that kind of thing here - see https://news.ycombinator.com/newsguidelines.html.

We detached this subthread from https://news.ycombinator.com/item?id=39710822.

the_gipsy3• 2 years ago

What about error stack traces? I find it crazy that you're supposed to grep error strings and pray they're all different.

pipe01• 2 years ago

You can do it yourself if you want using `runtime.Stack` and a custom error type

minzi• 2 years ago

Embarrassingly, I’ve been writing Go for a while but never really thought about it. Now that it’s been mentioned I’m curious why this isn’t baked in by default for errors. Does anyone know?

Xeoncross3• 2 years ago

Errors are different in every stdlib or 3rd party package I've ever used.

The ecosystem as a whole decided that errors should be useful, values, and properly bubbled up - and that your application should not be so hard to reason about that you need a stack trace to know where in the deep recesses of your code something went wrong.

I love stack traces in Java, but I don't miss them at all in Go.

the_gipsy3• 2 years ago

Sounds a bit apologetic. I don't think that the ecosystem decided. The language creators decided, and the ecosystem couldn't do much about it.

eweise• 2 years ago

"your application should not be so hard to reason about that you need a stack trace to know where in the deep recesses of your code something went wrong."

Curious how Go is better than other languages in this regard.

tbarbugli2• 2 years ago

we wrap all errors with a stack.Wrap function that adds stacktrace to the error. This allows us to add stacktraces to logs and err reporting tools like Sentry. Huge time saver

kjqgqkejbfefn• 2 years ago

I've started development of something similar for Clojure's core.async library, which implements Go-like channels. This is something I cobbled together to trace the origin of a subtle sporadic bug that would happen once every 10,000 runs in a program that makes heavy use of channels. I'm not really working actively on it, so it only covers a subset of the functions in core.async but I keep on expanding it, adding debugging support for functions every now and then when I need it.

It represents the flow of data between channels/threads as a sequence diagram using PlantUML. Today I implemented core.async/onto-chan! (it spins a thread that will take items from a sequence and put them onto a channel). Here's what it looks like:

https://pasteboard.co/VhvDroREOvOQ.png

It's especially useful in Clojure, as the experience with channels is not as polished as in Go (or so I heard): when a put or a take hangs, your only recourse is to sit and squint at your code. This tool will color them red, so you can immediately spot what's wrong. It also allowed me to spot a channel spaghetti plate I had not anticipated and wouldn't have remarked otherwise.

For now I've taken a "maximalist" approach which includes inspecting buffers, so it incurs a heavy runtime penalty that's only ok at dev-time (+ it keep tracks of the whole trace, so no flight recording in sight for now).

In the future I'd like to give it a proper interface (it's just a svg file with hoverable elements for now), maybe using constraint-based graph layout to lay out the sequence diagram using cola.js/adaptagrams or the sequence diagram from Kiel university Kieler project when it's integrated into the Eclipse Layout Toolkit. Thesis from the developer of this module:

https://web.archive.org/web/20220519080528id_/https://macau....

More powerful Go execution traces

Related Stories