Debugging an evil Go runtime bug: From heat guns to kernel compiler flags (2017)

https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/

by goranmoomin • 2 years ago

206 33 2 years ago

mseepgood• 2 years ago

(2017) Previous discussion: https://news.ycombinator.com/item?id=15845118

rollulus• 2 years ago

Ah, I remember 2017’s HN where the mere word “Go” in the title made posts skyrocket to 498 points. In 2024 the title should’ve contained “Rust” for the same outcome. Quite curious what will be required in 2031.

aapoalas• 2 years ago

Maybe Zig or Carbon? Just spitballing, personally it's of course Rust forever.

keybored• 2 years ago

Rust has always had a fair number of supporters on HN. Certainly since 2015.

freep1zza• 2 years ago

Try them you'll love it :)

BobbyJo2• 2 years ago

This is honestly wild. 99% of devs would have found a work around and moved on. Going so far as to create a multi-kernel test bench to narrow down the source of the instability is a level of dedication I have not personally seen, and I respect it.

randomdata• 2 years ago

By the same token, you might be the first person I have ever seen give respect to the 1x developer. I respect that. We could no doubt all learn a thing or two from the 1x developer that doesn't rush through everything with quick solutions.

sanbor• 2 years ago

If you check the issue[1] he reported the crash on November 7th and reported the issue is related to gcc and the kernel on November 8th. At least he was very quick going through the rabbit hole.

[1] https://github.com/prometheus/node_exporter/issues/730

eikenberry2• 2 years ago

Is the 10x thing really only about speed? I would say this guy is a perfect 10x example as he actually gets to the root cause of a difficult problem. When I think of 1x (or less) devs they are usually the type that don't get things done because they can't (without a lot of help), not because they are slow. I.E. overall technical chops, not just speed.

silisili• 2 years ago

I believe the 10x developer thing is stupid, and based on perspective. Watching someone work faster than you doesn't mean they are doing better work, and on the other hand could mean perhaps you are maybe a 1/10th developer.

That said, if a 10x developer does exist, Marcan is one of the few of them. What a ridiculous statement.

abbbi• 2 years ago

Marcan is an beast, this guy really loves to go down the rabbit holes.

toast0• 2 years ago

Problems like this tend to come back and haunt you though. Sure, you can set max threads to 1 and move on with what you're doing for a while... but a lot of people run Go so they can have a lot more than 1 thread.

I've run into some of these where it's a lot more rare to hit, and so then it's reasonable to not do the thing that hurts, but watch out for it in the future. Sometimes you get lucky and it magically fixes itself forever; sometimes the weird case that you only hit with internal traffic ends up getting hit by public tratfic a lot.

Crashes like this where a wild write breaks something at a distance are always a PITA to debug (especially here, where the wild write is harmless if there's no data race)

im3w1l• 2 years ago

I feel like I still didnt fully understand what's going on here. Is the following correct? "Threads hava a 'canonical' stack that the OS auto-grows for you as you use more of it. But you can also create your own stack by putting any value you want in RSP. This is what the Go program did, and the vDSO, assuming it ran on an auto-growing stack, tried to probe it, which lead to corruption."

derefr• 2 years ago

I believe that Golang, as a green-threaded runtime, is allocating a separate carrier thread for running syscalls on, so that a blocking syscall won’t block the green-threads. These syscall carrier threads are allocated with a distinct initial stack size + stack size limit than green-thread-scheduler carrier threads are, because it’s expected that syscalls will always just enter kernel code (which has its own stack.)

But vDSOs don’t enter the kernel. They just run as userland code; and so they depend on the userland allocated stack to be arbitrarily deep in a way that kernel calls don’t.

As shown in the article, Golang seems to have code specifically for dealing with vDSO-type pseudo-syscalls — but this is likely a specialization of the pre-existing syscall-carrier-thread allocation code, and so started off with a bad assumption about how much stack should be allocated for the created threads.

(I should also point out that the OS stack size specified in the ELF executable headers, only guarantees the stack size of the initial thread of a process created by exec(2). All further threads get their stacks allocated explicitly in userland code by libpthreads or the like calling malloc(2). Normally these abstractions just reuse the same config params from the executable (unless you override them, using e.g. pthreads_attr_setstacksize). But, as the article says, Golang implements its own support for things like this, and so can implement special thread-allocation strategies per carrier thread type.)

jeffbee• 2 years ago

Thread stacks in Linux are demand-paged: if you touch the next page then it magically exists, up to a limit. But the machine is not concerned with the convenient properties of this virtual memory area. To the CPU the register RSP is just an operand, expressed or implied, to some instructions.

saagarjha• 2 years ago

The problem is that the vDSO (which is compiled as part of the kernel but runs in userspace) does a stack probe for security reasons, trying to see if it will overrun the stack. It does this by checking if at least a page’s worth of data is accessible. If not, it will (typically) fault. However, Go programs use a stack size so small that they may have other data a page away, which means the problem may mess with that data and cause bad things to happen.

SPascareli13• 2 years ago

If the orq instruction was really a noop then there would be no problem as well?

wolf550e• 2 years ago

You can follow Hector Martin @marcan at https://social.treehouse.systems/@marcan/

He works on Asahi Linux, a Linux port to arm64 Apple hardware.

Thaxll• 2 years ago

One thing I learned from that post back then is that you can instruct Grub to ignore some part of your physical memory. Really nice trick, not sure this is doable on Windows / Mac?

dwattttt• 2 years ago

I was just looking at the bcdedit options earlier today, this is doable on Windows

Agingcoder• 2 years ago

This is very elegant. I’ve had my share of nasty system bugs ( compilers and kernels ) , but the dedication and the speed with which he went through it is quite remarkable.

The explanations are also very clear. Thanks for posting.

ncruces• 2 years ago

Related (for the hash based bisecting): https://research.swtch.com/bisect

fulafel• 2 years ago

I guess Gentoo Hardened meant the difficulty level in this case.

umvi• 2 years ago

I'm pretty confident in my computer abilities, but when I read stuff like this I feel like I have no skills at all compared to this guy. Like I'm still a high school athlete and he's an Olympian (also he was only 26 when he wrote the article).

switch007• 2 years ago

but when I read stuff like this I feel like I have no skills at all compared to this guy

Comparison is the thief of joy!

Terr_• 2 years ago

Over the course of 22 kernel builds, I managed to simplify the config so much that the kernel had no networking support, no filesystems, no block device core, and didn’t even support PCI (still works fine on a VM though!).

Flashbacks to a job where I was asked to figure out why a newer kernel was crashing. This was a very frustrating time, because I had (have) basically zero real C/C++ experience but I'd helped out with Bitbake recipes and everyone else was busy or moved to other projects.

To cut a multiweek tale of dozens of recompilations short: The kernel was fine. The headless custom hardware was fine. The problem was a hypervisor misconfiguration, overwriting part of the kernel address space. All of our kernels have been corrupt, but this was the first one where the layout meant it mattered.

A month of frustration, two characters to fix, the highest ratio I've encountered so far.

My reward for struggling through a complex problem I was unqualified for? "Great, now we need to backport security patches from the main Linux kernel to the SoC vendor's custom fork..."

Hajcnga• 2 years ago

Could someone explain why Google is obsessed with small stack sizes? The musl library also has an extremely small thread stack size, which makes many applications crash that are used to 8192K on Linux.

On Linux the 8192K aren't reserved unless the are actually used, so what is the point?

Ok, Golang will allocate green threads from its own allocator, but 104 bytes?!

neonsunset• 2 years ago

I wonder if they might be favoring spawning numerous instances of an application with few threads and little memory use to scale at the replica level over just scaling within an application? At least Go defaults do suggest that (and are also the reason for very poor performance on large many-core nodes).

samatman• 2 years ago

I don't believe that Google has anything at all to do with musl libc.

Applications should not be assuming a hard-coded stack size, that's a bug. They should be checking the value of PTHREAD_STACK_MIN. That is what it's there for.

Debugging an evil Go runtime bug: From heat guns to kernel compiler flags (2017)

Related Stories