Core dumps, feedback loops, and a game of telephone

https://dubroy.com/blog/core-dumps-feedback-loops-and-a-game-of-telephone/

by pdubroy • 4 years ago

81 12 4 years ago

I wonder what happened to Ray? Did the experience inspire him to learn more low level programming and switch job?

pdubroy2• 4 years ago

Author here. Unfortunately, I don't actually know what happened to Ray...and it was long enough ago that I've forgotten his name!

exikyut• 4 years ago

Y'know, I'm thinking of a company that starts with those letters, and might understandably have some high-security concerns...

markus_zhang4• 4 years ago

I'm sad this post is not bery popular. Does anyone else have their core dump story? What kind of jobs require such experience aside from compilers?

lazypenguin• 4 years ago

It's quite common to use crash dumps. I run a game server, we compile our release builds with debug info and deploy them to production. Sometimes the server runs for hours before crashing so we can end up with a pretty big dump. When the server crashes, we grab the dump and the correct debug symbols and we at least get a call stack to investigate and (hopefully) come up with a reproducible case.

At a previous job, we developed a desktop application. When the application crashed it would allow the user to submit the core dump to our crash system. We would then have a crash dump + application version and we could investigate customer issues to see if there was a bug or something like out-of-memory issues.

Fairly standard practice for a lot of shops. I haven't seen it for web applications since it's mostly dynamic languages with exception handlers that ultimately trap the exceptions (e.g. sentry).

markus_zhang4• 4 years ago

Thanks for sharing. I work as a data engineer who writes Python and SQL so I never go deep.

gp• 4 years ago

We are currently building a core dump based scheme to monitor crashes on our deployed devices. Devices use a different architecture than our development environments, and the deployed devices are often accessible only with limited bandwidth, so a live shell session can be more painful than waiting for the core dump to come back.

bitcharmer• 4 years ago

I don't think dealing with core dumps is that uncommon if you work close to the infrastructure level and the environment comprises many different technologies and high machine count.

In my case of high performance Java, core dumps were immensely useful for investigating issues like JVM bugs (C2 compiler bugs, GC bugs, classloader bugs) but also kernel bugs and hardware issues as well.

ollien• 4 years ago

What kind of jobs require such experience aside from compilers?

I work on an SD-WAN router (though on the "systems" side of things like configuration and upgrades, not routing or anything like that). The vast majority of the code we write in is C++, so when we get a crash, basically all we have are logs and core dumps; there's no stack traces in there for us!

markus_zhang4• 4 years ago

Thanks, looks like a must have skill for this kind of jobs.

saagarjha• 4 years ago

But the worst part was that it took a week or more to get a crash dump from them, because their lawyers insisted on examining everything to make sure it didn’t contain any sensitive information.

Coredumps can be pretty large and full of junk, so I wonder what the lawyers were doing when they inspected them…

pdubroy2• 4 years ago

The rumor at the time was that they actually printed them out and went through them with a black marker.

But now that I think of it, it couldn't have been true. With Courier 10pt you can fit about 4KB per page. So a 1GB core dump would would 256K pages!

Maybe they weren't examining anything, and it just took that long for them to sign off and allow the core dumps to be sent.

Core dumps, feedback loops, and a game of telephone

Related Stories