Time Traveling Linux Bug Reporting: Coming in Julia 1.5

https://julialang.org/blog/2020/05/rr/

by KenoFischer • 6 years ago

164 21 6 years ago

wallnuss• 6 years ago

I recently spend two weeks on and off hunting down a bug on a platform that didn't support `rr`. I am fairly confident to say that if I had `rr` available it would have taken me a couple of hours at most.

Being able to run backwards from the point of failure and understand where a value is coming from is very powerful.

Having this available in Julia directly is great, and will make it much easier to get bug-reports from users.

roca6• 6 years ago

This is very cool stuff. Keno is a great rr contributor, and I'm grateful for his Pernosco shout-out :-).

This sort of capability has always been part of my vision for rr/Pernosco. I want to see more projects like this doing record-and-replay capture of bugs in the field. With more projects depending on this sort of infrastructure, we can get more attention from OS/hardware people to make the record-and-replay systems better, and then I think we can reach a tipping point where it becomes best practice. Eventually we should be looking back and wondering why, as an industry, we didn't go in this direction earlier.

KenoFischer6• 6 years ago

No, thank you for all your work on this :). Building something like rr is only possible with some really solid engineering and I think you managed to accomplish that. I think we're aligned on the vision here. I bring up rr every time I speak to hardware people, but of course the first question they ask is "how many people need this right now?". With this I'm hoping to be able to increase the answer to that by an order of magnitude or so ;).

Veserv3• 6 years ago

If you don't mind me asking, you mention that you want more attention from OS/hardware people. What sort of things are you looking for from OS/hardware people that would make replay systems better?

As for a more technical question on implementation. Assuming the post accurately represents the performance characteristics of rr, what do you think is keeping the overhead so high in the average case described in the post (50%)?

Disclaimer: I work for a company that sells similar technology.

KenoFischer6• 6 years ago

I have done some experiments and it's reasonably possible to cut the average overhead to about 20% or so with known techniques. That said, I'm surprised you'd consider this high. I know of no system, including commercial ones that can even get close to these kind of low overhead numbers. Heck, I don't even know of one (other than rr) that actually works on applications of this complexity. If you have one, you should advertise it significantly more heavily ;).

roca6• 6 years ago

Some small-ish items:

-- Virtualization of rr's required PMU features on Azure and GCP

-- Virtualization of Intel's CPUID faulting on AWS and other cloud providers (Linux KVM has it, but it's not enabled in AWS)

-- Fix the reliability bug(s) in the retired-conditional-branches counter on AMD Ryzen so rr works there

-- Implement CPUID faulting in AMD Ryzen

-- Reliable instructions-retired counter on AMD/Intel (for more reliable/simpler operation)

-- Trap on LL/SC failures on ARM (AArch64 I guess) and other archs (so we can port rr there), with Linux kernel APIs to expose that to userspace

-- Similar traps for Intel HTM (XBEGIN/XEND) (less important now that's mostly been disabled for security reasons)

Bigger items:

-- Invest in kernel-integrated record-and-replay for Linux and other OSes

-- Implement QuickRec or some other hardware support for multicore record-replay on at least some CPU SKUs

(If anyone thinks they can help with any of that, talk to me!)

what do you think is keeping the overhead so high in the average case described in the post (50%)?

I assume you've read https://arxiv.org/abs/1705.05937 ? For many workloads the major unavoidable cost is context switching. With rr's approach every regular tracee context switch (intra or inter process) turns into at least 2 inter-process context switches (tracee to rr, rr to tracee). This is bad for thread ping/pong-like behaviour. For many other workloads the cost is simply the loss of parallelism as we force everything onto a single core. For other workloads there is a high cost due to system calls that require context switches to rr because we don't currently accelerate them with "syscall buffering". The last one can be mostly engineered around, the former two are inherent to rr's approach.

Who do you work for? :-)

KenoFischer6• 6 years ago

Hi all, I'm pretty excited to get this out there for you. I had published this this morning, so I could be around through the day to answer any questions, but things were a bit delayed by the HN algorithm gods (- thanks dang for rescuing it ;) ). That said, I'll check in periodically for the next hour or two if there's any questions I can answer.

tgflynn• 6 years ago

I'm surprised I haven't heard of rr before, it sounds like it could be a game changer for debugging many types of problems. How long has this project existed/been usable ?

Am I correct in understanding that rr can be used with any application (ie. the application doesn't have to be built specifically to support it) ? That's the impression the usage introduction on the website gives: https://rr-project.org/.

KenoFischer6• 6 years ago

How long has this project existed/been usable ?

It's been our (people who work on Julia) daily debugger since 2015 or so

Am I correct in understanding that rr can be used with any application

Yes

boromi• 6 years ago

is this coming to Windows or Mac?

Time Traveling Linux Bug Reporting: Coming in Julia 1.5

Related Stories