Mozilla's low-overhead open source replay debugger

http://robert.ocallahan.org/2014/03/introducing-rr.html

by bzbarsky • 12 years ago

272 42 12 years ago

robert.ocallahan.org

dman• 12 years ago

Another gem from Mozilla - I am astounded at how much good stuff is coming out of Mozilla in such a short timespan. Keep it up folks. Between this, rust, servo, asm.js and emscripten - you have my attention.

icarus127• 12 years ago

I don't know about the author but not all of us oppose Gay marriage.

malandrew• 12 years ago

Please don't confuse the personal views of the CEO outside of work with the views of the entire organization. There are a lot of genuinely good people at Mozilla doing good work, and I would bet that the overwhelming majority working there do not share the same views as the CEO. You're comment not only does a disservice to yourself and the HN community, but everyone else at Mozilla working to make the world a better place.

wmat• 12 years ago

This looks really interesting. I hope it doesn't get buried in the Facebook buys Oculus news tsunami.

I can't wait to try rr out.

moyix2• 12 years ago

This looks very cool, and will undoubtedly help with most run of the mill crashes. Unfortunately the machine model it uses is single-core, so you're not going to necessarily catch the really subtle concurrency bugs. Good multicore record and replay is hard, and has been the subject of tons of papers in the past 10 years. It would be cool to see some of that research make it out into practice.

sanxiyn5• 12 years ago

http://rr-project.org/rr.html is a presentation titled "How rr works" from Mozilla Research, with more details on the implementation.

mbrubeck• 12 years ago

It looks like the presentation is still a work in progress; many of the later slides just have "TODO: example" or similar. But it does contain some interesting content.

maninalift2• 12 years ago

...but that's great.

Wow, this is a wonderful thing.

I would go as far as to say it is the tool every developer wanted and most thought was impossible (the low overhead part at least).

Only recording the nondeterministic parts and replaying the rest is so obvious, once someone else has thought of it.

maninalift2• 12 years ago

except me of course cos I don't write bugs

gopalv• 12 years ago

This is just a bit too awesome, yet simple.

Think of it as the "git" for debugging. Take a snapshot, keep going etc ... and then backtrack back to the point where it all started going wrong.

Just very hard to automate easily, yet this uses gdb and all the other macro/command goodness (well, for power users) that gdb comes with.

grundprinzip• 12 years ago

"rr currently only supports x86 32-bit processes."

While the idea of a replay debugger is really nice, I don't understand where this limitation comes from. I thought, that since years we practically live in a 64 bit world.

chetanahuja• 12 years ago

Yes. This. This sounds like a great project and I would love to integrate this into our debugging workflow but this one limitation basically kills that prospect.

iSnow• 12 years ago

Well, it is version 1.0. If it is that important, maybe you could port it and issues some pull requests.

btown3• 12 years ago

There are a lot of low-level intrinsics that need to be called behind the scenes to do the recording, and I'm not surprised that they did their first experiments using the 32-bit versions, since they might be better documented, or there might be more expertise on the team if they've been working on 32-bit low-level stuff for decades. All those intrinsics should have 64-bit versions, though, so it's just a matter of time before 64-bit support lands. I'm sure they'd welcome pull requests!

masklinn• 12 years ago

I don't understand where this limitation comes from.

Time and money? Such as scheme requires a lot of very low-level ops (including things which are not available on all architectures), x86 is simpler and better-understood than x64, starting the project there seems as good a choice as not.

sanxiyn5• 12 years ago

You can use rr on 64-bit systems. rr requires 32-bit processes at the moment, but does not require 32-bit systems. 32-bit application on 64-bit kernel is a supported configuration.

polskibus• 12 years ago

Is this something similar to Intellitrace in Visual Studio?

sanxiyn5• 12 years ago

No. IntelliTrace records, but does not replay.

btown3• 12 years ago

An amazing project. Would it be possible to get to a certain point in a trace, then switch the process back into "live mode" so that further inputs could be made into the program? This could open up all sorts of interesting things, like recording interactions with web applications and then allowing a developer to open up high-level JS/DOM debugging tools to see exactly what went wrong without needing to worry about replicating the exact input sequence that led up to the bug.

It's unfortunate that the no-shared-memory limitation is so hard-baked in, so it would seem that things like Chrome rendering processes wouldn't play nicely with rr (I'm assuming they communicate with shared memory, correctly if I'm wrong!).

sanxiyn5• 12 years ago

As I understand, you can't "go live" with the current architecture. In replay, no system call is executed. Write is no-op. Read is done from the trace. This is how PTRACE_SYSEMU works. To "go live", you would need to actually execute system calls.

http://sysemu.sourceforge.net/

Re: Chromium IPC. "Channels are implemented using named pipes on Windows, and socket pairs (or in some special cases unix domain sockets) on POSIX."

http://src.chromium.org/svn/trunk/src/ipc/ipc_channel.h

btown3• 12 years ago

Hmm, was afraid of that re: SYSEMU. Still, I'm sure people will find ways to do higher-level debugging in rr with the current architecture!

e98cuenc• 12 years ago

The latest gdb also contains a reversible mode. Anybody can comment on how does it compare with rr?

I have only tried myself the reversible debugger in ocaml, and it was an eye opening experience.

sanxiyn5• 12 years ago

The critical word is "low-overhead". gdb's implementation is orders of magnitude slower.

On the other hand, rr can't do reverse debugging.

moyix2• 12 years ago

The lack of reverse debugging isn't an inherent limitation. One could index the replay and take micro-checkpoints along the way; then when you want to step backward, you rewind to the last checkpoint and play forward until you hit the previous instruction.

fijal2• 12 years ago

gdb reversible mode is useless for anything other than a toy example. We never got to the point where you can e.g. start the PyPy process, not to mention do anything with it. We had much more luck with UndoDB, but this one is still ~20x slower than native.

fulafel• 12 years ago

Was submitted yesterday: https://news.ycombinator.com/item?id=7458554

It seems to me that a large portion of cool stuff doesn't make it out from /new any more (not just this case, many others lately)

fijal2• 12 years ago

I was tracking a few of my own submissions - I think it's random. IF you happen to arrive on the frontpage and stay there for a few minutes, a reasonable amount of people will vote that up. However, if you stay on the front page for too short or not at all, the random fluctuation might kick you down. This is e.g. raspberry pi foundation linking to pypy blog vs submission of the pypy blog itself.

danmaz74• 12 years ago

I think it would be great to show one random "new" submission at the bottom of the front page on every page load (it could be marked as "just submitted"). Of course spammers would love that, but I think it would make the discovery process much better, and the end result would be positive.

nviennot• 12 years ago

I've been very interested in that topic for the past years.

We've implemented an in-kernel linux multicore record/replay engine. See for more: http://velvetpulse.com/2012/11/27/scribe-the-deterministic-t...

We also did a mutable replay engine, to replay on modified program (e.g. replay a bug that happened in a production environment, but this time with the logging verbosity turned into debug mode). Check out Dora: http://viennot.biz/asplos2013_dora.pdf

qwerta• 12 years ago

Replay tool is also for Java. It is called Chronon, is Eclipse plugin and was recently integrated into Idea Ultimate.

Dorian-Marie• 12 years ago

Link to the project: http://rr-project.org/

And the github: https://github.com/mozilla/rr

(On a side note, I think the gradient background is worse than a plain white background, when removing it, it become a lot more readable)

ArbitraryLimits2• 12 years ago

No one's going to rag on the post's author for labeling himself "Christian" right at the top? He might have donated to Prop. 8 after all.

Edit: That's sarcasm, people.

drderidder• 12 years ago

It's his personal blog and there's this thing called free speech.

gjm112• 12 years ago

Sarcasm fully understood. So, to address the point you're making underneath the snark:

No one (that I've seen, anyway) is ragging on Brendan Eich for being a Christian. I don't even know whether he is, although it's a fair guess given his decision to support Proposition 8.

Many people are saying that Eich's support of Proposition 8 is a bad thing, makes him a bad choice for CEO of Mozilla, discourages them from using Mozilla's products, etc.

Attacking someone for being a Christian: religious bigotry based on disapproving of someone simply on the basis of their identity. Generally a bad thing.

Attacking someone for supporting Proposition 8: disapproval based on what the person has actually done and its likely consequences. No reason why it shouldn't be a reasonable thing.

This doesn't seem to me to be a very subtle distinction.

(Aside: I do find it curious that of the four people who first come to mind when I think of Mozilla -- Mitchell Baker, Gervaise Markham, Robert O'Callahan, Brendan Eich -- two are rather vocally Christian (gerv's blog is called "Hacking for Christ"!) and a third has become infamous for an action that's probably the result of a similar religious position. Just coincidence?)

ArbitraryLimits2• 12 years ago

I'm confused, are you saying there's a link between people's beliefs and actions or not?

bzbarsky• 12 years ago

The question I have is why it's those four people who come to mind for you. Why Mitchell and Brendan is very clear, since they are the ultimate project leadership. But why specifically Robert and Gerv, as opposed to David Baron or Asa Dotzler or Johnny Stenback or Stuart Parmenter or Mike Shaver or various other people who were involved with the project since close to the beginning?

I have some hypotheses (e.g. Johnny doesn't blog, so isn't as "famous"), but I'm curious whether you have any yourself.

userbinator• 12 years ago

This has some similarities with a project I worked on that provided very advanced analysis and debugging capabilities including instruction-level "bidirectional execution". I can't say much more about that, other than the fact that it was for a whole-machine environment, and was a combination of special hardware support, but it's good to see some of the techniques becoming more widely available to the public.

jderick• 12 years ago

This reminds me of a similar idea I'd like to see. A smart tracer that keeps track of function call args and results and possibly local variables in a cache. Then when your program crashes, it can show you not just the backtrace but also recently called functions that may have influenced the crash. With such a tool you may be able to debug some crashes without a debugger at all.

haberman2• 12 years ago

Very cool! This looks a lot like something Michael Chastain was working on in the late 90s: http://lwn.net/1999/0121/a/mec.html

It also is similar to GDB's record-and-reply functionality, except that I believe GDB is limited to a single process. I suspect that GDB's approach is much higher overhead also, because I think it actually records state changes for every instruction: http://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-...

Unlike the GDB functionality, this doesn't seem to support reverse-stepping. Hopefully this can be added -- it makes a huge difference when you can reverse-step backwards from the failure to find out where things went wrong: https://sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution....

Tobu• 12 years ago

In theory, the replayer could be made to rerecord a full memory trace. Running the replayer in gdb would be enough for a single process.

haberman2• 12 years ago

In theory, the replayer could be made to rerecord a full memory trace.

Yes exactly, or if it's cheaper, create periodic compressed snapshots and replay from the previous one to recreate intermediate states. In my experience GDB's recording is very expensive and very rapidly runs into resource limits. Hopefully a snapshot-based approach would be more scalable.

Mozilla's low-overhead open source replay debugger

Related Stories