Rreverrse Debugging

http://huonw.github.io/blog/2015/10/rreverse-debugging/

by dbaupp • 11 years ago

142 46 11 years ago

huonw.github.io

wyldfire• 11 years ago

I’ve never been a huge user of debuggers. Being able to diagnose my code line-by-line, statement-by-statement sounded theoretically good to me, but I’ve never really “clicked” with it in practice.

Years ago, I would've been floored by a comment like this. But lately I find myself using them a little less often. These days, the codebases I interact with leverage more and more small-scope unit tests. This usually means that an unexpected test failure requires much less imagination to explain.

But debuggers remain a terribly simple way to identify the cause of a segfault/bus error. Even with code I've never seen -- if armed with just a stack trace I can identify a system misconfiguration or a workaround to avoid the error.

dbaupp2• 11 years ago

I definitely use a debugger for getting a backtrace from a hard crash, but it is (well, was) pretty much the only case where I'd reach for it as the first choice.

yoklov2• 11 years ago

Have you ever used a good visual debugger? I find that command line debuggers require you to keep too much state in your head at once, and even if you have a great memory and this isn't a problem, they still require you to query the new state after each step.

Sadly, most of the visual debuggers on linux don't meet my minimum standards for "good" (crashy, slow, typically have trouble inspecting certain types (mysteriously, since they're GDB frontends and GDB has no issue with it), etc...), and successive update to MSVC makes it's debugger worse and worse (although it's still the best I'm aware of at the moment)...

seccess• 11 years ago

Definitely agree for stack traces, but what about other debugger features like breakpoints, step-by-step execution, or memory introspection? When working with a large, multi-threaded system, I have found these features more trouble than they're worth.

EDIT: Grammar.

arthursilva• 11 years ago

Is there a good GUI wrapper for gdb or lldb?

josteink• 11 years ago

Ive tried using gdb with Emacs (via GUD[1]) if that counts, but it never worked out for me in the sense that I felt I got anything extra in return for the investment.

Maybe there's some special things I forgot to do, but out of the box GUD definitely felt (too) minimal for me.

Lately I've used LLDB when working on the .NET framework, and that seems like quite a reasonable thing to work with too at this point. I'd love some emacs-magic for this too :)

[1] http://www.emacswiki.org/emacs/GrandUnifiedDebugger

rkv• 11 years ago

https://cgdb.github.io/?

ihnorton• 11 years ago

Qt Creator's gdb integration handles the basics well (I haven't used it with lldb, but lldb supports the gdb MI protocol so it should at very least "work").

exDM69• 11 years ago

I generally use cgdb (or gdb -tui) but for some complex tasks, I use the ddd gui frontend. It's not a pretty gui and the usability is not great but the visualization it provides is excellent. I've used it for complex data structures and debugging numerical algorithms. Getting it to display structured data as you step through the code can be super helpful.

That being said, I hope that lldb will catch on and there will be some decent front ends for it. The GDB-MI (machine interface) used in debugger frontends is quite limited and clumsy. The lldb debugger should be a lot easier to embed in a project as a library (as opposed to using pipes and gdb-mi).

pzone3• 11 years ago

Qt Creator is great for Qt. KDevelop and Emacs gud-gdb also provide nice vanilla frontends.

rjbwork3• 11 years ago

"Imagine"... I've been doing it in Visual Studio for years.

Step 1. Find problematic code

Step 2. Step back to before problematic statement

Step 3. Change problematic data value and/or code to hypothesized good one (while debugger is active and paused)

Step 4. Continue execution and observe if output is as desired

Step 5. Write test case.

Easy peasy. I'm glad to see the Rust community doing something similar, as the benefits of ease of development are not to be discounted, and Rust seems like a pretty nifty language.

devit• 11 years ago

That seems to describe memory editing plus "edit and continue", which is distinct from record&replay and reverse execution.

Also, rr has nothing to do with Rust in particular, although this article shows how to use it with Rust.

ZenoArrow2• 11 years ago

"That seems to describe memory editing plus "edit and continue", which is distinct from record&replay and reverse execution."

Sounds more like IntelliTrace to me (if you're not familiar with it, it's worth mentioning it's only available in Visual Studio Ultimate).

https://msdn.microsoft.com/en-us/library/dd572114.aspx

rjbwork3• 11 years ago

Step 2 was explicitly stepping backwards i thought. The rest was just a continuation of what the ability gives you and what I often do after stepping backwards to observe the execution.

spott• 11 years ago

rr works for C++/C code too?

dbaupp2• 11 years ago

Yeah, for sure; as I say in "Removing the rose-coloured glasses", rr is definitely not the first tool to do this, it's just the first one I've been lucky enough to use. I've not done much development on Windows, so haven't had an opportunity to use/enjoy Visual Studio.

Also, I'm sure there's a large chunk of people like me, who don't use debuggers much and have no idea what a good one is capable of.

pjc50• 11 years ago

This sounds exciting so I thought I'd look it up: http://stackoverflow.com/questions/26431737/can-i-get-revers...

"Visual Studio (2010, 2012, 2013 Ultimate only .. does not support C++"

shinymark• 11 years ago

You can still move the program counter and modify arbitrary data in memory. It's not the same but I wanted to mention that as it comes in handy for me frequently.

interdrift• 11 years ago

Too bad this sometimes moves your focus from the problem you are solving and makes you an output matching machine.

rjbwork3• 11 years ago

HAH! True, but it's just one tool in a vast array of problem solving tools.

fredcy• 11 years ago

Elm has a "time traveling debugger", http://debug.elm-lang.org/

I'm eager to give it a try in a real project.

alphabetam• 11 years ago

IntelliJ's debugger can "drop frame", after which you can re-run a function.

pron• 11 years ago

For those interested, Java has a number of such "time-travelling" debuggers (I've used one to great effect):

* TOD: http://pleiad.cl/tod/index.html

* Chronon (commercial): http://chrononsystems.com/

* Jive: http://www.cse.buffalo.edu/jive/

* Whyline (research): http://www.cs.cmu.edu/~NatProg/whyline-java.html

* Omniscient Debugger (discontinued?): http://www.lambdacs.com/debugger/

karussell• 11 years ago

Thanks for this list!

And even without: an IDE like NetBeans (and probably any major Java IDE) can revert to a certain stack frame and even apply code changes on the fly - up to a certain degree of freedom only, but okayish for normal programmer life, where you need this only rarely if you have many smaller scope unit tests in place.

MichaelMoser123• 11 years ago

I just saw a presentation that describes the rr vm/recorder in more detail. I am very impressed (though i haven't got the chance to use/test it).

https://youtu.be/H4iNuufAe_8

pure science fiction - instead of recording the change produced by each instruction they are recording what changed between system calls (result of system call is recorded) and scheduling points.

They do record the result of each system call and somehow manage to count the number of instructions per scheduled thread (they can only do that on new intel processors - counting the number of branches not instructions as instruction counter is not reliable. They are doing their own scheduling since the VM is all running in a single thread. It is also very Linux specific - for general case they use ptrace to record the system calls and their result, but tracing of some operating system calls is optimized by injecting stuff into the kernel (!)) - that's the reason why the recorded data is of minimal size.

I wonder how they are dealing with epoll - here the result is passed via shared memory and not via the system call interface. Still i guess they are lucky that there is no kqueue like interface on Linux - with kqueue it would have been even harder to track when the event result comes in.

YorkianTones• 11 years ago

Windows has had this capability for at least 8 years with "time travel tracing" (or iDNA tracing), which produces a dump file that can be traversed back and forward in WinDBG or Visual Studio. Very useful for figuring out what caused unexpected state in complex integration scenarios with timing-dependent bugs. http://www.thewindowsclub.com/microsoft-time-travel-tracing-...

necessity• 11 years ago

It's called trace/execution replay and has been around for decades in various platforms, not just Windows.

moyix• 11 years ago

iDNA traces are huge though, and recording them adds a ton of overhead. rr is way more lightweight.

padator• 11 years ago

OCaml had a time-traveling debugger for 20 years now. Good job catching on. (to be fair it got full stack back trace only since 2000 so far later than Perl for that).

fulafel• 11 years ago

GDB has had it for many years too. I think Mozilla made rr to handle large apps better.

ZenoArrow2• 11 years ago

This might be impractical, but is it possible to record gdb traces using a different processor, as long as that processor has access to the RAM the program is running in?

For example, could you record gdb traces with a GPU in a PC with hUMA?

http://www.tomshardware.com/news/AMD-HSA-hUMA-APU,22324.html

That could allow for gdb traces to be recorded constantly without a CPU performance hit (traces could be truncated if/when the size grew too large).

chris_wot• 11 years ago

Oh man, this would have made my life troubleshooting a font layout issue in LibreOffice one hell of a lot easier :-(

pnathan• 11 years ago

My master's work was in the context of debugging. I spent a great deal of time reading descriptions of ancient debugging systems - most far in advance of the capabilities of the present day. The height of the art was reached in the early 90s - sophisticated logic structures over your program state were being held over your program state to automatically trigger actions - reverse debuggers were done back in the '70s as I recall; I believe the last major advance was done by an Italian chap in '92. Even SLIME and Common Lisp were quite well passed up by these capabilities. Most of those advances took place in the Lisp and Prolog world. I infer that this was due to the symbolic execution capabilities which the Fortran/Algol families have disavowed.

So it's always nice seeing advances in the industrial art come out, but a little wistful, knowing what once was.

EdiX• 11 years ago

I would really like reading more about this.

pcwalton• 11 years ago

To those saying "this has been done before in X language", "what's new in this", etc., see the presentation: https://mozilla.github.io/rr/rr.html

The real interesting thing with rr is how it's done at the bare metal with very low overhead, using deep hardware-level tricks involving the performance counters on modern x86 CPUs. It's relatively easy to do this when you have a VM, but doing it on native code running directly on the CPU is significantly more interesting.

helmut_hed• 11 years ago

A commercial product covering similar territory is UndoDB http://undo-software.com/undodb/

barteklev• 11 years ago

Yeah I love GDB. It's so mighty. The interesting that you can script your debuging and it's just mindblowing. It helped me much in debuging net communication. It was tough, but GDB let much! And It's for C/C++ too.

Rreverrse Debugging

Related Stories