Time Travel Debugging

https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/time-travel-debugging-overview

by clouddrover • 9 years ago

100 49 9 years ago

docs.microsoft.com

jasonwelk4• 9 years ago

This is cool stuff to see in the world of C++. It's great to see language communities feed off each other and drive innovation like this.

cma• 9 years ago

I think it has been in gdb before Swift.

userbinator3• 9 years ago

Yes: https://www.gnu.org/s/gdb/news/reversible.html

This has approximately nothing to do with C++ or even "language communities".

userbinator3• 9 years ago

TTD is efficient and works to add as little as possible overhead as it captures code execution in trace files.

...and then in the table below the screenshot...

Large overhead at record time. May collect more data that is needed. Data files can become large.

Based on past experiences with debugging using tracepoints etc., I'm more inclined to believe the latter.

Also, as a side-note, the "more modern visuals" of "WinDbg Preview" look horrible. It's a debugger, not a toy for the barely-computer-literate. Those who want the "friendly experience" will use Visual Studio instead, which amusingly enough continues to have menus instead of the disgusting ribbons as of the latest 2017 version. The screenshot also shows two lines of slightly-misaligned "Command" "Memory" "Source". Yuck.

(For those who don't understand, this what the original WinDbg looks like --- simple and functional: http://sandsprite.com/blogs/images/main_ui.png )

timmisiak8• 9 years ago

The overhead varies depending on how cpu/IO bound the application is. IO isn't really affected, so IO bound applications tend to not see a big slowdown. In theory, you could see a very large slowdown in the worst case, but in the average case for a "medium sized" process the slowdown would be noticeable but not affect the usability. This technology isn't based on tracepoints, it's based on in-process cpu emulation. The emulation overhead is on the order of 10-20x in many cases, whereas tracepoint overhead would be on the order of 1000x I believe (maybe worse).

If there is something specific you dislike about the visuals of WinDbg Preview, let us know through the feedback hub or emailing windbgfb@microsoft.com. We realize that folks that have been using WinDbg for 20 years are likely to not be interested in a new UI, but we face 20 years of legacy code every time we want to add a new feature to the UI. As an example, the new WinDbg UI has a javascript window for writing scripts that extend and automate the debugger. It took us approximately 8x less time to implement in WinDbg Preview than what we estimated it would cost in the legacy UI (and a much more junior dev was able to do it as well). We want to innovate without disrupting folks that have effective workflows in WinDbg, so we really want to hear feedback on the new UI. If there are specific things that we can change to make you more efficient in the new UI, please let us know.

userbinator3• 9 years ago

We realize that folks that have been using WinDbg for 20 years are likely to not be interested in a new UI, but we face 20 years of legacy code every time we want to add a new feature to the UI.

You can rewrite the UI code so it's easier for you to work on internally, but to the user it looks and acts like it was before --- but proceed cautiously[1]. I'm not against adding features and TTD sounds extremely useful, but having in the overview page some slight contradictions and a screenshot of a dumbed-down UI with obvious WTFs like the two duplicate lines really soured the first impression for me. (Looking at it again, I now see the path in the titlebar has been cut off, despite plenty of empty space after it...)

[1] https://www.joelonsoftware.com/2000/04/06/things-you-should-...

arbesfeld• 9 years ago

Very cool stuff! Microsoft also developed a TTD system for JavaScript/Node.js: https://www.microsoft.com/en-us/research/publication/time-tr...

At LogRocket (https://logrocket.com) we're building a TTD of sorts for recording production users in your web app. While we can't replay exact code, we can get you most of the way there with DOM recording/logs/network waterfall.

JoshTriplett• 9 years ago

Here's a review from the Mozilla folks: http://robert.ocallahan.org/2017/10/thoughts-on-microsofts-t...

alexashka• 9 years ago

Looks very nifty. Is this being developed for Windows only?

kuwze2• 9 years ago

There is also undodb[0] and gdb supports reversing[1]. There is also the rr[2] project from Mozilla. All of those support Linux.

I also recently discovered WDD[3] (which is for Windows) as well.

[0] https://undo.io/

[1] https://www.gnu.org/software/gdb/news/reversible.html

[2] http://rr-project.org/

[3] https://github.com/ipkn/wdd

nn3• 9 years ago

There is also PinPlay / DrDebug

https://software.intel.com/en-us/articles/program-recordrepl... https://software.intel.com/en-us/articles/pintool-drdebug

timmisiak8• 9 years ago

I believe all of those projects are based on single-core debugging. Restricting the trace to a single core is one way of introducing determinism in the process as a way of allowing replay later. Microsoft time travel debugging is a multi-core technology, which allows you to capture interactions between threads in multi-threaded processes.

Edit: Since I'm still "posting too fast", let me respond here to clarify. Both rr and undodbg support multiple threads, but do not record them simultaneously. They restrict execution to a single core. That's what I mean by recording interactions between threads. For instance, if you have shared memory between processes, that won't work with rr/undodb generally (although I know at least rr can work around this by recording both processes).

I'm not saying one is intrinsically better than the other since there are tradeoffs with both approaches. There is a high constant overhead for recording all cores, but it does have the advantage of scaling well to large numbers of cores.

kuwze2• 9 years ago

rr does support multithreading[0].

"rr doesn't record shared-memory multithreading, so it forces your application's threads to execute serially. Your application can see an additional slowdown if it takes advantage of multicore parallelism."

[0] https://github.com/mozilla/rr/wiki/Usage#recording-an-execut...

Game_Ender• 9 years ago

UndoDB supports threading [0]:

"Handles concurrency

Record and replay multi-threaded programs, and those that use shared memory and asynchronous I/O."

[0] https://undo.io/technology/features/

khuey• 9 years ago

Yes, there is definitely a significant tradeoff here. If you're recording something like Exchange Server having true multi-core execution is a big deal. If you're recording a web browser it's not as important, because most things are still gated on the UI thread. Hopefully y'all can get the constant factor overhead for recording down further over time.

With rr and Firefox we found that with sufficiently pathological scheduling we can eventually tease out many races and other bad thread interactions. This became rr's "chaos mode". I think the biggest drawback of not having true multi-core support in rr today is simply the performance hit that a parallel program will take during recording.

dmitrygr3• 9 years ago

We at VMware had this working and released in 2008. We supported windows and Linux. We even supported things like network applications that had lots of outside state. It was called replay debugging and besides Mozilla, few used it. Sadly it was killed when workstation 8.0 came out.

AlphaSite2• 9 years ago

Was it an open source project? If not do you know where it lives inside the company (I’m curious about how it works).

dmitrygr3• 9 years ago

It was based on record-replay capability of VMware hypervisor

timmisiak8• 9 years ago

That really was a cool feature of VMware (and I used to work on the vmware hypervisor briefly), although it used very different techniques and was limited to single core record/replay. Microsoft TTD is based on a different technology, and captures traces of individual processes. More useful for usermode folks, less useful for kernel mode folks compared to the vmware technology.

pmoriarty• 9 years ago

"Sadly it was killed when workstation 8.0 came out."

What ever happened to the intellctual property for that? Any chance it could be open sourced?

dmitrygr3• 9 years ago

Internal politics.

RyanRies• 9 years ago

Here is a recent CppCon video: https://www.youtube.com/watch?time_continue=3389&v=l1YJTg_A9...

I'm an escalation engineer at Microsoft, and we have been using this for many, many years to solve complex customer cases. Having it go public is a pretty big deal for us.

lj3• 9 years ago

Reminds me of Qira. http://qira.me/

timmisiak8• 9 years ago

I'm the dev lead for WinDbg and the windows debugging platform, and I'm a former dev working on time travel debugging (I wrote the cpu emulation for the project). Happy to answer any questions that anyone has.

Since I'm "posting too fast", let me put my answers here for now:

qod: Right now you can attach TTD to a process after initialization (or any point in time prior to the repro), but you can't trace starting from a breakpoint. We'd love to add that functionality though, so I'm sure you'll see it in a future version of the tool.

Glad you like the new WinDbg! It's been very polarizing (which you can see just reading the comments on this post!), so it's good to hear the positive feedback sometimes :)

We've got a ton of plans to make debugging even faster and more effective in WinDbg, so hopefully we'll win more people over as we make our tools easier to use and more powerful.

qod• 9 years ago

First I would just like to say how impressed I am with the new WinDbg and TTD, this is fantastic work and a huge improvement on the existing debugging tools.

My question is: if I want to start taking a TTD trace after hitting a particular breakpoint in WinDbg, can this be done with the current tooling? Use case would be if there is a large amount of initialization code that doesn't need to be traced because the bug or crash is known to occur at a later stage in the program.

timmisiak8• 9 years ago

Right now you can attach TTD to a process after initialization (or any point in time prior to the repro), but you can't trace starting from a breakpoint. We'd love to add that functionality though, so I'm sure you'll see it in a future version of the tool.

Glad you like the new WinDbg! It's been very polarizing (which you can see just reading the comments on this post!), so it's good to hear the positive feedback sometimes :)

We've got a ton of plans to make debugging even faster and more effective in WinDbg, so hopefully we'll win more people over as we make our tools easier to use and more powerful.

mncharity4• 9 years ago

For more than two decades now, I've been using the limited availability of reversible debugging as an example of just how dysfunctional software engineering is as a field. The shoemaker's family's tattered feet.

Reversible debugging been around for six decades (eg EXDAMS on Multics), been generally practical for two or three, been available in gdb for one, and now, finally, here we are. It's taken two bloody human generations to get this close to widely deploying TTD.

So yay. Maybe I can now retire TTD as an example of glacial progress. There's no shortage of others. Sigh.

csours• 9 years ago

I'm curious to hear some. Are you thinking of stuff like Plan 9 files/services?

mncharity4• 9 years ago

GP> example[s] of glacial progress

I'm curious to hear some.

How many keyboard keys can you hold down without interference effects? Some keyboards offer 10-ish. Or keycaps which are touch pads. Or change color. Or are little active displays. And touch pads which are pressure sensitive. Or detect proximity. Or have a nice stylus. But not mine. It's not space-cadet, or old thinkpad, but chicklets.

One nifty aspect of VR is that a lot of things get unstuck. Even if you're legally blind, VR brings you hand tracking, and vibrational haptics, and spatial audio. You might stroke home-row keys, and make little 3D gestures with fingers. But current Leap Motion hand-tracking software doesn't do linux, so you have to run it in a VM, and point a camera at the keyboard, and...

Here we are on HN. USENET had kill files - people you didn't want to see again. Recommendation systems track preferences and predict what might be of interest. Filters and hints - at lunch I only want to see high value content, but after work, I don't might conversational humor, but that varies by thread topic. But here we are on HN.

I was recently playing python 2 vs 3 games again. So that's an easy "from __past__ import python2", right? Not without scoped method dispatch. Like Ruby's refinements.

Here we are using browsers. Self language prototypical programming now available to the masses? Smalltalk-like live coding? Well, sort of. But try inheriting from a built-in "object" - "error: magical method X only works on magical pseudo-object type Y". Browser implementations have sometimes written the fast path, skipped implementing the general case, and considered themselves done. So does the dearth of prototypical-style code mean it's not a useful style, or is inaccessible, or still hasn't been widely tried? Shrug.

We increasingly attempt to decompose our big math, algorithm, and data structure libraries along orthogonal and algebraic lines. Sketching lattices of theories, with types and operators and laws. "1 is not an object; + is not a message" (1980's Smalltalk critique - does anyone remember where to find this? a technical memo?). If an algorithm only depends on + and a partial order, it says that. But it's not yet common. How often do you use units, instead of a primary school "what's the resistance? 3! Three what? 3!... volts!". And how many languages let you be explicit about laws? Or help you avoid "well, in wetware I model the problem domain, and in wetware I design the solution, and in wetware I compile the solution to an intermediate form... which is at long last understood by the idiot software compiler of narrowly circumscribed helpfulness". And how many such lattices support pushouts? Add a law, add an operator, and you get the same thing as if done in the other order. Haskell spent what, a half decade?, with lots of commitee and community discussion, and broken code, just to insert Applicative in its type hierarchy.

ARPA folks were once considering adding lots of little "lubricant" grants, analogous to the WWII troubleshooters which scurried around industry asking "what are you bottlenecked on? what might speed your work?" A new adminstration, and ARPA had more existential worries. But one strawman was something like "much of our knowledge about how to write good VM's is in person X's head; they've been meaning to write a book, but aren't getting to it; instead of waiting years for that knowledge to slowly diffuse, it would be inexpensive to pay someone to help, sitting outside their office, and stand on their desk". Instead we did the wait for years thing.

roca2• 9 years ago

As far as I can tell EXDAMS was never implemented. The RAND EXDAMS memo is more of a thought experiment than anything else.

Getting reverse-execution debugging to work adequately at scale is difficult. gdb's approach doesn't scale at all. I dispute that reversible debugging has been "generally practical for two or three decades"; as far as I know the first practical products were UndoDB and VMWare.

mncharity4• 9 years ago

EXDAMS [...] a thought experiment

Ah, good point.

"generally practical for two or three decades" [...] the first practical products

I was thinking of "we now know how to do this" (snapshot and replay (and input log)), rather than "and it's a product".

Which seems sometimes valid, and sometimes not. We know how to make a 4K/eye VR HMD with existing parts, and it's simply a market failure that one's not on my desk. But for other things, "know how to do it, sort of, in a lab" and "know how to do it, practically, for real" can be greatly separated in time and effort.

first practical products were UndoDB and VMWare

My very fuzzy recollection is DEC alpha had a reversible debugger in the late 1990's? - but I'm not finding it.

mncharity4• 9 years ago

As far as I can tell EXDAMS was never implemented. The RAND EXDAMS memo is more of a thought experiment than anything else.

The memo[1] mentions

Since EXDAMS is currently being debugged and is not operational, no performance statistics are available. [p12]

Suggesting implementation was attempted, though perhaps not successfully?

[1] "EXDAMS: extendable debugging and monitoring system" Memo RM-5772-ARPA APRIL 1969 http://www.dtic.mil/get-tr-doc/pdf?AD=AD0686373

DSingularity• 9 years ago

Mozilla has a similar tool, check it out:

https://github.com/mozilla/rr

libeclipse• 9 years ago

Just waiting for the rust/lisp squad to come along and tell us that we wouldn't need all this if we just joined them.

steveklabnik• 9 years ago

rr works well with Rust code.

roca2• 9 years ago

I'm a notorious Rust fan and also the lead author of rr :-).

revelation• 9 years ago

Oh my god they redesigned WinDbg with a Microsoft Office UI lookalike starter kit including ribbons. And at least one of the screenshots showed JavaScript.

So many fingers crossed hoping they don't ruin this indispensable basic tool. Every platform needs to have a no-nonsense debugger that doesn't fall over immediately when it has no symbols and source. WinDbg (was?) that.

timmisiak8• 9 years ago

The underlying engine is the same, so all of the functionality is fundamentally the same.

asveikau• 9 years ago

I like the 90s UI of old windbg too. Maybe stick to cdb.exe to avoid bells and whistles.

Time Travel Debugging

Related Stories