mark_undoio

Speaking as a Brit, our national trait is generally too understate things. So even saying what you mean, directly, comes off as a bit immodest and hyping it up in sales pitches sounds shady.

Americans generally say what they mean a bit more, so I think their mid point is just different.

There was a quote somewhere about Mercurial having a mental model small enough you can fit in your head - and that was the big win for me.

It was also fast and had very clean, easy to contribute to code. I remember submitting a patch and getting a bit of Python education from Matt, which was very useful.

Git is fine but it's inconsistent enough in the interface department, even after all this time, that I still get regularly frustrated. On the other hand, you can't just break a workflow that already exists and I very much appreciate it scales to work far beyond mine.

I do like that the git people are doing the difficult work of improving the UI over time - it's hard to change the engines while the plane is flying!

I've recently been thinking how AI agents could affect this.

If you're lucky enough to be able to code significant amounts with a modern agent (someone's paying, your task is amenable to it, etc) then you may experience development shifting (further) from "type in the code" to "express the concepts". Maybe you still write some code - but not as much.

What does this look like for debugging / understanding? There's a potential outcome of "AI just solves all the bugs" but I think it's reasonable to imagine that AI will be a (preferably helpful!) partner to a human developer who needs to debug.

My best guess is:

* The entities you manage are "investigations" (mapping onto agents) * You interact primarily through some kind of rich chat (includes sensibly formatted code, data, etc) * The primary artefact(s) of this workflow are not code but something more like "clues" / "evidence".

Managing all the theories and snippets of evidence is already core to debugging the old fashioned way. I think having agents in the loop gives us an opportunity to make that explicit part of the process (and then be able to assign agents to follow up gaps in the evidence, or investigate them yourself or get someone else to...).

Maybe this?

https://sites.google.com/site/616cellnet/ct2

Whilst I have seen the Rabbit fittings others have mentioned I do remember mention of a "zone phone" and suspect we're recalling the same thing...

Very excited to see this come out - though coding agents are impressive their UIs are a bit of a mixed bag.

Textual offers incredibly impressive terminal experiences so I'm very much looking forward to this.

I wonder how much agentic magic it'll be able to include though - Claude Code often seems like a lot of its intelligence comes from the scaffolding, not just the LLM. I'm excited to see!

I love the Patagonian Welsh. BBC Wales, which often has great comedy, has a sitcom based around the original emigration to Patagonia: https://www.bbc.co.uk/programmes/b060cd20

The whole thing feels very much like a Star Trek plot to me with a culture leaving on a ship to an unknown world to preserve their way of life - which later the crew would happen upon in some episode.

In Cambridge we've got a clock called the Chronophage which is intended to be a sinister "eater of time" - the designer has done a good job of making it feel uncomfortable to look at. There's some detail here: https://www.corpus.cam.ac.uk/articles/secrets-corpus-clock

My memories of what I've heard over time:

* The grasshopper escapement actually is the demonic insect that sits on the top, "walking" around the serrated ring.

* Although it's backlit electronically it's actually a fully mechanical design - including all of the weird things it does.

* The Chronophage itself blinks its eyes unnervingly.

* It sometimes pauses or ticks slightly backwards, then runs faster to catch up again.

* On certain special dates it does extra weird stuff.

* The "chime" is a metal chain dropping into a box.

There were three made in the series, this was the first one. I've always found it slightly unappealing aesthetically but also compelling - there's no arguing with the fact that there's always a crowd of fascinated observers looking at it.

Acorn was doing stuff in Cambridge UK until more recently than I'd realised - it effectively incubated a load of talent that went on to find other companies. Famously ARM span out of it but many others also went on to do cool things - my current company was founded by Acorn people.

This demo was so cool. There were lots of alternative OSes out there back then that felt very impressive.

Linux at the time was cool too but less polished than now. Lots of people were on the Win 9x series, which wasn't amazing - and Mac OS X was not yet fully baked.

These other OSes (QNX, BeOS) felt polished, amazingly fast - and slightly alien. The main sad thing from my perspective was that I couldn't get them online (my machine had a winmodem and nobody had open source drivers for those for ages).

Encarta 95 had it - I remember thinking how cool it was that it looked like Win 95.

I'm pretty comfortable with the agent scaffolding just restricting directory access but I can see places it might not be enough...

If you were being really paranoid then I guess they could write a script in the local directory that then runs and accesses other parts of the filesystem.

I've not seen any evidence an agent would just do that randomly (though I suppose they are nondeterministic). In principle maybe a malicious or unlucky prompt found somewhere in the permitted directory could trigger it?

Amp (ampcode.com) uses Sonnet as its main model and has GPT o3 as a special purpose tool / subagent. It can call into that when it needs particularly advanced reasoning.

Interestingly I found that prompting it to ask the o3 submodel (which they call The Oracle) to check Sonnet's working on a debugging solution was helpful. Extra interesting to me was the fact that Sonnet appeared to do a better job once I'd prompted that (like chain of thought prompting, perhaps asking it to put forward an explanation to be checked actually triggered more effective thinking).

I've heard various online stories, over the years, about how nobody nails the ergonomics quite like Maltron does. Which is amazing given how long they've been about.

I have an old Maltron that I got cheap (many years ago and it was old then!) and it's remarkable how unlike a modern consumer product it was. Thin, vacuum formed plastic, point-to-point soldered wire keyboard matrix. But that classic shape, keys with full sized key caps and travel, etc are all present.

The Kinesis I also have is much more mainstream - it feels more solid and looks more like a consumer product. But I understand it's just not quite as good, ergonomically.

It's quite weird having local footpaths and paved roads that turn out to have been constructed by the Romans originally - around here that also applies to canals, drainage ditches, etc. It just blends into modern reality.

I imagine some of the Roman stuff was built on even older roads and channels.

I don't understand this. Developer time is so much more expensive than machine time. Do companies not just double their CI workers after hearing people complain? It's just a throw-more-resources problem.

I'd personally agree. But this sounds like the kind of thing that, at many companies, could be a real challenge.

Ultimately, you can measure dollars spent on CI workers. It's much harder and less direct to quantify the cost of not having them (until, for instance, people start taking shortcuts with testing and a regression escapes to production).

That kind of asymmetry tends, unless somebody has a strong overriding vision of where the value really comes from, to result in penny pinching on the wrong things.

Process recording by time travel debug seems like a good fit for this problem - then you can capture 100% of process execution and then go back and investigate further.

We (Undo.io) came up with a technique for following a tree of processes and initiating process recording based on a glob of program name. It's the `--record-on` flag in https://docs.undo.io/UsingTheLiveRecorderTool.html. You can grab a free trial from our website.

For open source, with rr (https://rr-project.org/) I think you'd just `rr record` the initial process and you'll end up capturing the whole process tree - then you can look at the one you're interested in.

As others have said you could also do some smart things with GDB's follow-fork settings but I think process recording is ideal for capturing complicated situations like this as you can go and review what happened later on.

That's the thing though - they're using logs. My theory is that LLMs are intrinsically quite good at that because they're good at sifting text.

Getting then to drive something like a debugger interface seems harder from my experience (although the ChatDBG people showed some success - my experiments did too, but it took the tweaks I described).

My experiments are with Claude Opus 4, in Claude Code, primarily.

I'm fascinated by this paper because it feels like it could be a good analogue for "can LLMs handle a stateful, text-based tool". A debugger is my particular interest but there's no reason why it couldn't be something else.

To use a debugger, you need:

* Some memory of where you've already explored in the code (vs rooms in a dungeon)

* Some wider idea of your current goal / destination (vs a current quest or a treasure)

* A plan for how to get there - but the flexibility to adapt (vs expected path and potential monsters / dead ends)

* A way for managing information you've learned / state you've viewed (vs inventory)

Given text adventures are quite well-documented and there are many of them out there, I'd also like to take time out to experiment (at some point!) with whether presenting a command-line tool as a text adventure might be a useful "API".

e.g. an MCP server that exposes a tool but also provides a mapping of the tools concepts into dungeon adventure concepts (and back). If nothing else, the LLM's reasoning should be pretty entertaining. Maybe playing "make believe" will even make it better at some things - that would be very cool.

It will be interesting to know what challenges came up in nudging the model to work better with time travel debug data, since this data is novel and the models today might not be well trained for making use of it.

This is actually quite interesting - it's something I'm planning to make a future post about.

But basically the LLM seems to be fairly good at using this interface effectively so long as we tuned what tools we provide quite carefully:

* Where we would want the LLM to use a tool sparingly it was better not to provide it at all. When you have time travel debugging it's usually better to work backwards since that tells you the causality of the bug. If we gave Claude the ability to step forward it tended to use it for everything, even when appropriate.

* LLMs weren't great at managing state they've set up. Allowing the LLM to set breakpoints just confused it later when it forget they were there.

* Open ended commands were a bad fit. For example, a time travel debugger can usually jump around in time according to an internal timebase. If the LLM was given access to that, unconstrained, it tended to just waste lots of effort guessing timebases and looking to see what was there.

* Sometimes the LLM just wants to hold something the wrong way and you have to let it. It was almost impossible to get the AI to understand that it could step back into a function on the previous line. It would always try going to the line, then stepping back, resulting in an overshoot. We had to just adapt the tool so that it could use it the way it thought it should work.

The overall result is actually quite satisfactory but it was a bit of a journey to understand how to give the LLM enough flexibility to generate insights without letting it get itself into trouble.

I am appalled and delighted by this.

It feels like an AI cousin to the Python error steamroller (https://github.com/ajalt/fuckitpy).

Whenever I see this sort of thing I think that there might be a non-evil application for it. But then I think ... where's the fun in that?

Always fun to see what izabera has come up with - every single time I'm somewhere between delighted and terrified to see what she's made the computer do this time!

I assume you're aware but there are some open source assistants that can get you this (at least to some extent).

E.g. Codename Goose https://block.github.io/goose/docs/quickstart/

Which I think supports Gemini among with all the other major AI providers, plus MCP. I have heard anecdotally that it doesn't work as well as Claude Code, so maybe there are additional smarts required to make a top notch agent.

I've also heard that Claude is just the best LLM at tool use at the moment, so YMMV with other models.

Thanks for this - I've been using the MCP Inspector https://modelcontextprotocol.io/docs/tools/inspector but find it doesn't really fit my workflow.

I like the fact this mcp-debug tool can present a REPL and act as a mcp server itself.

We've been developing our MCP servers by first testing the principle with the "meat robot" approach - we tell the LLM (sometimes just through the stock web interface, no coding agent) what we're able to provide and just give it what it asks for - when we find a "tool" that works well we automate it.

This feels like it's an easier way of trying that process - we're finding it's very important to build an MCP interface that works with what LLMs "want" to do. Without impedance matching it can be difficult to get the overall outcome you want (I suspect this is worse if there's not much training data out there that resembles your problem).

What's limiting us is that Undo does need a Linux kernel - so traditional embedded programming wouldn't be a fit. Embedded Linux could work and we do support ARM64.

I've thought I bit about how you might support time travel on bare metal embedded - but actually there are hardware-assisted solutions (Lauterbach's Trace32 was one we came across) there sometimes.

This made me very happy

GDB does:

https://sourceware.org/gdb/current/onlinedocs/gdb.html/Proce...

But it's limited. It's really cool that it's integrated by default but it doesn't scale to big applications / workloads.

RR and Undo both use GDB as a user interface, though, so any skills you have there will carry over.

WinDbg's time travel debug is really cool and more people should know about it. I'm always a little sad that it's not (so far!) officially integrated in something like VS Code.

Before it was released publicly I believe Microsoft had been using it internally to share recordings on bug reports against massive pieces of software like Office. So it's a serious piece of tech.

Undo (where I'm CTO) has existed for longer than RR and its real benefit is that it scales to use cases where RR (for one reason or another) isn't a fit.

Technically:

* Doesn't need hardware performance counters - runs on more CPUs and on cloud systems (where performance counters are often blocked).

* Can attach and detach at any time - means you get to record just a subset of program execution that's interesting.

* You can our ship recording tech with your application and control it by API, so you can grab crash recordings on customer systems.

* Supports programs that share memory with non-recorded processes.

* Supports direct device access (e.g. DPDK).

* Accelerated debugging features - searching with recordings using parallel processing, accelerated conditional breakpoints a few thousand times faster than native GDB.

* We provide a stable, patched fork of GDB that we're occasionally told is more stable than the default.

For many people's use cases none of these really matter - they should use RR if they're not already.

But if you need any of these things then Undo can give you time travel debugging. In practice, it's usually big software organisations that we deal with because they have development pain and the extreme requirements we can match.

Ah, lovely blast from the past! I remember finding this DLL and being delighted that I could now put pretty icons on more things.

Also remember taking ages to figure out that it meant "more icons" rather than just a silly made up word.