Checking Out CPython 3.14's remote debugging protocol

https://rtpg.co/2025/06/28/checking-out-sys-remote-exec/

by ingve • 12 months ago

85 26 12 months ago

rtpg.co

BossingAround2• 12 months ago

So, IIUIC, new capabilities:

- It'll be possible to print stack traces without modifying or stopping the program.

- It'll be possible to exec into a program at runtime without modifying it.

I'm not sure why the author mentions remote_pdb - this has been with Python for some time, and works since Py 2.7? Not sure what changes in 3.14 for remote_pdb.

What I'm hoping though is improved tooling around debugging Python. Currently, in my experience, VSCode (more specifically, debugpy) provides pretty much unmatched remote debugging capabilities, and I'm really hoping we can have a standardized way to connect any IDE to remote Python processes with the same UX as VSCode.

I would love to use something like Zed, but without remote debugging abilities, the IDE is pretty useless for me. Perhaps better devs don't need remote debugging, but I depend on it more than a junior in college CS program depends on AI :)

jasonjmcghee3• 12 months ago

I thought the author mentioned remote pdb because it sounds like you can use it to attach to cpython now, and previously gdb would have been needed? I'm at last a few years behind on debugging cpython... But always used gdb.

rtpg3• 12 months ago

What I'm hoping though is improved tooling around debugging Python. Currently, in my experience, VSCode (more specifically, debugpy) provides pretty much unmatched remote debugging capabilities, and I'm really hoping we can have a standardized way to connect any IDE to remote Python processes with the same UX as VSCode.

This is what 3.14's remote debugging protocol gives us. And even more than that, because I believe that debugpy needs to be invoked beforehand!

The value of this tooling is you don't need to predict that you need debugging before running the program. All of your Python programs[0] can be debugged. No need to restart. No need to modify the launch parameters either: you don't need the "debug configuration" from vscode, you just need a PID.

[0]: so long as you're using the same version of Python and it has the remote debugging protocol enabled, and also you have the right OS permissions

maeln2• 12 months ago

If I understand correctly, this would be very useful to debug application running on GUnicorn, Celery, etc. Apps that usually have several workers / processes / threads. Currently, it is very annoying to use pdb for that. The Debug Adapter Protocol works well for this case, but the only fully feature, and not buggy, client right now is VSCode (last I checked, the nvim implementation was still a bit buggy. Didn't try the Emacs one).

I really like debugging in a simple shell (à la gdb) so this would really be nice for my workflow.

BossingAround2• 12 months ago

When debugging multi-threaded env with debugpy & VSCode, VSCode jumps to code that is active in currently active thread. Is DAP something different? From briefly looking at the docs, it seems like DAP calls debugpy for Python debugging, so we're probably talking about the same experience?

maeln2• 12 months ago

yes debugpy is the implementation of dap for python

frou_dh• 12 months ago

Seems pretty nice. I was getting worried when it was talking about requiring a third-party library (remote_pdb), but at least it sounds like you can now attach pdb to any running process on the same box using OOTB tooling only.

poulpy123• 12 months ago

I don't know if it's possible byt I would love to have a debugger that allows to go back in time from a breakpoint or an exception

dripton• 12 months ago

Yes, the term you're looking for is "reverse debugging". It exists, and it's better.

jasonjmcghee3• 12 months ago

The popular (Linux only) solution is https://rr-project.org/

magne_undoio• 12 months ago

https://rr-project.org/ is a great Time-Travel debugger and has the capability to record and replay the execution of the CPython interpreter, but since it depends on GDB (which just knows how to inspect compiled programs) it struggles to present the state of the Python program running inside of the interpreter.

I've found that the best way to inspect Python programs with GDB is to use the GDB extensions (https://github.com/python/cpython/blob/3.13/Tools/gdb/libpyt...) that the CPython project maintains. It's a bunch of code that tells GDB how to navigate the interpreter internals and how to find things like the local variables and the backtrace of the currently executing function. You can actually see quite a lot using this! It falls short if you need to do any kind of navigation or stepping, but it's great for understanding what your program is doing "right now".

At my employer https://undo.io we makes a Time-Travel debugger called UDB which is similar to rr and we've had some success introducing time-travel into this Python debugging approach. We can record and replay Python processes, and with some additions on top of the GDB extensions from CPython we can navigate around the execution history. It's a bit rough around the edges, but it's really powerful to be able to evaluate Python expressions at arbitrary points in time, and we're finding it valuable to use internally. Here are some more details if you're interested (you should be able to adapt this to rr): https://undo.io/resources/how-i-debug-python-code-with-a-tim....

I expect that this new remote debugging protocol will be useful to some of the approaches we're currently testing out.

jasonjmcghee3• 12 months ago

I found this to be a very good official resource on the topic https://peps.python.org/pep-0768/

whinvik• 12 months ago

Just to clarify, this requires both the client and the debugging script to be running 3.14? So I cannot have a script running an older version of Python but use a Python 3.14 debug script to attach to the running script?

joshlk• 12 months ago

Yes

rtpg3• 12 months ago

So Python 3.14 introduces this functionality. The process you want to debug needs to be on at least Python 3.14.

If you want to "just" use sys.remote_exec, you will need to use that from a matching Python version. If you dive into the implementation (check out the PEP), you could most definitely write a C or Rust program that implements remote_exec and "does the thing".

The protocol is basically "tell the program to read a python script from this file". That python script will be executed in the running program's environment, but you don't need for the software on the other side to be a python program!

sys.remote_exec itself only targets same Python versions though.

skeledrew4• 12 months ago

You can check out pyrasite[0], which does the same and works in theoretically all relevant Python versions, without need to version match.

[0] https://pyrasite.readthedocs.io

skeledrew4• 12 months ago

Wow, this pops up only a few days after I discover the existence of pyrasite[0]. I even created a wrapper around it to get a decent REPL (using ptpython[1]) and am planning to extend it over time into a kind of Pharo/Smalltalk live coding experience.

[0] https://pyrasite.readthedocs.io/en/latest/ [1] https://github.com/prompt-toolkit/ptpython

sczi2• 12 months ago

Interesting, I hadn't heard of pyrasite, it has a nice GUI with info about objects' memory usage, threads, open files and more. I'll definitely take ideas from it.

I'm working on a live coding environment for python[0], based on emacs' SLIME mode for common lisp. It's quite new and I haven't written documentation yet, but all the main SLIME features not covered by LSP are working.

- All results printed in the repl are presentations that can be inspected, copied around and used again -- as the actual object, not just it's str or repr text like in most repls.

- On any uncaught exception you get an interactive backtrace buffer where you can jump to source, see arguments and local variables for each frame, and eval code or open a repl in the context of any stack frame. And the arguments and local variables aren't just text but presentations you can open in the object inspector, copy to the repl and use, etc.

- A thread viewer where you can view stats on all threads, get the backtrace of any thread, spawn a repl in the context of any of it's stack frames, etc.

- An async task viewer with somewhat more limited functionality as async tasks don't keep a full stack.

- A pretty documentation browser using mmontone's slime-doc-contribs.

- The ability to trace functions, where again their arguments and return values aren't just printed as text, but as presentations, that you can open in the inspector, copy to the repl, etc.

- I took some code from IPython's autoreload extension, so interactive development without restarting and losing state mostly works.

If you want to collaborate or just talk ideas that'd be fantastic, I don't have any experience with the Pharo/Smalltalk world.

https://codeberg.org/sczi/swanky-python/

skeledrew4• 12 months ago

Nice set of features you have here, very similar to Smalltalk (particularly Pharo) ideals. Actually I'm also actively working on a Pharo VM simulator so I can ultimately get GToolkit[0], which I really like and is based on it, running in Python. Nothing published yet though, but can definitely get in touch via your project.

[0] https://gtoolkit.com/

rtpg3• 12 months ago

Yeah I think pyrasite would be able to vastly simplify its implementation if it chose to (it might not though! the "same python version" requirements can feel strict)

orbisvicis2• 12 months ago

The problem is running the injected code at a specific location... line number or definition.

pedrovhb• 12 months ago

You can presumably run code that calls `sys.settrace` for that. Which makes it somewhat underwhelming to realize that you pretty much could also do that before, but perhaps convenient that now you don't have to have the foresight to have set that up beforehand.

orbisvicis2• 12 months ago

Hmm, that's what pdb.set_trace() uses, right? That means monkey-patching live code will have an unavoidable performance hit, with the exception pointed out by sibling @skeledrew comment.

Which brings me back to my long-held belief that python functions are not quite first-class, not until you can rip them apart and put them back together as if nothing ever happened.

Not to say there haven't been improvements, as evinced by the decorator module rewrite from version 2->3 (or 3->4; can't remember) that completely changed how the resulting function was constructed.

skeledrew4• 12 months ago

Not really. The injected code will just be run in the context of one of the modules, probably "__main__". But you can make that code do anything, including decorating any class or function objects for whatever.

snthpy• 12 months ago

OT: This release should really be called CPithon.

Checking Out CPython 3.14's remote debugging protocol

Related Stories