Show HN: PyHeap – A heap dumper and analyzer for CPython based on GDB

by ivanyu • 4 years ago

3 1 4 years ago

ivanyu• 4 years ago

I've been working on a tool called PyHeap that allows making a running Python program (on CPython interpreter) dump its heap content on disk and later analyze and vizualize these files. This works without any modification to the program source code, CPython code, or even restart.

PyHeap uses GDB to attach to the target CPython process and inject a piece of Python code that writes the heap dump. The dump will not 100% complete, because the dumper relies only on what it can find in the run time using the standard modules `sys` and `gc` without support from CPython. For example, various native extensions like NumPy must be explicitly supported (not at the moment). However, it's good enough for many use cases.

GDB doesn't need to be installed "near" the target, only "near" the dumper itself. It mean, it will work even with e.g. dockerized Python apps without modification. Linux namespaces are generally supported.

A word on motivation. Java has nice tooling for saving (dumping) the content of the process' heap into a file. A running JVM or a JVM failing with out-of-memory (at least, HotSpot) can do a dump in a portable format. There are standard CLI (jmap, jcmd) and GUI (VisualVM) tools to initiate this [1]. Tools like Eclipse Memory Analyzer [2] or VisualVM [3] can read and query (OQL) these files. Literally see the values of fields, local variables, thread states, etc. This is a super powerful debugging technique. In my career I've figured out many tricky bugs by poring over a heap dump. I miss this debugging experience badly in the Python world. I decided to fill the gap and provide something similar to jmap + Eclipse Memory Analyzer or VisualVM heap dump viewer.

[1] https://www.baeldung.com/java-heap-dump-capture

[2] https://www.eclipse.org/mat/

[3] https://visualvm.github.io/

Show HN: PyHeap – A heap dumper and analyzer for CPython based on GDB

Related Stories