Python Debugging Techniques

http://aymanh.com/python-debugging-techniques

by edw519 • 17 years ago

141 19 17 years ago

jacquesm• 17 years ago

Talk about timely :) Thank you!

Debugging python in a server environment is a different kettle of fish though, I keep running in to situation where something goes wrong under water and there is absolutely no hint of where the problem lies. That's one of the most frustrating bits of django/python development as far as I can see.

In PHP it is a very rare occurence to get an error that does not immediately pinpoint the problem spot. In django/python if you get an error message at all chances are that it will send you off on an hour+ tour of the documentation trying to figure out what is up.

For fun have an alternate_name on a foreign key that ends on _id, you will get errors that have absolutely no bearing on the location of the problem.

spahl• 17 years ago

The first one with 'import code; code.interact(local=locals())' is priceless.

Is there a way to attach a python shell to a running process? I saw something similar for ruby but could not find a python equivalent.

amix2• 17 years ago

People that find code.interact useful should check out IPython* (which is an interactive Python shell with auto completion and a lot of other features).

Here is how I launch a shell for my projects (it tries to use IPython, if it isn't working it uses code.interact): http://paste.plurk.com/show/17110/

* http://ipython.scipy.org/moin/

timwiseman• 17 years ago

IPython is great and I used to be a huge fan before I found Spyder (formerly Pydee.) It is somewhat a matter of taste, but when working interactively I think Spyder is one of the best shells. It is available at http://code.google.com/p/spyderlib/

jamongkad• 17 years ago

Upvote for iPython, it quickened my learning of the language 10 fold. That and slime.vim is a match made in heaven!

Erwin• 17 years ago

Not that I know of. In the past I've allowed my process (which was essentially a HTTP-based app server) to evaluate commands from a secure source, then built a Python interpreter that sends what you type via to HTTP to it -- so you get a pseudo shell. At another point I've set up a signal, e.g. SIGUSR2 to essentially do the code.interact() thing like in the article -- so when you need to you can kill -USR2 the process and get into a shell. The first method lets you keep running the server while the second one freezes it while you debug it.

There are also macros for gdb floating around -- you can attach the gdb (C etc.) debugger to a python process and then examine the stack through some gdb macros, but the interactivity is limited (unless perhaps someone has built it up more). That lets you break in at any moment. See http://wiki.python.org/moin/DebuggingWithGdb

In many cases where a production process is doing mysterious things, using strace or perhaps ltrace on it can give a good hint to what it is doing, together with lsof to see what files it's reading/writing.

cool-RR• 17 years ago

You might be interested in this package I wrote for one of my projects:

http://github.com/cool-RR/PythonTurtle/

Check out the directory "/src/shelltoprocess".

It's a package for connecting a GUI shell, running on wxPython, to a Python process spawned by the multiprocessing package. It's documented and MIT-licensed.

TimothyFitz• 17 years ago

Twisted's manhole does exactly this. SSH into a process! http://www.devshed.com/c/a/Python/SSH-with-Twisted/3/

rarrrrrr• 17 years ago

WinPdb well let you do this. It lets you start a process, and then attach a graphical debugger to it over TCP. Then you can let the process run, break and examine whenever you want, set break points, conditionals, etc. Great for debugging things like FCGIs, FUSE plugings, etc.

amix2• 17 years ago

If you do WSGI web-applications then I can recommend following for debugging memory leaks:

* Finding and fixing memory leaks in Python's WSGI applications: http://amix.dk/blog/viewEntry/19420

This is a more general technique on debugging memory leaks using objgraph.py (that can be used in non-WSGI applications):

* Tracing Python memory leaks: http://www.lshift.net/blog/2008/11/14/tracing-python-memory-...

bcl• 17 years ago

hmm, it would be really handy if you could connect code.interact to a socket. Then when an error is raised you could telnet to the process and do a post-mortem on it...

danohuiginn2• 17 years ago

re: logging vs. print. If you do get stuck debugging something that uses print() for all its logging, remember that in extremis you can still redirect it elsewhere:

import sys

old_stdout = sys.stdout #store the real stdout

logfile = open('/some/file', 'a')

sys.stdout = logfile #redirect output to a file

#code you're debugging here

sys.stdout = old_stdout #restore the real stdout

[Obviously using logging from the start is _far_ preferable]

ivankirigin• 17 years ago

Can you run code.interact from a django view if you've set your server to run not as a daemon?

far33d• 17 years ago

I believe so - I use pdb.set_trace all the time in my django views (during debugging) and prefer it to code.interact

treo• 17 years ago

I just tried it and it works

danohuiginn2• 17 years ago

oh, and another one. pycallgraph is a good way of getting your bearings, visually, when confronted with a chunk of disorganized, unfamiliar code:

http://pycallgraph.slowchop.com/pycallgraph/ http://www.ohuiginn.net/mt/2009/01/pycallgraph.html

Grekker• 17 years ago

Winpdb supports embedded python and remote debugging, is free and cross platform. (http://www.winpdb.org/ is down for me right now, http://en.wikipedia.org/wiki/Winpdb has some info)

req2• 17 years ago

Though I don't know of an effortless way to disable debug prints, you can easily redirect to a file.

import sys print >>sys.stderr, "STDERR is fun."

There's not too much that suggests using this instead of logging.

Jebdm• 17 years ago

If for some reason you wanted to do this, you could just do

  import sys
  def debug(*s, sep=" ", nl="\n"):
    print >>sys.stderr, sep.join(map(str, s)), nl,
    pass

  debug("There was an error on line", 10, ":", nl="")
  debug(1,2,3,4,5, sep="-")

And then comment out the line with "print" to disable it.

Of course, by that point you may as well use the logging module.

Python Debugging Techniques

Related Stories