Get a Python subprocess' output without buffering

https://gist.github.com/thelinuxkid/5114777

by thelinuxkid • 13 years ago

7 1 13 years ago

gist.github.com

adamtj• 13 years ago

This is complex and inefficient. Using os.read() instead of the fileobject's read() method is better.

It's inefficient to read a single byte at a time. The problem with proc.stdout.read() is that, say, proc.stdout.read(4096) will block until it has all 4096 bytes or the pipe has been closed. "for line in proc.stdout" blocks until a full line has been read. That's the "buffering" this is trying to avoid.

Using os.read() is better. It acts like more like socket.recv(). It will block until data is available, but then return all data in the OS buffer up to a size of 4096 bytes. The key is that, unlike a python fileobject's read() method, os.read() will return less than 4096 bytes if that's all that is available.

For example:

  fd = getattr(proc, stream).fileno()
  data = os.read(fd, 4096)
  while data:
      yield data
      data = os.read(fd, 4096)

Edit: Now that I look more closely, "unbuffered()" actually buffers until it can produce an entire line. There's no reason to reinvent the wheel:

  for line in proc.stdout:
      print line

Get a Python subprocess' output without buffering

Related Stories