The Python GIL Visualized

And the concomitant disadvantages are that multi-process solutions eat memory like there's no tomorrow, and can share cached data only via IPC. So you get your scalability at the cost of a very large constant factor drop in overall performance.

Some problems don't care. Some do. But for those where threading is a more appropriate solution, the Python GIL is a significant limitation. And this is a great exposition as to why.

Announcing that threading should never be used just tells the world that you only work in one problem domain (probably web content generation) and haven't thought through the details elsewhere.

thorax2• 17 years ago

Sure, except when that doesn't make any sense.

I embed Python in my application. My highly graphical application requires XY,000 KB of memory. It's probably not wise to re-invoke my entire app Z times simply so that Python can properly take advantage of the user's multi-core OS while scripting inside my application.

Any time you embed Python (e.g. a game, a browser, etc), the multiprocessing support isn't going to work very well as it is today. Ideally we get to the point where threading works great in the core interpreter.

I asked Guido about this in front of his keynote audience a few Pycons ago, and his answer was a sad shrug and saying to use Jython or IronPython instead.

I'd love to see the GIL improved/removed because CPython-embedding apps don't have the same options as everyday pure scripting.

kilowatt• 17 years ago

Just because your frontend uses XY,000 KB of memory doesn't mean that worker processes have to too. The working set size for a fresh Python 2.6 run on my Windows 7 box is 6,620KB. See Chrome for a good example of this approach :)

scott_s• 17 years ago

I agree with you, but in practice, there won't be as much memory replication as you assume. Linux (and I assume most modern OSes) implement copy-on-write: no data is copied until a write occurs. So as long as the separate process only read that large data set, that large data set should exist in memory only once.

malkia2• 17 years ago

In case with Lua - I can simply create new context, and voila a whole new lua vm is there. Not sure whether that could be done with embeddable python.

tel• 17 years ago

Though this has nothing to do with GIL or Python, the presentation reminds me of threadscope:

http://raintown.org/?page_id=132

Take a look at it in action

http://www.youtube.com/watch?v=qZXq8fxebKU

diN0bot• 17 years ago

anyone know of a tool that will generate charts like that when i run my program? perhaps by instrumenting my code?

i'd be very curious about this, but i wouldn't want to invest a lot of time into it. right now i use python for my web site serving (django).

The Python GIL Visualized

Related Stories