Easy parallel Python with concurrent.futures

https://gist.github.com/mangecoeur/9540178

by edward • 11 years ago

50 11 11 years ago

gist.github.com

craigds• 11 years ago

If you need this in python 2.7, you can get it from pypi (pip install futures)

vegabook• 11 years ago

Another Python concurrent library?? To boot one which makes Python look like Java? This.that.that.this.whatever()?

How is this better than a few simple lines of multiprocessing module exactly?

Yes I'm being negative. People need to stop already adding yet more parallel libraries to Python. Contribute to existing 15+ options or stay away. Unless you're GvR (and even that's debatable) you're not helping the ecosystem - you're confusing it even further.

Python is not a good first choice for multicore anyway because it is at least 10x slower than compiled languages so the best you are hoping for is 8-core parallelism to get you back to the performance of single core C (or whatever compiled language you want). If you're serious about performance there is no other way than to abandon Python. Only Numpy (and Pandas) are keeping it competitive, and in a number of fields, it's looking more and more R-like as a prototyping-only language, with production in anything-but.

necrobrit• 11 years ago

concurrent.futures isn't really _another_ parallel library... its just the standard libraries higher level interface to both the multiprocessing and threading modules.

See the PEP: https://www.python.org/dev/peps/pep-3148/

DasIch• 11 years ago

concurrent.futures is not new, it has existed for years and is not an alternative to multiprocessing but an abstraction that can be used with threads and multiprocessing.

The purpose is primarily to have a solution to promises/deferreds in the standard library.

eloff3• 11 years ago

Parallel python is almost an oxymoron. Python has a GIL that prevents more than one thread from running Python code at the same time. You can get around that by spawning multiple processes each running an independent interpreter, but that's not as cheap or easy as spawning threads. You need to reuse those processes for multiple units of work, or chunk your units of work into big enough pieces to offset the startup costs. Sharing large amounts of memory between the processes requires copying or shared memory (which also isn't that friendly to use from Python.) Granted copying is harder to get wrong, so it's the recommended solution anyway. If you wanted extreme performance Python is a lousy choice in the first place.

wting• 11 years ago

CPython the implementation has a GIL; Python the language does not. Plenty of other implementations (PyPy, Jython, Cython) avoid having a GIL or have options to disable it.

If you wanted extreme performance Python is a lousy choice in the first place.

Using Python for glue code and C-modules for performance critical work is a popular option.

eloff3• 11 years ago

I'm aware of that, but CPython is by far the most popular implementation. Add IronPython to the list without a GIL as well.

However, Jython, IronPython and PyPy must all variously slow down even single-threaded code without the GIL because Python having the GIL made lists, dicts, and sets threadsafe. The locking (or complex lock-free algorithms or STM in the case of PyPy) is not free.

The Python language is just a poor choice for concurrency. It's also a poor choice for a glue language, as Lua is much better designed for that, but Python seems to have become popular in that role anyway (likely due to a richer ecosystem.) Python, even with all of PyPy's JIT magic is still one of the slower languages. If you're using it for real work (not just as glue) and you need extreme performance, then it's a poor choice in the first place.

Keep in mind that I have a Python day job, and Python has been my favorite language for over a decade. I'm not hating on Python, I'm just acknowledging its failings.

gsnedders• 11 years ago

Cython isn't a separate implementation: it's merely a language that compiles down to CPython API calls. Therefore, it's bound by the GIL insofar as CPython extensions are.

PyPy does really have a GIL, it's only the experimental PyPy-STM which doesn't.

davvolun• 11 years ago

If you wanted extreme performance Python is a lousy choice in the first place.

Agreed; assuming the problem is "we need blazing performance here," I don't think you should choose Python.

However, that doesn't preclude looking at improving performance. E.g. I have a python process that takes 10 minutes to run. If I rewrite it into pure C, I can get it down to less than a second, but if I (minimally) rewrite it to take advantage of concurrency, I can get it down to two minutes, I see that as potentially worth it. As in, I need to run this process 7 or 8 times a days, the hypothetical concurrent Python makes a lot of sense. If I need it to run 1000 times an hour though...

I see where you're coming from but I don't think parallel python is an oxymoron, I see it more as the old range() vs. xrange() issue. It's been awhile, but as I recall, there is no reason to ever use range(), but if I see it in someone's code, I'm not going to start freaking out. Similarly, I think rewriting a Python 'subroutine' to something faster when the optimization is found to be necessary is well-covered ground, and is incidentally more essential than parallel python, but I see value here.

I guess I can summarize my position to considering better performance (parallel python) versus extreme performance...

Easy parallel Python with concurrent.futures

Related Stories