Python Concurrency: The Tricky Bits

https://python.hamel.dev/concurrency/

by rbanffy • 5 years ago

39 15 5 years ago

python.hamel.dev

sleepysysadmin• 5 years ago

Threading is nice until you learn about the global interpreter lock. Then you go multiprocessing for everything.

dvfjsdhgfv• 5 years ago

OK, let's get things more straight:

* Threads are useful not just for speeding up, but also for parallel execution (storege/network io, graphics, ...)

* If you really care about speed and would like to parallelize CPU-bound code, you'll probably do better using another language like C/C++ (or Java, Go, D etc.) for this particular task

* In general, threads are used instead of processes because of the overhead involved in creating the latter

* Python multiprocessing module isn't the most efficient anyway, for example when passing large objects between processes because of the way they're serialized

* It also has some other quirks so I would definitely not advise to use it for "everything".

tasubotadas2• 5 years ago

Tricky bits? Thanks to GIL, it's just broken and doesn't work.

chrisseaton• 5 years ago

If you don't need parallelism, Python's concurrency model with its GIL is good enough and not broken.

helgie• 5 years ago

You can use numba to define functions that do not need to hold the GIL to run.

BerislavLopac• 5 years ago

"Thanks to GIL, multithreading in CPython is just broken and doesn't work."

Fixed that for you.

rbanffy2• 5 years ago

"Thanks to GIL, multithreading in CPython is just broken and doesn't work."

Thanks to the GIL, multithreaded Python code doesn't have to worry too much (you do, a little) about shared mutable objects.

targafarian2• 5 years ago

To say concurrency is broken in Python because of the GIL seems to be reiterated adage without actual wisdom behind it.[1]

Are you writing an IO-bound task? Then asyncio[2] (even if just viewed as an interface: e.g., you can make it even more performant by using uvloop[3]) is a great way to achieve performant concurrency. As Raymond Hettinger likes to point out, even if you throw away the GIL, you will have to implement locks on shared resources (unless you're writing embarrassingly parallel codes--in which case either you can invoke them separately or use the multiprocessing module to great effect, or see how I do it easily below), and by the time you get all the locks implemented in your GIL-less world (locks which are error-prone), you've given up the performance you thought you'd get due to throwing away the GIL.

Are you writing a CPU-bound task? I've only encountered this in scientific computing, as I'm a scientist, and for this (as another commenter mentions) I either use numpy[4] (strictly within a numpy call, the GIL is not held) or numba[5] (by default the entirety of your numba-fied functions do not hold the GIL). Therefore, I can either use Python threads or asyncio or Numba's built-in parallelism[6,7] (summary: some numpy routines for working with arrays of values are auto-parallized if called within a numba-fied function, and you can explicitly parallelize your own loops via `for i in numba.prange(N):`; in any case, then decorate the function with `@numba.njit(parallel=True)`) to run these numerical routines in parallel. I easily get 100% of all my CPU's threads doing work this way.

I have parallelized numba-fied functions both via numba.prange and via python threads (yes, threads) in the past few weeks to speed up 20-60 minute-long-running codes, and even though I've done this before, I was still very pleasantly surprised at how easy it was, and I got very nearly a 16x speedup on my 16-thread laptop, which made my code practical for interactive data exploration and building up simulations to help me understand some things I was observing, which is where Python really shines for me.

Also note that writing a routine in numba provides a straightforward path to putting your numerical codes on GPU[8,9], too, if the that model makes sense for your algorithm (though I've primarily used PyCUDA for this, which worked well).

Are you writing something that needs to use distributed (e.g., cluster) based resources? Then dask (dask.distributed) is a great way to go, with a similar interface to the Python standard-library `futures`. Runs all the same code and you get the same forms of parallelism, but distributed across computers.

In any case, do you get to FORTRAN or C speeds? Probably not, but close enough, and with the benefits (and costs) of using Python code.

[1] talk by Raymond Hettinger: https://www.youtube.com/watch?v=9zinZmE3Ogk [2] asyncio: https://docs.python.org/3/library/asyncio.html [3] uvloop: https://github.com/MagicStack/uvloop [4] numpy: https://scipy-cookbook.readthedocs.io/items/ParallelProgramm... [5] numba: https://numba.pydata.org/ [6] parallelizing numba: overview blog post: https://www.anaconda.com/blog/parallel-python-with-numba-and... [7] parallelizing numba, docs: https://numba.pydata.org/numba-doc/dev/user/parallel.html [8] numba on gpu, example: https://github.com/numba/numba-examples/blob/master/examples... [9] numba on gpu, docs: http://numba.pydata.org/numba-doc/dev/cuda/index.html

tasubotadas2• 5 years ago

Is this some kind of Stockholm's syndrome?

This has been discussed plenty of times and it basically boils down to "just get f*cked and work around it". In my case, I just use different programming languages for performance sensitive tasks.

But it's amazing how people can come up with all sorts of justifications for a broken system that they happen to love. And then people wonder how there are people who can support Trump :D.

pybo• 5 years ago

Anyone using Dask (dask.org)?

delduca• 5 years ago

https://nullonerror.org/2020/12/13/full-saturate-your-bandwi...

quasirandom• 5 years ago

A really good way to learn about concurrency is to write a SOCKS5 proxy. Start with a simple one using Python threads, then Python asyncio, and if you're brave C sockets with poll/select. The C parts are in Beej's guide to network programming. The SOCKS5 spec is very simple and has an RFC.

Python Concurrency: The Tricky Bits

Related Stories