Faster Parallel Python Without Python Multiprocessing

https://towardsdatascience.com/10x-faster-parallel-python-without-python-multiprocessing-e5017c93cce1

by Cyranix • 7 years ago

93 29 7 years ago

htfy96• 7 years ago

Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.

mushufasa• 7 years ago

Most of the time, if performance is a top priority, you'll choose a different language to begin with.

It's nice to have the option to speed up python in certain cases though.

riskneutral• 7 years ago

Because Python is not typed?

sametmax• 7 years ago

Python is strongly typed, by dynamically typed.

ATsch• 7 years ago

The thing is that there's really little benefit from this. Anything performance critical will be written as a native extension anyway, even if Python speeds up by 10x. If you're doing that, you can lift the GIL anyway and run as much in parallel as you want.

protomyth3• 7 years ago

Isn’t that a case of the tail wagging the dog? If Python code were 10x faster, the need to write performance sensitive code in a native extension would be a lot lower. There is a cognitive cost in having to switch between languages and knowing when to switch.

glandium2• 7 years ago

One problem is that because of the GIL, many python libraries don't use locking on their data, etc., because they don't need to. That makes them thread-unsafe without the GIL.

zaphirplane• 7 years ago

You mean libraries with c extensions

ris3• 7 years ago

If you look more closely, it often makes them thread-unsafe even with the GIL. Take a look some time at what operations the GIL can pre-empt a python thread in the middle of.

e12e• 7 years ago

There was/is stackeless with a(n initial?) focus on game engines iirc:

https://github.com/stackless-dev/stackless/wiki

It got less attention than pypy, but might be more pragmatic wrt parallel perf. There's also some work in pypy to remove the GIL, but not sure if there's been any news on that lately.

srean• 7 years ago

Stackless is wonderful, but its for concurrency not parallelism. Long time ago when I checked it out it was faster than go in concurrency but not so in parallelism.

mike_mg• 7 years ago

While I appreciate the efforts of authors and believe in long term mission, they seem to not mention anywhere some key shortcomings of Ray, while marketing it pretty hard (eg see the paper).

I have used ray (a year ago) in one of the advertised basic applications: parallelising the environments for RL. It was unusable back then, as it was clogging up the memory.

The plasma store which is backend for arrow was never cleaned which made the computation stop after 3 hours

Here’s the issue:

https://github.com/ray-project/ray/issues/2128

Or perhaps this has been fixed already?

tech_tuna• 7 years ago

I love Python but they best way to do do parallel Python is to use Go.

heyflyguy• 7 years ago

I am a terrible python scriptwriter. Still, I use my own scripts that I have written through trial and error and reading lots of stack overflow. I process about 500 images per day and each one takes about 30 seconds each. Adrian Rosenbrock has been a real lifesaver. I have a machine dedicated to this. I tried using multiprocessing once and could not get it to work. Being able to process in a parallel fashion would be a gamechanger for me.

The beauty of python for someone like me is that I can get my job done without actually having to do it. I free up my own time to leverage more of my creativity and have a multiplicative effect on my productivity.

The reason I bring all of this up is that so many of the examples for advanced libraries I see are geared towards seasoned software engineers. The examples include false arrays of data so that you can "just see" how to use it. I don't think anyone realizes how confusing this is to the guy who is a restaurant manager, or the gal who is a researcher that just needs to know how to make this work for them.

If it's an image, how about putting image = cv2.imread("C:\image.jpg") or whatever?

Anyway, a bit of a rant but there are people who are very thankful that smart people in this world like yourself write libraries we can use to make our daily lives better. Including example code that is stupid simple would make me so much happier.

Faster Parallel Python Without Python Multiprocessing

Related Stories