Progress on the Gilectomy

https://lwn.net/SubscriberLink/723514/4e850e6870991c76/

by etrevino • 9 years ago

86 34 9 years ago

krallja• 9 years ago

Thanks for posting this summary. The video of the talk was posted in May, but I never got around to watching it, because video always feels like it needs a much larger block of uninterrupted time than reading.

Animats4• 9 years ago

Multithreading in Python is all uphill. In Python, any thread can store into anything at any time. By name, even, using "setattr". There's no thread-local data. This implies excessive locking activity. That's why this guy is working so hard to do things that ought to be simple.

A few restrictions on the language would make it easier, but when this is suggested, Guido has a cow.

nine_k• 9 years ago

Is there a link explaining the suggestions succinctly?

Animats4• 9 years ago

- Multiple interpreters with mostly non-shared data in the same process.[1].

- VecPy, for number-crunching.[2]

- Newthreading, which I wrote in 2010.[3]

[1] https://lwn.net/Articles/650489/ [2] https://www.andrew.cmu.edu/user/dfarrow/proposal.html [3] http://www.animats.com/papers/languages/newthreadingintro.ht...

benhoyt• 9 years ago

"Guido has a cow". Hmmm, or maybe he knows this would be mean huge, backwards-incompatible changes to the language's dynamic nature, and especially after the experience with the changes (I think much lesser ones compared to this) from Python 2 to Python 3, he's very wary of that.

Still, if you were going to propose "a few restrictions on the language" to make it less dynamic and easier to do multi-threading / optimization, what would they look like?

VectorLock2• 9 years ago

From seeing the "Status of the Gilectomy" talk at PyCon it sounded more like the big blocker is not breaking the C API too bad. Now if somebody suggested a Python 4 with a totally backwards incompatible but thread-safe C API now that I could see him having a cow over (even though I like that idea)

omginternets3• 9 years ago

when this is suggested, Guido has a cow.

Like what?

My first reaction is to think "and he's [probably] right!"

Python's added value is it's very rich and flexible object model. Imposing restrictions on the object model is robbing Peter to pay Paul in an environment where Peter is more important, albeit for historical reasons.

stickfigure• 9 years ago

I don't really understand the problem with the GIL. Sure, people doing exotic high performance stuff probably hate it, but Python probably isn't their go-to language anyways.

For boring old web development and business processing, the GIL seems to be a very efficient way of utilizing a CPU+caches without resorting to the horrible event-driven programming model of Node. Want twice the throughput? Run twice the processes. I kinda wish Java would bring back the option of green threads, it was a perfect programming model for web development.

saghul• 9 years ago

The problem is with CPU intensive tasks. Being as expressive as Python is, it would be nice if CPU intensive algorithms could be written in it, without blocking all other Python threads while the algorithm runs. For example.

crdoconnor2• 9 years ago

it would be nice if CPU intensive algorithms could be written in it

It would be nicer if instead of trying to turn python into something it is not (a language for writing CPU intensive tasks), it was made better at what it is (glue code).

It would be especially nice if there were tighter, more seamless integration between python and rust so that hot paths in python can easily be migrated to rust.

xorcist• 9 years ago

If it's really CPU intensive, you can always fork.

toyg• 9 years ago

Python probably isn't their go-to language anyways.

That is the real problem. Any sufficiently popular language will inevitably be used in scenarios it is not suited for. Python has now reached that threshold, so all sorts of communities are trying to stretch it and pull it every other way. Sometimes this makes the language better, sometimes it's just a source of complication and frustration.

There is also the issue that some quarters feel python's role is being eroded by newer stuff like golang and nodejs, which were built with high-performance multithreading in mind from the start. I personally think this is not much of a concern (if you are willing to trade python for golang or nodejs, you were not really using python because of its expressiveness and conciseness in the first place), but clearly a lot of people do.

yumaikas• 9 years ago

What other tradeoff targets would be the considered as sharing python's expressiveness and conciseness? One of the python variants like stackless? C?

I'd be very curious as to the answer here. Some times you do like the pretty code from something like Python, but you get backed into a performance corner, at least I could see that happening.

Animats4• 9 years ago

Many machine learning people use Python, and they need all the CPU power they can get.

Python added the "horrible event-driven programming model of Node" in Python 3.6. The syntax is said to be better, though.

zodiac• 9 years ago

The python libraries ML people use (scikit-learn, tensorflow, numpy) all do the heavy work in C though

matthewmacleod• 9 years ago

Many machine learning people use Python, and they need all the CPU power they can get.

The compute code for ML is emphatically not written in Python – Python is essentially a lightweight API wrapper around it.

VectorLock2• 9 years ago

Its a perception problem. I think the simple perception of the GIL as making multithreading untennable in Python has given Go a big boost and has hurt Python considerably.

omginternets3• 9 years ago

has hurt Python considerably

Has it? Why the assumption that Go is a direct competitor?

My impression is that Go excels where Python does not, and vice-versa. I use both languages a lot for my day-job, and when using Go, I very much miss having an interpreter and a fully-interactive debugger (Delve is excellent considering the limitations imposed by Go, but much less flexible than pdb).

And let's not even talk about metaprogramming...

akx• 9 years ago

Previous thread (5 days ago): https://news.ycombinator.com/item?id=14432854

absoluute• 9 years ago

Getting rid of the GIL is easy - just move everything into the interpreter instance struct and add an interpreter pointer as the first parameter in every C API call except create. This should have been done in Python 3 since they broke the API anyway. Now everyone will have to go through another painful transition when they come to their senses. The current thread local storage hacks and PyThreadState/PyEval_SaveThread/PyEval_RestoreThread/PyThreadState_Swap are ridiculous.

Progress on the Gilectomy

Related Stories