Python PEP for Making the GIL Optional Doesn't Get Enough Support

https://old.reddit.com/r/Python/comments/143v676/update_from_nogil_python_pep_703/

by reanomality • 3 years ago

12 10 3 years ago

old.reddit.com

from the PEP author Sam Gross:

Hi Gregory and the Steering Council, Thanks for reviewing the PEP. The PEP was posted five months ago, and it has been 20 months since an end-to-end working implementation (that works with a large number of extensions) was discussed on python-dev. I appreciate everyone who has taken the time to review the PEP and offer comments and suggestions.

You wrote that the Steering Council's decision does not mean "no," but the steering council has not set a bar for acceptance, stated what evidence is actually needed, nor said when a final decision will be made. Given the expressed demand for PEP 703, it makes sense to me for the steering committee to develop a timeline for identifying the factors it may need to consider and for determining the steps that would be required for the change to happen smoothly.

Without these timelines and milestones in place, I would like to explain that the effect of the Steering Council's answer is a "no" in practice. I have been funded to work on this for the past few years with the milestone of submitting the PEP along with a comprehensive implementation to convince the Python community. Without specific concerns or a clear bar for acceptance, I (and my funding organization) will have to treat the current decision-in-limbo as a “no” and will be unable to pursue the PEP further.

Github Link: https://github.com/python/steering-council/issues/188#issuec...

viraptor• 3 years ago

I love this response. It summarises some issues I've experienced before and phrases them really well. It's one to bookmark for reference in the future.

coldtea2• 3 years ago

You wrote that the Steering Council's decision does not mean "no," but the steering council has not set a bar for acceptance, stated what evidence is actually needed, nor said when a final decision will be made. (...) Without these timelines and milestones in place, I would like to explain that the effect of the Steering Council's answer is a "no" in practice.

from the nogil author, and a comment from another:

Leaving aside the INCREDIBLE effort Sam went thru, the discussion around it is disingenuous with constant goal post moving and frankly myopic.

Reminds me of some discussions in FOSS projects were rejection is masked through various similar tactics.

One example would be golang proposals (and even code contributions), especially generics which were shot down for the better part of a decade, with a similar "we're open to it, as long as we find a proposal that meets our impossible/shifting standards", only for the core team to make their own idiosyncratic implementation when they finally accepted the need. Not really a better one or immune to the concerns used to shot others down, more of a NIH affair.

A similar story was going on (with a popular, bound-for-inclusion non-core-team implementation shot down too) for the depedency management case.

It's not just golang either, many other good proposals languish and PRs get forgotten and left to code rot. Even for bug fixes with no downside.

I wish project steering committees who get to decide/veto would be more explicit about "no" or "yes", as opposed to wanting to appearing open when such proposals aren't going anywhere.

shipmaker2• 3 years ago

Enough is enough. Sam (facebook) should fork his work. Get him a dedicated team working on this, it's too important to abandon due to some bs politics.

coldtea2• 3 years ago

Unless it's merged into mainline though, would it make a difference?

debugnik2• 3 years ago

They should consider copying OCaml's solution to the GIL: sets of one or more threads ("domains"), each set with its own global lock.

So old code spawns threads on the current domain, behaving as before, but new code can spawn new domains instead for proper concurrency. This lets you contain GIL-dependent threads while progressively migrating away from the GIL.

jononor• 3 years ago

PEP 684 – A Per-Interpreter GIL https://peps.python.org/pep-0684/

debugnik2• 3 years ago

This is a good and necessary step in the right direction, but if I understand correctly interpreters can't share memory directly. So this will unblock some uses cases for embedding Python and for replacing process-level parallelism, but it doesn't yet improve their existing threading model.

Nice to know anyway, I'll keep an eye on how they progress.

RcouF1uZ4gsC• 3 years ago

IMO, keeping the GIL is a good thing for Python.

A language shouldn’t try to be all things to all people. If you are really craving performance and fully exploiting all the resources of a CPU, Python is really not the language you are looking for. Multithreading brings in a lot of complexity, not only for the user, but also for the library implementer who may then have to worry about things they did not have to before.

Removing the GIL increases complexity and doesn’t provide any benefit to the vast majority of Python users and scripts.

Python PEP for Making the GIL Optional Doesn't Get Enough Support

Related Stories