500 Python Interpreters

https://izzys.casa/2024/08/463-python-interpreters/

by SAHChandler • 2 years ago

173 32 2 years ago

"No matter what happens there is going to be a lock occurring. No two PyThreadStates can execute Python bytecode at the same time. However, they can execute multiple C calls at the same time which is why for long running pure C operations extension and embedding developers are encouraged to release the GIL temporarily."

Very exciting! I wonder what the first set of motivating applications are and what kind of performance gains are they expecting.

staticautomatic• 2 years ago

Lxml frees the GIL internally and iirc the motivation was something like concurrent reads of different portions of large documents.

surfingdino• 2 years ago

I'm lo0king forward to it, but debugging this stuff is going to be another level of hard.

zahlman• 2 years ago

You posted this 11 days ago and it's been posted by two other people in the interim. (Why is this version of the post more popular?)

Jtsummers• 2 years ago

Sometimes articles just don't get traction and then they're invited to resubmit (by dang, the moderator). Since the submitter submitted it twice, that was probably what happened here.

As to why it's "more popular", I guess you mean is getting more upvotes, that's part of the trick with submissions. Post it on a slow Friday work day before a holiday and you'll get more clicks than some other days or times.

SAHChandler5• 2 years ago

It was entered into the second chance pool[1]. Interim posts were probably not as popular because I accidentally broke my site's CSS briefly that wasn't obvious in the preview deployments, but was present on the actual site. I only noticed about an hour ago

[1] https://news.ycombinator.com/item?id=26998308

MarkusQ• 2 years ago

CSS may still be broken. At least, I'm getting some really obnoxious color combinations (magenta on salmon, etc.)

layer8• 2 years ago

I was hoping for it to be an HTTP status code.

jacob019• 2 years ago

When they say per interpreter GIL in PEP 684, is that per thread? I wasn't aware of multiple interpreters in one process.

sitkack2• 2 years ago

Afaik, you could have multiple CPython interpreters per process before, but the GIL was global to the address space, not a per interpreter instance. I believe the GIL work also put interpreter state behind a single struct instead of a handful of global variables.

This could be totes wrong, all from memory, too tired to fact check.

gyrovagueGeist• 2 years ago

These were two separate and mostly unrelated PEPs but otherwise that's correct.

mkoubaa• 2 years ago

No not per thread. Each interpreter is capable of spawning threads, all of which share that interpreter's gil

quasarj• 2 years ago

Is 463 some joke I'm not getting?

takeda• 2 years ago

yes: https://old.reddit.com/r/TheOrville/comments/b6yk52/replicat...

vintagedave2• 2 years ago

// 500 interpreter states
static constexpr auto MAXIMUM_STATES = 463;

There is a joke here that I’m missing. Does anyone understand what it is?

SAHChandler5• 2 years ago

In the TV show "The Orville" a character has their matter replicator create "500 cigarettes", but it only created 463 after someone counted them all. This is a meme that's been making the rounds for a few weeks in some circles now.

vintagedave2• 2 years ago

Thanks!

StefanBatory• 2 years ago

Ah, so that's the origin of that meme TIL, thanks

jiggunjer• 2 years ago

So this means modules like concurrent.futures can create a thread pool object (as opposed to processpool) and run in parallel?

Or is the functionality as of 3.13 still limited to low level python embedding applications?

miohtama• 2 years ago

It is expected that many of the more complex Python libraries will break, as they assume no race condition on things that could have now a race condition when GIL is removed.

It will also mean that we can use threads instead of processes and a lot of pain related tot his should disappear in long term.

elnatro4• 2 years ago

Does this mean that even the Python 3.13 version is around the corner we have to expect some issues when running threads on a program?

Maybe I’m slow today, but are the author complaining (rightfully so) about the buggy no-gil implementation?

Should we wait until the feature is more stable before implementing new libraries? I was thinking on making use of it in new libraries I plan to develop.

What are the most blatant use case to test this feature?

PD: So many questions, I know, sorry everybody!

wrlanv• 2 years ago

The article says that embedding (calling Python from C/C++) is currently buggy. It is not clear whether the Python interpreter was compiled with --disable-gil.

I did not check the claim, since, as the article rightly points out, there are general issues in the Python space and other languages like Lua are far better for embedding.

elnatro4• 2 years ago

Thanks, I’m not an embedded developer and was losing the train of thought of the author.

SAHChandler5• 2 years ago

It might not have been obvious, but I used the currently available Python 3.12.5 release. However upon filing a ticket, and being asked for more testing, we found this issue persists into 3.13 and 3.14 alpha with and without the GIL disabled :(

funny_falcon• 2 years ago

Unfortunately, pocketpy moved to global reference to vm recently while changing implementation from C++ to C. Weird decision.

500 Python Interpreters

Related Stories