A green threading library with true concurrency for Python

by lukes386 • 13 years ago

30 39 13 years ago

pekk2• 13 years ago

Since this is a very new project, it is a good time to suggest that you abide by PEP8 (e.g., no 'waitAll') since this would be widely appreciated, and is not easy to fix later on.

mirman9• 13 years ago

waitAll gone. Before I put it in any package managers it will get style guided. This is currently on version -1.0.0.

derleth2• 13 years ago

version -1.0.0.

Version negative one? I don't think I've ever seen that before. Usually, the very earliest versions of software are numbered like 0.0.1 or something like that.

jacob019• 13 years ago

I don't really get what advantage this gives me beyond using gevent. There's still no parallelism. The readme describes it as "A solution to concurrency," but I already get that using gevent. Is it just to improve communication between green threads?

mirman9• 13 years ago

Fewer explicit yields & no monkey patching necessary for IO performing libraries.

MetaCosm3• 13 years ago

As a former python user, I wish this library had existed 18 months ago -- would have helped with some of the nasty cases you can get caught on with gevent.

jerf3• 13 years ago

It appears to add some amount of Python-bytecode level preemption to gevent, which allows you to hopefully avoid some of the pathological cases of cooperative scheduling. Said pathological cases are only a matter of scale... if your program becomes large enough, you will hit them, eventually.

That said, with no offense intended to mirman, I'd really hesitate before using this for anything serious enough to reach that scale in the first place. Gevent, frankly, visibly pushes Python to the limits (and occasionally a bit beyond), trying to also tack on some preemption on an environment not fundamentally expecting it would scare me another notch.

mirman9• 13 years ago

No offense taken - both of these are obviously visibly delicate.

There is a version in the history that used Greenlet instead of gevent which was potentially a bit less delicate, but it required wrapping of the main file and didn't work with time.sleep, and I didn't feel like it was worth writing my own locks, semaphores, mutexes, pipes and whatnot.

hogu• 13 years ago

what are the pathological cases one runs into with gevent?

erikb3• 13 years ago

It basically is 50 lines of code that implement some stuff from gevent... So yes, it's more of a joke I guess.

mirman9• 13 years ago

So number of lines of code makes a library a joke? One of the most commonly used Haskell libraries is only 25.

http://hackage.haskell.org/packages/archive/forkable-monad/0...

Also, this doesn't implement some stuff from gevent, it implements some stuff over and using gevent.

fusiongyro3• 13 years ago

Conpig threads still can only run on one core of a processor.

The disillusionment caused by having so many options for non-parallel "concurrency" in Python is, I believe, feeding the high defection rate from Python to Go.

mirman9• 13 years ago

The lack of options for parallel (and non-parallel) concurrency is, along with other things, feeding the high defection rate to many other languages.

pekk2• 13 years ago

We've had processes, threads and greenlets for a while now... if anything the problem isn't that there are no options, but too many options that require understanding to choose and apply.

Many of the people complaining about this issue don't have a demonstrated problem and could try any simple approach first (if the point is not just to slam Python in favor of something else, from the beginning).

MetaCosm3• 13 years ago

As one of those defectors... it isn't so many options, it is that they all have piles of gotchas and corner cases that can take weeks (months!) to debug in complex environments.

Yes, you absolutely can use all your cores by combining multiprocess, gevent and custom C code. But, debugging that stack of a level of hell I will never return too, ever.

gtani• 13 years ago

And the numpy/scipy/ipython notebook/scikit-learn axis is driving a lot of python adoption these days. Look in any blog about beginning data mining/machine learning, odds are it'll say learn python or R.

Different kind of user, but... just sayin

fusiongyro3• 13 years ago

That's true, and not incompatible with what I'm saying. I'm saying that the majority of defection from Python I hear about is defection to Go. A lot of that seems to be taking the form of loud and proud blogging, so we hear about it here. The defection rate from Python is almost certainly much lower than its overall growth rate, or it would be dying (Perl). I'm just saying, it's not an accident that people are leaving Python for Go. They want cheap and easy parallelism and concurrency. Python is great at many things, maybe even most things, but the state of affairs with parallelism is very weak. Python can expect to continue to lose people to Go until this is addressed in a real, tangible way that doesn't sound like excuse-making. I don't really expect the situation to change, because Python doesn't need it to, but the excuse-making is annoying.

mietek• 13 years ago

Conpig threads still can only run on one core of a processor.

This isn't true concurrency. Scaling to 20 million requests per second over 40 cores on a single machine is true concurrency.

functional_test• 13 years ago

You're confusing concurrency [1] and parallelism [2] -- this is addressed on the bullet point before the one you quoted.

[1] http://en.wikipedia.org/wiki/Concurrency_(computer_science)

[2] http://en.wikipedia.org/wiki/Parallel_computing

mercurial2• 13 years ago

Single-core concurrency is concurrency is concurrency (you have different computations occurring concurrently, even if they are not executing at the same time). What it's not is parallelism.

stonemetal2• 13 years ago

Concurrent means at the same time. You can't have things happening concurrently but not at the same time.

rektide• 13 years ago

The resounding question in my mind is why this, when there is Stackless Python? What's better about this greenthreading impl? http://www.stackless.com/

mirman9• 13 years ago

This is a library that can be used to supplement any implementation.

If I'd had the option to switch us to stackless easily, and I could guarantee it was as fast, worked with all the libraries, and was as stable, I probably wouldn't have written this. I imagine that there are a lot of people in the same boat, where switching interpreters isn't really an option.

erikb3• 13 years ago

Where's the library? I see about 50 lines of code that don't go far beyond a hello world of Gevent and some strangely written nose tests.

A green threading library with true concurrency for Python

Related Stories