Python 3.6 – Asynchronous Comprehensions

https://www.python.org/dev/peps/pep-0530/

by tbarbugli • 10 years ago

125 37 10 years ago

python.org

1st13• 10 years ago

Surprised to see this here :) I'm the author of the PEP, feel free to ask questions.

sagnew• 10 years ago

Just wanted to say excellent work. This is some convenient wizardy!

1st13• 10 years ago

Thanks ;)

orf• 10 years ago

Thank you so much for the pep, I've just finished writing a pretty neat tool using all the new 3.6 features and they are awesome, especially the comprehensions.

Thanks :)

1st13• 10 years ago

Thanks! Did you use new async generators in your tool?

jchassoul• 10 years ago

I just want to say thank you for make this possible! this are the progressive little things that are not always gloriously announce that keep making python what it is step by step.

lima• 10 years ago

Useful feature.

Another important async improvement in 3.6 are asynchronous iterators:

https://www.python.org/dev/peps/pep-0525/

(yielding from a coroutine)

jfaucett• 10 years ago

This is somewhat tangential but could some of you pythonistas opine on the status of concurrency in python as of 3.6? Are we talking JVM level goodness or even some message passing builtins like erlang or is python still meandering around problems other scripting languages like ruby have had in this area? Basically, how easy is it to build concurrent code that actually does improve with added processors assuming you structure your stuff correctly.

dom0• 10 years ago

~depends

Python is still fundamentally a n=1 language, where only one thread can run the interpreter at any time (GIL), like many scripting languages. Thus any gains in performance from using multiple threads has to stem from using external resources (syscalls, calls into C code) -- running pure Python code in multiple threads won't increase performance (but still provide concurrency).

Most libraries that do heavy lifting in native code and so on are written with this in mind, and allow thus ample performance gains, should these operations actually limit performance.

Of course, nothing keeps one from using entirely different means to concurrency, eg. message passing, like the ZeroMQ approach, RPC, shared memory and so on are all possible with Python.

So I'd say it's quite manageable, and that the GIL isn't really a significant limitation for most applications. However, it also means that in many cases process-level parallelism is preferable, eg. in the context of web applications this is the usual approach (and it doesn't have to cost a ton of memory - see uwsgi/pre-forking appservers), besides async.

paulddraper• 10 years ago

Everything true, though I would add that Python makes multiprocess parallelism fairly painless with the standard multiprocessing package.

jayajay2• 10 years ago

This is str8 dope... when you are accustomed to seeing node.js callbacks pre-es6.

jayajay2• 10 years ago

Trigger Warning, do not read if easily offended or if against introspection.

Downvotes? Would this have been upvoted?:

"Python 3.6 is really cool and looks fairly compact, especially when you're used to nesting callbacks, using promises, or fibers in Node.js. That said, arrow functions provide some of that compactness in ES6."

It's interesting how a certain verbal style resonates better or worse in certain communities, even if the content is held approximately fixed:

"Python 3.6 is really cool because of comprehensions."

"Python 3.6 is lit af bruh cuz dem list comps."

"dis lang is dope cuz ONE liners tho."

"python is fucking [i for its in me]"

For example, if I tried to teach a bunch of inner city kids about Python sounding like the average HackerNews user, they would never learn anything! For those of you that teach kids, you'll probably understand, everyone else will probably take offense to this, don't say I didn't warn you.

Communities like this (HackerNews) lack diversity -- we're all the same flavor of nerd, and that's boring as fuck, isn't it? We tend to agree with each other, and are diametrically opposed to people who try to burst our bubble. Clearly we're right about everything because we have been validated by our fancy BS's in honors physics, mathematics, chemistry, biology, and other sciences (oh, and engineering, if you're second-rate (shh! some people don't like mean jokes)).

There was a post on here about democracy after the Trump victory (or right before?). And the majority consensus of the nerds on this site was that government should not have unlimited power, and it should make itself easy to change, or replace, if needed. I.e., there should be no runaway governments, which we're stuck with. People should have the power to change things. For the most part, you guys agree with that, which I find incredibly ironic.

sametmax2• 10 years ago

I +1 for the overall insight, but it's logicial and desirable to have small communities of people alike. As long as you have many of them, that are all very different and their members are part of several of them. It know it's the case for me (I'm part active on HN, reddit, imgur, twitter but IRL in sport, charity and porn), and I believe a lot of people are too here.

So I think we are OK.

HN is indeed a bubble, but it's a fantastic one that would lose a lot of value if the liquid it came from was diluted.

zo1• 10 years ago

"For example, if I tried to teach a bunch of inner city kids about Python sounding like the average HackerNews user, they would never learn anything! "

I wouldn't try to do that myself, precisely because they're illiterate. They have much bigger problems than the accessibility of learning a programming language. They need to be functional members of society, and one important part of that is communication.

bushin• 10 years ago

HackerNews community is extremely diverse, that's why people strive to write more clearly, otherwise some C++ programmer from Russia or a hardware engineer from China wouldn't understand a thing.

coldtea• 10 years ago

Consider how you'd view a person you first meet wearing a normal t-shirt and jeans or whatever, and another dressed up and speaking like Vanilla Ice.

Sure, they might both be equally deep when you get to know them (yeah, right), but appearances do matter and/or can get in one's nerves.

PudgePacket• 10 years ago

Offtopic but the python website is looking nice, last time I visited it definitely wasn't so shmick.

noobermin3• 10 years ago

Hmm, I remember the design from a year or so ago if I'm not mistaken and it seemed the same.

Animats• 10 years ago

When is this useful? There's no real multiprocessing concurrency in Python. So using this on a computational example such as the one given:

    result = [i async for i in aiter() if i % 2]

won't speed anything up or put multiple CPUs to work on it.

Python's new "async" features are useful mostly for servers running many network connections simultaneously. Too many for one thread per connection. That use case used to be handled by "Twisted Python", but now there's language support. That's useful. But a list comprehension doesn't make sense for that use case. The PEP lacks any useful example.

The motivation seems to be to make async comprehensions have the same capabilities as regular comprehensions, even though that's not particularly useful.

yladiz• 10 years ago

I could see it being useful for gathering data via an API and returning it in an array, like if an API doesn't return an array of JSON objects but just an object, this could be useful for that use case. In general as long as it's not processor bound like computational tasks but may not be synchronous this could be useful. It also naturally extends the async for syntax to list comprehension.

It would be really nice if somehow these kinds of tasks could be tied into the multiprocessing package, so that async could handle processor intensive applications well.

rjbwork• 10 years ago

This. This is essentially what you can do with the AsParallel() LINQ method, or iterating over an IEnumerable of tasks generated by async methods does in C#.

spangry• 10 years ago

What about 'as_completed' in concurrent.futures? From my understanding, they fixed the performance issues with ProcessPoolExecutor in 3.4.

m_mueller3• 10 years ago

I recently tried to parallelize some non trivial code I have with multiprocessing.map/imap/imap_unordered - the state of parallel computing in python is really poor have to say. If the program's CPU runtime has gotten so long that you'd think about parallelizing, its datastructures tend to be too much to push through the pickle bottleneck and no gain would be had. Except of course number crunching stuff for which there's numpy. Bottom line is, you can do parallel computing, but trying to parallelize after the fact will usually fail, you have to plan for it from the very beginning due to no shared memory, which would add significant time until your first version is up and running

I long for a dynamic but strictly typed language with rich standard lib that gets this right. Nim?

bmh1002• 10 years ago

You should consider Clojure with its optional typing library. I used Clojure, and later converted a single-threaded application into multi-threaded with dramatic gains. I did not need to plan in advance for parallelization and did not do so. Despite that, the conversion was fairly pain free. The application did the following:

1. read csv 2. create data structure 3. calculate statistics for each column (max values, unique values, etc.) 4. combine those statistics with original columns to generate new columns 5. write new columns to new csv

Processing each type of statistics in its own thread, combined with promises, allowed the processors to independently work on the same data structure. Because of the need to handle non-numeric values in numeric columns, e.g. convert nulls to zeroes, matrix manipulations would not have been appropriate.

I just wanted to point out a type of situation from my experience that goes against your generalization, to give a more complete picture.

nimmer• 10 years ago

Although the standard lib is still improving, Nim is doing it right.

srean• 10 years ago

Its a bit immature at the moment, but here's a shout out for Julia. If you are used to python, numpy you will not feel out of place.

BTW, as far as I recall Nim is statically typed, please correct me if I am wrong.

rickycook• 10 years ago

even if you assume it's not useful (i'm sure it is; arrays of API requests you have to process for example), it's probably a laudable goal and worth the effort to make things standard. i was surprised when i saw this and thought "oh you can't await in a comprehension?"; i expected you to be able to.

sametmax2• 10 years ago

async / await for on anything that implement the interface, not just asyncio. Which means it works for Future tied to multiprocessing pools, so yes, you can use async / await for you CPU bound program.

dorianm2• 10 years ago

I find the Ruby solution more elegant: https://github.com/bruceadams/pmap

e.g.: Replace:

    require 'open-uri'
    (1..100).map { |i| open("http://httpbin.org/get?a=#{i}").read }

To:

   require 'open-uri'
   require 'pmap'
   (1..100).pmap { |i| open("http://httpbin.org/get?a=#{i}").read }

From 17.55s to 0.72s.

noobermin3• 10 years ago

As a non-rubyist, this took me a minute to guess through. I prefer list comps.

hint: I'm implying familiarity over elegance.

dorianm2• 10 years ago

I guess in Python it would be going from:

    list(map(lambda x: x**2, [1, 2, 3, 4]))

To:

   list(pmap(lambda x: x**2, [1, 2, 3, 4]))

lars512• 10 years ago

Parallel map is built in to Python already with pools.

    import requests
    import multiprocessing as mp
    
    pool = mp.Pool()
    urls = ["http://httpbin.org/get?a=#{}".format(i) for i in range(100)]
    pool.map(requests.get, urls)

tejasmanohar• 10 years ago

I can see a use case for `await` in comprehensions- e.g.

  responses = [await r() for r in requests]

... but, what's the 'practical' application of `async` in comprehensions? I can't think of one that shouldn't call an independent `async` function for readability.

Python 3.6 – Asynchronous Comprehensions

Related Stories