Waiting in Asyncio

https://hynek.me/articles/waiting-in-asyncio/

by BerislavLopac • 6 years ago

71 27 6 years ago

dilandau• 6 years ago

All these subtleties and gotchas signify, to me, a shit implementation and a fragmented interface. The exact opposite of the Zen of python.

hedora• 6 years ago

Can you give a practical example of a library or python subsystem that lives up to the “Zen of python” you are referring to?

j88439h845• 6 years ago

There are lots. `requests` is the most famous.

darksaints• 6 years ago

What would be the essence of that zen? A ridiculously slow interpreter? Littering your directories with .pyc files? The world's worst package and dependency management?

Don't get me wrong, python is a lovely language when the universe aligns perfectly. But buggy libraries with fragmented interfaces are everywhere in the python community, and any zen that could have been had by pythons expressiveness gets obliterated by the various other warts and faults of the ecosystem. It's just a tool, not a form of enlightenment.

j88439h845• 6 years ago

There is a document called "the zen of python".

true_religion• 6 years ago

Zen merely means meditation. It’s a practice, not a state of being or enlightenment.

I think the confusion comes about because many people use Tao and zen interchangeably. The Tao is supposed to represent a total understanding of the underlying universal principals, which can be seen as a form of enlightenment.

However Zen is not that.

The zen of python would represent a way to meditate while coding in python. Though I will admit the actual document reads more like a Tao, and it was originally called “the way of Python” (Tao also translates as the way).

https://www.python.org/dev/peps/pep-0020/

hynek2• 6 years ago

A more charitable view is that they signify an API that evolved from callback-based APIs like Twisted over many years and made nicer APIs like curio or trio only possible (whose features/insights slowly feed back into asyncio but backward compat keeps the warts around).

nall10• 6 years ago

Evolved? Twisted has a spectacular API, the only thing that is lacking is documentation.

But even without docs I understood it faster than asyncio and it is more pleasant to use.

Twisted is very elegant, but somehow elegant things get eradicated in the Python world. Time and again.

ghostwriter2• 6 years ago

What's the added value of curio and trio APIs, compared to gevent's Queues, Pools, and Groups?

https://sdiehl.github.io/gevent-tutorial/#data-structures

war1025• 6 years ago

We've been using Twisted at work for over a decade. We are finally completing our upgrade to python3 after much effort and pain. That means we are starting to look into the native async functions in python3.

What I realized the other day is that Twisted treats everything like an asyncio Task. That seems to make most of the more confusing gotchas just disappear. I suppose it's a matter of what you're used to though.

harpyeagle• 6 years ago

Thanks! I'm definitely going to keep this under my pillow.

j88439h845• 6 years ago

Trio is a much simpler design that's just as powerful as `asyncio`. It has exactly one way to wait for a task: await it.

https://trio.readthedocs.io/

For a group of tasks, use a nursery.

Edit: correction

nerdwaller2• 6 years ago

To save others the search regarding a nursery: https://trio.readthedocs.io/en/stable/tutorial.html#okay-let...

I don’t see much more useful here than understanding the asyncio primitives and available synchronization abstractions, just a different API mostly overlapping the same need

To be clear, trio also has more than one way to wait for a task - any await is the same (e.g. `await trio.sleep(N)` (e.g. `await asyncio.sleep(N)`). I think maybe you’re getting at waiting for a group of tasks?

j88439h845• 6 years ago

I'm on mobile so cant type a long response but IME its quite different in terms of ease of use, complexity, and correctness.

For a theoretical take see

https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

To be precise, to wait for a task its actually just await. Theres no ensure_future etc. Trio doesn't have the concept of futures or promises at all, and await f() is treated as a single piece of syntax, so in practice it doesn't have "awaitables" either.

0az• 6 years ago

Structured concurrency is why you should use Trio.

What do you get? For starters, Ctrl-C/KeyboardInterrupt just works. By restricting the design space, you end up with much more intuitive designs.

Obligatory Structured Concurrency essay link: https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

I recommend giving it a read.

justaguy88• 6 years ago

It's python. ..that took too long to figure out

amelius• 6 years ago

Yes, it should be in the title.

toolslive• 6 years ago

- wouldn't it be nice if you could figure out if a function/method could block?

- What's a good strategy to migrate a larger python codebase towards asyncio ? (Haven't found any)

ghostwriter2• 6 years ago

What's a good strategy to migrate a larger python codebase towards asyncio ? (Haven't found any)

use gevent instead

t-writescode• 6 years ago

There were 2 things about asyncio that I really didn't like:

1. the library constantly changed under our feet and that also meant that Stack Overflow articles were all over the place between Python 3.5 and 3.7.

2. I don't think I've ever successfully mocked an async function in python. This made testing async code in python, especially mocking responses from an async web call (an obvious use of async in python) very difficult, if memory serves me. It's been several months since I've used async in python (job change) though, so maybe it's gotten easier or maybe the exact mocking test case issues I had are different drom what I think.

xtreak29• 6 years ago

From Python 3.8 there is AsyncMock : https://docs.python.org/3/library/unittest.mock.html#unittes... . It's also available on mock backport from pypi for other versions. It's based on asynctest module. https://pypi.org/project/asynctest/

erdewit• 6 years ago

One other gotcha with asynio.gather is that is that it starts the awaitables in a random order.

Waiting in Asyncio

Related Stories