How Python asyncio works: recreating it from scratch

https://jacobpadilla.com/articles/recreating-asyncio

282 57 2 years ago

Completely insane that python decided to hijack "def" to be used to create an object that very much is not a function. At the least they could have gone was create a different keyword

maxbond2• 2 years ago

It's a function, it's just been transformed into a function with a different return type (that of a generator or coroutine, you can see the type signatures here [1]). You could do something similar in vanilla Python rather than language-level sugar, eg using decorators.

[1] https://docs.python.org/3/library/typing.html#typing.Generat...

As a caveat, I point out the type signature for academic interest, I prefer to use the simpler Iterable and Awaitable types in type signatures.

xmcqdpt22• 2 years ago

I think the type signature and example makes the parent's point, frankly. It has three types, on of which is contravariant, two different usage of the yield keyword, and two exit points.

Now granted most uses of generators are simpler than that, but still that's a lot of special syntax. And if you are writing Python without type-checking, that's a lot of bookkeeping you'd have to do when reading code.

throwaway815234• 2 years ago

They somewhat fixed that with "async def" but the whole thing is a huge tragedy anyway.

cuu508• 2 years ago

Do you mean they fixed the hijacking by adding the "async" keyword?

By the whole thing do you mean the async functionality in Python in general? What is the tragedy?

globular-toast• 2 years ago

What's the actual problem with it? Or is this just a language "purity" thing?

zo1• 2 years ago

Are you talking about decorators? Very confused trying to parse what you're saying in the context of python.

cqqxo4zV46cp• 2 years ago

What? There is no tragedy here. It sounds like it just put DX semantics ahead of…what? some sort of language purity? I can’t say that I’ve ever thought “gosh! that async function is…an async function, and that’s tripped me up!” before. Not even once.

hot_gril4• 2 years ago

The bigger problem with Python to begin with was that you have to use def for functions, meaning you can't inline them like with JS arrow funcs or C++ lambdas. And this isn't a language purity thing, it's just inconvenient.

krackers• 2 years ago

This seems to "busy wait" when sleeping, i.e. the event loop keeps running even if nothing is currently runnable. I remember reading about another toy implementation which handled it correctly (the way I believe it's actually done in asyncio) by tracking the next-runnable times of a task in sorted order, and if nothing is currently runnable you can sleep the event loop. And then this was extended so that instead of just next-runnable depending on wall-clock time, the task could have a dependency on a socket or something, so that you can use select w/timeout.

chippiewill2• 2 years ago

In asyncio itself you can have custom event loop implementations in addition to the out of the box one.

One popular implementation is uvloop (https://github.com/MagicStack/uvloop) which basically just implements the loop using libuv which takes care of doing stuff like `select` as you describe.

LtWorf• 2 years ago

I've encountered bugs with it. Unix sockets not being created at all. So I went back to the regular loop.

theelous32• 2 years ago

There's nothing really wrong with this on one level. Your event loop doesn't even have to loop, it can just start by running your main and end when that ends. Imagine starting a server which has a while true loop that waits over a socket, and on some shutdown condition or interrupt ends, ending the program. No busy waiting, no messing with sleeps or sockets from the event loops perspective. Run until complete vs run forever.

Better off, if making some toy loop, to not bother with the forever case.

immibis• 2 years ago

how do you wait for a socket if the event loop doesn't support waiting for sockets?

coolThingsFirst• 2 years ago

Whats the best book to learn about things like these?

sscherfke• 2 years ago

This sounds a lot like SimPy [1] / Simpy.io [2], but SimPy predates asycio by many years.

[1] https://simpy.readthedocs.io/en/latest/

[2] https://gitlab.com/team-simpy/simpy.io

kodablah6• 2 years ago

Asyncio allows you to replace the event loop with an implementation of your own. For Temporal Python we represent workflows as a custom, durable asyncio event loops so things like asyncio.sleep are durable timers (i.e. code can resume on another machine, so you can sleep for weeks). Here is a post explaining how it's done: https://temporal.io/blog/durable-distributed-asyncio-event-l....

The biggest problem with asyncio is how easy/common it is in Python to be able to block the asyncio thread with synchronous calls, gumming up the whole system. Python sorely needs a static analysis tool that can build a call graph to help detect if a known thread-blocking call is called directly or directly from an async def.

guappa3• 2 years ago

Good luck with that. A simple read() might block or not, depending on what the descriptor is and how it's configured. How would you statically detect this?

kodablah6• 2 years ago

Like many static analyzers in ambiguous situations, you have to decide whether the possibility of false positives is worth the safety. In this case I'd flag it knowing that it could be marked ignored as needed (same with open() if it was used).

I have toyed with a MyPy plugin to do this, but the arbitrary state storage/cache is a bit limited to do good call graph and function coloring (metadata can be on types but not functions). And there's just not many other good _extensible_ static analysis libraries. Language like Go have static analysis libraries that let you store/cache arbitrary state/facts on all constructs.

chippiewill2• 2 years ago

Temporal is seriously cool.

When I found out how it had implemented the asyncio event loop it was a real mind blown moment.

swyx• 2 years ago

how? is there a source to read?

hot_gril4• 2 years ago

Maybe this is a bad idea, but... What if, instead of the current way where every call (including native stuff) is sync by default, it were the other way around? You'd quickly whitelist basic things like arithmetic and data struct access as sync, then you could maybe detect other things that should be sync if the event loop is spinning suspiciously fast.

9999000009996• 2 years ago

Why not use threads ? I'm still trying to understand if Python is really meant for concurrency. Asyncio always felt like it's barely hanging on, maybe it's me, by C# has a cleaner async implementation in my opinion.

kodablah6• 2 years ago

We need hundreds or thousands and they need to be cheap and, in Temporal's case, the cooperative multitasking needs to be deterministic which native threads are not. Temporal's C# SDK uses .NET tasks the same way and tasks have a lot of parallels with asyncio (though they combine with threads in one static thread pool which can be confusing in itself).

dnadler• 2 years ago

In the workflow example, is ` Purchaser.purchase` supposed to be `do_purchase`?

kodablah6• 2 years ago

Fixed, thanks! (it had undergone revisions hence the function name mismatch)

tda• 2 years ago

There is a brilliant talk by David Beazley on asyncio. I used it to write a Discrete Even Simulation tool. Pretty cool to be able to make your own implementation of asyncio and replace the system clock with a simulated time

JosephRedfern• 2 years ago

Link to David Beazley’s asyncio talk: https://youtube.com/watch?v=Y4Gt3Xjd7G8

theelous32• 2 years ago

dabeaz's async talks are imo the most important things you can consume to understand events loops and async, followed closely by njsmith's analysis and criticism of different methods of async invocation and call structures: https://vorpus.org/blog/notes-on-structured-concurrency-or-g...

guappa3• 2 years ago

No mention of poll()? It's not at all how asyncio works then.

xmcqdpt22• 2 years ago

I would have preferred if the article did the same without using yield at all, because frankly that's where I feel the real magic is.

See here for a much more in-depth description of coroutines in python,

https://aosabook.org/en/500L/a-web-crawler-with-asyncio-coro...

tusharsadhwani• 2 years ago

This is brilliant stuff, a very nice high level explanation that skips everything that would make it too boring for a first time reader.

Adding other resources at the end which explain how it _really_ works under the hood would be great.

goon12• 2 years ago

This is a similar article that I thought was pretty good: https://tenthousandmeters.com/blog/python-behind-the-scenes-...

How Python asyncio works: recreating it from scratch

Related Stories