Guido van Rossum Deconstructing Twisted's Deferreds

https://groups.google.com/d/topic/python-tulip/ut4vTG-08k8/discussion

by thezilch • 13 years ago

225 62 13 years ago

groups.google.com

pkinsky• 13 years ago

I'm probably missing something, but isn't Idea 4: Chaining Deferreds just flatMap over Futures? Scala may have spoiled me somewhat.

zeckalpha• 13 years ago

And errbacks are really just the Maybe monad.

seanmcdirmid• 13 years ago

None of these future/aync systems are very novel, but there is demand for them. Eventually most of them will fail and a few of the good ones will survive. And ya, they will resemble functional programming language X and Y in some of their features, because they aren't meant to be novel.

nostrademons• 13 years ago

Errbacks are the Error monad. Failure is the Maybe monad (or errbacks with callback=None).

noelwelsh3• 13 years ago

Yes, the Twisted Deferred is a not-quite-a-monad. The API would be simpler if it was, but I guess the original developers didn't have that insight at the time they were writing it.

dons• 13 years ago

Calling monadic bind "flatmap" makes me cry inside.

DanWaterworth• 13 years ago

Indeed, it's like snatching defeat from the jaws of victory.

On a related note, Guido van Rossum gives monads 5/5 stars... sort of.

aidos• 13 years ago

I've only looked at Twisted briefly before but I feel like I learnt a lot from this run-through.

Guido writes so well; he is able to thoughtfully deconstruct ideas with a critical eye while still praising the bits he likes. I think we could all learn something from the way he has approached this. It's definitely a model I will aspire to follow while appraising unfamiliar code.

Pxtl• 13 years ago

The bit with passing the result along in the chain of deferrals reminds me a bit of some of the fun stuff you can do with Events in C#.

Locke1689• 13 years ago

Yes, although that's considered bad practice with async.

I previously used Twisted to implement a speculative model of new Internet architectures for a graduate class when I was in my distributed systems phase in college.

One thing that Guido mentions is that Twisted Deferred computations generally rely on adding callbacks, such that you say add_this, add_that, etc. and all of these callbacks are meant to be executed asynchronously, but sequentially.

My feeling about Twisted was that Deferred's are almost too easy -- to the point where I found myself returning a deferred to another function, adding more callbacks, passing it to someone else to add more callbacks, etc.

At the end of the computation I felt like I had inadvertently brought back a lot of the pain of callbacks -- it's often difficult to figure out what gets called when and you have to look at a lot of places in the code to figure it out.

In contrast, my recent experience with async is that it encourages having asynchronous code that looks synchronous. That alone feels like a benefit to me, because it encourages the same grouping that one would normally do in their code, which just seems harder to lose control of.

C# async is certainly capable of the same "spaghetti" nature of Deferred (or callbacks in general, actually), but my experience has been that the structure encourages the opposite.

parennoob• 13 years ago

I don't understand any of the ideas alluded to therein, but the tone of the discourse in there and the thread that led to this (https://groups.google.com/forum/#!msg/python-tulip/EgpBV5-sI...) is very respectful, and people apologize if their tone was too snarky or contemptuous.

I like this a lot about python, and it makes me excited and eager to perhaps contribute a little to its development some day (I'm currently at the level where I have a decent idea about the OOP aspects of python, and know some tricks that I think are nifty, such as overloading __setattr__ and the like. So quite a long way to go, but hopefully I will get there some day).

takluyver• 13 years ago

You probably don't have as far to go as you think - the standard library is largely written in straightforward Python. The limiting factor is more finding a bug that no-one else is already fixing.

chrismonsanto• 13 years ago

Could fix this one I submitted 3 years ago: http://bugs.python.org/issue9226

d0mine2• 13 years ago

Most code don't need __setattr__ level of magic https://github.com/lihaoyi/macropy (search for "Levels of Magic")

thezilch• 13 years ago

If you happen to not make it through the thread to the following link or would prefer to join Guido and others discuss further edits to the post, before he makes it into a blog or a contribution to Twisted docs, the following is a Google Doc with several comment threads about quotes throughout the article: https://docs.google.com/document/d/10WOZgLQaYNpOrag-eTbUm-JU...

eagsalazar2• 13 years ago

Looks a lot like promises. Cancellation is missing from Q.js anyway.

One thing that always surprises me though is how people rightly are disgusted by callbacks but think promises/deferreds/futures/etc are nice. They are nicer but doing a lot of async stuff stills sucks big time even using these tools. You will write way more code and it will be way tricker than an otherwise comparable synchronous app.

Obviously sometimes you have no choice but I think one should always strive to minimize the async parts of your app as much as possible.

sethammons• 13 years ago

Or maximize the async parts of your app as much as possible if you are wanting to build a scalable, distributed system and don't want any blocking calls :)

coldtea• 13 years ago

One thing that always surprises me though is how people rightly are disgusted by callbacks but think promises/deferreds/futures/etc are nice. They are nicer but doing a lot of async stuff stills sucks big time even using these tools. You will write way more code and it will be way tricker than an otherwise comparable synchronous app.

Well, nicer is all that matters...

spartango• 13 years ago

Promises, Deferreds, and Futures are all names for the same general idea, that which uses a monad to stand in for the result of operation that may or may not have been completed.

There are implementations of this concept for many languages, from Java's futures to C# Tasks to Twisted Deferred to various JS Promise/Future implementations.

One thing that I find amusing in the callbacks+/-futures discussion is Google Guava's ListenableFutures (http://code.google.com/p/guava-libraries/wiki/ListenableFutu...):

The JDK ships with Futures under java.util.concurrent, and they do what you would expect. What Guava adds is the ability to use Futures in the "pure" sense (get/check result) and with callbacks or delegates. The neat thing here is that you can use whichever paradigm is appropriate for the design of different modules of your system, and safely mix and match as needed.

Guava isn't the answer to all the questions in the Futures world, but I do think it brings a neat idea to the table here.

hcarvalhoalves2• 13 years ago

OT: Isn't passing an object around that defines a series of callbacks to be applied essentially a... monad?

jerf• 13 years ago

No. There's too many differences in all sorts of directions for this to be a useful way of approaching them.

hcarvalhoalves2• 13 years ago

Right, I'm stretching. For a moment I had a glimpse of it being the same design pattern, since Deferreds are composable.

noelwelsh3• 13 years ago

Yes, futures/promises/whatever-you-want-to-call-them are monads in Scala and Haskell. The interface to these Deferreds are not, but it would be quite simple to make them so.

voyou• 13 years ago

Yes: here's an interesting discussion of promises (in JavaScript) in terms of monads: http://blog.jcoglan.com/2011/03/11/promises-are-the-monad-of...

YZF2• 13 years ago

I made lots of micro-edits while thinking about this, let me try and combine to something more cohesive... Also of note is that I use Twisted in my day to day.

In common Twisted programming a deferred can not fire before you add the callback since the only way a deferred can fire (for typical async work!) is through the proactor (which Twisted calls the reactor) and you haven't yet returned control to the proactor. The bug comment seems to imply a more fundamental issue but if Twisted had such a fundamental bug it just wouldn't work.

I think Guido's point is that you can still potentially add callbacks to a Deferred that has fired which opens an opportunity for bugs. He has a point there but you don't usually use Deferred that way. Generally when you add callbacks you know the Deferred hasn't fired yet.

I think Twisted isn't that bad but there are a few things I dislike about it: It's harder than I'm used to to combine synchronous and asynchronous. Twisted is "contagious".

Because the Twisted application "wants" to be single threaded, everything goes through the reactor and therefore everything has to be pure async. You can deviate from that but it has some amount of built in inflexibility. If I compare with C++ Boost::Asio there's much better multi-threading support in Asio with its worker threads (which is important for scaling across cores) and you can do more interesting combinations of async/sync IMO.

Twisted's @inlinecallbacks decorator allows you to write linear looking code you it also encourages a too-serialized way of doing things. The power of asynchronous is to do things in parallel but in Twisted you often either end up with very hard to follow "pure" asynch code or less efficient "chained" asynch code. At least that's my experience.

I personally prefer having callbacks to the Deferred returns because very often the only thing you do with a Deferred is add callbacks and it makes your code messier, more clutter/boilerplate. Returning Deferred allows for the "generator" style @inlinecallbacks which actually can be an anti-pattern from a performance perspective and also many people find it confusing to read... Explicitly chaining callbacks is what I prefer. To elaborate on callback vs. Deferred you can relatively easily build more complex structures of any kind you wish over the callback when you need it. In a sense it's the most basic way of doing things. I like basic :-)

Related to combining async and sync styles you end up seeing things like isinstance(ret, Deferred). There's also maybedeferred for dealing with async vs. sync. Still not great.

rdtsc10• 13 years ago

it's like Twisted is contagious. I can't quite put my finger on it.

I can. Twisted is viral (not in a good way always). Once you start using Twisted you are doomed and stuck in a parallel reality of Twisted-only libraries forever. I have used Twisted professionally for 4+ years, this is not something I just read on a blog.

You first say "Oh, I know look at the cute example of parsing a line protocol with it, let my try that. That was easy, I should adopt it". And then it got you. You want to fetch data from a particular database. That's easy, github has a project in Python that does it. Oh but you need it to return a deferred now. You can't simply make a connection and block because it blocks the reactor. This library is no good, you either have to pray someone who knows Twisted wrote a Twisted version of it, or you have to write your own.

As you pointed out, you can use code to combine threaded and deferred code but it is not fun or easy. Make sure you think twice before adopting it.

BTW the best way to learn it is to use it. Don't read about it. Reading about it won't help much. It is one of those subjects, like riding a bike if you don't know how. Once you get it you get it and it becomes easy. What is not easy is library fragmentation.

You have been down-voted and perhaps I'll be too for my negativity. But I am worried about infatuation with Twisted and Promises in recent (last year and on) Python design emails and moving them more to be part of the standard library.

I think this is a not healthy move for Python and a step forward. I'll speak my mind and if you feel this is too negative, just downvote, I'll understand, but I think it has to be said. They are copying what they are seeing in other language hint: node.js (Javascript). I feel there is a bit of jealousy, Python is maturing, its last exciting thing was 3.0. PyPy has been relegated to the magic academic research corner (unfairly I think). But, node.js and Javascript on the server is not successful because it has a callback style of concurrency. There are other reasons. So copying that aspect won't make Python more appealing. Heck, Python had an awesome callback library for ages -- Twisted, when Javascript was still use to make buttons flash and Java applets were the hot thing. Did it take off? Somewhat, but not really. It was famously hard to grok. The freshest breath of air so far in practical Python use on the server is using greenlet. But that also has been pushed to sort of "ah it is dirty, we don't like, how will you ever know if your code switches co-routines". Well it is not perfect but it is most practical and performant to date. I wish that became part of core Python. I wish they copied more from Go, Rust or Erlang rather than a callback style concurrency with some extra sauce on top. A

Anyway, ending my rant here, and hoping I didn't upset too many people already.

ekimekim• 13 years ago

Agreed. I make heavy use of greenlet/gevent, and one of the primary advantages is that you can import and use naive sequential libraries, and they won't block your process (assuming they're pure python so the monkey patching works properly).

As for "not knowing when you might switch coroutine" - I contend that if it matters, you're doing it wrong. Note that, unlike proper threading, things can't happen at the same time / interleaved with simple statements, so it's still very easy to do atomic actions without special locking - eg. there's no way self.x += 1 could switch in the middle (unless someone is badly abusing the __add__ method).

cdavid• 13 years ago

I only have around ~1 year of twisted experience, but I felt like the viral aspect of twisted was even worse than that.

Not only everything needs to be asynchronous, but using deferred leaks everywhere in your API. Deferred callback firing mechanism 'recursively' passing the result of the previous callback is the most egregious aspect of it. Yet, I think deferred is by far the most readable/maintanable method to write async in twisted. For example, inlineCallbacks looks nice on paper, until you start using them and realize stacktraces are often wrong, and that you loose a lot of concurrency by forcing things that could run in // to be serialized.

I think explicit async is simply the wrong model for a dynamic language. Often, async is just an optimization, and more efforts should be spent on how to make it happen under the hood.

lvh6• 13 years ago

I can. Twisted is viral (not in a good way always). Once you start using Twisted you are doomed and stuck in a parallel reality of Twisted-only libraries forever. I have used Twisted professionally for 4+ years, this is not something I just read on a blog.

This seems to be a common sentiment. I'm writing a talk which I'm submitting to PyCon 2014 that hopes to demonstrate that that isn't (or doesn't have to be) true.

I've made the repo[1] and preliminary slides[2] available :)

[1] https://github.com/lvh/TwistedMixing [2]: http://blog.lvh.io/TwistedMixing

adamtj• 13 years ago

The viral nature of twisted is not limited to twisted and shows up in lots of places. It's actually a consequence of the problem that "concurrency is Hard".

There is a tradeoff to be made between pre-emptive and cooperative multitasking. Every platform must choose one or the other or a mix of both, and then deal with the consequences. Threads, of course, have a well-known class of very dangerous bugs that make using them difficult. Processes make pre-emptive multitasking much safer by erecting certain barriers, but at the cost of being much more tedious.

These days, callbacks, events and other forms of cooperative multitasking are the new shiny. They avoid the big problems of threads and multiple processes, but have some problems of their own. First and simplest is that you can only use one CPU per process without mixing in threads or multiple processes. Second, if something doesn't cooperate, it blocks everything else. You have to be very careful about when and where you yield the CPU. Because of that, everything must be event-driven or use callbacks, or you risk trashing your throughput and/or latency.

Threads have a similar problem: if you use them, you're stuck using thread-safe libraries. It's not as bad as the cooperative case though. The thread-safety of a library can be independant of the code that uses it. In the cooperative case, you don't have that independence. You're not just stuck with cooperative-safe libraries. You are restricted to events or callbacks that are compatible with the cooperative framework used in your application. If your application uses events, then the library must use events instead of callbacks, and it must use the same types of events to plug into your event loop. If not, then you need a compatibility layer. Some cooperative languages reduce the problem by mandating how events and/or callbacks work for all libraries. Python doesn't have that, which where the viral nature of twisted comes from. That's why you're talking about twisted-compatible libraries and not just python libraries, and why this a non-issue with node.js database libraries.

edit: wording

thristian3• 13 years ago

In common Twisted programming a deferred can not fire before you add the callback since the only way a deferred can fire (for typical async work!) is through the proactor (which Twisted calls the reactor) and you haven't yet returned control to the proactor.

    d = Deferred()
    d.callback(42)
    return d

Hey, I just made a Deferred that fired before any callbacks were added! In fact, this is such a common occurrence that there's helper functions in the "defer" module to do it: defer.succeed() and defer.fail(). In particular, this sort of thing is very useful for unit-testing Deferred-based APIs.

rdtsc10• 13 years ago

You are right. I _think_ what they meant to say was that the deferred can only fire once. You cannot call .callback() twice. Or adding deferreds to it after it fired won't get those functions called.

However, even with respect to your example, it is not idiomatic. Yes you can do anything on anything in Python. And in testing you often monkey-patch and mock things. Which doesn't mean you'd do often in production code. The idea of deferreds is to help with highly concurrent IO applications. It will be a file descriptor (socket) or a timer that will fire your callback more than often.

YZF2• 13 years ago

Yes and that's why I added typical async work. From the docs "callback: Run all success callbacks that have been added to this Deferred.". Let's see your calling code to this snippet. Is it adding callbacks? Why?

lvh6• 13 years ago

The bug comment seems to imply a more fundamental issue but if Twisted had such a fundamental bug it just wouldn't work.

Exactly. It works just fine.

I think Guido's point is that you can still potentially add callbacks to a Deferred that has fired which opens an opportunity for bugs. He has a point there but you don't usually use Deferred that way. Generally when you add callbacks you know the Deferred hasn't fired yet.

I still really don't see the problem with that.

I think Twisted isn't that bad but there are a few things I dislike about it: It's harder than I'm used to to combine synchronous and asynchronous. Twisted is "contagious".

I'm hoping to demonstrate that that doesn't have to be true: see the end of this comment :)

Because the Twisted application "wants" to be single threaded, everything goes through the reactor and therefore everything has to be pure async. You can deviate from that but it has some amount of built in inflexibility. If I compare with C++ Boost::Asio there's much better multi-threading support in Asio with its worker threads (which is important for scaling across cores) and you can do more interesting combinations of async/sync IMO.
Twisted's @inlinecallbacks decorator allows you to write linear looking code you it also encourages a too-serialized way of doing things. The power of asynchronous is to do things in parallel but in Twisted you often either end up with very hard to follow "pure" asynch code or less efficient "chained" asynch code. At least that's my experience.

Yes, I think that's definitely true :) @inlineCallbacks has its uses: if you really really want to do this bunch of things sequentially. Of course, since @inlineCallbacks decorated functions return deferreds, you can then use that with tools that let you manage a bunch of jobs in parallel. E.g. if a given task is a few steps, you can write it in an @inlineCallbacks decorated function, and then if you have to do a bunch of them, just call it a few times (and maybe pass the resulting deferreds to gatherResults if you want all of the results).

I often end up with some code in @inlineCallbacks that wraps some regular callback code, or the other way around. They're different tools, and while on average my experience has given me an anti-inlineCallback bias, it has its uses :)

Related to combining async and sync styles you end up seeing things like isinstance(ret, Deferred). There's also maybedeferred for dealing with async vs. sync. Still not great.

I'm really interested in this, because I'm writing a talk about mixing Twisted with other stuff. What about maybeDeferred do you take issue with? FWIW, I know of some shops, e.g. Vumi in South Africa, that mix sync and async just like this.

I've made the repo[1] and preliminary slides[2] available :)

[1] https://github.com/lvh/TwistedMixing

[2] http://blog.lvh.io/TwistedMixing

jasonmoo2• 13 years ago

I think the entire go spec would fit into the same number of pages as Guido's explanation of deferreds. zing!

jasonmoo2• 13 years ago

Omg you guys have no sense of humor.

iopq• 13 years ago

I can write what's new and exciting about Go on the back of a napkin.

thristian3• 13 years ago

I first came across Twisted Python and Deferreds years ago, and it was my first introduction to the world of asynchronous programming. It was a little bit difficult to wrap my head around the intricate braid of callbacks and errbacks, but once I realised they mapped directly onto sequential statements and try/except in synchronous code, I got a lot done.

In the intervening years, I've seen a lot of projects re-implementing asynchronous programming (A perfectly valid idea, Twisted certainly has enough backwards-compatibility warts to warrant an occasional rebuilding), but the early attempts used bare callbacks for everything, which I found frustrating because I felt Deferreds were clearly a better system. Later attempts introduced "Futures" or "Promises" which were slightly better than bare callbacks but did silly things like pass the same initial result to every registered callback instead of being composable. These almost-but-not-quite-entirely-unlike-Deferreds were even more infuriating, in a "somebody is wrong on the Internet" kind of way.

I'm really glad to see Guido picking the best ideas out of Twisted's Deferreds and trying to make them accessible to a professional-developer audience. If your language/runtime doesn't support some kind of co-routines, then your only hope for dealing with asynchronous code to tame callbacks, and Deferreds are the best model I've come across for doing that.

(Guido mentions he won't be using Deferreds or Deferred-analogues in his Tulip async framework, for undisclosed reasons. I'm betting those reasons are that modern Python does have workable co-routines, so callbacks and callback-taming aren't necessary.)

lazyfunctor• 13 years ago

I am new to this. Can you give me any pointers to differences between promises and deferreds. I have some idea about promises in JS. http://promises-aplus.github.io/promises-spec/ How is twisted's deferred different?

thristian3• 13 years ago

Just the first difference that caught my eye: section 3.1 of the document you linked to says that once fulfilled or rejected, a Promise must not change to any other state, and its value may not change. In Guido's exposition of Deferreds, under the heading "Idea 2: Pass results from callback to callback", he says:

The most interesting bits are the last two lines: the result of each callback is passed to the next. This is different from how things work in concurrent.futures and Tulip, where the result (once set) is fixed as an attribute of the Future. Here the result can be modified by each callback.

This enables a new pattern when one function returning a Deferred calls another one and transforms its result, and this is what earns this idea three stars.

noelwelsh3• 13 years ago

I think this distinction should be framed in terms of Python's futures, not futures per se. Python's futures (http://docs.python.org/dev/library/concurrent.futures.html#c...) are very limited. They lack the equivalent of the then method in Javascript promises. This means there is very limited ability to compose futures in Python.

Twisted's Deferred allow some kind of composition, though IMHO it is rather messy.

You'll note in Guido's emails he says Python's futures are based on Java's. Java's futures (I can say from experience) are a completely useless crock. All you can do is poll them, which makes the whole point of asynchronous operation moot (http://docs.oracle.com/javase/6/docs/api/java/util/concurren...)

Javascript's promises are a better implementation, as are those in Scala and Haskell.

penguindev• 13 years ago

that modern Python does have workable co-routines

I think its called 'yield from'. Not sure if I'm sold on the name, but surprised nobody has mentioned it yet. From what I've seen, it looks really nice.

A comment on the post has an example of it:

def sync_and_read_bookmarks(): if (yield from sync_bookmarks()) > 0: return (yield from read_bookmarks())

annnnd• 13 years ago

This bit sums my experience with Twisted nicely: "...[the available docs] are either aimed at absolute beginners or at experienced Twisted users." I was highly frustrated trying to find any useful docs on Twisted.

This is where I dumped Twisted for Tornado (and I'm quite happy about it too ;). Looking back Twisted seems over-engineered too, but that is strictly my personal opinion and could be due to lack of understanding.

rkangel• 13 years ago

What interested me is the discussion about naming (re-use of callback, "errback", "Failure" etc.).

That sort of discussion quite often smacks of bike-shedding, but Guido's points here made me release how clear and simple a lot of the naming in Python is, and how much that adds to readability and ease of understanding of code. Clearly a lot of thought along the lines of "this should be a verb" has gone into the standard library and choice of language keywords, and this has spread to code built on top. Without realising it, this is one of the things that has drawn me to Python.

I'll stop taking it for granted now, pay attention to how it's done, and maybe my own code will benefit.

dustingetz• 13 years ago

Here is a much better explanation, in Javascript: http://domenic.me/2012/10/14/youre-missing-the-point-of-prom...

robert-zaremba• 13 years ago

Interesting thing is that when using geenlet + proper event framework (gevent / eventlet) you don't have all this issues about different callback styles / methods / objects, and what is more important, IO libraries support (like DB drivers). I know there are ways to use twisted drivers in tornado, or use `deferToThread` - but they doesn't seems as pythonic as monkey patching + python drivers (like with eventlet + <your pure python DB driver>)

It's worth nothing that PyPy has a great support for greenlets with his JIT.

Guido van Rossum Deconstructing Twisted's Deferreds

Related Stories