The Twisted Way

http://glyph.twistedmatrix.com/2012/12/the-twisted-way.html

by pablohoffman • 14 years ago

27 29 14 years ago

glyph.twistedmatrix.com

pekk• 14 years ago

It's a library, not the Tao. And it is a big, complex library which introduces plenty of its own unique jargon and abstractions, which is why people find it confusing.

The fact that Twisted advocates have to keep browbeating others to accept it is just one indication that it's not perfect and it's not for everyone.

vosper3• 14 years ago

Twisted is the least accessible of the big Python libraries that I've come across. I've come to it a couple of times when it seemed like it would suit a problem I had, only to throw up my hands in frustration and go look for a solution somewhere else.

Unfortunately this article did nothing to enlighten me - it has some pretensions of being philosophical without explaining any of the practicalities.

Does anyone use Twisted in production? Do you consider it worth the learning curve, and would you use it again if you were implementing your system now?

influx2• 14 years ago

I use Twisted and love it. What really helped me was reading the source code instead of trying to decipher the documentation. The code is really readable and things started clicking once I made that investment.

vosper3• 14 years ago

If you don't mind - what did you build with it? Was Twisted the only suitable library - what about Tornado?

I'm curious because Twisted has been around for ages and seems to be in active development but I've never talked to anyone who actually uses it.

wsc• 14 years ago

> What really helped me was reading the source code instead of trying to decipher the documentation.

That's an awful excuse. People rely on libraries in order to do their jobs. A "worse" library with better documentation helps more people.

cturner2• 14 years ago

I've used it, and had some success with it, but now use my own stack for network programming.

I struggled with twisted several times without getting it. Then I did some non-blocking socket/select programming and thought, "I wish I had a library that.." and then realised what twisted did.

There's a few reasons I'm not using it now. Main one is that I've turned against frameworks. I find it's generally easier to understand a problem and write the code than to really learn a generic framework and coax it to your needs. And by just writing the code yourself you don't have to deal with dependencies, you have more freedom to make changes, debugging is easier, it's easy to deploy in a restricted environment.

Also, I have a rule of thumb when I'm writing python that if I'm using inheritance I'm doing something wrong. I have huge trouble understanding duck-type code that uses inheritance. Encapsulation is always possible, and I find it a lot easier to refactor. Twisted uses inheritance extensively. The docs encourage you to extend their classes. In places the application developer is expected to reach into framework objects and poke variables.

lvh6• 14 years ago

Your restricted environments don't allow virtualenvs?

It's interesting that inheritance vs composition comes up, since one of the most common criticisms I've heard is actually the opposite (that there's too many different objects to get things done).

However, I do understand why you might think this: protocols, which is probably the first thing you touch in Twisted, are usually implemented through inheritance. For resources, the default implementation that you subclass (twisted.web.resource.Resource) contains a bunch of behavior that you almost certainly want.

The important thing to remember is that this inheritance is only to get default behavior. If you want to implement it without inheritance, Twisted very clearly and explicitly defines the interface you have to implement:

https://twistedmatrix.com/documents/current/api/twisted.inte...

https://twistedmatrix.com/documents/current/api/twisted.web....

These are just two examples: Interfaces are all over twisted.

And, to quote from the article:

  It doesn't need to be that way, though.  When you create
  an object, it is best to create it as independently as
  possible; to test it in isolation; to discretely separate
  its every interaction with the outside world so that they
  may be carefully controlled, monitored, intercepted an
  inspected, one iteration at a time.

This has recently come up in the async-pep conversation on Python-ideas. If you have a significantly better solution for e.g LineReceiver, I'd love to steal it!

lvh6• 14 years ago

Yes, I use Twisted in production. I absolutely consider it worth the learning curve. And I've started new projects in it in the last 3 weeks, so I guess that answers your final question.

cmccabe4• 14 years ago

GREAT SOLUTIONS IN ENGINEERING

Problem: bicycle seats are hard. They hurt.

Analysis: there must be something wrong with your pants.

"Solution": dorky pants.

from http://dilbert.com/strips/comic/1994-07-18/

Problem: Python has poor support for threads, and concurrency in general.

Analysis: you must need more middleware.

"Solution": incomprehensible callback-based frameworks.

readme• 14 years ago

If by incomprehensible, you mean you need to read more than 10 pages of documentation to understand it, then sure.

pyalot2• 14 years ago

There are two problems with callback based frameworks.

#1 In python specifically, we do not have proper closure and no anonymous blocks, this makes operating it tedious as the definition of the logic for a callback is always somewhere else from where the callback is setup.

#2 In any language supporting proper closures (javascript for instance, hello node.js) excessive, nested use of callback based frameworks leads to the phenomenon of the pyramid of death, whereas you keep nesting closures and you have to keep them all in one place because each depends on its outer scope.

I intently dislike twisted and node.js for these very reasons. I've written my own little framework based entirely on greenlets in python, which is delightful and easy to use and entirely avoids both dislocality of action and the pyramid of death.

cmccabe4• 14 years ago

Callbacks are bad for the same reason non-local gotos are bad. They destroy locality and make the code hard to read.

rhettg• 14 years ago

I've used twisted for semi-major project (https://github.com/Yelp/Tron).

It's certainly a difficult thing to get your head around. The parts of the project that touch twisted is always the most complex to explain to anyone else. However, async code is always hard. It also doesn't help that the code base shows it's age.. non-PEP8, strange naming conventions etc.

However, one thing that I've noticed with recent projects using Tornado (https://github.com/rhettg/Dynochemy) is that some concepts that I thought was overly complex in Twisted makes sense eventually. For example, handling errors in async code is really hard. The framework twisted came up with with Deferred, callbacks and errbacks is pretty smart. You almost have to re-invent twisted before you can appreciate it.

I'd recommend reading the original paper for more information: http://www6.uniovi.es/python/pycon/papers/deferex/

sneak• 14 years ago

Twisted is great. The only real problem comes from their utter unwillingness to accept less-than-100%-perfect ipv6 patches. It's 2012 for fuck's sake.

leoliu• 14 years ago

I am more interested in a post about what's wrong with twisted. Anyone care to comment?

w0utert• 14 years ago

Having used Twisted for a few projects and now using Pyramid for another, I'd say there are two big problems with Twisted that kill the experience for new developers:

1) The documentation is terrible.

Even though there's a lot of it, and even though the API documentation is pretty good, the 'narrative' and tutorial-like documentation is downright horrible. It tries to cram so many things into the same contrived examples at once, that reading it creates more confusion than giving insights. Take for example the 'evolution of a finger client', which is one of the first things you get directed to when you want to start 'learning twisted'. Instead of introducing some basic concepts that affect design decisions and then working them out in simple, individual, small examples, it starts out with a server example so trivial that it's completely un-representative of how a typical Twisted service would look like, and then, in a few pages, crams so many things into it at once that you have to go back, re-read and actually try to implement and run the example yourself to figure out what (and especially: why) the heck things are working like they are working. In one swell swoop it 'teaches' you Twisted by throwing around component-based programming, Zope interfaces, adapter classes, the factory pattern and how you can basically use it everywhere even though there's no real reason to do so, how you can expose 4 different protocols talking to the same service at once, how you can add asynchronous versions of the same services, and so on. All of it combined in a single 'Twisted application' that reads like some kind of intentially obfuscated crypto-program that somehow magically does all these things.

Compare to that the Pyramid documentation, which is absolutely fantastic and got me up to speed and writing very clean and stable applications in just a few hours. Where reading the Twisted docs the main thing going around in my head was 'wtf do I have to do here and why', while with Pyramid it everything immediately made sense. Note that I'm not saying this is a problem with Twisted itself, just with the way the documentation tries to explain how to use it.

2) Twisted doesn't advertise its specific benefits and downsides very well.

Superficially looking, many people might think Twisted is just like other web-application frameworks like Pyramid, Flask or Django, but it isn't. You can do most of everything using Twisted, but it's much lower-level than those. Just like this article mentions, Twisted is more like a toolbox to create event-based applications than it is like a web-app framework. This makes it a lot more flexible, but also more complex. While this allows all kinds of use cases that nobody would ever consider e.g. something like Flask or Django for (e.g. game servers, chat servers, etc), it also means that if your application is 'yet another web application', the other options are probably going to be a lot simpler and cleaner than using Twisted.

I think because Twisted doesn't want to choose and declare what it's main strengths and purposes are, it's attracting many developers of applications that are a much better fit for the more specialized web-app frameworks, leaving them with a really bad experience, spending too much time figuring out the framework and ending up with cryptic gibberish that doesn't really implement any actual application logic.

Don't misinterpret this as a critique of Twisted itself, I still like it a lot, exactly because it is so flexible and generic. It's just not the best tool for most web applications, but if if you are implementing some kind of event-based application that doesn't fit the philosophy of the other popular frameworks, it's extremely powerful and well-designed. I think the Twisted community should make this difference more explicit for new developers.

vosper3• 14 years ago

Your analysis of the problems with the tutorial is spot-on. It also really doesn't help that it's presented as "this is a description of what's changing on this page, and here's wall of updated code - go figure out the differences from the previous page"

inglesp• 14 years ago

For anybody wanting to learn Twisted, I can only really recommend the excellent introduction to asynchronous programming (with Twisted) here: http://krondo.com/?page_id=1327

ishbits• 14 years ago

I used Twisted in 2005 to develop and server and agent infrastructure.. Portions of the system are now in Java, but the distributed agents are still in Python and out there in production..

I am currently not using Twisted for anything, but often think that things I'm currently working on would be easier if done in Twisted.

I came to Twisted after 5 years of doing non-blocking servers in C with libevent (or hand rolled event loops). So I didn't have to wrap my head around don't call us, we'll call you aspect. I'm currently using Tornado for a websocket based service, but will probably use Twisted again for my next Python project.

The Twisted Way

Related Stories