Twisted Python's lead hacker discusses Tornado

http://glyph.twistedmatrix.com/2009/09/what-i-wish-tornado-were.html

by thristian • 17 years ago

101 77 17 years ago

glyph.twistedmatrix.com

davidw2• 17 years ago

Nice article. As someone who has been involved with open source for a while, I think one of the most annoying things is indeed when people don't even tell you about their problems. I'm already doing work for you for free; I don't have money to go do market research, so I'm counting on you to step up and say something if there's a problem.

acangiano2• 17 years ago

He makes excellent points, but let's compare the two scenarios for a moment.

A) Helping out Twisted, they'd have gotten:

* Kudos from the Twisted community for their contributions to the project;

* Less code required.

B) Creating Tornado, they got:

* To avoid other people's code they didn't like in the first place.

* All the credit. They are the authors of the framework, not just contributors;

* A fresh start. A project designed to their needs;

* Ability to lead the project in the direction they like;

* No stigmas attached (e.g., "Twisted is too complex");

* Major kudos from a large community of developers interested in "real-time" web services and applications;

* Tons of free publicity to the project and FriendFeed/Facebook's open source initiatives.

pvg3• 17 years ago

This is deeply silly. More fame and glory? So one day, Bret Taylor leaned back, rubbed his hands and whispered to himself 'Yesss, I shall write Tornado for fame and glory. Before I start I must briefly consider how I'll keep all the screaming teenage fans away and what to wear at the ticker-tape parade'. Or perhaps he just wanted to solve a problem he and his team had. And having done that, was nice enough to open source the solution.

acangiano2• 17 years ago

I removed my "fame and glory" comment as it was misleading and didn't truly represent what I think. I'm not arguing that he/they premeditated it and did it for the glory. Their priority was getting FF done, undoubtedly. However, there are concrete advantages (and visibility) deriving from their choice.

Confusion2• 17 years ago

To avoid other people's code they didn't like in the first place.

'Not Invented Here' syndrome. A bad reason.

A fresh start. A project designed to their needs

That's shortsighted. Maybe the short term goals had different accents, but in the long run, Twisted and Tornado have the exact same goals.

Major kudos from a large community of developers interested in "real-time" web services and applications

For rewriting something that already exists and not doing an significantly better job and arguably a worse job? Applying those resources to Twisted would also have yielded this result.

Tons of free publicity to the project and FriendFeed/Facebook's open source initiatives.

Also belongs under A. Large contributions to OSS would yield that same effect.

A fresh start. A project designed to their needs Ability to lead the project in the direction they like

You are assuming, without any reason, that they wouldn't have had that chance with Twisted. If that wouldn't work out, they could easily have forked Twisted, effectively taking over the project.

encoderer3• 17 years ago

"'Not Invented Here' syndrome. A bad reason."

That seems a little strawman-ish, honestly. On the contrary, they embraced the technology and architecture of the app.

What's wrong with looking at something like this.. an important infrastructure piece... liking the idioms and architecture but having issues and thinking .... We have a benefit the Twisted guys didn't have... we have prior art... a map of this territory ?

I don't think anybody here has a strong conviction that FriendFeed was _right_. But you and others seem to have a strong convication that they were wrong, and all I think....and I think others here too.... all I think is that it is silly to attack them and be so certain they were wrong.

These are talented developers. Smart guys. With a little bit of business talent, too.. All of us have been in complex project cycles. Who are you or anybody to pick them off like they somehow caused somebody harm? Why the intensity? What gives?

scott_s4• 17 years ago

When the invention in question is core to your product, sometimes it does make sense to do it yourself. See Joel Spolsky's essay "In Defense of Not-Invented-Here Syndrome": http://www.joelonsoftware.com/articles/fog0000000007.html

mcantelon2• 17 years ago

If you don't like someone's code, and the time cost of communication with them to change the code upstream is likely to stop you from meeting deadline (or they have a different final vision), creating something tailored to your own needs is the only solution.

far33d• 17 years ago

> That's shortsighted. Maybe the short term goals had different accents, but in the long run, Twisted and Tornado have the exact same goals.

Actually they don't. Twisted is "an event-driven networking engine written in Python"

Tornado is "an open source version of the scalable, non-blocking web server and tools that power FriendFeed"

Note that the scope of these things are completely different.

pvg3• 17 years ago

There seems to be a rather unrealistic assumption that people who don't like your product (be it website, framework, sandwich) owe you well-reasoned feedback. It's a great and useful thing to get but the fact is, most people are interested in solving their problem and unless they are quite sure your product is the best way to solve their problem, they'll simply move on.

Perhaps this is an artifact of how seriously the Twisted people and their 'lead hacker' take their project. But the twisted-centric view of the world leads to petulant posts full of odd statements:

My main point here is that if you're about to undergo a re-write of a major project

They wrote Tornado, they didn't 'undergo a rewrite of Twisted'.

For a tiny fraction of the effort invested in Tornado, FriendFeed could have worked with us to resolve many of the issues creating that chaos.

Interfacing with an another team to solve problems in a framework they didn't like (even if just aesthetically) would have probably taking significant resources for close to zero return.

There's quite a bit more in the same vein. It's wonderful that the Twisted people care. But perhaps a little too much.

ellyagg• 17 years ago

I'd be pretty damn upset too if some other project got a lot of press for an announcement which included comments undermining the quality of my project. Sorry, but if you're going to strongly imply that my project's ease of use and performance are subpar, them's fighting words.

pvg3• 17 years ago

The comments were really in passing. 'Looked at it, wasn't for us', more or less. Perhaps them's fightin' words. But getting huffy about them probably helps your project even less, given they'd have otherwise gone largely unnoticed.

encoderer3• 17 years ago

Ok, but did you click thru to that listserv thread here https://garage.maemo.org/pipermail/brisa-develop/2009-August... ?

I'm not familiar with the project or this guy in particular, but man, he seems awfully abrasive. Sarcastic and bitter came to mind as I read them. I have no dog in this fight, and one could fairly assume that these 2 discussions are just not his best moments. But from what I've seen...

Confusion2• 17 years ago

The assumption isn't that they owe you feedback: the assumption is that it was not in their best interest to create another instance of something that already exists. That assumption is supported by the fact that Tornado has the exact same goal as Twisted. Effectively, it is a rewrite, especially if they have obviously looked closely at Twisted's code for inspiration.

A CTO wasted money here.

jnoller• 17 years ago

No, a CTO didn't waste money; he got exactly what he needed, cleaner, simpler. No - it doesn't handle every single edge case, nuance, platform - but it does exactly what they needed. They didn't need to spend time patching and grokking twisted, or needing to deal with any rough edges to ramping personnel up on it. They had a job to do, and they did it.

Now that it's out in the wild, maybe someone will cross-pollinate, maybe changes will go in both projects. But the good thing is that they got their job done.

scott_s4• 17 years ago

In that sense, Firefox is a "rewrite" of IE.

lucifer• 17 years ago

If you accept the validity of the cathedral vs. bazaar premise, then the pattern of OSS fragmentation ultimately works against the general community's interests.

On the other hand, competition in the course of establishing best of breed constructs for emerging paradigms is healthy, as a preliminary stage before community building per Raymond's model.

davidw2• 17 years ago

Ok, but there are products, and there are products. This is a product that you can hack to your heart's content, not something sitting on a store shelf that you can take or leave. Oh, yeah, and you get it completely for free. And with a license that you can base your N-million dollar company on it with no ifs ands or buts. And could potentially save you thousands of dollars worth of work. And with developers willing to contribute their own time to make it work better for you. And yet you can't write an email saying "hi, this almost works for us, but has a few issues I'd like to talk about (and potentially help fix)".

dirtyaura• 17 years ago

Interfacing with an another team to solve problems in a framework they didn't like (even if just aesthetically)

I think dislike for aesthetic reasons is one of the best reasons to do your own rewrite. I can imagine that proposing radical changes to e.g. API style of an existing FOSS project would be the most difficult up-hill battle.

tdavis• 17 years ago

One thing to remember about Twisted is that it is basically worked on by two people. It is an enormous code base for those two to maintain, develop on, and document at the same time, even if they are ridiculously smart (which they are).

I gave kudos to Tornado, but Twisted deserves some appreciation too -- and contributors! I tried to take twisted.web into the present (the client anyway), but I unfortunately wasn't sufficiently TDD to get it contributed. Perhaps next time!

codexon2• 17 years ago

I submitted a patch to Twisted to fix their SOCKS implementation.

It was "accepted" after a month, and by accepted it means put into a branch where it will take up to a year to be incorporated into the mainline depending on whether the few maintainers feel like taking a look at it.

andrewtj• 17 years ago

Two people is news to me — their website lead me to believe otherwise: http://twistedmatrix.com/trac/wiki/TwistedMatrixLaboratories

moe• 17 years ago

I agree that FF should have asked and should have used twisted under the hood. I hope the porting efforts will be successful and eventually replace the original Tornado impl.

I cannot help, though, but yet again in this blog post notice the unfortunate, defensive tone that is so omnipresent in the twisted community.

The post starts out by (politely and rightfully) pointing out FriendFeed's communication mistake and then goes on to describe how green the grass would be in twisted land.

What I miss is a bit more critical self-reflection about the reasons for FF skipping twisted, even without FF spelling them out in detail. It's not likely that FF found completely new and unseen problems with twisted. I'm more inclined to believe that they simply stumbled over the standard, long-known twisted problems; first and foremost the lack of documentation and (probably) the verbose coding-style that many parts of twisted demand.

I have the highest respect for the effort that went into twisted and for their often cited code quality. But they need to realize that they can't display a "your own fault if you're not smart enough for this"-attitude towards documentation and at the same time mock anyone who doesn't want to put up with that.

Like any OSS project twisted is ofcourse not obliged to anything. But neither are people like the FF guys obliged to choose twisted.

rg123• 17 years ago

I 2nd this completely re: Twisted documentation. When Twisted's communication via their documentation has been so lacking/off-putting/etc., it is hard to feel sympathy for a Twisted guy complaining about the lack of communication from people who chose a different route.

andrewvc• 17 years ago

Well, if this benchmark from yesterday is accurate, then Twisted can't handle as many concurrent connections as Tornado. I'd say that's a good reason not to use Twisted.

http://www.apparatusproject.org/blog/2009/09/twisted-web-vs-...

roder2• 17 years ago

If you read the comments in the blog, it's likely just related to the platform. I did the test on my mac, which certainly isn't the final say... I'll redo the test today different hardware today.

roder2• 17 years ago

I thought that glyph's response was appropriate and on the mark. It seemed to me that Bret had taken a few jabs at Twisted without really providing quantitate evidence. Particularly the comments about being "demo-quality" and "tons of bugs".

It would have been nice to see innovation happen collaboratively between FriendFeed and the Twisted team. Everyone would have benefited more IMHO.

Twisted has certainly done a lot for event-based architecture. As Glyph points out, the use of Deferreds have made their way into javascript frameworks like dojo. To me, that's credible and proof of twisted's impact in the development community.

boucher• 17 years ago

I really think that calling Deferreds a "de-facto standard" in JavaScript is a huge stretch.

I'm having trouble pinning down exactly what makes Deferreds different from any asynchronous callback system, but it seems to be the chain of callbacks?

dlsspy9• 17 years ago

One of the frustrating differences I ran into is the callback mechanism in tornado. You can't attach more than one callback, and the one kind of callback does two different things (meaning the callback has to determine whether or not the call was successful before it does its work).

To me, this makes things more complicated. What if I want to use the same error handling code for all bits of code (really common for me is `d.addErrback(log.err)` to just log it), but each successful call has to do something different?

Twisted deferreds also allow you to compose chains.

This is a somewhat trivial example, but by having the callback do both success and error cases, I had to build an adaptor to pass to callbacks to include the "is this a failure" information.

The easy way to do that is to create a wrapper class and construct it one way or another based on whether a success or error occurred. I ended up just composing it like this:

    deferred.addCallback(lambda x: Response(body=x))
    deferred.addErrback(lambda e: Response(error=e))
    deferred.addCallback(lambda r: callback(r, *args, **kwargs))

The first callback and errback create a Response (wrapper) object that contains either a body or an error. The second callback receives the response that was made by one of the above and calls the initial callback.

This is sort of unrolling complexity and sweeping it away. The real problem is that you can only have one callback in tornado (and you can't easily chain them). That means that what actually happens when this deferred's callback fires is that it ends up calling the original callback, which is an inner wrapper function inside of another function that calls a curried function that calls your original callback -- and you have to create these things by name when you're building out your auth class.

To me, that's complicated.

deferreds seem like there's more involved than you get with a simple callback until you actually try to use the simple callback.

The interesting properties of deferreds are

1) You can add multiple callbacks and errbacks and build chains as complicated as your system needs including composing common cases (such as success and error counters).

2) If a callback (or errback) returns a deferred, you spin off another dimension of the chain.

#2 is why the above code and about twenty lines of web.py exist for a single use case.

Here's a concrete example (close to something I do in the real world™):

Imagine a simple twitter API. You go grab a tweet and then give it to something else: `t.grabTweet(something).addCallback(processTweet)`

Now someone adds a requirement that between `grabTweet` and `processTweet` you want to perform short URL expansion. In twisted, I'd just toss an `addCallback(expandURL)` in the middle and call my awesomely well-tested URL expander -- which makes an async http call to another service and doesn't call the next callback in the chain until that service returns the data I need to fill satisfy the request.

And then after I did that, I found that the URL expansion stuff I was using was blowing up the upstream and causing me to fail to expand some, so I added async cache support to those inside the API (not as an additional toplevel callback). For this new step, the toplevel API doesn't change, but now I can do more async stuff that may lead to further async stuff without having to write additional glue code.

smoody• 17 years ago

I understand the response at some level, but I also totally get the idea of making things work the way one's mind works. It makes coding much more enjoyable and speeds the process of app development once the core work is done.

And it was nice of the author to do some QA work for Tornado, too. Hopefully he'll go the extra step and submit those bugs he found as tickets so that he's walking the walk and not just talking the talk. :-)

rams• 17 years ago

At one time the Twisted guys were accused of walking away instead of trying to improve Python's async module. History repeats.

nostrademons• 17 years ago

I find it interesting that nearly all the comments here are on Tornado vs. Twisted rather than the larger issue of communicating vs. coding that the author brings up.

There've been a lot of times - most recently yesterday - when I've hacked up a solution, mailed it off, and got back a response of "I just did this a couple days ago. With a one-line change, my code could've covered your needs too." And then of course I'm like, "Oh. Shit. Let's modify your code instead and I'll throw away my changelist."

I get the feeling this happens a lot outside of my own personal/professional sphere as well. Google's famous for its NIH syndrome. Python's got about 5 different competing web frameworks. Lots of people still write their own machine code generators instead of using LLVM. I'm kinda curious how much coding people are willing to do to save one e-mail.

Ironically, the folks who insist on building their own stuff are often right. Google's certainly done well for itself, and it's great that we can use Django instead of Zope now. So maybe the problem is that communication really is so expensive that people are better off rewriting large chunks of code to avoid it. But if that's true, we should be looking for ways to reduce the cost of communication, rather than glorifying the folks who go to heroic measures to avoid it.

It's curious - yesterday, Coders at Work was at the top of the hotlist, and most of the comments said, "I'd rather learn about the business & management aspects than the technical aspects." Yet here's an interesting business & management problem, but everyone's commenting on the technical aspects.

lacker2• 17 years ago

I think the author is misinterpreting what's going on. It's not that it's too hard to communicate with the Twisted folks, it's that the FriendFeed folks didn't think Twisted was good enough. Twisted is too buggy and too complicated.

zaphar• 17 years ago

The reason I think that folks who build their own stuff are right is that they understand their problem domain better than anyone else.

The author says communication could have cut their time down considerably. That may or may not be true. Getting into a debate with an external project about what should or shouldn't be the correct way to fix/change the external projects code can be a huge drain on time and mental energy. You can't fault a group for deciding not to bother.

Not to mention that if you become the person responsible for a piece of an external project you are suddenly not just supporting your use of that piece you are supporting everyone else's use of that piece. That leads to more debating/discussion about the right way to do it. If you have to find a way to scale fast it might actually in the long run be less time to just write it yourself for your specific needs not an entire communities needs who have different scaling issues than you and different goals. Many times One size does not fit all and convincing someone else of that is just too time consuming to try.

The author did bring up an interesting point about friendfeed developers not even trying to communicate though so I can understand what prompted him to write the article.

johnbender• 17 years ago

One reason I wouldn't have reached out, as a private company, is tipping my hand to competitors. Maybe an NDA would solve that?

n8agrin2• 17 years ago

An observation: Python coders seem to believe that when someone has solved a problem in Python and open sourced that problem every Python coder must use their solution or else they are somehow doing an injustice, perhaps even undermining the community!

On the contrary, 1) Friendfeed can do whatever they want. There is no part of the Python license that says you must use Twisted or else. 2) I doubt FF's statements really caused damage to Twisted's user base, just as we use Cherrypy at work and didn't drop it on Thursday because of Tornado. 3) All open source contributions are a Good Thing. If Twisted devs are really feeling screwed FF is basically saying "here's how we did it, use it if you want". Sounds pretty win win to me.

Instead of complaining and accusing I would suggest Twisted devs take the publicity opportunity to show why Twisted is still great, benchmarks, what the do better than Tornado and what they can learn from it.

cturner• 17 years ago

    An observation: Python coders seem to believe

That's a poor generalisation, and I assume you've built it on the back of a single observation. There are dozens of python template libraries that solve problems with different levels of complexity, and there's no animosity over it. Indeed, one of the reasons I like python is that there's a strong standard library, and then a healthy diversity of tools so you can mix and match to get what you want.

    I doubt FF's statements really caused damage to Twisted's user base

They implied that it had bottlenecks.

Your suggestion in the last paragraph might cause benefit.

General comment on difficulties learning twisted - I found it difficult to grasp until I'd practically reimplemented the patterns by accident in the course of writing my own client/server application. I got stuck on a concept at one point in my non-twisted socket application, and the guys on #twisted helped me past it. For those who are interested, the O'Reilly book is obsolete but a better starting point than the online docs.

n8agrin2• 17 years ago

That's a poor generalisation, and I assume you've built it on the back of a single observation.

On the contrary, my observation is based on comments I've read in many Python community forums, especially when applied to Python vs (your other high-level language of choice). For example, I've read many comments to the effect of "I like Python over (Ruby, Perl, etc) because it promotes one right way of doing things and having one package for handling problem X." I am not stating that this belief is a Python language requirement, nor do I believe that this observation adheres to all Python coders so yes, it is a generalization. But my generalization is certainly not based on a single observation. Finally, I will gladly play the devil's advocate and agree that there is little way to prove this generalization, short of spending several days collecting emails on Python mailing lists, which is why it is offered as an observation only. It is an observation of the practices of a community not something codified into a PEP, at least as far as I know based on my limited research.

I completely agree with the rest of your comment and value your critique. Python is a great language that provides a diverse set of tools, which is why I also choose it to handle many tasks.

lacker2• 17 years ago

The Twisted people don't seem to get that their framework is too complicated for many uses. You can't just work on someone else's framework and make it simpler, if they don't agree that it should be simple in the first place.

dlsspy9• 17 years ago

"You can't just work on someone else's framework and make it simpler"

Are you suggesting that python is more complex to use than x86_64 assembler?

Similarly, when I removed code from tornado to make it work with twisted, I neither tornado more complicated to use, nor did I introduce any limitations.

The "too complicated for many uses" thing kind of bugs me, though. It's the same argument people use to use for postgres vs. mysql. Something along the lines of "it has features I don't need" (where need probably means "understand" or "need today").

That's where tornado left me. What you call complication, I call features. I use some of these features.

For example, I'd be interested in knowing how something like my tdf web server could be written in tornado. It attaches processes to resource paths, captures and transforms the output from these processes (in a stream) and streams replicates of that data to whatever connections come in.

The answer, of course, involves reimplementing reactor.spawnProcess on top of ioloop. Then you find some of the subtleties contained therein. Then you begin to understand why it seemed complicated to begin with.

evgen• 17 years ago

> Are you suggesting that python is more complex to use than x86_64 assembler?

No, but twisted is much more complicated than regular python. The inversion of control and the fact that until quite recently it was almost impossible to write code for using the framework that was easy to read/understand by experienced python ciders who were not familiar with the idims and conventions used by the project. Twisted generates more WTFs per minute than any other bit of python kit. Some of this is due to a tradition of poor documentation within the project and some due to the nature of the code. Twisted is a complicated system, and sometimes this complexity is not worth embracing.

crad4• 17 years ago

I like the idea of the performance and simplicity of Tornado, as I primarily use web.py. After reading this blog entry I decided to give Twisted another look. It took me about 2 minutes to get to the info about the web module and the examples were not very clear. I've walked away wondering a few things:

If I use twisted, should I use web or web2? How do I map URI's to functions?

When I go to the tornado homepage, I scroll and in < 1 minute I see how I would code for it and can be off and running. To develop web apps, I shouldn't have to spend time figuring out an async network communications framework, I should be worried about programming my app.

My perception from the Twisted site is that I would need to spend time figuring out the twisted way of doing things, then figure out the web add-on way of doing things, then worry about my app. With Tornado, I'm very close to a drop in replacement for web.py and I can be off to the races.

dlsspy9• 17 years ago

Your perception of twisted may be as it is because you can do SO MUCH with twisted. If it's a web.py replacement you want, it doesn't add complexity.

Take a look at the diff to go from a friendfeed ioloop to a twisted reactor loop (with my adaptor) in tornado:

http://pastebin.com/f707e5ae9

You replace two imports with two other imports, and three lines of startup code with three other lines of startup code. Your app doesn't change.

scott_s4• 17 years ago

Which is the same as saying it's a failure of documentation and explanation.

crad4• 17 years ago

I was eyeing up the code earlier for this. This is great and is a good counter for my argument about Tornado's near drop in replacement. However, that's not my main point.

Ultimately, I think Twisted, due to the fact it does so much, is intimidating. The documentation lacks succinct examples of how to implement the very basic things people want to implement. The documentation to figure out how to make an app is buried under at least 3 clicks and even then isn't clear. The comments, when there are comments, in the examples don't truly explain the why of the actions just that they are there.

If twisted wants to be considered as a serious contender as foundation to develop web apps on, they need to take a look at how important very good documentation is to Django's success or how the simple, uncommented example at http://www.tornadoweb.org/ is more clear than the first bit of web example code at twisted: http://twistedmatrix.com/projects/web/documentation/examples....

I'm not knocking twisted, I'm trying to explain why when I went around evaluating what I would use in my projects, I did not choose Twisted. If you find my decisions and observations valid or not is up to you.

nuggien• 17 years ago

Is the code for tornado.twister available anywhere?

waleedka• 17 years ago

You might be missing the point. Had FriendFeed worked on improving Twisted and it's documentation to make it easier to use, they would have achieved what they needed in less time and would have avoided fragmenting the resources of the open source community among two projects.

crad4• 17 years ago

I do get that point, but to that end, perhaps they made their decision, in part, based upon the same type of thought process.

This is progress, and as a result, the Twisted people, if they're listening, have gotten valuable feedback out of the process from many people other than myself as a result.

There has also been much more about Twisted on places like Hacker News than I've seen, relatively speaking. With Dustin porting the meat of Tornado on top of Twisted's IO handler, there is much more interesting things to come out of it all.

It is in no way a negative thing for Twisted for there to be competition in an area of cross-over such as this.

mbrubeck2• 17 years ago

I think you underestimate the effort it would take to write concise, accurate documentation for a project the size of twisted. I bet it took far less time to write the 5000 lines of code in Tornado than it would to rewrite the documentation for Twisted and its web modules, and get that accepted and integrated upstream. And it wouldn't fix the fact that Twisted has all sorts of functionality that Tornado developers don't need, aren't interested in documenting, and that web developers would have to either avoid or wade through.

calambrac2• 17 years ago

Twisted has always been an "emperor has no clothes" project to me, a project that's no doubt very useful but at the same time seems to revel in its unnecessary complexity (just read its documentation - how many times do you really have to tell us that something might seem overly complicated but is worth it for its flexibility?) I'm honestly glad to see a no-bullshit alternative that doesn't have that same sense of worshipping complexity as a sign of quality.

enneff• 17 years ago

That viewpoint seems quite narrow to me.

The documentation of Twisted attempts to introduce several event-driven programming patterns. Most people have a 'wtf' moment or two when they come across design patterns that approach things seemingly backwards.

The further I got into my Twisted app, the more I found myself reaching for things and finding that were already there for me. It was a great experience. You should try it!

calambrac2• 17 years ago

I've done plenty of work with Twisted. My complaint was not that it lacked functionality, my complaint was that the culture of the project favors 'flexibility' over everything else. It's like the C++ of networking libraries. I'm sure there are plenty of times and places where its approach is exactly what you need, but it's just rarely a good fit for what I need.

Tichy• 17 years ago

It sounds very good, but I must admit I immediately remembered lots and lots of comments from open source developers that amounted to "not happy with our project (a ka 'free toilet')? well duh, hack it yourself, we owe you s*". Obviously not the fault of the Twisted developer, but I am not surprised if people actually prefer to hack their own stuff.

paul3• 17 years ago

The great thing about open source is that people are free to share ideas and code. If the Twisted team likes parts of Tornado, they are free to replicate those parts back into Twisted while ignoring the parts that they don't like.

Meanwhile, users are free to choose whichever approach suits them better. Twisted is no doubt more complete in terms of protocols and such, but many people also value simplicity. One important point that I think has been overlooked here is that the core of Tornado is only a few hundred lines of code. For me, the fact that I can actually read _all_ of the code in just a few minutes is very valuable. I'm sure many others don't care though, which is why it's great that we live in a world where there doesn't always have to be a single "right" answer.

dlsspy9• 17 years ago

"Twisted is no doubt more complete in terms of protocols and such, but many people also value simplicity. One important point that I think has been overlooked here is that the core of Tornado is only a few hundred lines of code."

I think this is a bit misleading. The twisted port involved removing almost 1,400 lines of code and didn't change the tornado API. That is, tornado is not more simple without twisted.

I do understand the value of being able to read all of the code quickly, but there are other turtles below these. For me, it's very valuable to know that the part upon which I'm building is a well-tested codebase whose semantics I understand, and for which I can write tests when things begin to violate my understanding.

If tornado is used in more applications, it will have to grow in complexity to meet the needs of those applications. I see tornado as having two faces. The web portions can certainly continue to grow, but the network IO handing just means that people will have to port over things that already exist within twisted, and expand it in complexity (such as the kqueue work that is in progress) to handle more low-level stuff.

At some point, you can't just easily sit down and read all of it (at least, not any more easily than you can with twisted).

paul3• 17 years ago

Tornado relies on a small number of well defined system calls (such as epoll). For me, the difference between reading (and debugging) a few hundred lines of code down to the system call level, and reading (and debugging) a big library like Twisted is rather significant. For you, the difference may not be important, and that's fine. I also don't think that it's necessary for Tornado to replicate all the functionality of Twisted -- if people want Twisted, they should simply use Twisted and ignore Tornado (and hopefully the Twisted team will have borrowed the best bits of Tornado anyway).

randallsquared2• 17 years ago

That is, tornado is not more simple without twisted.

That's not exactly what I'd say. :) Tornado without twisted is simpler than tornado with twisted, because you can't ignore the huge system you'd otherwise be able to avoid.

Edit to remove mild flame.

mbrubeck2• 17 years ago

"The twisted port involved removing almost 1,400 lines of code and didn't change the tornado API."

Or, looking at it another way, you removed Tornado's simple 1400-line core, and replaced it with a 150,000-line dependency.

(Yes, I know that Tornado On Twisted doesn't depend on the entire Twisted framework. But the port does involve a lot more code in total than the original.)

frognibble2• 17 years ago

There are several async programming libraries for Python including Allegra ( http://laurentszyster.be/blog/allegra/doc/ ), libevent-python (and other libevent bindings to python), the standard asyncore module and more. Why should Facebook use Twisted instead of one of these other libraries?

imbaczek• 17 years ago

more importantly, why should it roll it's own given a plethora of more mature alternatives?

frognibble2• 17 years ago

Perhaps there are many alternatives because they all have problems.

Twisted Python's lead hacker discusses Tornado

Related Stories