Python and the Real-time Web

http://mrjoes.github.io/2013/06/21/python-realtime.html

by joecarpenter • 13 years ago

152 17 13 years ago

mrjoes.github.io

csense• 13 years ago

I like the blog post format better than a Google Hangout. When there's a conversation, unless (or perhaps even if) it's heavily moderated, things get lost in the noise.

I was really confused about the word "polyfill," which the author used without explanation. Since I've written code to render filled polygons before (concavity is a fun corner case), I assumed this was what it was referring to, and I was really confused about how that's even remotely relevant. Wikipedia finally straightened me out [1].

[1] http://en.wikipedia.org/wiki/Polyfill

joecarpenter3• 13 years ago

Thanks for feedback, I added link to the wikipedia article.

pronoiac• 13 years ago

When did "polyfill" replace "shim," I wonder?

drakeandrews• 13 years ago

I've always taken it to be that it's a polyfill if it's somewhat elegant, or where a newer technology encompasess a superset of an older one and a shiv if it's somewhat hackish or where older technologies are made to stand in for newer ones. I am probably wrong.

arscan• 13 years ago

http://remysharp.com/2010/10/08/what-is-a-polyfill/

"Shim, to me, meant a piece of code that you could add that would fix some functionality, but it would most often have it’s own API. I wanted something you could drop in and it would silently work (remember the old shim.gif? that required you actually inserted the image to fix empty td cells – I wanted something that did that for me automatically).

I knew what I was after wasn’t progressive enhancement because the baseline that I was working to required JavaScript and the latest technology. So that existing term didn’t work for me."

the_cat_kittles2• 13 years ago

I think of "polyfill" as filling many different holes to make it so you can forget about the different cases (browser compatabilities in this case). A polyfill pours concrete over a bumpy surface and makes its smooth. "Shim", to me, is essentially synonymous with monkey patch, i.e. something you shove in there to make something work- shoving a folded napkin under a table leg to make it flat is shimmy to me.

rdtsc• 13 years ago

Since I've written code to render filled polygons before (concavity is a fun corner case), I assumed this was what it was referring to,

I was told it is part of the web startup, full-stack hipster vocabulary, because "shim" is just not cool enough and a new word was needed.

emehrkay• 13 years ago

I haven't read the whole post yet, but is the author saying that a WSGI+Gevent is better at real-time than Tornado alone?

pekk• 13 years ago

I don't think so. For example,

'I prefer out-of-process approach, where separate set of processes/servers are responsible for realtime portion.'

and

'Tornado is the framework I stopped on.'

and

'Greenlets make everything "easy" at cost of possible issues and allow implicit context switching.'

joecarpenter3• 13 years ago

I'm just sharing my personal experience with both. And no, I prefer Flask (WSGI) and Tornado working side by side, in separate processes.

rubiquity• 13 years ago

Wow this is an incredible wealth of information in regards to the Real-time Web. I recommend anyone read this regardless of whether you use Python or not.

jisaacstone• 13 years ago

I'm working on porting a node/websockets game I wrote to python using the twisted websockets branch. (not integrated in core yet but fully functional so far)

Instead of using socketio I rewrote the client side code in pure js. It was not difficult at all, and nearly all browsers support websockets now anyway.

Anyway - if you've written a twisted server before then writing a websocket backend is trivial, and the frontend is not much more difficult. I might do a full writeup when the project is complete.

the_cat_kittles2• 13 years ago

After stumbling around this problem for ~6 months, making half-solutions, this is such a great article. Thank you!!!

lvh• 13 years ago

Nice, thorough article. Kudos for showing all the alternatives. That said, I think you had a strong preference for one, and that shone through. For example:

With greenlets, you say: Why is greenlets are great? Because they allow writing asynchronous code in synchronous fashion. They allow using existing, synchronous libraries asynchronously. Context switching magic is hidden by greenlet implementation.

Greenlets don't magically make synchronous libraries asynchronous! That's gevent! Greenlets are just the coroutines. Plus, you're forgetting that "magically" making sync libraries async doesn't actually work in plenty of cases, and the way that it works generally involves monkeypatching half the IO bits in the standard library!

Also, there's a bit of a false dichotomy. You claim that there's two options: coroutines or callbacks. That might be true, but then you need to keep in mind that callbacks doesn't necessarily mean something-looking-like-the-Tornado-callback-demo-code. That mind seem like a nitpicky implementation detail, but that's the sort of thing that lets Twisted do inlineCallbacks (where you write generators instead of coroutines) or Corotwine (a third party package where you get actual coroutines) or geventreactor (a third party package where the event loop is done by gevent -- since the other way around isn't actually supported).

The inline callbacks equivalent, given the same API, would be something along the lines of:

    data = yield get_more_data()
    return make_response(data)

The twisted equivalent of this wouldn't even look like data = yield get_more_data(); Twisted's API calls you when there's data, so it looks even simpler:

def dataReceived(self, data): return make_response(data)

Also, txsockjs now integrates great with the Twisted Resource API, so it will live neatly side by side with existing web stuff you're serving from twisted, which can include e.g. WSGI apps (since twisted comes with a wsgi server :))

I have prepared a talk that deals with Twisted Mixing, which I hope to submit to PyCon. The work in progress is here: https://chiselapp.com/user/lvh/repository/TwistedMixing/inde...

I'd love to have a cogent argument for the things you didn't like about Twisted, but two out of three appear to be more or less the same thing ("complex", "hard to learn"), and, on top of that, subjective. I'm sorry you had a hard time. It shouldn't have been. If you have any specific issues, I would like to address them. I'm assuming that by "not PEP-8" you mean mostly "uses camelCase", in which case I'll give the usual apologist answer:

- Twisted predates PEP-8's recommendation of snake_case ;-) - It's actually PEP-8 compliant: the PEP says to do what the code around you does, and something about consistency being a hobgoblin ;-)

Also, you mention that you can run Twisted on Tornado, but not Cyclone. Is there a particular reason for that? From all points I can tell, they get you the same result (mixing Twisted and Tornado code), but the reactor that ships with Tornado just gives you fewer event loop options (and generally inferior ones).

Disclaimer: I'm a twisted dev. Can you tell? ;)

joecarpenter3• 13 years ago

Greenlets don't magically make synchronous libraries asynchronous! That's gevent!

That's right - I wrote that they allow writing code in synchronous fashion and I didn't say that it happens automagically though.

That might be true, but then you need to keep in mind that callbacks doesn't necessarily mean something-looking-like-the-Tornado-callback-demo-code.

Yep, I also provided how Futures and generator-based coroutines can be built on top of the callbacks.

Just in case, Tornado supports both out of the box.

The twisted equivalent of this wouldn't even look like data = yield get_more_data(); Twisted's API calls you when there's data, so it looks even simpler

Very similar to some Tornado API too - method of a class that gets called when something happens. For example, sockjs-tornado follows this convention.

- Twisted predates PEP-8's recommendation of snake_case ;-)

Yes, I'm aware of it, but before I started playing with asynchronous libraries, I had some Python experience and got used to underscore naming convention. While it is easy to switch between both, I'd prefer consistent code style. Especially when I mix typical Flask Web application with real-time portion in same application.

No doubt, Twisted is mature and featureful framework. When I was investigating different options, Twisted was first one I tried. However, I also tried Tornado and found it easier to start with. And because Tornado worked for me, I decided to stick to it.

VeejayRampay• 13 years ago

As a Ruby programmer, I face the same sad state of affairs in Ruby frameworks unfortunately.

Nice writeup about all the alternative, well done.

Python and the Real-time Web

Related Stories