WSGI and the Pluggable Pipe Dream

http://lucumr.pocoo.org/2011/7/27/the-pluggable-pipedream/

by DasIch • 15 years ago

121 20 15 years ago

lucumr.pocoo.org

That was quite a rabbit hole, but a good read nonetheless.

I was pretty surprised at all the criticisms that Pump received yesterday (http://news.ycombinator.com/item?id=2810373). The general response was defensive and critical just because it was trying to replace WSGI. However, it would seem that WSGI isn't perfect for all applications and indeed there is a place for Pump. My question is: how can we resume encouraging people to contribute? Quite honestly, if I had posted that thread and received the response it got, I would be utterly discouraged from contributing to the community in the future.

sirn• 15 years ago

But the thing is the problem Pump solves, is already solved by all WSGI wrappers at application level: Werkzeug and WebOb. I don't think it's really worth to have another standard just to solve something that has been solved at the application level (and may led to even more confusions in the future.)

Armin Ronacher (who also is the author of Werkzeug) did designed something similar, called Webapi[1] and as he mentioned in the article, it's not a very good idea.

[1] https://github.com/mitsuhiko/super-secret-wsgi-replacement

endlessvoid942• 15 years ago

That is a good point. However, the thing that's attractive to me about Pump is how exceedingly simple it is. A lot of things (like Django) try to do a LOT. The complexity of frameworks like WebOb seem to be growing, and that's a turnoff.

I did take a look at Werkzeug and it's one that is definitely awesome. But I think there's a false expectation that choice is bad. There's nothing wrong with Pump. It's another choice, and the response from the community was "WE ALREADY HAVE THIS" instead of "good job".

There's nothing wrong with Pump (at first glance). And instead of giving it a chance, the reaction is "no thanks". And that's sad.

adamtj3• 15 years ago

WSGI is _not_ a framework. It's a gateway interface, like CGI. Django is a web application framework. You don't (usually) write apps using a raw gateway interface like WSGI or CGI. You write web applications in a framework like Django or Pump.

The framework then needs to work with the web server -- but how? To be portable and useful, a framework it should support CGI, SCGI, Fast-CGI, ISAPI, HTTP, etc. That lots of work to do for every single framework. Big ones and little ones all need to support all those interfaces, and you know they won't. That's the whole point of WSGI: to be one single abstraction to bind all web servers to all frameworks.

If you want to write a framework, WSGI sets the bar pretty dang low. It supports pretty much everything you need to do HTTP without too many awkward quirks. You don't need to worry about how to implement a fast-cgi->nginx adapter. You just make your framework talk WSGI and plug it into existing nginx->fast-cgi and fast-cgi->wsgi libraries. Done.

WSGI was carefully designed to be as simple as possible, but no simpler. It is simple, but powerful and flexible. It's possible to write powerful composable middleware that works with any webserver and any framework, because middleware is just a WSGI app that wraps a WSGI app. Middleware can transparently add sessions, caching, compression, chunking, etc, that don't need to be tightly coupled to the framework or the webserver. So, just like WSGI can bind all webservers to all frameworks, any middleware can insert itself between any framework/webserver pair (I'm sure there are exceptions, of course). The framework shouldn't have to screw around with chunked encodings or compression. WSGI's design allows one good library to easily do that work regardless of the rest of your stack.

This article spoke about how awkward send_request() is. What it does is make it trivial for middleware to read, add, and modify headers without having to parse HTTP. Suppose you want to write a javascript obfuscator. One way to do that is to write some middleware that detects Content-Type="text/javascript" and obfuscates the response. Because of the way send_request() works, you detect javascript by looking at a variable, not writing an HTTP parser. Also, if you write that middleware for Django, it'll work unchanged when you get fed up and switch frameworks.

The whole write() vs yield vs return thing can is awkward, but it makes it easier to write a wsgi adapter for older cgi-based frameworks that respond via sys.stdout.write().

Only framework, middleware and gateway authors have to care about WSGI. If a framework author is complaining about WSGI, he or she almost definitely don't understand where WSGI came from or what problem it's really solving.

WSGI is minimal, composable, and elegant, even considering that it's backwards compatible with older frameworks. If everything supports WSGI, then everything can just be plugged into everything else without much screwing around. Composability and loose coupling are very powerful, especially at the levels of the stack where WSGI lives.

pyninja2• 15 years ago

I agree with you, but Pump is not a web framework. You are not supposed to write your application with Pump. Picasso is a framework I built on top of Pump (https://github.com/adeel/picasso) that's more suitable for that.

(I can see how you could get a mistaken impression from reading the discussion on HN, though.)

nikcub• 15 years ago

it was still a good contribution because it prompted this discussion, this blog post and it might lead to something new again

mclin• 15 years ago

The last part was the most insightful for me.

Personally I am pretty damn sure that WSGI no longer carries the importance it had a few years ago. I think it no longer makes sense to merge different applications on the WSGI level together, it should be done on a higher level and JavaScript is a nice way to do that.

Just think about Google's gray bar. You can totally throw such a bar on top of different independent parts of the application by emitting a tiny piece of JavaScript that generates that bar and handles your user session.

I started thinking about it and was like "Right, you can make multiple async requests to hit multiple apps!"

afhof• 15 years ago

"But before you blindly throw it away you have to understand why it was created in the first place."

This is an excellent piece of advice that I wish more people would use. If you find something that you disagree with (a law for instance) it is a good idea to first see how it came to be. The decisions people usually have some reasoning behind them that can't just be discarded when being evaluated.

perfunctory• 15 years ago

There are a few things I don't understand in this post

> neat nod to async systems that can use this neat trick to yield empty strings to signal that they are not ready yet.

Can't the system just do nothing to signal it's not ready yet?

DasIch• 15 years ago

An empty string is "nothing".

the_mitsuhiko• 15 years ago

> Can't the system just do nothing to signal it's not ready yet?

In an async system that means you either have to suspect execution (for instance by yielding something) or to block. And blocking is bad.

79522• 15 years ago

I like the conclusion about using http. This goes against the approach advocated in monolithic frameworks such as django. It would be interesting to see a web framework that separates components and connects them with http.

adamtj3• 15 years ago

That's exactly what WSGI does. Well, except for one brilliant improvement: It doesn't pass around raw HTTP, it passes around pre-parsed HTTP data structures. There are three structures: a response code ("200 Ok"), a list of headers, and an iterable that yields the body of the response (or older frameworks write to a filelike object that you can reimplement if necessary). Dealing in pre-parsed (or rather, not-yet-serialized) HTTP makes it much easier to write middleware.

If you really want to plug stuff together with HTTP, there exist HTTP servers that serve wsgi apps, and I'm sure there is a wsgi app somewhere that is an HTTP client. So you plug your framework's wsgi-app (which you didn't write and don't have to worry about) into an HTTP server. Then you point a WsgiHttpProxy at your HTTP server. Plug that WsgiHttpProxy app to your middleware and plug your middleware into another instance of your HTTP server. Rinse and repeat for as many middleware layers as you want. Now everything is HTTP, like you want.

Or, you know, just plug all your middleware together, avoiding parsing, serialization, IPC, context switching and all that crap.

Wilduck• 15 years ago

The author isn't arguing that this is a bad idea per se, just that it doesn't actually work that easily.

>> But what I do not believe in is that magical plug that is called “framework independent pluggable application”. I don't know where this idea came from that it might work, but it does not. The idea that you can reused code on top of WSGI to work with Framework 1 and Framework 2 is not working out.

WSGI and the Pluggable Pipe Dream

Related Stories