Requests: HTTP for Humans

http://docs.python-requests.org/en/latest/index.html

by DanielRibeiro • 15 years ago

404 66 15 years ago

sayrer• 15 years ago

This library seems like a good idea. I hope it is or will be based on Joe Gregorio's httplib2, so it gets close to correct.

Also, I can't believe it includes PATCH support. Surprising, but good, to see that out in the wild.

StavrosK2• 15 years ago

Why not just include arbitrary verbs? The rest of the protocol doesn't change at all...

izak302• 15 years ago

There is support for arbitrary verbs, just more direct API support for the common ones.

here's the api for a generic request, get is right under it: https://github.com/kennethreitz/requests/blob/develop/reques...

sdfjkl• 15 years ago

This was much needed.

rubergly• 15 years ago

I'm almost frustrated to see this. I just learned to use urllib2, and found that it took a great deal of effort to get any reasonable behavior. I wish I had known about this before. It really looks great and I will try to use it in any future work.

kenneth_reitz5• 15 years ago

You could be a contributor then ;)

abecedarius• 15 years ago

From the title I thought this would be some kind of mechanical-Turk webserver.

dickeytk• 15 years ago

Wow I was just about to start going through the pain of using urllib2 for the millionth time right before I read this.

waterside81• 15 years ago

Awesome stuff. How many hours I've wasted futzing around with urllib2 and creating my own Request handlers and all that other nonsense. This lends itself nicely to working with 3rd party APIs (Facebook, Twitter) in particular.

espeed• 15 years ago

I used httplib2 (http://code.google.com/p/httplib2/) for the REST module in Bulbflow -- httplib2 is by Joe Gregorio of Google, and it's not bad.

RyanMcGreal• 15 years ago

httplib2 has a very nice API, but be warned that it's not thread safe.

notmyname• 15 years ago

how does this compare with webob (http://docs.webob.org/en/latest/index.html)?

ericflo• 15 years ago

It's completely different.

StavrosK2• 15 years ago

It doesn't. Webob is a server framework, requests is a client.

ianb• 15 years ago

People will rightly tell you that it does not. But I must admit that this did get me to thinking about a few small changes to WebOb along with the integration of WSGIProxy, at which point there would be some real similarities.

dshah• 15 years ago

Have been using this for a little while now. Totally awesome.

The simple example on the home page should have one more line of code that shows that you use r.content to get to the content that came back from the request. That would seem to be the most common use case.

beaumartinez• 15 years ago

So that's why I couldn't get it to work—have a look at this[1].

Thanks for teaching me what the documentation failed to—it's neither in the front page example, or in the API documentation for Response.

[1] https://github.com/kennethreitz/requests/issues/105

kenneth_reitz5• 15 years ago

Fixing now :)

danielpal• 15 years ago

Fascinating, I am really glad that "we" the new wave of programmers are not only conserned about code quality but are also deeply concerned with "usability". Everyday, as I go back and forth using old libs vs new libs, its obvious that the quality of old libs is pretty damn good but they have crappy "usability". New libs have a bit more bugs, being that they are also "newer", but they are a pleasure to use.

maxogden• 15 years ago

equivalent in node: http://github.com/mikeal/request equivalent in jquery: https://github.com/iriscouch/request_jquery

losvedir• 15 years ago

How do those handle SSL and user authentication? I skimmed your links but didn't see anything.

Python's urllib2 isn't that bad for the simple stuff shown in your links, but it's a mess when you have to try to log in somewhere. Seeing in the example that logging in is as straightforward as

    requests.get('https://api.github.com', auth=('user', 'pass'))

just blew me away.

EiZei• 15 years ago

Yeah the authentication parts of urllib2 remind me of the horrible 20-letter classes glued together in Java, just the thing I wanted to escape from by using Python. Seeing that I'm trying to make this script interface with a website that uses both cookies and http auth this library seems like a godsend.

izak302• 15 years ago

For those interested in getting to know requests vs urllib2, I gave a talk @ PyOhio, covered some simple examples in each (and why requests is super great) http://python.mirocommunity.org/video/4388/pyohio-2011-consu...

stock_toaster• 15 years ago

This looks neat. I have been using restkit[1] for a little while now, but I am certainly going to check this out.

[1] http://benoitc.github.com/restkit/

jokull2• 15 years ago

+1 for restkit. It has a Pythonic interface, shortcuts and completely replaces urllib, urllib2 and httplib all of which I've had some beef with before. Love restkit!

smiley325• 15 years ago

I've noticed that the layout of the site is uncannily similar to flask's documentation (http://flask.pocoo.org/docs/), down to the "Fork me on Github" in the top right corner. Is there some sort of software producing this kind of documentation?

paxswill• 15 years ago

I don't have much more info than this, but it looks like they're both generated with Sphinx (http://sphinx.pocoo.org/), going by the footer from flask, and the common source (both have `<div class="sphinxsidebar">`).

jmreardon• 15 years ago

It's pulling in a copy of the css file the flask site uses.

akavlie• 15 years ago

The "Fork me on Github" is pretty common on a lot of sites (Github made those graphics), but that aside I agree -- I was thinking it was another Pocoo project until I saw the author.

Looks like he just co-opted the theme (Pocoo might have it available somewhere, but I couldn't find it). Nothing wrong with that, it's a great theme.

kenneth_reitz5• 15 years ago

My modification of the Flask theme is available here:

    https://github.com/kennethreitz/kr-sphinx-themes

The original is here:

    https://github.com/mitsuhiko/flask-sphinx-themes

forsaken• 15 years ago

Yes. They're both created using Sphinx: http://sphinx.pocoo.org/

Requests is hosted on a site called Read the Docs, which does documentation hosting for Sphinx projects: http://readthedocs.org/

Sphinx has become the defacto documentation tool for Python projects, and a lot of other projets as well. Notably a number of PHP projects, and Varnish the web cache use it also.

ludwigvan2• 15 years ago

It should also be noted that Sphinx was developed by Pocoo, the same group that develops Flask, Jinja, Werkzeug.

http://pocoo.org

watmough• 15 years ago

I don't know much Python, but I'd likely be correct in assuming there are quite a few proven http client libraries in existence.

This looks like another reason to believe that if you build a better solution, even when the space is well-covered, that you can still win adoption, so maybe, a lot of people can stop worrying about being second.

perfunctory2• 15 years ago

> so maybe, a lot of people can stop worrying about being second.

Indeed.

SoftwareMaven• 15 years ago

Being second can often be beneficial. You get to learn from the mistakes of the first without being locked into those mistakes.

marcamillion• 15 years ago

Not to start a language pissing war...but that syntax looks awfully like Ruby.

That means it's pretty elegant and nice :)

bialecki2• 15 years ago

Quick question on the code. I'm looking at this commit:

https://github.com/kennethreitz/requests/commit/773630b0109c...

and wondering what "from .status_codes import codes" does?

Specifically the fact that it's not "status_codes" and has a dot prepended. Is that something special in Python (guessing no) or some convention that is used? Just curious.

bialecki2• 15 years ago

Just needed to look at the docs: http://docs.python.org/reference/simple_stmts.html#the-impor....

The relevant bit:

"When specifying what module to import you do not have to specify the absolute name of the module. When a module or package is contained within another package it is possible to make a relative import within the same top package without having to mention the package name. By using leading dots in the specified module or package after from you can specify how high to traverse up the current package hierarchy without specifying exact names. One leading dot means the current package where the module making the import exists. Two dots means up one package level. Three dots is up two levels, etc. So if you execute from . import mod from a module in the pkg package then you will end up importing pkg.mod. If you execute from ..subpkg2 import mod from within pkg.subpkg1 you will import pkg.subpkg2.mod. The specification for relative imports is contained within PEP 328."

Cool stuff.

dheerosaur• 15 years ago

Really cool stuff indeed. Thank you.

pangram• 15 years ago

It's a relative import -- where "relative" here is like relative vs. absolute paths. See:

http://docs.python.org/tutorial/modules.html

anoopks• 15 years ago

Does anyone know of such an API in Java - HTTP for Humans in Java?

ollysb2• 15 years ago

Restlet is a good attempt at writing an api that matches the http spec well. You can find docs for a client at http://wiki.restlet.org/docs_2.1/13-restlet/21-restlet/318-r.... It's still more verbose than it might be, would be a good starting point for writing a decent api though.

nl• 15 years ago

Restlet is a server framework.

I think the parent poster is looking for a client API that is nicer to use than java.net.Url and/or the various versions of HTTPClient

deno• 15 years ago

Jetty's HTTP client is pretty decent.

http://wiki.eclipse.org/Jetty/Tutorial/HttpClient

perfunctory2• 15 years ago

This seems to be a fantastic project. I wish more libraries were designed this way. Library's usability is just as important as performance or feature completeness.

alexatkeplar• 15 years ago

Using this with DictShield (https://github.com/j2labs/dictshield) or Colander (https://docs.pylonsproject.org/projects/colander/dev/) would be very cool methinks

pestaa• 15 years ago

Quickly judging by the look, I thought "hey, Armin Ronacher just released another library!", but this one seems as well thought out as if it was designed by Armin himself.

ludwigvan2• 15 years ago

Another useful Python library by Kenneth Reitz is tablib, "Python Module for Tabular Datasets in XLS, CSV, JSON, YAML, &c. ". See https://github.com/kennethreitz/tablib

ez77• 15 years ago

That looks extremely useful. Thanks! Project webpage: http://tablib.org

tsycho• 15 years ago

Does any one know of a good Ruby equivalent of such a library?

kennystone• 15 years ago

https://github.com/archiloque/rest-client

st3fan• 15 years ago

This is a great library. I've used it in a couple of projects now and it is great.

This is the kind of stuff that I would love to see in the Python standard library.

joeshaw2• 15 years ago

A coworker introduced me to Requests a couple of weeks ago. It makes life a whole lot easier.

It does highlight my biggest gripe about Python, though. I love the language, but the class library is hit-and-miss, and anything to do with HTTP is a pretty big miss. httplib, urllib, urllib2, cookielib -- yuck. I like that Requests has a nice API, but it's still built on top of pretty bad class libs underneath. (And the Requests API does expose some warts, like having to use cookielib's CookieJars.)

I'd really love to see someone's take on a clean break from the standard modules with a simple, maleable API.

jokull2• 15 years ago

You are describing restkit. Check it out. Goes past the broken standard lib stuff and straight to socket. Very pythonic.

joeshaw2• 15 years ago

looks interesting! thanks for the pointer.

(Here's the URL for the project: http://benoitc.github.com/restkit/)

kevingessner• 15 years ago

If you're looking for a "clean break", Google's httplib2 [1] might be what you want. It's somewhat lower-level than Requests, but adds a bunch of advanced stuff like caching support and keep-alives. As you might guess from the name, httplib2 is more like httplib, while Requests appears to be more like urllib2.

1: http://code.google.com/p/httplib2/

kenneth_reitz5• 15 years ago

It should be noted that httpbib2 is not a Google project. A lot of people prefer it though.

acdha• 15 years ago

httplib2 is part of why you should use requests: it's far more respectable as a client but not as well documented and it still takes way too much code for basic operations. I appreciate what httplib2 is trying to do, that there's a ton of hard low-level annoyances in building a modern HTTP client, but really, just use requests instead. Kenneth Reitz is very motivated and he gets the degree to which simple things should be simple whereas httplib2 feels more like an academic exercise than something people should use to build production systems[1].

Disclosure: I'm listed in the requests AUTHORS file but can claim credit for, oh, about 0.0001% of the awesomeness.

1. http://code.google.com/p/httplib2/issues/detail?id=96 is a good example: an annoying bug which affect many people, there was a fix available for months, which worked great when I applied it in a fork and pounded a couple TB of data through it, but it took over a year to make it into trunk and even longer to make it onto PyPI where any other project which required "httplib2" would get the working version.

serichsen• 15 years ago

By the way, Common Lisp has Drakma for this: http://weitz.de/drakma/

simonw• 15 years ago

I wish someone would write an equivalent of this for Python's myriad of horrible process management functions. popen, popen2, popen3 etc were a nightmare, and the newer subprocess module still doesn't have an API that I'd call memorable or obvious.

kenneth_reitz5• 15 years ago

I am planning on this actually! HTTP lends itself well to running a process.

It will likely be very call+result oriented at first (no interaction w/ a running process)

llimllib• 15 years ago

For such a new module, the API for subprocess is remarkably bad. I have to look it up absolutely every time.

pault• 15 years ago

Check out psutil (http://code.google.com/p/psutil/). I discovered it the other day when I needed to get a list of open ports used by an ssh client running in a subprocess, and it was as easy as this:

    sshc = psutil.Process(pid)
    sshc.get_connections()

It allows you to upconvert process ids, and it has its own wrapper of Popen that returns Process objects. It also seems to have well thought out windows compatibility, if that matters to you.

:D <- My face when I found it.

eddieplan9• 15 years ago

The example written in urllib2 instantly reminds me of Java and the classy query joke [http://ejohn.org/blog/classy-query/]. Just a thought: are most Java-related APIs over-engineered, and does Java have a bad influence on many API design work?

Requests: HTTP for Humans

Related Stories