Python Libraries you should know about

http://doda.co/7-python-libraries-you-should-know-about

by trueduke • 14 years ago

483 69 14 years ago

mochizuki• 14 years ago

I have a few scripts that could benefit from PyQuery, didn't know about that one. Also path.py looks like it will save me some time to. Thanks!

klibertp• 14 years ago

path.py is a must have, I am amazed how often it's being rediscovered, despite the fact that I try to advertise it whenever it's relevant. But it's quite old at this point, maybe we should be more excited about efforts like this one: http://www.python.org/dev/peps/pep-0428/

denzil_correa• 14 years ago

Thanks for sharing. These are a nice set of libraries I never used. I am bookmarking this page.

marcofucci2• 14 years ago

You should probably add requests http://docs.python-requests.org. Anyway great list! I'm already using most of them and they are awesome.

3amOpsGuy• 14 years ago

Yeah. Also pretty much anything by Kenneth Reitz (although i can't warm to the args lib for some reason) is worth a good look.

Could probably do a top 7 just by Reitz:

  From: https://github.com/kennethreitz

  1. Requests
  2. CLINT - Easy CLI tools inc cross platform colour
  3. Envoy - "subprocess for humans"
  4. Tablib - csv, excel and plenty others, tabular data
  5. python-guide - A work in progress book
  6. dynamo - Amazon Dynamo as a python dict
  7. gistapi.py

kami884510• 14 years ago

Hey. Author of the blog post here.

I specifically excluded awesome libs like requests, SQLAlchemy, Flask, fabric etc. because I thought them too "main-stream". If you know what you're trying to do, it's almost guaranteed that you'll stumble over the aforementioned. I tried to compile a little bit of a list of libraries that SHOULD be better known, but aren't.

bmelton• 14 years ago

That's what I thought you'd done, and it was a great idea. Except for dateutil and sh, I hadn't heard of any of these. In response to the 'Python for Humans' post, this was a perfect compendium.

Thanks a million.

marcofucci2• 14 years ago

I see your point but in my opinion, requests is quite different from SQLAlchemy, Flask and fabric. I wouldn't put them at the same level.

takluyver• 14 years ago

The very first suggestion complains that BeautifulSoup is too slow, but as of version 4, it's actually just a navigation layer on top of your preferred parser. So it's as fast as lxml, and as easy to use as, well, BeautifulSoup.

kami884510• 14 years ago

Hey before BS 4 got released I looked forward to the very same thing you're talking about, using BS on top of lxml. But it seems the performance hasn't really improved:

http://www.crummy.com/2012/1/22/0

vivek-infosec• 14 years ago

A DOM parser will always be slower than a stream parser ...

pmorici• 14 years ago

I used BeautifulSoup for a project once because of all the accolades it gets here but found it to be less than robust. It might be sufficient in a scenario where you have a single site you need to scrap but I found it was totally unreliable when used across a wide range of sites and esp. sites with foreign language content.

ludwigvan• 14 years ago

I started using PyQuery yesterday, after using BeautifulSoup for a long time. It seems much easier to use.

  pq_page = pquery(url=PAGE_URL)

Note that PyQuery has some encoding issues too (or rather the sites I were scraping were too bad, showing two different encodings in meta tag!), here are two different things I have done to workaround:

  page = requests.get(PAGE_URL)
  pq_page = pquery(page.text)

If that doesn't cut it (because requests detects it wrong too), try forcing the encoding in requests:

  page = requests.get(PAGE_URL)
  page.encoding = 'utf-8'
  pq_page = pquery(page.text)

Tloewald• 14 years ago

Speaking as someone with approximately 4h of Python experience -- great list. I'll be using the sh lib right away.

CodeMage4• 14 years ago

Unless you're using Windows :(

The docs got my hopes high, but they fail to mention sh isn't supported on Windows -- I saw that by looking at its source on GitHub. I'll take a look at pbs, but this kinda bummed me out.

shill• 14 years ago

Scratch that itch. :)

pbreit• 14 years ago

Do yourself a huge favor and develop on Linux. VirtualBox + Ubuntu 12.04 = Free.

bjourne• 14 years ago

When compiling lists of the best Python libs, one definitely has to check out Pocoo: http://www.pocoo.org/ They are a bunch of dudes who are incredibly skilled at putting together great API:s. All their libraries, from pygments, jinja2 to sphinx are well-documented and extremely simple to use.

Permit• 14 years ago

Is PyQuery fast? The whole premise of the first point was that BeautifulSoup was too slow, but then he didn't provide a comparison between them.

tiramisou• 14 years ago

It uses lxml, so yes it is fast.

streblo• 14 years ago

So does BeautifulSoup

kami884510• 14 years ago

Hey Permit. This is a really good point. Looking at PyQuery's source code [0], it really does nothing in terms of parsing other than calling lxml's fromstring function and then works with the result of that when evaluating queries. So it would probably be a tad slower than pure lxml since it also does other bits (like checking for URLs and then fetching them for you) but from looking at the source code, I'd think that the overhead is minimal.

[0] https://github.com/dsc/pyquery/blob/master/pyquery/pyquery.p...

Jabbles• 14 years ago

Do you have any more up-to-date data than a 2008 comparison?

_yfoe• 14 years ago

The comparison should be moot, because Beautiful Soup 4 uses the lxml parser when it's available: http://www.crummy.com/software/BeautifulSoup/bs4/doc/#specif...

leftnode• 14 years ago

One I'd like to add is Docopt: http://docopt.org/ and https://github.com/docopt/docopt

It makes it very simple and intuitive to build command line apps.

IgorPartola3• 14 years ago

And if you want to have both command line args and config files, check out my project: https://github.com/ipartola/groper. It even supports creating a sample config file out of the options you have defined.

iang• 14 years ago

Dang, wished known about that a week ago when I was kludging it for a couple of utilities (thinking "someone must have a library for this but I can't find it!"). Thank you.

CJefferson• 14 years ago

This might just be your examples, but it seems a bit verbose.

For example for:

define_opt('server', 'daemon', type=bool, cmd_name='daemon', cmd_short_name='d')

Can I just write define_opt('daemon') ?

llambda• 14 years ago

I'm only surprised to not see any of Kenneth Reitz' work on this list, e.g. Requests. In fact in that vein is rauth, an OAuth client lib built on top of Requests (https://github.com/litl/rauth). Full disclosure: I'm the author of rauth. :)

slurgfest• 14 years ago

Kenneth Reitz has fantastic publicity already. Can I please just hear about the other things going on in the Python world without bringing him into it, sometimes? It is like a cult of personality.

ominous_prime• 14 years ago

author mentioned in another comment that he left off many things like Requests, since they are very well known, or easily discovered.

kami884510• 14 years ago

Hey. Lots of people suggested this so I added a notice to the blogpost to further explain why I chose these libraries:

> I specifically excluded awesome libs like requests, SQLAlchemy, Flask, fabric etc. because I think they're already pretty "main-stream". If you know what you're trying to do, it's almost guaranteed that you'll stumble over the aforementioned. This is a list of libraries that in my opinion should be better known, but aren't.

think-large• 14 years ago

I <3 you OP. So much right now. I didn't even know i needed these until I read this post and now I know and I'm so happy.

I know matplotlib comes with Python(x,y) but that's a pretty awesome one too.

the1• 14 years ago

watchdog is pretty buggy on linux at least.

kami884510• 14 years ago

Hey, can you talk more on that? I found it to work quite well on my machine and servers, it's also an integral part of another open source library that I'm building. [0]

[0] http://github.com/doda/imagy

rabialam• 14 years ago

Great list. In particular, fuzzywuzzy and pattern caught my eye in a "how-have-I-not-heard-of-these" kind of way.

jnazario• 14 years ago

for fuzzy date work i tend to relay on the parsedatetime module:

http://code.google.com/p/parsedatetime/

it seems to accept syntax similar to the 'at' command does (and obviates the need for my python C module to do that parsing based on the scheduler parser for 'at'). examples include "1 day ago", "ten hours from now" and the like. very useful.

mercuryrising2• 14 years ago

Since everyone is asking (and no one is doing), I put together a simple benchmark for pyquery, bs4, and lxml (cssselect/xpath).

https://gist.github.com/4061368

All it does is grab paragraphs from python.org's html a couple thousand times.

    ==== Total trials: 100000 =====
    bs4 total time: 31.6
    pq total time: 9.3
    lxml (cssselect) total time: 5.4
    lxml (xpath) total time: 4.3
    regex total time: 8.9 (doesn't find all p)

What does it mean? Unless you're running thousands of queries for parsing, it doesn't matter which library you choose. My computer old and slow. Pick which one is the easiest, that you'll fight the least with. Don't put energy into unnecessary optimization. Using a good library is like choking someone, they'll fight for a little while until they pass out. (you'll remember this analogy next time you want to switch libraries, do you really need to choke someone to get your job done?) After they pass out, it's smooth sailing and you don't have to worry. Don't rock the boat unless you have to.

kami884510• 14 years ago

Haha, thank you mercuryrising. That's quite the interesting comparison. A few thoughts:

- Python.org is quite a simple web app, it would be interesting to run this against something a little more complex like a long wikipedia article or http://www.nytimes.com/ or even the Alexa Top 500

- It would also be interesting to split the times between parsing and selecting as I feel that's where the difference between pq and lxml comes in.

All in all it seems BS4 is quite a bit faster than I gave it credit for (especially factoring in parsing+selecting instead of just the former)

mercuryrising2• 14 years ago

Python.org was pretty simple. bs4 really hits the brakes on the NYT page.

    ==== Total trials: 100000 =====
    bs4 parsing time: 0.6
    bs4 selecting time: 146.3
    pq parsing time: 0.0
    pq selecting: 15.7
    lxml parsing time: 0.0
    lxml (cssselect) selecting time: 12.4
    lxml parsing time: 0.0
    lxml (xpath) selecting: 11.5

robocaptain• 14 years ago

I'm only upvoting for the choking analogy.

That one is going into my permanent memory file.

nathan_f77• 14 years ago

This is quite a cool idea:

>>> path('a') / 'b' / 'c' path('a/b/c')

It would be fun to have that in Ruby!

mattdeboard• 14 years ago

Alright, let me say that I do not like this part of path. Operator overloading is fine, I mean the + is overloaded all to hell. But seeing the division symbol in weird places in a project I took over was distracting first, then irritating, and by the time I figured out what was going on I did not want it despite seeing some benefit.

draegtun• 14 years ago

I agree. It reminds me of FancyRoutes which I also found off putting - http://news.ycombinator.com/item?id=1100248

Path.py may already do it (docs are limited so I'm not sure) but something like below would be better:

  my $foo = dir('a', 'b', 'c');

This is how Path::Class works in Perl (https://metacpan.org/module/Path::Class). This way the path is a filesystem agnostic directory object.

NB. Twisting operator overloading isn't all bad though. For eg. IO:All nearly gets it all right and being a little mad I will sometimes use it :) https://metacpan.org/module/IO::All

ekimekim• 14 years ago

I actually wrote up a module a while ago, what it did was in ways similar to path:

>>> print SimpleFS('a').b.c

a/b/c

The bigger thing (and original purpose) was uses like this:

>>> root = SimpleFS("/")

>>> sda_size = root.sys.block.sda.size

>>> for fd in root.proc.self.fd:

... os.close(int(fd))

>>> root.var.log[sys.argv[0] + ".log"] += "Size of disk: %d" % sda_size

brunoqc• 14 years ago

You have a typo. Search for 'PyQyery', twice.

kami884510• 14 years ago

Thank you, fixed.

roryokane• 14 years ago

Here are some Ruby libraries I’ve used that are similar to those Python libraries:

pyquery equivalent: Nokogiri (http://nokogiri.org/). Lets you select elements with jQuery-like selectors. Uses libxml2 as its parser.

watchdog equivalent: watchr (https://github.com/mynyml/watchr). Run code when the filesystem changes.

path.py equivalent: rush (http://rush.heroku.com/). Provides a far better API to the filesystem than the standard library.

I also found this equivalent to fuzzywuzzy, but I’ve never used it: amatch (http://flori.github.com/amatch/)

JeffJenkins• 14 years ago

The coolest part of dateutil isn't the parser, it's the recurrence rules and recurrence rule sets. Doing that on your own is extremely error-prone if you have a non-trivial recurrence.

kami884510• 14 years ago

Cool, thank you for making me aware of that. I haven't had to use something like that but it's good to know. I'll put it in the blogpost.

EDIT: I added it below the `parse` example. Thanks again!

reinhardt• 14 years ago

If you do any non-trivial work with decorators, the `decorator` module is a must: https://micheles.googlecode.com/hg/decorator/documentation.h.... Think of it as @functools.wraps on steroids (though this probably doesn't do it justice). FWIW I think it should be in the standard library.

chewxy2• 14 years ago

The latest BeautifulSoup uses lxml (version 4+). How does it compare to PyQuery?

dm8• 14 years ago

Excellent collection! I will add one more.

Python Imaging Library - Today's web is full of images and PIL makes it easy for image manipulation. Although, it's not extremely performance efficient at very large scale.

joeshaw• 14 years ago

In a similar vein, I recently discovered Wand (http://dahlia.kr/wand/) which is a gorgeous API built on top of ImageMagick. It's a lot more limited in functionality than what PIL offers, but for common operations like scaling, cropping, or extracting EXIF data it's much nicer to work with.

chewxy2• 14 years ago

actually, Pillow is a much better fork of PIL. It's much more friendly

http://pypi.python.org/pypi/Pillow/

polm23• 14 years ago

No mention of pandas or nltk?

I'd never heard of pattern before, and while it looks like it's a nice bundle of features, I'm concerned by the fact it references pyWordNet by name even though it hasn't been an independent project since 2006 (http://osteele.com/projects/pywordnet/). Has anyone actually used it?

andrewcooke• 14 years ago

i have reservations about dateutil. it's certainly true that the built-in time and date routines in python are ugly, but dateutil's parsing doesn't try to give a single, consistent interpretation of an input date/time - instead it parses it in chunks, where one chunk can effectively overwrite another. so you can have input that is illogical or inconsistent and dateutil will happily give a single "right" answer instead of flagging an error.

for example, see this bug on so - http://stackoverflow.com/questions/10575919/strange-date-par...

in short: it's ok for non-critical cases where you just need "some date" from input. but don't use it if you would rather have an error than an incorrect interpretation.

Python Libraries you should know about

Related Stories