Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip

http://clemesha.org/blog/2009/jul/05/modern-python-hacker-tools-virtualenv-fabric-pip/

by iamelgringo • 17 years ago

157 41 17 years ago

grandalf3• 17 years ago

as someone who has been exploring python lately but who has spent the past few years using mostly ruby and javascript, i have really enjoyed observing how the python community approaches problems ...

- code readability is generally very important (unlike ruby, where magic-fu is quite important)

- there is more heterogeneity

- there is less of an obsession with testing and boasting about it.

Most importantly, I haven't read any blog posts promoting automated code quality measurement tools (are any of your methods over 3 lines?) -- those things are incredibly backwards in my opinion, and the gravitational pull of the ruby community toward them is unfortunate, and suggest that too many ruby programmers are squandering their gains in productivity refactoring code into 3 line methods...

amelim2• 17 years ago

I know this may be hearsay, but I've been using pylint and have been generally pleased with the results. I don't take everything it says as the ultimate source of coding standards, but it will pick up on some silly mistakes one might make and generally make my code more maintainable.

Granted, I'm still in University, but I think your disdain for automated code quality tools may be misplaced. Instead blaming the tools, why not blame the community as a whole for engaging in arcane coding practices.

jacobolus• 17 years ago

You mean heresy. Hearsay is information you learned from a friend. :)

grandalf3• 17 years ago

There are definitely legitimate uses of such tools... I myself occasionally use jslint, to minimize the chances of obscure browser compatibility issues.

However some poeple run these "code quality" measures as part of a regular test suite, so if you write a method that has more than 3 lines you get a warning or failure.

In my opinion this leads to less readable code, as methods are prematurely refactored simply for size, and coders tend to do things like putting a hash literal on a single long line rather than formatting it for maximum readability.

This is just my opinion... I don't hold it against someone to use them, I just think they can make code less readable, not more.

roder• 17 years ago

I think testing is an ok thing to be obsessed about... Oddly i just wrote that in my profile just a few minutes ago.

I like to view the Ruby VS Python discussion as pointless. There's a lot to learn from each other. In my opinion those two languages are poised to, if not already, lead web-development into the future.

grandalf3• 17 years ago

Totally. They are both awesome. It's fun to be a part of both "cultures" :)

stcredzero• 17 years ago

Repetition leads to boredom, boredom to horrifying mistakes, horrifying mistakes to God-I-wish-I-was-still-bored

    It is by will alone I set my mind in motion.
    It is by the juice of sapho that thoughts acquire speed,
    The lips acquire stains.
    The stains become a warning.
    It is by will alone I set my mind in motion.
    -- David Lynch (Piter de Vries from Dune movie)

"It is caffeine alone that sets my mind in motion. It is through beans of java that thoughts acquire speed, that hands acquire shakes, that shakes become a warning... I am... IN CONTROL... OF MY ADDICTION!" -- From the Minicon Graffiti Wall, 1989

oyving2• 17 years ago

I am probably missing something about Fabric, but it seems to me it solves a problem already solved by most unix distributions.

Why not utilize the host packaging system for deploying your code and applications?

delano3• 17 years ago

There are classes of problems that can't be solved at the system level, like deploying to more than one system type or across multiple machines at the same time.

moe8• 17 years ago

Sorry, but I wonder what you're talking about. Package managers are designed exactly for this purpose.

It's not even hard to leverage the power of apt (the most advanced of the bunch) for your own deployments.

In essence you setup a local mirror and add that to the sources.list of all your hosts. Then you learn how to roll your own debs and push them to the mirror. The actual deployment happens via apt-get - which can be triggered by cron, a shell-script, puppet, a sweaty admin at 4am, or whatever fits your bill.

Working with the system this way instead of along with it (or even against it) has various advantages. Most importantly you get proper dependency management. Need to roll back to app version 2 from version 3, whereas version 2 depends on an older version of foo? No-brainer, apt takes care of that for you, both ways, also in much more complicated cases. Need a package or package version that's not in the official repos? No problem, roll your own and make your application package depend on that.

With a bit of elbow grease you can also have it mangle your database and other auxiliary infrastructure appropiately, within the respective pre-/post-install scripts.

Fabric and capistrano are just expressions of the old "if you don't understand you're doomed to reinvent, poorly" meme.

tvon• 17 years ago

If you're distributing software to end users a packaging system is probably the way to go, but if you're looking to deploy code from development environments to staging and production servers then something like Fabric or Capistrano is the way to go.

oyving2• 17 years ago

Why? With the host packaging system I get versioned packages, through host management tools I can control which packages my production, staging, and development hosts should have, and I can describe my dependencies on host libraries and software through the host system's own tools.

Together with distribution systems like apt I can also significantly ease deployment.

I can see that executing some commands over a set of hosts at the same time could be useful, but doesn't sound like a killer feature for me.

As for deploying from staging to production servers, it sounds more tidy to build proper packages to deploy in staging and test before deploying the same packages to production.

beza1e1• 17 years ago

It's not the packaging, but Fabric (and Capistrano and friends) all look like shells scripts to me.

troels3• 17 years ago

Think of them as shell-script frameworks. They include commonly used tasks/libraries that are useful for deployment and build tasks. Yes, you could use shell script, but these frameworks gives you a more convenient environment. They are also easier to set up and are more portable, since they have fewer external dependencies.

ubernostrum2• 17 years ago

Because it's sometimes simpler not to, especially if you're planning to do more than just internal deployment.

You may end up dealing with multiple operating systems or package tools, for example, at which point Python's ubiquitous setup.py (and tools which work with it on every OS) look very attractive.

You may end up needing to install App A with Version 2.0 of Dependency B, and App C with Version 3.5, on the same server, at which point virtualenv starts looking really attractive.

These sorts of situations are extremely common in the real world, and "use the OS package tool" isn't a one-size-fits-all solution for them.

pibefision• 17 years ago

I'm using Capistrano to deploy python code and works fine.

Can fabric get the lastest release from a git repo?

simeonf• 17 years ago

Yes - to the extent that fabric is just a framework for building shell deployment commands and could run the shell command.

I think there's a lot of (forgive me) synergy from pairing pip, virtualenv and fabric together though: pip can easily install a project into a virtualenv by checking it out of a source code repository (currently git, bzr, svn, hg I believe). Rebuilding the environment then is just a matter of "freezing" the current virtualenv, transferring the list to the host and building an env with the appropriate packages (down to revision #) that's a "put" and a "run" - two lines in a fabfile.

The article links to one of my blogs which links to the slides from talks I did on pip, virtualenv, and fabric for Baypiggies a few months ago: http://simeonfranklin.com/blog/2009/mar/28/baypiggies-presen... if you're interested...

jwecker• 17 years ago

The tool he's missing that I end up using constantly is Memoize - http://www.eecs.berkeley.edu/~billm/memoize.html - enormously simplified my make processes, and works very well with Fabric and the others.

ajross• 17 years ago

Beware: talks about the benefits of using utility code to solve subproblems without even a nod to the complexity introduced by adding dependencies to a project and the costs thereof. This always tells me I'm reading someone who views software development as writing code only, and doesn't spend much time maintaining old stuff. All the "hacker" verbiage notwithstanding, I think the author is missing some important points about what makes a good hack.

That said, I don't know a thing about any of these tools. They might be fantastically worth it for all I know. But talk about the tools, not about how they make you a great hacker.

ubernostrum2• 17 years ago

I do real-world deployment of Python applications. I use two of these three tools daily, and will be using the third once it hits API stability.

I use them because -- even though they represent added dependencies -- they deal with tough problems in clean ways.

For example, every platform has its equivalent of DLL hell; in Python it's not unheard-of for Application A to need version 1.0 of some library, while Application B needs version 2.0, they both have to be on the same server and the two library versions are incompatible.

You can solve this by careful manual management of import paths, or you can use virtualenv, which does it for you and ensures complete isolation of each application's dependencies. I opt for virtualenv.

Add pip and you get a reproducible build process: create a virtualenv, and install whatever's needed to turn it into a working environment for your application. Then 'pip freeze' and you have a requirements file which can reproduce that environment for you on-demand (and which is also human-readable documentation of the needed software).

Fabric's in the middle of a big rewrite and so I'm not presently using it, but I will be once it's stable, because it adds the final piece of the puzzle: deployment to multiple servers, with support for multiple targets (e.g., I can differentiate "deploy this to the staging area" from "deploy this to the production environment").

Yes, there are tools on popular operating systems which can do these sorts of things, but you don't always have a homogeneous deployment target where you can rely on the same tool being available everywhere. These tools, on the other hand, are pure Python and so get to be cross-platform largely for free.

davepeck• 17 years ago

Fabric is in a wild state of flux right now. I used it a bit -- then a bit more -- then I backed off entirely. It has promise, but it seems a bit young at the moment.

diN0bot• 17 years ago

>" [deploying production code from local dev] always involves several steps like packaging up the source from you source code management system, putting the source in the correct place remotely, and the restating the remote web server. This can be very tedious by hand, especially for a couple of frequent, small changes. "

did i get this right? commit/push local code, up/fetch on server, reload web server.

am i deploing wrong or naively? you know some people honestly don't see the point in version control? yeah, i don't want to be that person.

s3graham• 17 years ago

I keep meaning to use virtualenv. Good pointer to howto with modpython?

inklesspen• 17 years ago

Don't use modpython. It's pretty obsolete.

snprbob86• 17 years ago

I took a brief look at virtualenv, but decided not to use it for some reason that currently escapes me. The solution that I came up with instead has been working out quite well so far...

My main package has a sub-package called "external". When I run manage.py, it immediately imports external. External's __init__.py adds a bunch of things to sys.path

Whenever I want to add a new package, I simply add the egg, or source, or whatever to the external folder, add an entry to the external/__init__.py and check it in. This process can even selectively load packages by platform with a simple if-statement. Now when I checkout a new enlistment from SVN, I immediately get the full set of dependencies at their exact versions.

Simple, but effective.

Thoughts?

Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip

Related Stories