Python Application Deployment with Native Packages

http://hynek.me/articles/python-app-deployment-with-native-packages/

by craigkerstiens • 14 years ago

180 26 14 years ago

othello• 14 years ago

Deploying web apps as self-contained binaries reminds me of Go [1]. Except in Go you get this for free as a simple (and swift) compile step.

Of course, this is true of any compiled language that can produce self-contained binaries, but Go has the added benefit of being well adapted to web app development.

1: http://news.ycombinator.com/item?id=3722514

mbreese• 14 years ago

This isn't the same as a self-contained binary. This is packaging up your application, all associated files (javascript, css, etc) and virtualenv setup into a .deb (or .rpm). Then the package can be added to a local repository and installed like any other program.

Even with Go, you'd have to package up your single binary in order to get the same experience. (Although, that package would be pretty small!)

lifeisstillgood2• 14 years ago

from vrmd.fabric.ubuntu.deploy import Deployment

How do you differentiate builds on say redhat / BSD / windows as opposed to Ubuntu ? I would guess vrmd.fabric.fedora.deploy contains a lot of the same code except for pathname changes?

hynek8• 14 years ago

TBH, we currently just convert the debs to rpms when necessary. But RPM-based servers are only legacy here so it doesn’t happen too often.

We have no windows and no BSD _yet_. There might come some FreeBSD for ZFS's sake soonish, then I’ll have to refactor. :)

lifeisstillgood2• 14 years ago

Thank you.

obtu• 14 years ago

I'm a bit wary of calling an fpm-generated deb a native package, those are about as well integrated as a checkinstall (make install -> deb) or an alien (rpm -> deb) package. fpm may still be useful for this use case, but it doesn't take me much more effort to get a proper source package.

mattyb• 14 years ago

How are FPM-generated debs any less integrated than native packages? You have the same control over the package metadata using its flags as you do with a Debian control file, and using `-s dir`, you have complete control over the package contents. You can ship rc/Upstart scripts, use pre/post-inst/rm scripts to add users and backup data, etc. These are all the same tasks that lots of Debian-maintained packages perform.

Please look at the FPM wiki, you'll see that it offers many more options for integration than checkinstall (which I used to use, and hated):

https://github.com/jordansissel/fpm/wiki

And why would you want to create source packages? One of the benefits to generating binary packages is that you compile once and roll it out to hundreds of servers. I'd rather not have to compile, say, Erlang on every target server.

hynek8• 14 years ago

We used to use native tooling, the usage of fpm is rather new in fact. It simplified the code big time without compromising on anything.

j-kidd3• 14 years ago

I think native packages make things too rigid. A private PyPI server hits the sweet spot for us. The steps to deploy an update is as simple as:

1. activate virtualenv

2. run `easy_install -U app`

and no root permission is required. It also has the added benefit to allow having multiple copies of the same package in the same machine [1], which can't be done with native package.

[1] In one deployment, we have staging instance and production instance in the same machine...

sciurus• 14 years ago

Sure you can have multiple version installed with native packages; for examples, see the discussion at https://groups.google.com/group/devops-toolchain/browse_thre...

j-kidd3• 14 years ago

Of course, it has always been possible to have multiple versions installed with native packages. Gentoo supports that since day 1. What I was talking about though (and I didn't make it clear), is having the same version installed multiple times in different locations (i.e. separate virtualenv).

oasisbob• 14 years ago

For some environments, this may be the case.

I find native packaging to be really helpful in mixed environments. On a daily basis I need to deploy Rails/PHP/Python, and using a native package manager is a nice common denominator.

When I first heard about this strategy I thought it was crazy -- but have since come to really like it.

hynek8• 14 years ago

I think you missed one point here: The package _contains_ the virtualenv, so you even _must_ have multiple copies of the same package for every app that needs it.

That’s the whole point of it: have a consistent and reliable way to deploy whole applications including all dependencies.

j-kidd3• 14 years ago

That is exactly my point - you have to build multiple packages for the same app, even for the same version (app1_venv_a-1.0, app1_venv_b-1.0), if there is a need to have more than one instances of it running on the same machine.

Until emerge/apt-get/yum/etc add support for virtualenv, I will stick with .egg packages and be python-native rather than system-native.

onedognight• 14 years ago

> Sometimes git pull fails halfway through because of network problems

Separate the pull into "git fetch" into a shared bare git repository (if you care for efficiency) which is atomic and then a "git checkout" into a new directory using a single symlink(2) and rename(2) which is also atomic because rename is.

> Imagine you’re pulling from git and at the same time, the app decides to import a module that changed profoundly since its last deployment. A plain crash is the best case scenario here.

> On the other hand, deploying using self-contained native packages makes the update of an app a near-atomic,

You can't have your cake and eat it too. Native packages have the exact same race conditions as an (inplace) git update.

hynek8• 14 years ago

You make fair points here however I was arguing against the widespread “git pull” culture.

It's clear that some of the points can be mitigated using more sophisticated git+fab magic. Nevertheless, the other points remain true and the combination of them all makes me believe that native packages are the better way.

akheron• 14 years ago

In postinst:

> virtualenv /vrmd/$APP_NAME/venv

This step could be skipped if you built different debs for different distro versions (in chroots for example, like sbuild does it).

Might be too much of a hassle, though, for a little gain. It just looks slightly wrong to me that you first install a deb and then go and overwrite many of the files the deb installed.

hynek8• 14 years ago

I believe the reason is that the position of the virtualenv changes (from /home/buildbot/package-version/vrmd/venv to /vrmd/package/venv) and some links didn't match. I'm not 100% sure though, it's some time ago. :)

nakkiel2• 14 years ago

I personally use a good old Makefile to create debs. I'm not sure using Fabric for that gives much of an edge. I also handle base system configuration through a hierarchy of debs that perform tasks ranging from user configuration and database setup to application setup.

I've used that sort of setup in various configurations and it's solid. After all, that's what our favorite distos use (whether it's deb-based or not is irrelevant, concepts are similar).

If and when configuration needs to be "shared" or "propagated" between machines, I store in a store of some kind (RDBS, KV).

cageface2• 14 years ago

I've said this before here but I think the ability of Go to create a server as a single statically linked binary is a HUGE deployment advantage. It's so much easier to just compile exactly what you want and push out a single file.

hynek8• 14 years ago

Does it also help with deployment of non-code related files (eg. static files in web apps)?

cageface2• 14 years ago

No but I think the dependencies among those files tends to be less intricate and brittle than code.

nakkiel2• 14 years ago

Definitely a problem in Go. It's still possible to embed resources in the binary but if you're talking about a lot of such resources, the resulting binary size can be an issue as well. Not to mention resources won't be available outside of the binary.

Python Application Deployment with Native Packages

Related Stories