A Better Pip Workflow

http://www.kennethreitz.org/essays/a-better-pip-workflow

by ycnews • 10 years ago

171 84 10 years ago

xuan• 10 years ago

Nice post as alway from Kenneth.

However, this workflow has a little drawback. If you have a dependency not from pipy, e.g. `pip install git+ssh://github.com/kennethreitz/requests.git@master`, it won't work.

shadowmint• 10 years ago

yes. Its a continual irritation to me 'pip freeze' doesnt support this.

89vision2• 10 years ago

I've given up trying to use vcs links as dependencies with pip. I deploy everything to an internal devpi server and use --extra-index-url

K0nserv3• 10 years ago

The lack of separation between requested direct dependencies and a pinned resolved dependencies(including transitive) has been one of the great confusions for me learning Python. Having used Bundler, CocoaPods, Composer, and NPM before, which all have this separation builtin, pip feels broken.

However there are a few project that tries to solves this, but the fact that the Python community has not decided on one cripples any initiative to fix it.

mattnewton• 10 years ago

I can second this, I wish every time I interact with pip that it was npm. I never appreciated the KISS approach npm uses of just dumping all the dependencies in local folder exhaustively until I visited Python versioning hell upon myself by not using a virtualenv (which is an ugly hack itself).

true_religion3• 10 years ago

Not to mention that NPM can resolve different version dependencies so if A requires v2 and B requires v3 of a module, they both can live separate and happy lives.

The above is possible for Python as well; I sketched out an implementation which patches __import__ to handle dependency resolution by version, but.... I'm afraid it's a bit unpythonic.

StavrosK2• 10 years ago

How does NPM do it? My requirements.txt looks like this and I've never had any problems:

  foo ~= 1.8.2
  bar ~= 2.4.1
  baz

(These are only requested dependencies, resolved are not specified).

amiraliakbari2• 10 years ago

Interestingly, the separation was the main problem I encountered deploying a Rails application for the first time. I still prefer the PIP's approach, as it promotes always using latest versions.

pak2• 10 years ago

Promoting the use of newer versions is a little nice, but you will rue the day you deploy to production and it automatically pulls some flaky new package version that you didn't have a chance to test locally.

K0nserv3• 10 years ago

How does it promote using the latest version?

akx• 10 years ago

The `pip-tools` utilities automate this nicely.

thejash• 10 years ago

They automate that, plus more. Everyone interested in the original link should really check it out: https://github.com/nvie/pip-tools

nikolay6• 10 years ago

... but it's notorious for not working on latest Python and pip.

njharman3• 10 years ago

A limitation you are not bound to cause they are open source.

bobinator606• 10 years ago

like it or hate it, this is exact how its supposed to be done.

castis• 10 years ago

That doesn't mean its the way it should always be done though.

sprayk• 10 years ago

I was thinking about this the other day when I was helping a friend with his method #1-style requirements.txt, and how I wish there was something similar to composer's "lockfile".

The author's proposed method is basically the same as how php's composer does it, with its composer.json and composer.lock. Specify your application requirements by hand in composer.json, run composer install, and composer.lock is generated. Check both in so you can have consistent deploys. When you want to upgrade to the latest versions within constraints set by hand in composer.json, run composer update to pull latest versions, updating composer.lock. Run tests, and commit the new composer.lock if you are satisfied.

Legion• 10 years ago

The author's proposed method is basically the same as how php's composer does it

Composer merely cloned Ruby's Bundler and its Gemfile/Gemfile.lock in that regard. Which is a good thing. It's beyond puzzling that Python has spawned multiple dependency managers, none of which have replicated the same golden path.

servilio• 10 years ago

That it hasn't been adopted as a core functionality doesn't there isn't one, zc.buildout first stable release predates bundler's 0.3.0 by at least one year:

https://pypi.python.org/pypi/zc.buildout/1.0.1

bpicolo2• 10 years ago

This is sort of the same thing setup.py is intended for.

The way I've seen this successful in practice across many projects:

setup.py: specify top-level (i.e. used directly by the application) dependencies. No pinned deps as general practice, but fine to put a hard min/max version on them if it's for-sure known.

requirements.txt: Pin all deps + sub-deps. This is your exactly known valid application state. As mentioned, a hasty deploy to production is not when you want to learn a dependency upgrade has broken your app.

requirements-dev.txt: dev dependencies + include requirements.txt

neuroid• 10 years ago

This. Using setup.py for specifying min versions (and occasionally max versions) is definitely the way to go.

However, manually maintaining a requirements file in addition to setup.py is quite tedious in the long run. It is much better to freeze requirements during the build process and use the generated requirements file for deployments. But, decent test coverage is key here.

falsedan6• 10 years ago

This is easily the best option, barring how painful it is to write a setup.py.

chishaku• 10 years ago

Filling out setup.py and other package boilerplate used to annoy me as well but that all changed after I started using cookiecutter.

https://github.com/audreyr/cookiecutter

snuxoll4• 10 years ago

It's really not that bad with setuptools - as an added bonus it becomes easy to package your application up in an RPM.

jsmthrowaway• 10 years ago

I've bookmarked Donald Stufft's opinion on this for a couple years, which contrasts interestingly with Kenneth's and your own (and which makes a lot of sense): https://caremad.io/2013/07/setup-vs-requirement/

I like the concrete-vs-abstract, library-vs-application ideas there. Most opinions on this overlook the distinction.

bpicolo2• 10 years ago

I don't at all disagree with that, actually. I do consider that a part of setup.py, but it rarely matters in practice. The python ecosystem doesn't have similar programmatic api specs compared to, say, PHP where they get together and rigidly define these things: http://www.php-fig.org/psr/psr-7/

Not to say the PHP strategy is necessarily ideal (feels restrictive to me), rather just the comparison between the two. That's the sort of ecosystem that would make the abstract part matter more

ianbicking• 10 years ago

This in fact was my intention when creating this feature in pip: setup.py has the abstract dependencies, requirements.txt is a recipe for building something exact. And I quickly found requirements-dev was useful for adding tools (some people have done that with setup.py extras, but the ergonomics on that never seemed great).

The one thing I wish was part of this is that there was a record of conflicts and purportedly successful combinations. That is, you distribute a package that says "I work with foo>=1.0" but you don't (and can't) know if the package works with foo 1.1 or 2.0. Semantic versioning feels like a fantasy that you could know, but you just can't. Understanding how versions work together is a discovery process, but we don't have a shared record of that discovery, and it doesn't belong to any one package version.

This sense that package releases and known good combinations are separate things developed at separate paces is also part of the motivation of requirements.txt, and maybe why people often moved away from setup.py

foxhop• 10 years ago

Hi Ian, long time fan of your work.

I'm supporting both using some silly logic to pull in requirements.txt and supply that to setup.

https://github.com/russellballestrini/botoform

e12e• 10 years ago

A pacakage/system that handled this would be great. Zope and Plone have the notion of "known good sets" -- and it's not really a pleasure to use (but much better than nothing). As far as I can tell, with Plone 5 - the recommended way to install Plone is from the unified installer - leaving known good sets, pinning and buildout to manage plugins:

http://docs.plone.org/manage/installing/installing_addons.ht...

For an example of what bootstrapping a full Plone 4 site via buildout entails, have a look at the (defunct) good-py project:

http://good-py.appspot.com/release/plone/4.2rc2

http://www.martinaspeli.net/articles/hello-good-py-a-known-g...

As long as one is able to manage to keep projects small, and in a virtual-env of it's own, managing "known good sets" (in buildout, or for pip) shouldn't really be too hard. But as projects grow, a real system for managing versions will be needed. As far as I know there are no good systems for this... yet. Ideally you'd want a list that people could update as they run into problems, so that if projectA runs fine with foo=1.0, and bar=1.1, maybe projectB discovers a bug in foo<=1.0.4rc5 and can update the requirement.

It's not a trivial thing (see also: All the package managers, apt, yum, etc).

crdoconnor• 10 years ago

While the Method #2 format for requirements.txt is best practice, it is a bit cumbersome. Namely, if I’m working on the codebase, and I want to $ pip install --upgrade some/all of the packages, I am unable to do so easily.

$ pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs ./venv/bin/pip install -U

Ought to be a first class command, though.

After running that the first thing I do is run the tests. Then I freeze, commit and push.

This is usually a good thing to do at the beginning of a new sprint so that more subtle bugs caused by upgrading packages can be teased out before releasing a new build.

For my projects that I release on pypi I don't want to use pinned dependencies, but for them I run periodic tests that download all dependencies and run tests so that I'm (almost) instantaneously notified when a package I depend upon causes a bug in my code (e.g. by changing an API).

tibbon• 10 years ago

I'm probably going to get downvoted to hell on this; but as a Rubyist who has been working on a Python project, I've been finding pip to be really weak against rubygems/bundler.

With RubyGems/Bundler I love the ability to point to github repos, lock versions (or allow minor/patch versions to update), have groups, etc.

requiresments.txt and pip just feels, awkward and weird. Especially when combined with virtualenv, in comparison to Ruby this is just stiff and strange.

I've had nothing but problems with more complex packages like opencv and opencl as well.

scandinavian2• 10 years ago

With RubyGems/Bundler I love the ability to point to github repos

You can do this with pip.

mrharrison• 10 years ago

I've had nothing but problems with more complex packages

I could say the same things about rubygems. With coding a lot of it is what you are familiar with. To me python and pip is clean and simple, ruby and rubygems is overly complex. But that's because I'm familiar with python, so yeah.

infecto• 10 years ago

I am fairly positive you can do everything you have described. Just doing a quick google search returns the answers I need.

1. Point to github repos. `pip install git+ssh://git@github.com/echweb/echweb-utils.git` http://stackoverflow.com/questions/4830856/is-it-possible-to...

2. Lock minor versions `Django>=1.3,<1.3.99` http://stackoverflow.com/questions/6047670/pip-specifying-mi...

What is a group? My google search did not turn up anything relevant.

brbsix2• 10 years ago

The only bummer is that you can't declare git repos as project dependencies, AFAIK they have to be installed manually as you've described.[0][1] Does ruby handle this well? This is a real PITA when the Cheese Shop / Warehouse either don't have a particular project or only have an outdated version. In such cases you pretty much just have to create your own mirror.

FYI groups allow you to group various dependencies. E.g. you have one group of dependencies for development, another for testing, and a minimal one for production.

[0]: --process-dependency-links will permit you to pull in VCS repos as dependencies locally, but it does you no good when distributing packages for third-party consumption

[1] https://groups.google.com/forum/#!topic/pypa-dev/tJ6HHPQpyJ4

falsedan6• 10 years ago

Same here, it's so weird coming for Perl (with its healthy admiration of Ruby's tooling) and to be stuck with manually tracking my package's (sorry, distribution's) requirements + pinning their versions in a Make target.

pak2• 10 years ago

Agree, isn't the difference between method #1 and method #2 precisely the reason that Bundler has a distinction between Gemfile and Gemfile.lock? http://bundler.io/v1.3/rationale.html Programmers want to handle and edit method #1, but sane deployment or collaboration requires tracking resolved versions via method #2.

The Gemfile contains the top-level dependencies that the app needs; the Gemfile.lock ensures that everybody developing or deploying is using the same gem versions for top-level and resolved transitive dependencies. Periodically one can `bundle update` to upgrade gem versions: http://bundler.io/man/bundle-update.1.html

It does continue to surprise me that distinctions like these are not handled by Python tooling, which has this absolutely sordid history around packaging, which continues aplomb in the wheel vs. egg wars... http://lucumr.pocoo.org/2014/1/27/python-on-wheels/

blaze332• 10 years ago

I would additionally recommend to use :

  pip freeze -r requirements-to-freeze.txt > requirements.txt

instead of just:

  pip freeze > requirements.txt

So you can keep your file structure with comments, nicely separating your dependencies from the dependencies of your dependencies. Actually I only used one requirements.txt file, removing everything below "## The following requirements were added by pip --freeze:" and regenerating it when I changed my dependencies.

And sure, beware of git urls being replaced by egg names in the process.

nikolay6• 10 years ago

Nice! Didn't know this was even possible!

nikolay6• 10 years ago

Duh! That's how I was using it along with the -l option!

true_religion3• 10 years ago

Another tool you might find useful is pipreq.

    pipreq --savepath gen.requirements.txt /

The above will run through all your source code, and generate the requirements that you actually use via imports.

https://github.com/bndr/pipreqs

Disclaimer: I contribute to it.

StavrosK2• 10 years ago

Does it work well? I can't imagine it would be very accurate for things like a Django project...

edavis• 10 years ago

With this workflow, what purpose does requirements.txt serve? Would the file ever be used directly?

Only thing I can think of is you'd track top-level packages in requirements-to-freeze.txt during development, while your deploy would use requirements.txt to get a deterministic environment.

fredsir• 10 years ago

That is indeed the idea.

david-given• 10 years ago

I still find myself automatically thinking of this PIP.

http://www.shaels.net/index.php/cpm80-22-documents/using-cpm...

Diederich• 10 years ago

Pinto (https://metacpan.org/pod/distribution/Pinto/lib/Pinto/Manual...) nails this problem in the Perl ecosystem:

Pinto has two primary goals. First, Pinto seeks to address the problem of instability in the CPAN mirrors. Distribution archives are constantly added and removed from the CPAN, so if you use it to build a system or application, you may not get the same result twice. Second, Pinto seeks to encourage developers to use the CPAN toolchain for building, testing, and dependency management of their own local software, even if they never plan to release it to the CPAN.

Pinto accomplishes these goals by providing tools for creating and managing your own custom repositories of distribution archives. These repositories can contain any distribution archives you like, and can be used with the standard CPAN toolchain. The tools also support various operations that enable you to deal with common problems that arise during the development process.

goerz2• 10 years ago

I would argue that the information in "requirements-to-freeze.txt" could simply be expressed in the "install_requires" list in setup.py. That is, the "general" dependencies (with only vague version information) should be in setup.py, whereas requirements.txt should pin the exact "guaranteed to work" versions. For small/non-commerical projects, requirements.txt may not even be necessary.

mixmastamyk2• 10 years ago

So, you use `python setup.py` instead of pip?

goerz2• 10 years ago

No...? AFAIK (and based on my daily experience), pip processes setup.py, so it definitely installs the dependencies listed there.

smnrchrds• 10 years ago

What does "requests[security]" do? I don't remember ever using the bracket syntax.

krallin• 10 years ago

This means "install the `requests` package with the `security` extras.

In this particular case, this installs 'pyOpenSSL>=0.13', 'ndg-httpsclient', 'pyasn1'. See: https://github.com/kennethreitz/requests/blob/46184236dc177f...

antonagestam• 10 years ago

From the docs: extras_require A dictionary mapping names of “extras” (optional features of your project) to strings or lists of strings specifying what other distributions must be installed to support those features. See the section below on Declaring Dependencies for details and examples of the format of this argument.

luhn• 10 years ago

Brackets are used to install recommended dependencies. See http://pythonhosted.org/setuptools/setuptools.html#declaring...

In the case of requests[security], it installs some extra packages that allow for more secure SSL. http://stackoverflow.com/questions/31811949/pip-install-requ...

amiraliakbari2• 10 years ago

Practically, it is used to prevent newer versions of pip from complaining about "insecure platform" and have cleaner CI build logs.

nikolay6• 10 years ago

I personally use requirements.txt for source and requirements.txt.lock, which I find more standard.

Or maybe use the pip tools' standard: requirements.in for source and requirement.txt for compiled at least.

njharman3• 10 years ago

Instead of doing that manually, use pip-tools https://github.com/nvie/pip-tools/

Some posts on theory and use http://nvie.com/posts/pip-tools-10-released/

blaze332• 10 years ago

He explicitly wrote: "I thought long and hard about building a tool to solve this problem. Others, like pip-tools, already have. But, I don’t want another tool in my toolchain; this should be possible with the tools available." so it's not like he's not aware of pip-tools.

kenneth_reitz• 10 years ago

/me has commit access to pip-tools.

jbmorgado• 10 years ago

I would also suggest using anaconda if you are using python for science: https://www.continuum.io/downloads

It makes managing the python packages and even the python versions quite easy.

ekmartin• 10 years ago

We built Doppins (https://doppins.com) to be able to use pinned PyPI dependencies and/or ranges and still keep them up-to-date continuously. Still another tool, but it's quite quick to enable on a repository and doesn't require any maintenance afterwards.

89vision2• 10 years ago

I've been using pip-tools to manage my requirements. It allows you to specify a top-level requirements file, and from that it builds you a requirements.txt with specific versions and all dependencies. It has really streamlined my pip workflow.

https://github.com/nvie/pip-tools

conradev• 10 years ago

Yep, I'm surprised no one knows about pip-compile. It does exactly as the OP suggests, but with the ability to specify a range of versions.

deepwalker• 10 years ago

Hey, I have wrote Pundle to solve this problem! Idea is simple — we just have package to install other packages with versions freeze. Pundle maintaine frozen.txt alongside with requirements.txt.

What more — pundle does not use virtualenv and install all packages to user directory ~/.Pundledir and import frozen versions on demand.

It have all nice commands like install, upgrade, info etc

Check it https://github.com/Deepwalker/pundler

darkerside• 10 years ago

Another method is to use pip constraints to enforce package versions https://pip.pypa.io/en/stable/user_guide/#constraints-files

itissid• 10 years ago

I found myself using tox for managing the general case of this. But the spirit to the blog is in the same direction.

p4wnc6• 10 years ago

If you don't mind committing to Anaconda Python distributions, then you should simply use conda.

You can still pip-install things within a conda environment, and conda can manage more dependencies than just Python dependencies (a common use case is managing R dependencies for a Python statistical workflow).

You can do

    conda list -e > requirements.txt

then

    conda create -n newenv --file requirements.txt

to create a Python environment from the frozen requirements.

I believe that conda makes it easier to selectively update, but even if you don't enjoy those features of conda, the same two-file trick as in this post will work for conda as well, since you can use `conda update --file ...`. Conda's "dry-run" features are more useful than pip's as well.

The perfect feature for conda to add is the ability to specify alternative Python distributions, either by a path to the executable, or by allowing alternative Python distributions to be hosted on Binstar.

I can understand why Continuum wants conda to heavily influence everyone to use only Anaconda, but I think the goodwill of making conda work for any Python distribution would bring more to them than keeping it focused solely on Anaconda. (For example, I know some production environments that still use Python 2.6 and are prevented from updating to 2.7 -- and even if they did update, they'd need to keep around some managed environments for 2.6 for testing and legacy verification work).

odonnellryan• 10 years ago

I agree. Only used conda on a few projects just recently, so not too battle tested, but I've found it much easier to use, especially with "difficult" libraries.

drcongo• 10 years ago

Vaguely related, I hacked together a little tool called Olaf[1] for people who like to pip freeze but also like to have multiple distinct requirements files like requirements-dev.txt – Needs a little work still but I've been using it on projects quite happily.

[1] https://pypi.python.org/pypi/olaf

A Better Pip Workflow

Related Stories