The State of Python Packaging

https://www.bernat.tech/pep-517-and-python-packaging/

by BerislavLopac • 7 years ago

133 42 7 years ago

bernat.tech

mistrial9• 7 years ago

> we wont be covering conda or OS specific builds

so 'azure-pipelines.yml' is not operating-system specific ?

dataking• 7 years ago

Not really. It controls continuous integration builds (or Pipelines in Microsoft parlance) on Azure DevOps. You can test on Linux, macOS, and Windows hosts.

kgwgk• 7 years ago

It says (now, at least): “A heads up, I will focus mostly on the Python Packaging Authorities systems (pip, setuptools, so no conda or operating system specific packagers).”

iandanforth2• 7 years ago

I have literally never seen a python project organized the way the maintainer does. This does not bode well.

nmalaguti• 7 years ago

The src directory is becoming a popular option. See https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-str... for an enumeration of the reasons.

Another resource might be https://hynek.me/articles/testing-packaging/

eesmith2• 7 years ago

My package has both Python code ("module") and a C extension ("_module"). The C extension code is under src/ following a long Unix tradition.

With the Python-module(s)-under-"src/" option, where do people usually place their C extension code?

It's clear that I could use any name, like "pysrc", and not just "src/" for the Python code. What I'm asking about is if there is a growing consensus about how to organize a package with both Python and C modules.

yyyymmddhhmmss• 7 years ago

I think using the src directory is a great idea. The tests can’t just reach over into the package anymore so you have to do an editable install, but the payoffs are big, as noted in the articles posted in a sibling comment.

alexbecker• 7 years ago

It's a fairly new paradigm, and I've noticed these things spread slowly in the Python ecosystem (Python 3 was released over 10 years ago). But I use it, and I've started seeing it in some other python packages lately.

Also keep in mind it is only for python packages, not applications--the primary benefit is that it is impossible for the tests to access files that are not part of the package build.

j88439h84• 7 years ago

I think it's a great fit for applications too.

iandanforth2• 7 years ago

Having One Punch Man as tier 3 invalidates all your points. ;)

kough3• 7 years ago

It's funny, I used to have a very popular (~2k pageviews a month, 1st or second search result on google) tutorial on how to get a package onto pypi. I deleted it because since then the PyPI team have written a good guide on how to do it: https://packaging.python.org/tutorials/packaging-projects/ . If you're reading this and wondering how to create a python package, follow that. Everything else is going to be wrong, or out-of-date, or worse.

saagarjha2• 7 years ago

It might have been better to leave the content up, but put a large disclaimer recommending the official guide. Is there any reason you didn’t do this?

warmwaffles• 7 years ago

Speaking for myself, when I skim an article first, I look for code examples and work my way backwards. I have missed disclaimers before.

kough3• 7 years ago

Change starts at home.

andyonthewings2• 7 years ago

I just happen to notice some discussion on Twitter among the Python data scientists side: https://twitter.com/pwang/status/1095173073341505536

I have the impression that conda and specifically how Python, Python packages, and native (e.g. C/C++) libraries interact is still an important problem not yet perfectly solved.

nyberg• 7 years ago

Wouldn't something along the lines of Guix be suitable? It's already used in HPC for pipelines and deals with all dependencies.

andyonthewings2• 7 years ago

It's kind of like suggesting to use system package manager for development package management. It will work, as long as the development material (e.g. a Python library) is available to the package repo, or the package manager is able to use the language-specific repo (e.g. Pypi).

But most system package managers are platform specific, which is not really suitable to be adopted officially by cross-platform languages like Python. I mean, Python can't adopt Guix as "the" Python package manager since Guix has no Windows support.

kristjansson• 7 years ago

Yes, but the effort to package the sort of software that's causing issues in python for the Guix/Nix/Spack/Easybuild class of systems is herculean[0] and idiosyncratic.

The groups maintaining and promulgating 'bad' packages would have to relinquish control over some of the dependencies they're currently venturing in or statically linking, or packagers have to reverse engineer their build systems, do that work themselves, and maintain it going forward.

[0] https://www.youtube.com/watch?v=NSemlYagjIU

Q6T46nT668w6i3m• 7 years ago

It’s mostly solved.

pip install numpy

should work on most platforms.

Nevertheless, there’re problems (that might be confusing to non-Python authors since they are not viewed as problems) like pip not installing system dependencies (e.g. matplotlib doesn’t install freetype).

jhayward• 7 years ago

Conda also solves the multiple versions of Python problem, in addition to SAT-solving the dependency map, and handling binary distributions.

atoav• 7 years ago

When I started learning Rust a year ago, I thought I was going the hard route, but I was surprised how much sense it made.

Especially the whole packaging thing with cargo makes so much sense. Compiling your application on a new machine is a breeze, you just need cargo and the rest happens automatically.

In the meantime I always find myself wondern how I was meant to do dependency managment again in python. Moving projects between machines can be a horror compared to Rust.

pipenv is a step forward, but nowhere near enough. We need something that just works and is part of python.

daveFNbuck• 7 years ago

For larger projects, I like to use Pants [1]. You just include an installer script in your repo and anyone with a Python installation should be able to build your project on their system with a simple command. Of course, plenty of issues pop up in practice, but it works remarkably well.

[1] https://www.pantsbuild.org/

emj• 7 years ago

When you start having multiple incompatible ABIs active at the same time, that's when dependencies get hard. You get automagical solutions that only work for some people because it's hard to see all possible combinations of versions. I think Rust is still too young.

richardwhiuk• 7 years ago

Rust's solution is 'build everything', so you can't have incompatible ABIs. That might not scale, and it means initial build time is pretty appalling, but unlike Python, Rust compiles to a binary, so it's less manky.

rullopat2• 7 years ago

The package management system, is probably the main motivation I always preferred Ruby to Python. Package management on Ruby is so clean, managed by few libraries done right (gem and bundler) that looking at the clusterfk of Python ways to package and manage dependencies I'm getting histeric.

klibertp• 7 years ago

For reference, can you list the most important differences/things done right in Ruby-world packaging as compared with pip?

rullopat2• 7 years ago

The problem is that there is NO one good standard to install packages in Python, neither there is one standard to create a package that is promoted by the language developers. Plus, in Ruby with Bundler, what is done by virtualenv in Python (environment isolation) is basically integrated in Bundler itself.

BerislavLopac• 7 years ago

Any thoughts about PEP 582? https://www.python.org/dev/peps/pep-0582/

gaborbernat• 7 years ago

not a big fan of it myself, it's great for people new to programming, but it's easy to quickly grow out of it :thinking:

hjk05• 7 years ago

Well you see actually Python packaging is extremely simple, people just don’t get it. You just use pip! Well actually distutils is what you use not pip, except don’t use distutils use setuptools instead, distutils is outdated, but not really because setuptools uses distutils internally... but anyways you should use setuptools is what I mean. Anyway you just set up a setup.py file with all the details of your package, except don’t do that, instead use a minimalistic setup.py and write your metadata in setup.cfg, except that’s silly instead put your version in the module, and your description in a file and use hooks in the setup.cfg to get them. Except actually you should maybe use pipenv instead, or poetry actually or what was it this week again? Err so where were we. Ohh yea, python packaging is really simple, people just don’t get it. But wait we haven’t really touched on virtual environments and dependency management yet....

My favorite vote for “Completely break backwards compatibility in python4” is packaging. Tear it all out in one go and completely forget it then start fresh with one and only one way to do it, under completely new names kill pip, setup.py and everything connected to them and let the Phoenix rise again under a new name. Google is a graveyard of the last 6 different “the new way of doing it” and everyone learns by doing it “wrong” 3-4 different ways before they get even close to what’s considered “the right way” this quarter... and then they bikeshed their own “fix” over that and share it on their blog to confuse everyone else. It’s frustrating to no end.

All that and we havn’t even touched on distributing projects that mix in c++/c, or just pure python projects that need to be compiled to an executable pyinstaller works great for that though! Until it doesn’t for some obscure unknown reason you’ll then have to figure out.

Building c++ projects is an absolute breeze compared to the hell that is Python.

mercer2• 7 years ago

I went from Ruby, to Node, to Elixir, and my annoyances with the package/project managers have decreased with every step. I'm honestly surprised that Python's 'approach' is so much worse than any of them. Are there any particular reason why there hasn't been a good package/project manager that became dominant in the Python ecosystem?

To elaborate on my current situation. With Elixir, I run mix new <project-name> and I get 1) a project structure, including test helpers, config stubs, formatter config, gitignore, and a mix.ex file where I can add dependencies. With some flags, I can get a supervised setup (for long-running stuff, basically), or an umbrella app that allows me to put multiple projects under one 'umbrella project'.

Aside from potential issues where the apps in an umbrella rely on different version of the same package, dependency is local to the project and doesn't conflict with another. I also get formatting and test 'tasks', with DocTest support.

PeterCorless• 7 years ago

This seems related: https://news.ycombinator.com/item?id=19164986

The State of Python Packaging

Related Stories