Thoughts on the Python Packaging Ecosystem

https://pradyunsg.me/blog/2023/01/21/thoughts-on-python-packaging/

by samwillis • 4 years ago

21 9 4 years ago

pradyunsg.me

eesmith• 4 years ago

I enjoyed this systems level description, and think it is fundamentally sound.

zdw• 4 years ago

The point that `pip` is the only "ships with python" tool can't be understated.

Most projects I've worked that have the "I need a few python tools" just generate the venv and such in a Makefile or similar, then source the venv's activate file and run whatever is needed in a make target. This is good enough in nearly all cases, and doesn't require that one of the other tools be installed at the OS level.

No matter how good the other tools get, not being in the default distribution == won't get used, no matter how good it is, because it can't be relied on always being installed.

disgruntledphd2• 4 years ago

Yeah definitely. That being said, conda competes pretty effectively in the scientific/numeric space, because pip does not solve the problems that these people have, and for quite some time would cheerfully break the environments of said people.

tdido• 4 years ago

Very much so. As a bioinformatician, for me conda has become a central part of my workflow as a generic package manager through the conda-forge and bioconda channels. I know a few university HPC clusters that use it to handle software instead of e.g. environment modules.

subradios• 4 years ago

To add to your point, even "professional" software engineers are a lot of times in a role where they are in 9 ecosystems in an average workweek.

I've used poetry, even loved it - but without being able to control the project the only thing I can rely on is pip.

AtlasBarfed• 4 years ago

Which one, pip aliased to pip3, pip that is actually pip, or pip3?

As a forced user of python, python packaging is a dumpster fire.

The fact I can reliably only run python apps from docker containers is ridiculous.

carapace2• 4 years ago

(With apologies to those who've heard it before) the solution is to model the domain (of packaging/distribution) using Python itself and provide a simple Pythonic API for it, like what pathlib does for filesystems.

zbentley• 4 years ago

I think that's not the problem. Modelling isn't the issue. Half of these tools have good, usable ways of making an API for development and packaging. The issue is that they all make slightly different decisions. "Just make a domain model" falls apart when it encounters users with different subjective preferences or objective requirements--to say nothing of vastly different skill levels, which the article touches on.

carapace2• 4 years ago

Hmm, if that's the case then I'm not sure there's a good solution. The situation reminds me of the old web frameworks situation ("Python, the language with more web frameworks than keywords." har har) which was resolved when BDFL blessed Django.

Thoughts on the Python Packaging Ecosystem

Related Stories