Python Wheels Crosses 90%

by groodt • 6 years ago

223 218 6 years ago

groodt8• 6 years ago

325 of top 360 Python packages are now distributed as Wheels.

navait• 6 years ago

I don’t know a lot about Python tooling, but in general my experiences with pip have been pleasant, so I appreciate all the work done by the maintainers to make it pleasant.

ciarannolan• 6 years ago

Could someone give a beginner's tl;dr of wheels vs. eggs?

I use Python extensively but the "Advantages of wheels" section on this site is way over my head.

edit: Thanks everyone :)

rantwasp• 6 years ago

http://packaging.python.org/discussions/wheel-vs-egg/

use wheels

thundergolfer• 6 years ago

This page does a good comparing them: https://packaging.python.org/discussions/wheel-vs-egg/

Is anything in that unclear?

In practice, this bit I think it's pretty important:

Wheel has a richer file naming convention. A single wheel archive can indicate its compatibility with a number of Python language versions and implementations, ABIs, and system architectures.

polm23• 6 years ago

This is a pretty short overview.

https://packaging.python.org/discussions/wheel-vs-egg/

Having packaged wheels but not eggs, my understanding is somewhat limited, but eggs being unversioned and having weaker filename conventions sounds like it would be a pain.

I mainly use binary wheels for projects that include compiled C extensions, so I need a different wheel for each supported platform. There's a straightforward filename convention for that, and all the tools support it, so I don't have to even think about that when generating or uploading wheels.

groodt8• 6 years ago

Wheels are newer and have an official PEP (PEP 427)

Eggs contain compiled Python bytecode which requires publishing egg versions for every version of Python. While Wheels are distribution formats that can support more than a single Python version.

u801e5• 6 years ago

One thing I found when converting our python application packaging from RPM to wheels is that wheels don't properly handle the data_files parameter in the setup call. That is, it places files under the python library directory instead of in the absolute path as specified. This means that sample configuration files and init scripts end up in the wrong place on the file system. In order to get around this, we had to upload the source distribution to our devpi instance and run pip install with the --no-binary option which would then place those files in the correct directories.

The other issue is that there's no equivalent of the %(config) RPM spec directive to prevent the config file from being overwritten if it already exists on the file system.

So, for libraries, wheels are a good cross-platform packaging solution, but not so much for applications that require configuration files and init scripts.

groodt8• 6 years ago

Agree.

I don't think the Wheel specification was intended to replace application distribution. It is aimed at the distribution of individual Python libraries.

enriquto3• 6 years ago

What's the difference between "applications" and "libraries"? Why can't they be distributed in the same way?

jeanvalmarc• 6 years ago

Is it desirable for a distributed Python package to write files to a hardcoded absolute path? Wheels seem to respect the `package_data` setup key, which is definitely inside the library path and seems to work pretty well for library data. I'd be a bit horrified if a package I installed from Pypi attempted to write stuff in `/etc/` or whatever. I guess I'm just asking, aren't these two separate problems (config management and Python packaging)?

u801e5• 6 years ago

Is it desirable for a distributed Python package to write files to a hardcoded absolute path?

The established behavior of python setuptools when creating an RPM spec file via the bdist_rpm command was to create a spec file that would invoke the build command in the %build section and the install command in the %install section. The install command would place those files in the absolute paths specified in the data_files named parameter.

It has worked this way even back when distutils.core, the predecessor of setuptools, was used.

As for whether it's desirable, if you're missing the config file, init script, and other things necessary for an application to work, then you essentially have an incomplete installation and will have to copy those files from other sources and update them yourself via config management before you're able to run it.

I guess I'm just asking, aren't these two separate problems (config management and Python packaging)?

In my view, config management should update a config file that's already there and also restart the service as needed (either when the package is updated, the config file is updated, or both). It shouldn't be responsible for placing the config file and init script itself.

For example, if I install an application like redis or rabbitmq-server, the package manager will place the included sample config file in the correct directory (e.g., /etc). I can further modify that file as necessary before starting the service. It will also include the scripts necessary to start the service as a daemon. Config management could be used to update the config file for environment specific concerns, but it doesn't need to essentially duplicate what the package manager has already done (create the directories, set permissions, place the file, etc). Python applications, regardless of how they're packaged, shouldn't really be different in this regard.

kstrauser• 6 years ago

Python has a couple of mechanisms for fetching files from inside the wheel files (or eggs or...) without you having to know or care about exactly where they’re located. See https://stackoverflow.com/questions/6028000/how-to-read-a-st... for examples.

sandGorgon• 6 years ago

You need to reopen this https://github.com/pypa/wheel/issues/92

pjc503• 6 years ago

it places files under the python library directory instead of in the absolute path as specified

How is this supposed to work in virtualenv environments? What happens if you have two of them?

raziel2p2• 6 years ago

Wheels don't, and shouldn't, replace RPM or deb packages, and I don't think anyone advertised them as such. They replace eggs or installing from source .py files, you should still build RPMs or debs on top of that if you want to do any system-level stuff like set up systemd services, config files in /etc and so on.

u801e5• 6 years ago

Wheels don't, and shouldn't, replace RPM or deb packages

Part of the motivation for using wheels in our case was to allow for pulling in later versions of certain python packages as part of the dependency resolution process when installing or updating the package. Using RPM while attempting to get the updated dependencies would mean having to repackage every single dependency (direct and indirect) from pypi as RPMs and upload them to our yum repo so that yum would be able to handle dependency resolution.

I don't think anyone advertised them as such. They replace eggs or installing from source .py files,

The setup method from setuptools does have the necessary features for handling system-level files and the bdist_rpm command leveraged that when building RPMs. It just seems that if this was a bug rather than a feature, then it should have been noted when transitioning from egg based distributions to wheels. Nothing in the documentation for eggs or wheels states that they don't support deploying system files in absolute file paths.

The fact that doing something like:

pip install git+ssh://git@github.com/org/project.git@1.2.3

and

pip install project==1.2.3

have different results for where files listed in the data_files named parameter of the setup end up on the file system suggest that the lack of absolute path support when installing wheels is a bug rather than an intentional change.

downerending3• 6 years ago

So far it's just a bit of smoke on the horizon, but I'm noticing some packages abandoning 'pip' installs entirely in favor of 'conda'. It's a bit early to tell if this trend will take off, but it does seem plausible.

dijksterhuis3• 6 years ago

I really hope conda stays on the user end.

Like, I understand it's great for managing package dependencies/setting up environments etc for a single project. But I've found it's an absolute nightmare for building docker images/generally doing build stuff when it's involved.

Not to mention conda installs seem to take waaaay longer than pip.

apt + pip works far more sensibly and reliably in my experience.

blt• 6 years ago

conda seems to spend a lot of time in a SAT solver or something to deal with the known NP-complete problem of dependency resolution [1]. Maybe pip doesn't do the same thing for requirements.txt setups?

[1] https://research.swtch.com/version-sat

lsorber2• 6 years ago

My experience has been the opposite: building Docker images is much easier with conda than it is with pip. With conda you can start from miniconda3, copy an environment.yml, and then conda create it. With pip, you might need to take additional steps to install system dependencies like build-essential first, and you'll need different tools to manage your virtual environment and Python installation too.

mbar84• 6 years ago

I've built my project template around conda, primarilly so that a new user can just do `make install` and have all environments created. The docker portion also works fine, the only issue being that the images are perhaps a bit larger than I would like.

https://gitlab.com/mbarkhau/bootstrapit

closeparen2• 6 years ago

As I understand it, conda has more sophisticated handling of native components, and is largely seen in scientific environments where Python is just the frontend to libraries that are actually in C++ and Fortran.

0az• 6 years ago

Conda does their own thing, so I hope that this doesn't become commonplace.

It's a nightmare of interop. Yes, it works for one person on a single laptop, but my experiences with conda outside of the happy path are universally terrible.

I'll stick with the standards.

lsorber2• 6 years ago

Can you give an example of a case outside of the happy path?

pas2• 6 years ago

Conda is "pip" for people who know _nothing_ about pip/operating-systems/IT. It is basically a way to have a working environment without understanding that maybe pandas and numpy and scikit are not Python built-in modules, and that maybe you'd need a working C/C++ (and even a Fortran compiler too) to install it.

It's great for that purpose, but - as far as I know - it has not much to do with general Python dependency management.

downerending3• 6 years ago

Conda does (in theory) handle python module dependencies well. In practice, like pip, sometimes you win and sometimes you lose.

Note that you can use 'pip' within a 'conda' environment as well, so you can sort of have it both ways.

dimatura2• 6 years ago

I think this trend may be limited to fields relying heavily on natively compiled dependencies, which may not even have any python at all. Conda packages can be C++, Fortran, pretty much anything. That's the main draw for me -- otherwise, I'd stick with pip, which is definitely more mainstream.

fortran772• 6 years ago

But snakes don't have wheels. Why not call it "skins?"

xapata• 6 years ago

You get wheels of cheese from the Cheese Shop. https://www.youtube.com/watch?v=Hz1JWzyvv8A

fortran772• 6 years ago

Aha! Thanks.

reedwolf• 6 years ago

I wonder how long until nobody remembers Python isn't named after the snake species?

haberman3• 6 years ago

Today I spent at least an hour fighting with Python packaging. The more I think about it, the more I feel that self-contained static binaries are the way to go. Trying to load source files from all over the filesystem at runtime is hell. Or at least it's hell to debug when it goes wrong.

I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit.

rtpg• 6 years ago

PyOxidizer might be of interest to you.

haberman3• 6 years ago

Yes, that looks like exactly what I had in mind. Thanks for the reference.

shoo• 6 years ago

Around a decade ago I worked for a place that packaged and shipped python applications to windows desktops using a similar bundling tool, py2exe. It worked pretty fine. Pretty much just unzips into a directory: there's your self contained python app hiding inside an exe and some associated data files/dlls in the directory.

We still used wheel archives etc for managing some python dependencies during our dev and build process, but it was never a complete solution: some python packages depend upon non python shared libraries so you need to install them using an operating system level package manager. Then when shipping the software to users you cannot assume the user has a working version of python, or even if they do you don't want to increase your support burden by having your application running inside some arbitrary python environment.

pansa22• 6 years ago

Thanks, I hadn’t heard of PyOxidizer before. Looks like a significant improvement over tools like PyInstaller and Shiv, which produce self-extracting executables that are slow and/or leave files hanging around after running.

m4rtink9• 6 years ago

Its fun and games until you find out one of the things you bundled into your static binary has a remote code execution CVE that is being actively exploited in the wild. Good luck tracking down all the artifacts and making sure they are rebuild with the patched version - till the time next CVE hits and you can do it all again for every static binary bundling the affected component.

marcinzm2• 6 years ago

I mean, containers are basically just that and people seem to like them just fine.

closeparen2• 6 years ago

Rolling out a new binary is a happy path process that gets used all the time; what's the problem here?

mdoms• 6 years ago

There's no reason whatever binary packaging format you use can't store metadata on all of its internal dependencies. "Tracking down" vulnerable dependencies is almost trivial in this case.

pjc503• 6 years ago

But this doesn't necessarily work in the "loose binding" world either - because people pin their dependencies to a specific version to avoid breakage.

There's a fundamental unresolvable tension in automatic update systems between "not updating breaks things" and "updating breaks things"; you cannot solve for both at once and either one will get users mad at you, although not upgrading at least gives a pushback argument. See how all this has played out with Windows and Mac updates, for example.

earthboundkid3• 6 years ago

What’s the scenario where dynamic libraries are regularly upgraded but static binaries are not? Just seems totally non-sequitur as a reason to prefer dynamic libraries.

zaphar• 6 years ago

This issue has different risk profiles based on the target user base.

If the install is on a consumer machine for regular usage then the right answer is shared libraries for the machine. It adds a lot of complexity for the packagers for that OS but you get a lot of safety for that consumer.

If the install is an inhouse app deploying to servers a company controls/rents whatever then the right answer is probably a static application. Consumers are more likely to want to chase the latest version of everything. Company developed software is far less likely to want that. The problem then becomes less about packaging and more about deploying fixes quickly. A static binary that is rebuilt, tested, and deployed is going to be a smoother path to fixing that error than figuring out how to deploy your new shared library and avoiding any issues in our production environment caused by clashes to that library. This is made more problematic by the likelihood that you will need multiple versions of that shared library with the fix to meet the needs of applications that can not yet be upgraded. Static binaries make things easier in this scenario and thus quicker to resolve.

vmchale4• 6 years ago

Some ecosystems allow you to do that, no?

u801e5• 6 years ago

Today I spent at least an hour fighting with Python packaging. The more I think about it, the more I feel that self-contained static binaries are the way to go. Trying to load source files from all over the filesystem at runtime is hell.

Most python packages I have seen will put their python source in the python site-packages directory. Where were the source files you were trying to load located on the file system?

est• 6 years ago

Trying to load source files from all over the filesystem at runtime is hell
"static" binaries that package everything together into a single, self-contained unit.

Ummm, except debugging "static" binaries without symbol file is hell.

If there's a .py source somewhere on the disk, I can at least try edit the source file from the package and debug internal private variables as last resort.

josefx• 6 years ago

Python can execute zipped files and attaching a zip to an executable is trivial. With a decent zip tool you also should be able to edit the zipped source files directly.

Spiritus2• 6 years ago

I’m curious. Can you explain the problems you were having? And how they would’ve been solved by a static binary.

Trying to load source files from all over the filesystem at runtime is hell.

This sounds like a mess unrelated to Python. Why are you trying to load source files from "all over the filesystem" at runtime?

I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit.

Compiling source files from all over the filesystem would be equally annoying.

IshKebab3• 6 years ago

Assuming you are genuinely asking... Python programs are almost always run from source, and they usually import third party libraries. Those libraries are installed by Pip... somewhere on your system. And then Python has to find it somehow at runtime. That relies on environment variables being set correctly, and works differently on different systems and is basically a huge mess even before you consider things like virtualenv and the fact that non-Linux systems usually have multiple copies of Python installed (and Python 2 and 3!).

It's basically a huge mess. There's even an XKCD about it.

Consider the alternative: Go compiles programs to a single statically linked executable with no dependencies. You literally just copy one file to your target machine and run it. It basically can't fail. That's part of the reason Go is so popular for server stuff.

And it's not just because Python isn't compiled. Other scripting languages handle this much better. Even JavaScript - for all the hate node_modules gets for being enormous, at least it works reliably!

PeterisP• 6 years ago

The python package requires a C extension that fails to compile because it requires some (non-python) dependency C library which it can't find (or use) despite it being installed to the system. Often because it expects version X but the system has installed version Y and I can't or don't want to install the particular version that this python package needs to compile its C code.

jensgk• 6 years ago

I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit.

Isn't that why Docker is so popular? If we had Python static binaries (or the ability to have several versions of the same Python library installed) would we use Docker so much?

agumonkey• 6 years ago

and docker is (was) written in go who kinda put more light on static, the circle is complete

gorgoiler5• 6 years ago

In the past week my coding colleague (whose age is only just into double digits) and I have had to:

* insert lines of code into asciidoctor Ruby to debug how the command line interface differs from the library interface;

* read through Raspberry Pi SenseHat code to figure out that the drivers won’t load via SSH (you have to plug in an HDMI cable!);

* rummage through Python’s PY Sequence List to answer the question “what kind of sort does Python actually use?”.

The problem with shipping binaries is it lowers the standard for being able to build from source.

If a package is easier to ship as a binary because the source is hard to distribute reliably, we give up hope of end users being able to modify and build their own versions.

This is obviously bad for freedom. It’s also hard for debugging and learning.

executesorder66• 6 years ago

You have a 10 year old coding colleague?

kelnos• 6 years ago

read through Raspberry Pi SenseHat code to figure out that the drivers won’t load via SSH (you have to plug in an HDMI cable!)

I'm really curious about this. Why would a driver care about whether you load it on a "real" tty vs. over the network? How does it even know?

stinos• 6 years ago

If a package is easier to ship as a binary because the source is hard to distribute reliably, we give up hope of end users being able to modify and build their own versions

You might be giving up hope too soon, because what you describe has in my experience been the life story of source code from the start, to some extent. And I'm not blaming it for that. Distributing source code (well, not the distribution itself, but getting it to build) on all possible systems out there is hard and impossible to get 100% correct. It's simply inevitable to encounter systems on which it doesn't build because of their specific configuration, or user error.

Despite that, end users have been building their own versions and still do. Users who really want to don't let themselves hold back by a compiler error. Granted it's possibly that in a perfect world where everything would just work there would be somewhat more users doing that, but there's no such thing as a perfect world.

Moreover I do have the impression things are actually much better these days, mainly thanks to CI which makes it easy to get out of the bubble and build on multiple platforms. At least I don't have that feeling anymore of 2 decades ago where I sort of was prepared to waste hours getting anything to build which got handed to me on the internet. Still, it would be very interesting to go and see the percentage of projects out there which just build according to the instructions.

yodelshady• 6 years ago

read through Raspberry Pi SenseHat code to figure out that the drivers won’t load via SSH (you have to plug in an HDMI cable!);

What? I can assure you this is not true.

My R Pi has never touched an HDMI cable and runs a sense HAT just fine. What OS?

cowsandmilk• 6 years ago

rummage through Python’s PY Sequence List to answer the question “what kind of sort does Python actually use?”.

If you needed to dig through source code to find out that python use Timsort, I genuinely am curious about how bad you are at search engines.

I believe in reading source code to answer some things, but that example is really odd for anyone who has researched sorting algorithms since Python literally invented a sort used by many other languages now.

abathur• 6 years ago

You might find the Nix package manager fruitful. (Fair warning: Nix isn't "easy", but it is powerful. It takes some work to find your sea legs.)

It isn't always possible, but Nix places a lot of emphasis on building software from source (or downloading a pre-built copy from a binary cache), and it has fairly good support for overriding/customizing the expression that builds one or more pieces of software.

A few months ago I was trying to debug a test in the Oil shell project that broke only on macOS for very unclear reasons. I eventually figured out that, in a specific mode, bash (which both Nix and macOS use as sh by default) was filtering out a specific environment variable that this specific test was trying to set, before running a target program with a sh shebang. I only needed to add a few lines to the project's build expression to patch bash and use the patched version for the tests (all without affecting the system/user bash).

haberman3• 6 years ago

I agree that being able to read source and modify the code for experimentation are important.

In my experience, the best way to read source code and experiment are to get the source from the upstream repo (eg. GitHub) and read/build from that. I do this with code written in C or C++ all the time.

The code is always out there (unless you're talking about proprietary software, which none of your examples above are).

sulZ• 6 years ago

* read through Raspberry Pi SenseHat code to figure out that the drivers won’t load via SSH (you have to plug in an HDMI cable!);

I've suspected that for a long time without confirming it personally, so thank you for your time.

oefrha2• 6 years ago

zipapp? It’s not like a self-contained solution didn’t exist for a long time. (If you mean the Python installation also needs to be contained in that self-contained binary — sorry you chose a scripting language.)

And yesterday I spent at least an hour fighting with cargo build. Without knowing exactly why we fought both our anecdotes are quite useless.

kkapelon• 6 years ago

I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit.

You mean Docker containers? :-)

drran2• 6 years ago

VMWare images or even hardware images: it's easier to buy new phone than to patch old one.

mturilin• 6 years ago

Deno (https://deno.land) solves this problem very well for JavaScript. I’d like so see more languages adopting “dependencies are URLs” approach. No more package hell.

julianwachholz• 6 years ago

Like GoLang did?

aequitas• 6 years ago

How does this solve the problem? As I can tell it replaces a centralized dependencies (requirements.txt) containing references to a known and trusted registry (pypi.org) with url's in all files across your project (and 3rd party library's files). Making maitenance only harder and security updates even more so. And making it harder to debug things as you now keep in mind that multiple versions of the same packages can be used in your source. Also, requirements.txt already support's urls if you want your sources from other places.

For me, most of the pain with the Python packaging went away after I started using Pip-tools[0]. It's just a simple utility to add lockfile capabilities to Pip. Nothing new to learn, no new filosophies or paradigm's. No PEP waiting to be adopted by everyone. Just good old requirements.txt + Pip.

[0] https://github.com/jazzband/pip-tools

atoav2• 6 years ago

The best thing would be if python had something like a cross between pip and python-poetry, but better integrated with the interpreter.

Python has the most unpythonic package experience ever. As elegant as the language is, packaging is a convoluted nightmare. It still might work for some project, but certainly is no fit for the average python user.

necrotic_comp2• 6 years ago

I started using the pipenv this weekend and it seems like it's pretty good for at least maintaining an environment within a project. I found it friendly at least.

That being said, I'll admit I don't know much about what it means to be pythonic. What do you think the drawbacks are to pipenv-type approach and what would be more pythonic in your mind ?

ashishb• 6 years ago

Even more so in a world where storage is cheap.

jillesvangurp2• 6 years ago

Agreed. I'd go one step further and call that docker container. If it needs to run on a server, the preferred format these days is a docker container. I too spend a bit of time fighting pipenv lately and generally trying to figure out the various ways python tries to virtualize barfing all over the filesystem by default.

IMHO python, node.js, ruby, and other scripting languages share a historical confusion of concerns related to development tooling and attempts to counter the unfortunate choice of installing dependencies globally that tend to leak to the server side in inappropriate ways. With Docker.

ClumsyPilot3• 6 years ago

Except we solved the problem at a higher level of abstraction, creating a more complex system. If every language used LLVM (for example) we'd have almost the same effect.

There are other things docker can do (like mapping volumes) but I would be much happier if we solved it at the level of the executable.

mintplant• 6 years ago

Just use Poetry. I haven't fought with Python packaging in a long time.

ghshephard4• 6 years ago

I see a lot of references to Poetry, but I've yet to see any uptake with the "professional" python developers that I know - which could be entirely to inertia, rather than "best tool for the job" - most of them see to be still in the pip/virtualenv/requirements.txt world for all of their development/docker/k8s deployments.

I wonder if there have been any surveys on poetry, and whether it's trajectory (presumably exponential) suggests that it's going to become a dominant package management utility in the next 10 years or so?

nh22• 6 years ago

You may enjoy this StackOverflow answer on The Return of Static Linking, summarising advancements on the topic:

https://stackoverflow.com/questions/3430400/linux-static-lin...

vmchale4• 6 years ago

The more I think about it, the more I feel that self-contained static binaries are the way to go. Trying to load source files from all over the filesystem at runtime is hell.

One need not depend on the filesystem for such things. And as bad as python packaging is, others get it better. One can have less-than-static-binaries without being quite that bad.

I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit.

Plenty of problems with that too!

blaisio• 6 years ago

I do prefer static self contained binaries, but I think it's important to note that Python's package management system is horrible, and making self contained binaries is not the only way to fix it.

developer22• 6 years ago

IMO: Python is a garbage language, and should never be used for professional applications. Try to write an app whose very first block is a try/except for KeyboardInterrupt, with or without an if __name__ == '__main__' block. Now run the program a couple dozen times, and press ctrl+c as fast as you can. You will wind up with a debug-level stack trace dump for an uncaught KeyboardInterrupt exception from the core libraries that is thrown before your first line of userland code is parsed or executed. Python runs so much internal setup code before your first line of code is even considered. It's literally impossible to write a Python program that doesn't dump debug garbage to its output, without writing a separate bash script that uses trap to proxy SIGINT and SIGTERM to the running process. Python is a fucking joke in my eyes; and this is before discussing the way async/await was implemented by gutting the core of the language in a way that makes zero fucking sense. Python is a total… fucking… joke. It has some nice features when all you care is about the syntax and ease of development, but the fact is it's a terribly broken language. Use C. Or C++. Or Rust. Or Go. Or Ruby, or literally no joke… even PHP. Python is literally THAT HORRIBLE once you understand its internals. It absolutely disgusts me to say, but PHP with its single-threaded model is put together better than Python.

tldr; Do not use Python. It makes certain tasks seem easy, but all it truly does is make you look like an idiot developer. Pick literally any other language, as Python has screwed up too many basic concepts (related to POSIX and expectations from C) to be taken seriously.

shadeslayer_• 6 years ago

comments like this make me want to ask for a downvote button.

jessaustin2• 6 years ago

This seems like a really effective way to set up a page for shaming "laggards". It would be interesting to track over time how many github issues are just links to this page.

rlayton2• 6 years ago

As noted on the page, but this idea (at least in the python world) started with such a "Wall of shame" for libraries that hadn't updated to Python 3 support. I'm not sure how much progress was directly attributable to that site, but it was fairly widely referenced.

jessaustin2• 6 years ago

Haha python is so authoritarian.

thehiddenbrain• 6 years ago

For anyone working on python wheels requiring to compile c/cpp/fortran, this may be useful. See https://scikit-build.org

(Disclaimer: I am one of the maintainer)

dralley• 6 years ago

Thanks for your work! I've used skbuild to package a couple of C library dependencies we have that would otherwise only be available to us as RPMs.

awinter-py3• 6 years ago

have always wondered why pypi doesn't generate whl files for pure-python sdists

and why companies like travis / github aren't more active in language-level packaging work

github gives away so much free docker time -- faster installation would save them money directly

X-Istence3• 6 years ago

Because there is no way to know if something is pure-python or not.

Even more so with pep517 and being able to use different build systems.

awinter-py3• 6 years ago

wait then how does bdist_wheel know to make platform-dependent vs platform-independent whl files?

wirrbel2• 6 years ago

underrated comment.

pypi could probably generate a lot of wheels. In fact, `pip` generates wheels from packages it downloades before installing them, so wheels can be generated for a lot of packages already directly from sdist.

Why it isn't done already? Good question. I guess it has to do with a majority of python packages installing just fine from source (those that aren't a problem for packaging anyway), and then those packages that need wheels because they use native extensions kind of are high-maintenance to bundle and roll out, especially if upstream maintainers haven't invested in building their projects for that.

I would love to see auto-bundled wheels. but I guess for this to fly, package maintainers would need to start standardizing a bit better (that also involves tooling). Easy things are missing, for example in setuptools you cannot even define smoke tests that distributors could use to verify successful bundling.

usr11062• 6 years ago

That doesn't seem to be a particular fast adoption. I remember seeing the first wheels when working at a job I quit early 2010. So it must be over 10 years.

Edit: A web search points to 2012, so maybe it's "only" 8 years?

Edit 2: Pip came in 2008, so something changed somewhat before 2010 as I remembered. But what did it install if not wheels?

groodt8• 6 years ago

If you were using PyPI you would have encountered source archives or Wheels. If you were not using PyPI you could have encountered source archives or Eggs.

usr11062• 6 years ago

If you were using PyPI you would have encountered source archives or Wheels.

Before 2010? That's what I believed to remember. But it looks like wheels did not exist before 2012, so that can't be true.

jacobush• 6 years ago

Eggs?

foldr• 6 years ago

Ah, so I've been confusing PyPI with PyPy. Someone made a great naming decision there.

young_unixer2• 6 years ago

I confuse PyPI with PyPA, the maintainers of pip, which allows you to interact with PyPI.

wodenokoto2• 6 years ago

A too little known fact: PyPI is pronounced Py-pee-ai, whereas PyPy is pronounced Py-py.

If everybody knew that, I think the confusion would mostly disappear.

JTon• 6 years ago

A too little known fact: PyPI is pronounced Py-pee-ai

So I was reading this comment and thinking to myself, how does one even pronounce "ai"? Language is not my strong suit. After searching the web, I found out that "ai" is a diphthong and it sounds like "eye" or the letter "i". Leaving that info here for others who may be confused. Sound: https://www.youtube.com/watch?v=uyKgPH0kmrU

dmix• 6 years ago

That’s a great way to track tech adoption within a community.

fouc• 6 years ago

Is Python Wheels the equivalent of Ruby Gems? Or is it better?

groodt8• 6 years ago

Gems

quietbritishjim• 6 years ago

Honest questions from non-user of Ruby:

* Do Ruby gems contain all binary depedencies statically linked into them?

* Is there a list of exceptions that are guaranteed to be present on target platform (see manylinux [1]).

The fact that you answered "Gems" rather than "something better" means the answer to both must be "yes" but I wanted to check.

[1] https://github.com/pypa/manylinux

arusahni• 6 years ago

Any idea if musl support is on the horizon?

groodt8• 6 years ago

I think still a while off https://mail.python.org/archives/list/distutils-sig@python.o...

dingdingdang• 6 years ago

Since pip is used to install Wheels it would probably be best to have a new separate 3rd party meta tool to install package managers themselves to avoid the confusion. Preferably this should have it's own additional PEP and integrate PyPI and PyPy along with other packages that could make life simpler for the (hopefully now happier) end user.

schwag09• 6 years ago

At one point in time I created a Python package to highlight this benefit of wheels: "Avoids arbitrary code execution for installation. (Avoids setup.py)" - https://github.com/mschwager/0wned

Of course Python imports can have side-effects, so you can achieve the same results with 'import malicious_package', but the installation avenue was surprising to me at the time so I created a simple demo. Also consider that 'import malicious_package' is typically not run as root whereas 'pip install' is often run with 'sudo'.

ghshephard4• 6 years ago

I thought the rule was never run `sudo` with `pip install` or you'll screw up the permissions on your system.

sigjuice2• 6 years ago

For other reasons, it is just as bad when using python from Homebrew on macOS where sudo isn't necessary. `pip install` will install modules in `/usr/local` where they will get mixed with Homebrew-provided python packages. I was hoping there would be a way to make `pip install --user` the default, but I couldn't figure it out the last time I checked.

daveisfera• 6 years ago

Any news on supporting Alpine Linux with wheels?

Python Wheels Crosses 90%

Related Stories