PyPI now supports digital attestations

https://blog.pypi.org/posts/2024-11-14-pypi-now-supports-digital-attestations/

by miketheman • 2 years ago

218 186 2 years ago

blog.pypi.org

woodruffw37• 2 years ago

I'm very excited this has come to fruition!

PyPI's user docs[1] have some more details, as does the underlying PEP[2].

[1] https://docs.pypi.org/attestations/

[2] https://peps.python.org/pep-0740/

ashvardanian• 2 years ago

Here is the GitHub issue you can subscribe to for automatically generated attestations in the GitHub CI PyPi upload action: https://github.com/pypa/gh-action-pypi-publish/issues/288

woodruffw37• 2 years ago

Just a note: that issue is for connecting the already-present attestations to GitHub's attestations feature[1]. Attestations are already automatically generated and uploaded to PyPI regardless of that issue.

(That issue will still be great to resolve, but just to avoid any confusion about its scope.)

[1] https://docs.github.com/en/actions/security-for-github-actio...

dlor• 2 years ago

This is awesome to see, and the result of many years of hard work from awesome people.

antononcube3• 2 years ago

It looks like another process certain software engineers want to program and facilitate. I hope it is and stays optional.

LtWorf9• 2 years ago

I'm pessimistic about that.

trishankkarthik• 2 years ago

Very cool, and congrats!

The corresponding ToB blog post says the following:

Longer term, we can do even better: doing “one off” verifications means that the client has no recollection of which identities should be trusted for which distributions. To address this, installation tools need a notion of “trust on first use” for signing identities, meaning that subsequent installations can be halted and inspected by a user if the attesting identity changes (or the package becomes unattested between versions).

Agree: signing is only as good as verification. However, trust-on-first-use (TOFU) is not the most secure way to map packages to attestations because nothing stops attackers who have taken over PyPI from tampering with the _unsigned_ mapping of identities to attestations in package lockfiles for new clients (certainly in containerized environments where everything could look new), and even just new versions of packages.

Although [PEP 458](https://peps.python.org/pep-0458/) is about signing the Python package index, it sets the foundation for being able to securely map packages to signed in-toto _policies_, which would in turn securely map identities to attestations. I think it is worth noting how these different PEPs can work together :)

cpburns20096• 2 years ago

Great, now how do you use attestations with Twine when publishing packages on PyPI outside of the Github ecosystem?

hifromwork2• 2 years ago

You need to rely on one of the four trusted publishers. You can't do it yourself: https://docs.pypi.org/trusted-publishers/adding-a-publisher/

guappa23• 2 years ago

You don't. The whole point is that you can no longer sign anything. Microsoft signs for you.

And of course the signature means "this user can push to github" and nothing more.

remram• 2 years ago

Hopefully the attestation is bound to a specific commit, so you can know the binaries came from the source?

Otherwise I don't get it.

krnavy• 2 years ago

After 2FA, the previous PyPI buzzword that was forced on everyone, JFrog discovered a key leak that compromised everything:

https://news.ycombinator.com/item?id=40941809

JFrog also discovered multiple malicious package exploits later.

Now we get a Github centric new buzzword that could be replaced by trusted SHA256 sums. Python is also big on business speak like SBOM. The above key leak of course occurred after all these new security "experts" manifested themselves out of nowhere.

The procedure remains the same. Download a package from the original creators, audit it, use a local repo and block PyPI.

SethMLarson3• 2 years ago

Hello! I believe I'm one of the "manifested" security experts you're hinting at :)

Good security doesn't demand perfection, that's why security is both prevention and preparedness. The response from our admin was in every way beyond what you'd expect from many other orgs: prompt response (on the order of minutes), full audit of activity for the credential (none found), and full public disclosure ahead of the security researcher's report.

JFrog also discovered multiple malicious package exploits later.

If you're referencing malicious packages on PyPI then yes! We want to keep PyPI freely open to all to use, and that has negative knock-on effects. Turns out that exploiting public good code repositories is quite popular, but I maintain that the impact is quite low and that our ability to respond to these sorts of attacks is also very good due to our network of security engineers and volunteers who are triaging their reports. Thanks to the work of Mike Fiedler (quarantining packages, API for reporting malicious packages, better UI for triagers) our ability to respond to malicious packages will become even better.

Now we get a Github centric new buzzword that could be replaced by trusted SHA256 sums.

In a way, this feature is what you're describing but is easier to automate (therefore: good for you as a user) and is more likely to be correct because every attestation is verified by PyPI before it's made available to others (which is also good for users). The focus on GitHub Actions is because this is where many Python projects publish from, there is little incentive to create a feature that no one will use.

Python is also big on business speak like SBOM.

Indeed, there is legislation in many places that will require SBOMs for all software placed in their markets so there is plenty of interest in these standards. I'm working on this myself to try to do the most we can for users while minimizing the impact this will have on upstream open source project maintainers.

zahlman6• 2 years ago

In a way, this feature is what you're describing but is easier to automate (therefore: good for you as a user) and is more likely to be correct because every attestation is verified by PyPI before it's made available to others (which is also good for users).

How long should I expect it to take until I can automatically generate an attestation from `twine`? Or does someone else have to sign off on it through some OpenID mumbo-jumbo before I can qualify as "trusted"?

Automating the creation of SBOMs sounds even further out, since we're still struggling with actually just building sdists in the first place.

zahlman6• 2 years ago

After reading the underlying report (https://jfrog.com/blog/leaked-pypi-secret-token-revealed-in-...), I can't help but think: "where is the defense in depth?" Since `.pyc` files are just a cache of compilation that's already generally pretty quick, this could have been prevented by systems that simply didn't allow for pushing them into the Docker image in the first place. Or by having `PYTHONDONTWRITEBYTECODE=1` set on the developer's machine.

(Also, now I'm trying to wrap my head around the fact that there's such a thing as "Docker Hub" in the first place, and that people feel comfortable using it.)

akx• 2 years ago

now I'm trying to wrap my head around the fact that there's such a thing as "Docker Hub" in the first place

Unless you build all of your images `FROM scratch` by default (or use in-house registries or quay or whatnot for all of your base images), you've used Docker Hub too...

pabs33• 2 years ago

Download a package from the original creators

Don't download packages from PyPI, go upstream to the actual source code on GitHub, audit that, build locally, verify your build is the same as the PyPI one, check the audits people have posted using crev, decide if you trust any of them, upload your audit to crev too.

https://reproducible-builds.org/ https://github.com/crev-dev/

belval4• 2 years ago

I have a bit of uneasiness about how this is heavily pushing GitHub actions as the correct way to publish to PyPI. I had to check PEP740 to make sure it was not directly supported by Microsoft.

The generation and publication of attestations happens by default, and no changes are necessary for projects that meet all of these conditions: publish from GitHub Actions; via Trusted Publishing; and use the pypa/gh-action-pypi-publish action to publish.

If you then click on "The manual way" it adds a big disclaimer:

STOP! You probably don't need this section; it exists only to provide some internal details about how attestation generation and uploading work. If you're an ordinary user, it is strongly recommended that you use one of the official workflows described above.

Where the only official workflow is "Use GitHub Actions".

I guess I am an idealist but as a maintainer this falls short of my expectations for the openness of Python and PyPI.

woodruffw37• 2 years ago

Where the only official workflow is "Use GitHub Actions".

The standard behind this (PEP 740) supports anything that can be used with Trusted Publishing[1]. That includes GitLab, Google Cloud, ActiveState, and can include any other OIDC IdP if people make a good case for including it.

It's not tied to Microsoft or GitHub in any particular way. The only reason it emphasizes GitHub Actions is because that's where the overwhelming majority of automatic publishing traffic comes from, and because it follows a similar enablement pattern as Trusted Publishing did (where we did GitHub first, followed by GitLab and other providers).

[1] https://docs.pypi.org/trusted-publishers/

silverwind• 2 years ago

Why does this need to allowlist CI providers in first place? Why not publish an open interface any CI provider can integrate against?

belval4• 2 years ago

I get that, that's why I didn't go "This is Embrace Extend Extinguish", but as constructive feedback I would recommend softening the language and to replace:

STOP! You probably don't need this section;

In https://docs.pypi.org/attestations/producing-attestations/#t...

Perhaps also add a few of the providers you listed as well?

The only reason it emphasizes GitHub Actions is because that's where the overwhelming majority of automatic publishing traffic comes from

GitHub being popular is a self-reinforcing process, if GitHub is your first class citizen for something as crucial as trusted publishing then projects on GitHub will see a higher adoption and become the de-facto "secure choice".

benatkin• 2 years ago

That's good. IIRC the feature was launched with GitHub as the only option, which is the same state of affairs of https://jsr.io/ right now.

IMO the launch should have been delayed until there was more than one trusted publisher.

freeone30003• 2 years ago

It requires an OIDC IdP, though… with PGP, I can verify that my identity is constant, but now, I’m reliant on some chosen third-party. And it has to keep being the same one, to boot! I don’t like the lock-in this causes, and I definitely don’t like the possibility of my right to publish being revoked by a third party.

guappa23• 2 years ago

Except that also for trusted publishing, they only allowed github in the beginning and eventually added a couple of other providers. But if you're not google or microsoft you won't be added.

zahlman6• 2 years ago

"Ordinary user" here.

It comes across like the goal of this system is to prove to my users a) who I am; b) that I am working in cooperation with some legitimate big business. I don't understand why I should want to prove such things, nor why it's desirable for b) to even be true, nor why my users should particularly care about a) at all, whether I can prove it or not.

I think my users should only care that the actual contents of the sdist match, and the built contents of the wheel correspond to, the source available at the location described in the README.

Yes, byte-for-byte reproducible builds would be a problem, for those who don't develop pure Python. Part of why sdists exist is so that people who can't trust the wheel, like distro maintainers, can build things themselves. And really - why would a distro maintainer take I take "I can cryptographically prove the source came from zahlman, and trust me, this wheel corresponds to the source" from $big_company any more seriously than "trust me, this wheel corresponds to the source" directly from me?

If I have to route my code through one of these companies ("Don't worry, you can work with Google instead!" is not a good response to people who don't want to work with Microsoft) to gain a compliance checkmark, and figure out how someone else's CI system works (for many years it was perfectly possible for me to write thousands of lines of code and never even have an awareness that there is such a thing as CI) so that I can actually use the publishing system, and learn what things like "OIDC IdP" are so that I can talk to the people in charge...

... then I might as well learn the internal details of how it works.

And, in fact, if those people are suggesting that I shouldn't worry about such details, I count that as a red flag.

I thought open source was about transparency.

didntcheck• 2 years ago

I might be reading this wrong but doesn't it include four workflows, including Gitlab? The "manual way" section is grouped under the Github tab, which indicates to me that it's only explaining how the Github-specific workflow works?

https://docs.pypi.org/trusted-publishers/creating-a-project-...

belval4• 2 years ago

I was referring to the "Get Started" section that links to a producing attestation page that can be found here: https://docs.pypi.org/attestations/producing-attestations/#t....

Support for automatic attestation generation and publication from other Trusted Publisher environments is planned. While not recommended, maintainers can also manually generate and publish attestations.

woodruffw37• 2 years ago

You're on a slightly different page: Trusted Publishing supports four different providers, but this initial enablement of attestations (which are built on top of Trusted Publishing) is for GitHub only. The plan is to enable the others in short order.

hifromwork2• 2 years ago

Even more, the previous way was to use GPG signatures, which were recently deprecated and removed. So you don't really have a choice.

Where the only official workflow is "Use GitHub Actions".

Well you can do it manually with other solutions... as long as they are one of the four trusted publishers (see "Producing attestations manually does not bypass (...) restrictions on (...) Trusted Publishers":

https://docs.pypi.org/trusted-publishers/adding-a-publisher/...

This means that you literally can't do it manually, you have to rely on one of:

* Github

* Google Cloud

* ActiveState (I'm not familiar with it)

* Github.com (not just github, only that one instance)

Really surprising development, IMO.

burnt-resistor• 2 years ago

It looks a lot like reinventing the wheel, but as an octagon.

ryanisnan• 2 years ago

I think you're being overly critical. When it says that it adds support for Trusted Publishers, it links directly to this page: https://docs.pypi.org/trusted-publishers/.

This page clearly explains how this uses OIDC, and uses GitHub Actions as an example. At no point in my read did I feel like this was shilling me a microsoft product.

guappa23• 2 years ago

Try to run your own gitlab instance and use this to publish. See what happens…

At this moment this only supports github.

doctorpangloss3• 2 years ago

On the one hand, you are totally right, GitHub Actions are the VS Code of automation. People choose them because they are broke, they work well enough, and the average person chooses things based on how it looks. GitHub Actions looks easy, it's cozy, it's comfy, VS Code looks like a code editor, everything has to be cozy and comfy.

On the other hand, considering all of that, you can see why Python would arrive at this design. They are the only other people besides NPM who regularly have supply chain attack problems. They are seemingly completely unopinionated about how to fix their supply chain problems, while being opinionated about the lack of opinions about packaging in general. What is the end goal? Presumably one of the giant media companies, Meta, Google or Microsoft maybe, have to take over the development of runtime and PEP process, but does that even sound good?

LtWorf9• 2 years ago

I think the end goal is to only allow github/google whatever accounts to publish code on Pypi, so importing whatever will be USA-sanctions safe.

The burden to ban russians/koreans/iranians will be on those big companies and pypi will be able to claim they respect the rules without having the resources themselves to ban accounts from sanctioned countries.

So in my opinion, in the future tooling will have some option to check if the software was uploaded via microsoft et al and can be considered unsanctioned in USA.

Or they might just say that's how you upload now and that's it.

LtWorf9• 2 years ago

Funny how I have a talk about the evolution of pypi's security at the upcoming minidebconf this weekend.

I guess I'll have to update my slides :D

But well as a debian developer my advice is to just use debian and completely ignore pypi, so I might be slightly biased.

mistrial9• 2 years ago

except that is impractical at best. The Debian (and Ubuntu) python packaging groups have been overwhelmed since Python2.7. no way at all does the Debian packaging ecosystem reasonably satisfy python packages.. And there is more.. the Debian system does what it must do -- ensure system Python has integrity with major software stacks and the OS use of Python. Super! that is not at all the needs of users of python. As you know, Python has passed Javascript as the most used language on Github. There is tremendous pace and breadth to several important Python uses, with their libraries.

Please reconsider this position with Debian and python packaging.

zahlman6• 2 years ago

Does a .deb file even allow for installing for an arbitrary Python (i.e. not the system installation)? Am I supposed to find them through Apt and hope the Debian version of the package has a `python3` prefix on the name? Is that going to help if there are dependencies outside the Apt system?

ikiris• 2 years ago

I think y’all are being overly aggressive against what is from an outside perspective looks like a mvp targeting the majority use case.

If the dev’s lower in the comments claim that other support is soon to be added doesn’t pan out then sure this is a valid argument, but it feels very premature and ignores the reality of launch expectations of most agile based teams.

immibis3• 2 years ago

And the in-detail explanation is full of references to specific third-party services, like Fulcio and SigStore...

HdS84• 2 years ago

While I understand that, are more deep problem is that their main competitor (gut lab) is too busy to implement the fad of the day. Got labs cicd is already an unholy mess of scripting in yaml, and there is no good way to provide encapsulated pieces of code to gitlab. GitHub has actions which have a defined input and output, gitlab has nothing comparable. That makes it very difficult to provide functionality for gitlab.

riffraff• 2 years ago

Is it really that hard to provide a gitlab workflow?

AFAIU it'd be enough to have a docker image and use that in a standalone workflow which is a few lines of yaml in both GitHub and gitlab, what'd be the problem with that?

myworkinisgood• 2 years ago

CI/CD components and steps.

SethMLarson3• 2 years ago

Congratulations to everyone involved in this work! This is an incredible building block for not just supply-chain security both inside and downstream of the PyPI ecosystem but also for data analysis of open source projects (through strong linkage back to source repositories). Thank you all <3

jonnycomputer• 2 years ago

Supply chain security is very important, and this seems like an important step. Seems absolutely essential that something like the Mozilla foundation, or EFF, or some other open-source friendly entity help provide such a service, instead of corralling users into companies with exploitative business models.

I am in no hurry to be pushed into using Github, Gitlab or whatever else. Programmer's open source code has been monetized by these companies to feed AI LLM beasties, and it's fundamentally objectionable to me. I self-host my code using Gitea for that reason.

Comma2976• 2 years ago

Great, now all that is missing is a decent packaging system for Python.

ris• 2 years ago

I'm not really convinced of the value of such attestations until a second party can reproduce the build themselves on their own hardware.

Putting aside the fact that the mechanisms underpinning Github Actions are a mystery black box, the vast vast vast majority of github workflows are not built in a reproducible way - it's not even something that's encouraged by Github Actions' architecture, which emphasises Actions' container images that are little more than packaged installer scripts that go and download dependencies from random parts of the internet at runtime. An "attestation" makes no guarantee that one of these randomly fetched dependencies hasn't been usurped.

This is not to go into the poor security record of Github Actions' permissions model, which has brought us all a number of "oh shit" moments.

mhils• 2 years ago

Fully reproducible builds would of course be nicer from a security standpoint, but attestations have vastly lower implementation costs and scale much better while still raising the bar meaningfully.

amelius4• 2 years ago

I'm curious what would happen if a maintainer's PC is compromised. Is there any line of defense left at that point?

guappa23• 2 years ago

None.

Developer machine will have ssh keys and github tokens that can be used to push a commit on github, that will be built, signed, and uploaded on pypi.

amelius4• 2 years ago

That sounds like a gigantic attack surface then ...

pabs33• 2 years ago

Your own source code audits (including just watching git logs), and or social code audits using crev.

https://github.com/crev-dev/

pabs33• 2 years ago

Wonder when PyPI will be doing bootstrappable and reproducible builds.

https://bootstrappable.org/ https://reproducible-builds.org/

stavros• 2 years ago

Given that it doesn't do builds, never?

OutOfHere• 2 years ago

Are tools like uv, rye, hatch, etc. going to facilitate this?

HelloNurse5• 2 years ago

They could if there was a way to create an attestation that the tool (and you) can use.

Uptrenda• 2 years ago

Despite crypto getting easier and easier + more main stream. You often don't see it used that often. Even in ((('"```blockchain'"```))) projects (wink emoji; skull emoji; frown; picture of a chart crashing.) I welcome these additions to PyPI. I am a proud Python Chad and always will be.

guappa23• 2 years ago

They even got some public money from Germany's sovreign tech fund to couple uploads with gigantic USA companies.

This is probably deserving a criminal investigation since it appears the funds were probably misused?

Well done guys! Good job!

tzlander• 2 years ago

Thanks for pointing this out! In general, the sovereign tech fund should not fund Python or the PSF, since the administration is U.S. dominated and the whole organization is captured by U.S. corporations.

Unpaid developers from Europe (and Asia) just serve as disposable work horses for the U.S. "elites".

guappa23• 2 years ago

I'm just sad I didn't think to write this earlier and now it won't be seen by many.

kps• 2 years ago

You can count me among those that are suspicious that this is a frog-boiling step, but it doesn't appear to me that STF money went to this, from https://www.sovereign.tech/tech/python-package-index#what-ar...

Maybe there is a case to be made for STF to fund making Codeberg (a German-headquartered organization) one of the PyPI trusted hosts. If Codeberg were supported, that would go a long way to addressing fears. And conversely, if Codeberg can't meet PyPI's bar, that suggests complete commercial capture.

guappa23• 2 years ago

Yes I agree. It'd be nice if codeberg got some funding and could be used to upload stuff in the same way github can.

Then pypi wouldn't just me a means to the end of keeping people on github rather than going to open platforms.

jeroenhd• 2 years ago

From the [docs](https://docs.pypi.org/trusted-publishers/internals/):

Reliability & notability: The effort necessary to integrate with a new Trusted Publisher is not exceptional, but not trivial either. In the interest of making the best use of PyPI's finite resources, we only plan to support platforms that have a reasonable level of usage among PyPI users for publishing. Additionally, we have high standards for overall reliability and security in the operation of a supported Identity Provider: in practice, this means that a home-grown or personal use IdP will not be eligible.

From what I can tell, this means using self-hosted Github/Gitlab/Gitea/whatever instances is explicitly not supported for most projects. I can't really tell what "a reasonable level of usage among PyPI users" means (does the invent.kde.org qualify? gitlab.gnome.org? gitlab.postmarketos.org?) but I feel like this means "Gitlab.com and maybe gitea.com" based on my original reading.

Of course by definition PyPI is already a massive single point of failure, but the focus on a few (corporate) partners to allow secure publishing feels like a mistake to me. I'm not sure what part of the cryptographic verification process restricts this API to the use of a few specific cloud providers.

Ferret7446• 2 years ago

I wonder why they aren't using SLSA attestations. It'd be good to have a single standard than every language having its own format.

woodruffw37• 2 years ago

This standard does support SLSA attestations. We’re enabling upload support for them pretty soon.

cpburns20096• 2 years ago

The root point of this seems to be PyPI does not have the resources to manage user identity, and wants to outsource that component to Github, et al. That sounds fairly reasonable. But why deprecate GPG signatures? The problem with GPG signatures as I understand it is it's difficult to find the associated public key. That's fair. Why not host and allow users to add their public keys to their accounts? Wouldn't that solve the problem?

dale_glass• 2 years ago

GPG is an ancient bit of tech with numerous problems:

* An extremely complex, byzantine packet format with security problems of its own.

* Decades of backwards compatibility, which also harms security.

* Extreme unfriendliness towards automation.

* Way too many features.

* Encouragement of bad security practices like extremely long lived keys.

* Moribund and flawed ecosystem.

Lots of cryptographers agree that PGP has outlived its usefulness and it's time to put it out of its misery.

And really there's little need for GPG when package signing can be done more reliably and with less work without it.

I was a fan of PGP since the early days, but I agree that at this point it's best to abandon it.

cpburns20096• 2 years ago

I'll take your word that GPG is outdated. GPG is just the one that PyPI used to support. I don't particularly care what public key signing suite is used.

ldng• 2 years ago

So, let me get this clear, after PyPA enshittifying the package expérience for more than a decade, and now mess has been clean by uv, the security FUD (and let not be naive here, the power struggle) move onto enshittifying PyPI and make Python user miserable for a decade more ?

PyPI now supports digital attestations

Related Stories