notatallshaw

Their infrastructure (pip) getting worse and worse

Hi, I'm a pip maintainer, can you please report any issue you think is happening in recent versions of pip: https://github.com/pypa/pip/

I particularly work on performance and stability and think we've been making big strides over the last two years

getting everyone to install private venv's for every app, leading to missing security updates, because updates just break everything.

venv's stop your OS tooling from breaking, Linux distro maintainers pushed Python packaging to not allow installing into the system packages by default, and they weren't wrong.

All these other problems are solved with proper use of a lock file (which are now standardized, and you can generate and install from them in the latest version of pip), and/or use a project tool (uv, poetry, etc.).

Running pip install lightning is all that is needed to activate

FYI, pip added cooldowns in 26.1:

  * https://discuss.python.org/t/announcement-pip-26-1-release/107108
  * https://ichard26.github.io/blog/2026/04/whats-new-in-pip-26.1/

  * CLI: pip install --uploaded-prior-to=P1D ...
  * Env Var: PIP_UPLOADED_PRIOR_TO=P1D pip install ...
  * Config: pip config set global.uploaded-prior-to P1D

The note at the end of the post indicates the user asked Claude to review their own chat logs. It's impossible to tell if Claude used or built a a performance harness or just wrote those numbers based on vibes.

Pip maintainer here, to do this in pip (26.0+) now you have to manually calculate the date, e.g. --uploaded-prior-to="$(date -u -d '3 days ago' '+%Y-%m-%dT%H:%M:%SZ')"

In pip 26.1 (release scheduled for April 2026), it will support the day ISO-8601 duration format, which uv also supports, so you will be able to do --uploaded-prior-to=P3D, or via env vars or config files, as all pip options can be set in either.

The threat actor was sophisticated enough to spam GitHub issues with dozens of different accounts. I imagine they could completely overwhelm PyPI with unauthenticated reports.

I had no idea how to contact anyone related to PyPI

https://pypi.org/security/:

If you've identified a security issue with a project hosted on PyPI Login to your PyPI account, then visit the project's page on PyPI. At the bottom of the sidebar, click Report project as malware.

As a pip maintainer I don't think that's really true. The resolver in both pip and uv are fundamentally sequential and single threaded, you can't really queue up or split out jobs.

What uv does is parallelize the final download of packages after resolution, and batch pre-fetch metadata during resolution. I don't think these benefit from async, due to their batch nature classic multi-threaded download pools are probably the better solution, but I could be wrong!

Experiments have been done on the former in pip and didn't find much/any improvement in CPython, this may change in free threaded CPython. For the latter we currently don't have the information from the resolver to extract a range of possible metadata versions we could pre-range, I am working on this but it requires new APIs in packaging (the Python library) and changes to the resolver, and again we will need to benchmark to see if adding pre-fetching actually improves things.

pixi offloads PyPI ecosystem stuff to uv, but pixi is conda first. The team were actually the first to build a Rust based Python package resolver (rip), but after uv was released they migrated to uv's resolver (Python package resolvers are hard and a lot of work to build and must be tested against the whole ecosystem).

On a widely used open source project I maintain I've been seeing PRs in the last month that are a little off (look okayish but are trivial or trying to solve problems in weird ways), and then when I look at their account they started opening PRs within the last few weeks, and have opened hundreds of PRs spread over hundreds of repositories.

My understand is Astral's focus for ty has been on making a good experience for common issues, whereas they plan for very high compliance but difficult or rare edge cases aren't are prioritized.

Compliance suite numbers are biased towards edge cases and not the common path because that's where a lot of the tests need to be added.

My advise is to see how each type checker runs against your own codebase and if the output/performance is something you are happy with.

AI is somewhat helpful but I'm not interested in a company finding a way for me to pay to do my volunteer OSS work. GitHub Copilot offers a permanent free subscription for OSS maintainers.

I previously ignored a free offer when Claude reached out to me as an open source maintainer as it was a glorified free trial. I hope this one continues beyond the listed 6 months, I am not interested in a glorified free trial and if it requires entering credit card details I won't be signing up.

I found the diagrams on the Wikipedia pages help build an intuitive understanding of each step: https://en.wikipedia.org/wiki/Conflict-driven_clause_learnin...

Also, the pubgrub-rs guide I find has a gentle ramp in introducing complexity: https://pubgrub-rs-guide.pages.dev/internals/intro

I assume lock and dependency files are in the training data, so predicting version number tokens have high probabilities associated with them.

As a pip maintainer I primarily think about resolution and resolver performance. I got involved when pip introduced it's current resolver around late 2020, and got my first PR landed in mid 2021.

I recently became a packaging maintainer, from working on fixing edge case behavior around specifiers and prerelease versions.

When I did some recent profiling I noticed that A LOT of time was being spent in packaging, largely parsing version strings. I found a few places in pip and packaging that reduced the number of Version objects being created, Henry really ran with the idea of improving performance and made big improvements. I'm excited for this to be vendored in pip 26.0 coming out at the end of January.

If anyone is interested the next big improvement for pip is likely to implement a real CDCL (Conflict-Driven Clause Learning) resolver algorithm, like uv's use of pubgrub-rs. That said, I do this in my spare time so it may be a year or more before I make any real traction on implementing that.

At one company I worked at, we had a system where each deploy got its own folder, and we'd update a symlink to point to the active one. It worked, but it was all manual, all custom, and all fragile.

The first time I saw this I thought it was one of the most elegant solutions I'd ever seen working in technology. Safe to deploy the files, atomic switch over per machine, and trivial to rollback.

It may have been manual, but I'd worked with a deployment processes that involved manually copying files to dozens of boxes and following 10 to 20 step process of manual commands on each box. Even when I first got to use automated deployment tooling in the company I worked at it was fragile, opaque and a configuration nightmare, built primarily for OS installation of new servers and being forced to work with applications.

We're getting there https://pip.pypa.io/en/stable/cli/pip_lock/ !

Pip has been a flag bearer for Python packaging standards for some time now, so that alternatives can implement standards rather than copy behavior. So first a lock file standard had to be agreed upon which finally happened this year: https://peps.python.org/pep-0751/

Now it's a matter of a maintainer, who are currently all volunteers donating their spare time, to fully implement support. Progress is happening but it is a little slow because of this.

Unfortunately that was posted 1 month before the Faster CPython project was disbanded by Microsoft, so I imagine things have slowed.

Python has about 40 keywords, I say I would regularly use about 30, and irregularly use about another 5. Hardly seems like a "junkyard".

Further, this lack of first class support for lazy importing has spawned multiple CPython forks that implement their own lazy importing or a modified version of the prior rejected PEP 690. Reducing the real world need for forks seems worth the price of one keyword.

I had been hoping someone would introduce the non-virtualenv package management solution that every single other language has where there's a dependency list and version requirements (including of the language itself) in a manifest file (go.mod, package.json, etc) and everything happens in the context of that directory alone without shell shenanigans.

Isn't that exactly a pyproject.toml via the the uv add/sync/run interface? What is that missing that you need?

You let people make the mistake and have the library throw an exception if they do that, not through type checking but just through something eventually calling a method that doesn't exist.

Exceptions or crashes would be annoying, but yes, are manageable, although try telling that to new users of the language that their code doesn't work because they didn't understand the transitive dependency tree of their install and it automatically vendored different versions of a library for different dependencies, and how did they not know that from some random exception occurring in a dependency.

But as I explain in my example, the real problem is that one version of the library reads the data in a different layout from the other, so instead you end of with subtle data errors. Now your code is working but your getting the wrong output, good luck debugging that.

I wouldn't mind a codebase where numpy objects created by dependency B can't be shared directly with dependency A without me first running some kind of conversion function on them

Given there's no compiler to enforce this check, and Python is dynamic language, I don't see how you implement that without some complicated object provenance feature, making every single object larger and every use of that object (calling with it, calling it, assigning it to an attribute, assigning an attribute to it) impose an expensive runtime check.

But maybe I'm missing something obvious.

I agree, but as a type checker it is a subjective choice, whether to be explicit and not make assumptions or whether to infer from certain certain patterns as probably correct and not be noisy to the user. Very glad to see they plan to fix this.

I assume in your example if you update the foo declaration to the following it solves the complaint:

    foo: list[str] = []

  * What can I infer from an empty collection declaration?
  * What do I allow further down the code to update inferences further up the code?

A lot of the tools you list here are composable, mypy checks type hints, black formats code, because of the purpose and design ethos of those two tools they would never think to merge.

So which is it that you want, to just reach for one tool or have tools that have specific design goals and then have to reach for many tools in a full workflow?

FWIW Astral's long term goal appears to be to just reach for one tool, hence why you can now do "uv format".

Hmm. Presumably mypy and pyrefly use the same ones, but then I don't understand why pyrefly is complaining and mypy isn't:

…where is it even puling "tuple[Any]" from…

Perhaps it's a bug in pyrefly, perhaps mypy or pyrefly is able to infer something about the types that the other isn't. I would strongly suggest checking their issues page, and if not seeing a report already report it yourself.

While there is an ongoing push to more consistently document the typing spec: https://typing.python.org/. It does not actually cover all the things you can infer from type hints, and different type hint checkers have decided to take different design choices compared to mypy and will produce different errors even in the most ideal situation.

This is one of the reasons why I am waiting for these libraries to mature a little more.

There is one part of Python that I consider a design flaw when it comes to packaging: the sys.modules global dictionary means it's not at all easy in Python to install two versions of the same package at the same time. This makes it really tricky if you have dependency A and dependency B both of which themselves require different versions of dependency C.

But it solves the problem that if A and B both depend on C the user can pass an object from A to B that was created by C without worrying about it breaking.

In less abstract terms, let's say numpy one day changed it's internal representation of an array, so if one version of numpy read an array of a different version of numpy it would crash or worse read it but misinterpret it. Now if I have one data science library produces numpy arrays and another visualization library that takes numpy arrays, I can be confident that only one version of numpy is installed and the visualization library isn't going to misinterpret the data from the data because it is using a different version of numpy.

This stability of installed versions have allowed entire ecosystems build around core dependencies in a way that would be tricky without that. I would therefore not consider it a design flaw.

The standard library does not directly include type hints, they are stored in typeshed: https://github.com/python/typeshed

You can take a look yourself if you think some of them are wrong: https://github.com/python/typeshed/tree/main/stdlib/asyncio

The advantage is type hints can be fixed without needing to release a new version of Python. The disadvantage is there's a lot of code in the standard library that doesn't really consider how to represent it with type hints, and it can be really tricky and not always possible.

I'm surprised to see so many people moving to pyrefly, ty, and zuban so quickly. I was going to wait until some time in 2026 to see which has matured to the point I find useful, I guess some users really find existing solutions actually unworkable.

There is, the web. The web distributes code directly to end users at a much larger scale. To distribute the bandwidth costs, the web is federated: to depend on a script you refer to its url, and whoever hosts this url foots the bill.

But the Web is notorious for the problems I listed, you end up with standards around not following standards. It leaves almost all the responsibility on the client tool (browser or whatever) to do validation to stop malicious sites, name squatting, accepting and "fixing" poorly constructed metadata etc.

Deno is a Javascript implementation for the backend that attempts to mimic this pattern (it later introduced a more npm-like centralized repository, but afaik it's optional). Deno is of course less popular than Python, but its url-centered model can really scale imo.

I was not familiar with Deno, I've done some shallow reading on this now and it's certainly interesting. I don't know enough about the JavaScript world to make a comment on the pros or cons.

But I don't think can work for Python, as transitive dependencies would immediately conflict as soon as dependencies required a different version of the same transitive dependency. And the guarantee of Python packaging is you only have a single version of a library installed in an environment, while it can cause some dependency solver headache, it also solves a lot of problems as it makes it safe to pass around objects.

Taking PyPI as a central place of packages, it is known that their bandwidth bill would be $1.8+M per month (https://dustingram.com/articles/2021/04/14/powering-the-pyth...) were it not for Fastly giving them a 100% discount.

Are there any reliable decentralized package distribution systems operating at within 2 orders of magnitude of that scale? How do they handle administrative issues such as malicious packages or name squatting? Standards updates? Enforcement of correct metadata? And all the other common things package indexes need to handle.

I'm clearly skeptical, but would be very interested in any real world success stories.

Seems like this is a fairly recent change, for Pnpm at least, https://socket.dev/blog/pnpm-10-0-0-blocks-lifecycle-scripts...

What has been the community reaction? Has allowing scripts been scalable for users? Or could it be described as people blindly copying and pasting allow commands?

I am involved in Python packaging discussions and there is a pre-proposal (not at PEP stage yet) at the moment for "wheel variants" that involves a plugin architecture, a contentious point is whether to download and run the plugins by default. I'd like to find parallels in other language communities to learn from.