Should you use upper bound version constraints?

https://iscinumpy.dev/post/bound-version-constraints/

by BerislavLopac • 2 years ago

46 51 2 years ago

dannyz• 2 years ago

I agree, I think upper bound constraints go against what is commonly accepted and used in the Python ecosystem. What I try to do on my projects now is to always have a nightly CI test step, in theory if an updated package breaks my package it will be caught fairly quickly

bruce343434• 2 years ago

This whole problem is because in python you can not specify interfaces and say "I need a function called f that takes an x of type T and returns a U", so instead you encode that in an indirect way, like "I know version x.y.z works so I'll just require that".

Any other way risks runtime errors. And to people about to mention types in python: those are also checked at runtime.

People keep using these hyper dynamic languages and then running into these robustness issues and scaling limitations brought on by their very dynamism. It makes me mad and sad.

teaearlgraycold• 2 years ago

Just use TypeScript when possible.

317070• 2 years ago

There are static type checkers, though: https://github.com/google/pytype

So you can specify interfaces (protocols) and check them in your installation process.

leni5362• 2 years ago

Function interfaces are better, but not necessarily sufficient. Semantics of functions can also change in a breaking way.

LegionMammal978• 2 years ago

Since when has static typing eliminated backward-compatibility problems in interfaces? Not all visible runtime behavior can be encoded in the type system, unless you're using a proof assistant or formal-verification system.

AlotOfReading• 2 years ago

Even formal verification misses visible behaviors, the line is just a little farther down. Correct runtime visible behavior can also include side effects like the amount of time taken (e.g. constant time cryptography), or even the amount of heat generated [0]. You're not going to encode everything you could possibly want in a type system without modeling the entire universe, so draw an opinionated line somewhere reasonable and be happy with it.

[0] https://news.ycombinator.com/item?id=39751509#39761349

hot_gril• 2 years ago

Dynamic typing can make importing libraries riskier, but the benefits can outweigh the costs. Also, it's not hard to just not break APIs (unless you're node-redis), and you should have tests anyway if you really care.

oivey3• 2 years ago

It makes me pretty mad and sad that people think static languages solve this problem pretty much at all. If you do, I have a version of liblzma for you to install. If you do, do you release versions of your libraries without version numbers because the compiler will catch any mistakes?

VS1999• 2 years ago

It solves the issue of finding out a function signature changed at compile time instead of runtime, which is infinitely better. The real answer is that serious software developers don't just leave their packages on auto-update and rarely, if ever, update dependencies unless there's a good reason.

o11c• 2 years ago

Theoretically static languages don't solve this problem, but in practice, programmers writing packages in a static language don't gratuitously break their API every release or so. Which seems far too common in Python-land.

cardanome• 2 years ago

And to people about to mention types in python: those are also checked at runtime.

They are not checked at runtime at all. Type declarations are only used for static analyzing tools and not by the runtime.

So types are checked BEFORE runtime by the tooling just like they would be in TypeScript or any other language that offers gradual typing.

Yes, the dynamic nature of Python does make type safety and certain performance optimizations very difficult but then again it is the dynamic nature that allows for the high productivity of the language. A static language would be far less ergonomic to use for the typical prototyping and explorative programming done in Python.

int_19h3• 2 years ago

A static language would be far less ergonomic to use for the typical prototyping and explorative programming done in Python.

A static language without type inference, sure. But that's not the only option.

OCaml, for example, will infer object types for you based on what methods are called with what kinds of arguments inside the body.

flakes• 2 years ago

They are not checked at runtime at all. Type declarations are only used for static analyzing tools and not by the runtime.

This is the common usecase, but types certainly are used at runtime by many libraries. Frameworks like FastAPI use the type annotations to declare dependency injection which is resolved during application startup. In other cases like Pydantic, they are used to determine marshalling/unmarshalling strategies.

cqqxo4zV46cp• 2 years ago

What!? A function’s signature does not completely describe its behaviour. This doesn’t remotely address the problem. Have your preferences all you want, but this is blatantly a case of being blinded by some silly programming language culture war.

burntsushi• 2 years ago

No. It's a problem because you can only have one version of any given package in your dependency tree. You can't have `foo 2` and `foo 3` in your dependency tree. Without that limitation, there is a release valve of sorts where you can incur two different semver incompatible releases of the same package in your dependency tree in exchange for a working build. The hope is that it would be transitory state until all of your dependencies migrate.

Rust, for example, has precisely this same problem, except that it is limited to public dependencies. For example, if `serde 2` were ever to be published, then there would likely be a period of immense pain where, effectively, everyone needs to migrate all at once. Even though `serde 1` and `serde 2` can both appear in the same dependency tree (unlike in Python), because it is a public dependency, everyone needs to be using the same version of the library or else the `Serialize` trait from `serde 1` will be considered distinct from the `Serialize` trait (or whatever) in `serde 2`.

But if I, say published a `regex 2.0.0` tomorrow, then folks could migrate at their leisure. The only downside is that you'd have `regex 1` and `regex 2` in your dependency tree. Potentially for a long time until everyone migrated over. But your build would still work because it is uncommon for `regex` to be a public dependency.

(Rust does have the semver trick[1] available to it as another release valve of sorts.)

This problem is definitely not because of missing interfaces or whatever.

[1] https://github.com/dtolnay/semver-trick

Chris_Newton3• 2 years ago

It's a problem because you can only have one version of any given package in your dependency tree. You can't have `foo 2` and `foo 3` in your dependency tree.

That does seem to be the fundamental problem with the Python model of dependency management.

If your dependencies have transitive dependencies of their own but your dependency model is a tree and everything is clearly namespaced/versioned, you might end up with multiple versions of the same package installed, but at least they won’t conflict.

If your dependency model is flat but each dependency bakes in its own transitive dependencies so they’re hidden from the rest of the system, for example via static linking, again you might end up with multiple versions of the same package (or some part of it) installed, but again they won’t conflict.

But if your dependency model is flat and each dependency can require specific versions of its transitive dependencies to be installed as peers, you fundamentally can’t avoid the potential for unresolvable conflicts.

A pragmatic improvement in the third case is, as others have suggested, to replace the SemVer-following mypackage 1.x.y and mypackage 2.x.y with separate top-level packages mypackage1 x.y and mypackage2 x.y. Now you have reintroduced namespaces and you can install mypackage1 and mypackage2 together as peers without conflict. Moreover, if increasing x and y faithfully represent minor and point releases, always using the latest versions of mypackage1 and mypackage2 should normally satisfy any other packages that depend on them, however many there are.

Of course it doesn’t always work like that in practice. However, at least the problem is now reduced to manually adjusting versions to resolve conflicts where a package didn’t match its versions to its behaviour properly and/or Hyrum’s Law is relevant, which is probably much less work than before.

eesmith• 2 years ago

I need a function which, given a graph, returns the graph diameter as an integer.

I need another function which, given a graph, uses a faster but approximate method to return the graph diameter as an integer.

I need a third function which, given a graph, returns the graph radius as an integer.

All three of these functions have an identical type signature.

Oh, now I need something which takes a regex pattern string and a haystack string, and returns 1 if the pattern is found in the haystack, otherwise 0.

And the regex "|" pattern must match longest first, not left-right.

And it needs to support "verbose" mode, which allows comments.

And it supports backreference matches.

How do you express that type?

Now I need to numerically integrate some arbitrary function "f" in the range 0.0 to 1.0. Which of the many numeric integration methods should I use which prevents runtime issues like being unable to converge?

leni5362• 2 years ago

Capping at major version level is just common sense, IMO. It almost shouldn't be even possible to leave the major version out when declaring package dependencies. A new version might as well be an entirely unrelated library/application from the viewpoint of compatibility, so it might as well be part of the name of the package. Not unlike how the python major version is part of the name of the interpreter.

Major versions going EOL'd and unmaintained is unfortunate, but that's not a purely technical problem. Releasing a new major version and breaking compatibility with existing users is as much a social decision as a technical one.

I'm not sold on the "semver doesn't work anyway" angle here either, although I admit that it's not perfect.

quotemstr4• 2 years ago

Right. Semver makes no sense. A new major version is just a new package. Model it as one. Want the new package to conflict with the old one? Model the conflict explicitly. Major version numbering is total nonsense that everyone buys into for some reason.

eyelidlessness3• 2 years ago

Model the conflict explicitly.

If you reject relying on major versioning as the mechanism of detecting that conflict (as you declare it “nonsense”), how do you go about “modeling” the conflict. Like you say, a major new version is a new package. What is the model for that, apart from the thing designating it as that? The API? The behavior of your calls to that API? The infinite unknown scope of internal changes not exposed directly in the API?

I’m not trying to dismiss your comment, I sincerely don’t understand what you’re suggesting people do.

svieira• 2 years ago

If you haven't seen the "SemVer rant" in Spec-ulation you're missing out: https://youtu.be/oyLBGkS5ICk?si=ZL7MXObP7sWk7JaZ&t=1798

croemer3• 2 years ago

Not for Python where it screws you. This article is about Python - the title might have benefitted from that being mentioned.

saila• 2 years ago

The general rule I use is that libraries should not specify upper bounds on dependencies but applications should.

I use Poetry for all my projects, but I agree that it exacerbates the issue somewhat with its default npm-style version syntax.

jbmsf• 2 years ago

The most sane thing I've found is:

- Pin all application versions - Don't pin or set upper bounds in libraries. Lower bounds may work. - Use automation to continuously upgrade and test new versions of everything

If you just pin, you fall behind and eventually it becomes expensive to catch up. If you don't pin, you lose repeatability. If you don't automate, the upgrade work doesn't happen reliably.

ijustlovemath2• 2 years ago

What's your automated upgrade test setup?

globular-toast3• 2 years ago

This is what I do:

1. Use pip-tools to generate a pinned requirements list. This is used to build artifacts like docker images, installer bundles etc. All pins are upgraded periodically (manually). Any version changes are inspected (this requires "knowing" your dependencies a bit and which ones might cause problems). Finally the resulting artifacts are manually tested.

2. An automated build of master every 24 hours from completely unpinned versions. That is installing the package fresh from pyproject.toml. The automated test suite will reveal any upcoming breakage that would happen if we continue without upper bounds on certain packages. If there is breakage we decide whether to accommodate the new breaking dependencies or constrain them with an upper bound (perhaps recording a tech debt).

stouset• 2 years ago

I just keep everything up to date as often as possible.

It’s rarely a pain in the ass. When it is, it’s because we’ve chosen dependencies poorly and that’s a good signal to move on to something else. Occasionally major versions require a little more time and effort, but just eat the cost and do it.

What’s painful is updating rarely and calcifying. Do the hard thing often and it stops being hard.

kazinator• 2 years ago

GNU programs have been doing this for decades w.r.t. Autoconf and Automake versions (which you run into when you need to "boostrap" them to work on them rather than just build them as a downstream user).

croemer3• 2 years ago

I struggled to refind this post when searching on hn.algolia.com so I'll write a comment that'll help others find it: This very long article describes the pitfalls of capping dependencies in Python libraries. It's specific about Python, it discusses the bad precedent that poetry is setting.

croemer3• 2 years ago

This was previously posted by same user in 2022: https://news.ycombinator.com/item?id=29507681

And by another user 54 days ago: https://news.ycombinator.com/item?id=39486552

classified• 2 years ago

Of course you should. But only when necessary.

Should you use upper bound version constraints?

Related Stories