Rust Solved Dependency Hell

https://stephencoakley.com/2019/04/24/how-rust-solved-dependency-hell

by weinzierl • 7 years ago

41 18 7 years ago

I thought npm did this first.

sansnomme• 7 years ago

They did. Reduced it from a NP-class problem (IIRC dependencies constraint solving required a SAT solver) to mere tree traversal.

ameliaquining2• 7 years ago

I think it's still NP-complete because you can still specify arbitrary dependency constraints. (Of course, just because it's NP-complete doesn't mean it's not tractable in practice for the kinds of instances that come up in real life.)

lugg• 7 years ago

Kept trying to find issue with this but the title says it all.

_nhynes2• 7 years ago

Basically. Cargo is so good that the only things I could possibly wish for are specifying binary dependencies and easier use of Cargo as a library (and maybe a more robust `patch` section, but there's always `paths` in .cargo/config).

IloveHN84• 7 years ago

I hope that also conan.io will support it for C++

mshroyer• 7 years ago

(Not a Rustacean yet.)

Are there any circumstances in which this approach can have undesirable behavior, for example if the dependency has to deal with some global state?

This seems like the kind of thing that could create difficult-to-understand problems if Cargo gives you two copies of a dependency only under specific and rare circumstances. (But I'm happy to be proven wrong, and I'm sure the Rust folks have thought this through more carefully than I have.)

_nhynes2• 7 years ago

Nah. The Rust compiler's warnings are wonderful, as ever: https://github.com/rust-lang/rust/blob/master/src/librustc/i...

zaarn• 7 years ago

That is actually a fairly recent addition. I wasted like 2 days of my time trying to find the bug until someone pointed out it could be differing versions of the crate.

bjorn2k• 7 years ago

If you rely on global state, this global state is also mangled to a different symbol, this might be undesirable. For example, if you use a logger which relies on a extern defined variable. Your singleton is duplicated. Also all your code is duplicated.

I agree that how rust does it is a very nice solution.

babyloneleven• 7 years ago

The rayon crate has a global state: the entire program must use the same thread pool, even if different crates depend on different rayon versions.

It uses a Cargo feature meant to be used by native libraries: you typically can link them only once (or else the build will fail when it finds repeated symbols), so Cargo must ensure only one is present on the entire dependency graph. And IIRC Rayon depends on an auxiliary crate that fakes a native lib and provides the thread pool.

If this seems very hacky and convoluted, it's because it is.

deepsun• 7 years ago

Hasn't Java also solved it using Modules?

moring• 7 years ago

Java has a very similar solution to the problem using OSGi. It uses class loaders to encapsulate sets of classes to isolate them and also uses the class loader for disambiguation.

Compare the solutions: "It seems there's four basic components incorporated in a mangled symbol name:"

* "The fully qualified name of the symbol." -- same in Java

* "Generic type parameters." -- missing in Java because of type erasure

* "The name of the crate containing the symbol." -- represented by the class loader in Java

* "An arbitrary “disambiguator” string" -- not present in Java, and I'm not sure why this is needed at all

It even comes with almost the same downsides: "This solution isn't perfect though:"

* "...we can't pass objects around between different versions of a library" -- similar with OSGi (you can pass objects using the opaque "Object" type), but you might be able to have libraries agree on a common interface, defined by an "API library", which has the same version for both and allows to pass objects.

* "Any static variables or global state will be duplicated" -- same with OSGi * "Our binary size increases necessarily" -- same with OSGi

(edit: formatting)

wst_• 7 years ago

Even without OSGi there are plugins allowing so called package names shadowing and, as a consequence, for multiple versions of the same dependency coexist in the same classpath. Rust, though, have this as a part of the language. This is a whole new level of user (or developer in that case) experience.

nwmcsween• 7 years ago

Version strings are sort of a hack, while it does communicate API breakage, etc it doesn't communicate what. Given a language with an effect system and powerful enough type system you could verify for input x result is y and store it as a hashed sym alias, etc

iamcreasy• 7 years ago

I thought you can have different version of the same library for different parts of the project in Java, but didn't know about the non deterministic behavior about which one to use. I wonder how Gradle handles this situation.

panpanna• 7 years ago

Yeah, but rust also created it's own kind of dependency hell.

Need random numbers? Then you need this create. Is it first party or written by some random dude? Hard to tell.

leoh• 7 years ago

The author missed another option to resolve the dependency issue — to run the root dependency (and its transitive dependencies) in its own classpath.

Rust Solved Dependency Hell

Related Stories