Experimental Dependency Vendoring in Go 1.5

https://groups.google.com/forum/#!msg/golang-dev/74zjMON9glU/4lWCRDCRZg0J

by NateDad • 11 years ago

174 45 11 years ago

groups.google.com

wumbernang2• 11 years ago

That's really nice.

This is a royal PITA with .net and nuget and I've get to find a half decent solution that allows you to actually have the source rather than a bunch of symbol-less dll dependencies or a ridiculous build stack.

romanovcode• 11 years ago

Since MVC6/VS2015 beta you can actually get the source instead of dlls. However your build times will increase dramatically so it's a bad idea unless very needed.

wumbernang2• 11 years ago

Interesting - thanks for pointing it out. I was completely unaware of that. Going to go and play with it on my win10 test box now :)

NateDad2• 11 years ago

I really like this proposal. It's basically the best of all current dependency tools - you get reproducibility without needing external configuration files, without hugely long vendor import statements, and still supporting raw go-get.

AYBABTME• 11 years ago

This is pretty much what we've been doing at DO, but we were dealing with GOPATH manipulation to achieve it. By having `GOPATH="third_party/:docode/"`, `go get` pulls packages into `third_party` automatically, and `go install` finds our binaries in `docode/src/services/foosvc/cmd/foosvcd`.

Migrating to use `vendor/` instead of `third_party/` will be trivial, and not having to modify GOPATH will be nice.

Having a GO_VENDOR=1 env var on top of the `go get -vendor` feature flag would be nice, so that the feature works with minimal disruption.

technoweenie• 11 years ago

How do you handle a custom GOPATH like that when the current repository needs to be in there too? We've tried manually symlinking the repo dir into the custom '$GOPATH/src/github.com/user/repo' dir, but it feels really hacky.

anezvigin• 11 years ago

You can separate multiple paths with a colon.

lstoll• 11 years ago

We just keep all our go code in one repo.

vruiz2• 11 years ago

This is nice. Finally it seems like we are going somewhere. The call for proposals had not yielded anything groundbreaking so far. Tools like gb are nice but as Russ says it's not gogettable. This seems like a good compromise.

lobster_johnson4• 11 years ago

Am I correct in thinking that this proposal only lays the foundation for other tools to actually manage the vendor folder?

In other words, you would still want something like godep or gb, because manually copying dependencies into a folder (or using git submodules, ugh), and then tracking their versions through their lifecycle, isn't a workable solution in most situations.

This is clearly an improvement over "go get", but I don't understand why Go is so insistent that dependency management is something for the community to solve, and not something that should be provided by Go itself.

Go is so opinionated about so many things, and yet so little interested in having a strong opinion here. The Unix philosophy is great, to be sure, but the Unix philosophy is about increasing modularity through simplicity, and this problem is about modularity.

vruiz2• 11 years ago

I don't understand why Go is so insistent that dependency management is something for the community to solve, and not something that should be provided by Go itself.

They don't want to try solving a problem they don't really have. As RSC says:

the community understands and encounters these problems more than we on the Go team at Google do.

lobster_johnson4• 11 years ago

I find it worrying that Go is seemingly entirely stewarded by a company whose technical workflow is so different, to the extent that it curtails technical decisions. It's designing for the lowest common denominator in an equation that emphasizes Google.

Even if your company doesn't need or want Bundler/Cargo-style dependency management doesn't mean that you are intellectually unfit to design out a good solution together with the community. Dogfooding is great and all, but it's actually possible to design technical solutions that you won't personally use.

aikah2• 11 years ago

They don't want to try solving a problem they don't really have.

yet Dart has an official package manager ,which is called pub.

So the Go team don't want to solve this problem because they don't have this problem , but the Dart team wants to solve this problem even though they don't have this problem? Do you know what is the name for that? it's not technicality is is called POLITICS.

EDIT: and both teams work at Google.

enneff5• 11 years ago

Am I correct in thinking that this proposal only lays the foundation for other tools to actually manage the vendor folder?

Yes.

I don't understand why Go is so insistent that dependency management is something for the community to solve

Well, they solved it, and now we are beginning to adopt their solution. Seems to be working.

lobster_johnson4• 11 years ago

Given that you still need an external tool not provided by Go, I'm not sure it's "solved" until the community rallies around a single tool, like Ruby did with RubyGems and later Bundler.

rsc2• 11 years ago

Is the criticism here really that we have declined to express a strong opinion about something we don't know a lot about?

ngrilly3• 11 years ago

A quick comment to tell you that I really really appreciate how you solve problems step-by-step, without rushing. The improvement you proposed for supporting vendoring in Go 1.5 is great.

twblalock• 11 years ago

I don't understand why Go is so insistent that dependency management is something for the community to solve

The expectation that every new language should come with a dependency management ecosystem is a bit questionable. A lot of major languages' dependency management solutions were created by the community:

Java: Maven, Gradle

Python: PIP

Ruby: Gems

Node.js: NPM

Some of these have been adopted by the official language maintainers, but as far as I know, they originated in the community.

coldtea• 11 years ago

The problem is that Go implements a half-arsed way of dependency management (go get) and enforces things like GOPATH that clash with community created solutions.

0xdeadbeefbabe• 11 years ago

A vendors B and C, but C vendors D and E. According to this proposal do you lay out the files like this: (A (vendor (B C D E)))?

Edit: I guess HN tree markup is not working :)

jooon3• 11 years ago

Yes. That would work. However, I also believe this will work: (A (vendor (B (C (vendor (D E)))))).

https://groups.google.com/d/msg/golang-dev/74zjMON9glU/LjrVU...

https://groups.google.com/d/msg/golang-dev/74zjMON9glU/0e7M7...

https://groups.google.com/d/msg/golang-dev/74zjMON9glU/qGzsS...

jooon3• 11 years ago

and if you have both: (A (vendor (B (C (vendor (D E))) D E)))

C would pick D E in its C/vendor before D E in A/vendor before D E in $GOPATH/src

ps. I have edited these comments about 5 times to get the parentheses right. :)

drewolson3• 11 years ago

Am I understanding correctly that this proposal again side steps / ignores pinning dependencies for libraries? To me, this is the biggest current problem in the ecosystem and, as a library author, this doesn't solve any of my issues. I hope I misread somehow.

enneff5• 11 years ago

With this scheme libraries can vendor their dependencies, if necessary.

drewolson3• 11 years ago

Can you reference the part of the proposal that states this? I'm reading the following points:

$GOPATH/src/foo will be imported as foo by non-main package $GOPATH/src/mypkg/p.
$GOPATH/src/mypkg/external/foo will be imported as foo by a non-main package p anywhere, when p is being built as a dependency of command $GOPATH/src/mypkg/c.

It seems to me this is saying that libraries will use external packages only when built as a dependency of a main package. Again, this means library authors are out of luck.

groks• 11 years ago

You're pushing the Go ecosystem in this direction:

    https://spot.livejournal.com/312320.html

Lethalman• 11 years ago

How we solved it in Nix: http://lethalman.blogspot.it/2015/02/developing-in-golang-wi...

We don't need go get or any other tool, and we have about 174 packaged libraries in our repository.

ngrilly3• 11 years ago

This is a great and necessary improvement. Being able to vendor dependencies without having to rewrite import paths is like having the best of both worlds.

The proposed mechanism looks similar to what Node.js does with the node_modules directories:

https://nodejs.org/api/modules.html#modules_loading_from_nod...

Experimental Dependency Vendoring in Go 1.5

Related Stories