Go's old $GOPATH story for development and dependencies

https://utcc.utoronto.ca/~cks/space/blog/programming/GoTheGopathDevelopmentStory

by ingve • 2 years ago

60 57 2 years ago

utcc.utoronto.ca

stairlane• 2 years ago

I struggle to see the authors point.

His last sentence:

It could have worked, at least in theory, in a world where preserving API compatibility (in a broad sense) is much more common than it clearly is (or isn't) in this one.

Is he implying that using vendor packages is more clear that using version tags in a go.mod? That would certainly be the first I’ve ever heard of someone preferring dependency management via vendor packages over a go.mod & ‘go get’.

Personally I absolutely love the way go manages dependencies. I have way more confidence that my old go programs will run with minimal dependency related issues compared to programs written in something like python.

mdaniel• 2 years ago

There is a small asterisk about "more clear" in that if one vendors the deps, they are guaranteed to be available, and not subject to github outages, which seems to be 99.5% of all go deps in the world

It for sure is easier to reason about a declarative text file than 18 quadrillion .go files in a subdirectory, but using go.mod does come with tradeoffs

Yasuraka• 2 years ago

Go uses $GOPROXY which points to https://proxy.golang.org/ by default with direct sources as fallback, very reliable

siebenmann2• 2 years ago

My badly communicated overall point is that I don't think it's right to say that Go started without any thought about dependency management. Instead, the Go developers had a theory for how it would work (with $GOPATH creating workspaces), but in practice their theory didn't work out (for various reasons). For me, this makes the evolution of Go modules much more interesting, because we can get a window into what didn't work.

(I'm the author of the linked-to entry. I wrote the entry because my impression is that a lot of modern Go programmers don't have this view of pre-module Go, and especially Go when you had to set $GOPATH and it was expected that you'd change it, instead of a default $HOME/go that you used almost all the time.)

quectophoton2• 2 years ago

but in practice their theory didn't work out (for various reasons)

For me, one of the reasons I disliked $GOPATH is that source code was not in $GOPATH; it was in `$GOPATH/src`.

The fact that they are very strongly against relative imports also didn't help, even if I get why they do it.

gwd2• 2 years ago

It's certainly an interesting angle. It might have been more clear if the documentation and how-to's had explicitly used that sort of terminology: e.g., saying that step 1 of creating a project would be to create a new workspace directory for all the dependencies, then create a package directory (with a git tree) inside that.

There are certainly nice things that the current Go module system buys; but one thing I miss is that under the old system, if one of the packages wasn't working the way you expected, the code was right there already; all you had to do was to go to that directory and start editing it, because it was already cloned and ready to go. The current thing where you have to clone the repo locally, add a "go.work" to replace just that package locally to do experimentation, and then remove the "go.work" afterwards isn't terrible, but it's just that extra bit of annoying busy-work.

But being able to downgrade by simply changing the version in go.mod is certainly a big improvement, as is having the hashes to make supply chain attacks more difficult.

dfawcus• 2 years ago

Is he implying that using vendor packages is more clear that using version tags in a go.mod? That would certainly be the first I’ve ever heard of someone preferring dependency management via vendor packages over a go.mod & ‘go get’.

Isn't it the case that Debian builds of go 'packages' (i.e. making a .deb package') are using the pre modules mechanisms? If so, then they would seem to prefer it for some reason.

guilherme-puida• 2 years ago

Isn't it the case that Debian builds of go 'packages' (i.e. making a .deb package') are using the pre modules mechanisms?

As far as I know, yes: https://salsa.debian.org/go-team/packages/dh-golang/-/blob/d...

vetrom• 2 years ago

They prefer it for a number of reasons, for better or worse.

The big reasons I usually call out are:

- packages should be buildable without calling out to any network resources. (this is the primary reason Debian dh-make-golang disables 1.11 style gomod support)

- Debian specifically avoids vendored copies of code. This does create functionality and miantenance challenges, but it also produces a leaner installed code base. One again, for better or worse, this is a standing part of Debian package policy and warts from this particular choice stand out all over, not just golang. (newer Debian python, for example outright disables pip for non-venv python)

There are other nits and warts that become apparent, but I think those all waterfall out from the two points I've called out here.

(The Debian package policy in question: https://www.debian.org/doc/debian-policy/ch-source.html#embe...)

leaveyou3• 2 years ago

sorry for the slight off topic but I still don't understand what is the advantage of having the domain hard coded in the name of the module

go mod init example.com/hello

and also hardcoded in all the import statements

import "example.com/hello"

what if you move the code to other domain ? you need to change the module and all the references to it instead of only the references

I find it quite cumbersome/counter intuitive even after all these years.

ionut-maxim• 2 years ago

I think that the domain is hard coded so you can import the module from any git repo and not depend on a central repository.

_ph_2• 2 years ago

One big disadvantage of using the domain name in the module specification is, that as a package provider, you are kind of permanentely chained to the hosting provider of your choice, e.g. github.com. Moving your package would destroy its "identity".

It is bad enough how much of a monopoly and tie-in github today has (probably the reason Microsoft bought it) and a language environment shouldn't contribute or even aplify that role.

foldr2• 2 years ago

The alternative (as with e.g. npm) is being tied to the hosting provider for the package ecosystem. If you want to, you can use your own domain name for your Go package URL and then link it to a repo hosted elsewhere. There's really no namespace on the internet that you have a more permanent claim on than a domain that you yourself have registered.

treve• 2 years ago

The benefit is that you're not creating a new directory of names, they reuse an existing well established one.

Creating a central authority for names is a lot of work.

Im not that experienced with Go, but I believe it's possible to create a vanity package name on a domain you control. If you want to change hosts, you can just point your domain to something else.

I may be wrong there though!

iainmerrick• 2 years ago

Yes, if your module URL is on a site you control, you can serve a <meta name="go-import" ...> tag to redirect it to your source repo. The module URL is permanent but you can move the repo.

The go-import tag is documented here: https://go.dev/ref/mod#vcs-find

It's a bit more fiddly if your module is part of a monorepo and doesn't live at the root. In that case your go-import tag needs to point to a GOPROXY server. I have a proxy server here: https://github.com/more-please/more-stuff/tree/main/gosub

arccy4• 2 years ago

Indeed not creating a new directory of names is an advantage, plus it also means all the short names are effectively reserved for the standard library, and there's no scramble from developers to squat on all the "good" names.

quectophoton2• 2 years ago

Yep, it's unfortunately a bit hard to find, but it's well explained here:

https://pkg.go.dev/cmd/go#hdr-Remote_import_paths

Also AFAIK there's still no good UX for private packages that require auth.

raverbashing• 2 years ago

I completely agree with this and this really smells of someone thinking about the problem for 3 seconds and moving on

It's the part of Go I find the weirdest

Node/Npn does this best.

sunaookami• 2 years ago

How many times does a module switch the domain? And in that case you can still search+replace over the whole project in one second. Non-issue.

leaveyou3• 2 years ago

I'm a simple man, I experiment a lot, I create a module locally and bam, straight from the beginning I have to decide where I will host this module and very often I don't want to make it public so I only keep it locally on my machine, but often I need to share my module among my several machines (laptop, mini desktop) but I still don't want to share with github publicly and it's annoying that there is no easy way to do this AFAIK. In a better world I would create a "module" in a folder, give it a symbolic name at most (like 'ShinyModule') and share it in various ways; like I could share the folder using samba, ftp, ftps, sftp, https and in the consuming side, you would just import 'ShinyModule' and have a single file per consuming module which says:

'ShinyModule' https://github.com/shinyAuthor/ShinyModule or

'ShinyModule' ../../ShinyModule or

'ShinyModule' \\something\something\ShinyModule

and if there is no 'download' description for an import, the default should be ..\ShinyModule relative to the current/importing module

Joker_vD3• 2 years ago

The company I am currently at has changed the domain name for the inner Gitlab server three times in two years. The older domain's are tentatively supported but they behave differently in respect to authorization so... we had to switch our imports whole-sale, or the builds would just keep breaking for no apparent reason.

arccy4• 2 years ago

That's why there's https://go.dev/ref/mod#vcs-find . Your import path doesn't have to map directly to vcs repos, as long as you can serve an html meta tag to redirect it to where your current repo is:

    <meta name="go-import" content="root-path vcs repo-url">

leaveyou3• 2 years ago

Thanks for the link. When I started learning Go modules that spec was not available (or it was a lot smaller). After a brief look I feel the Go modules spec is longer and more complex than the (original) Go language spec which is quite ironic :-D. I still like Go although it's sad seeing it straying from the 'one true' path of simplicity.

Joker_vD3• 2 years ago

But you can't serve this meta tag if you don't control the URL/domain anymore, and you can't force all of your users to use e.g. an intercepting HTTP proxy for such requests.

CraigJPerry• 2 years ago

An advantage is typo squatting gets eliminated (see js or python worlds for why this is an issue).

If you want to move to another domain the “go mod edit —replace” cli tool is designed for this.

If you want to host somewhere else, or maybe change to a vanity url or something, you can just publish a go-import meta tag and it all just works https://pkg.go.dev/cmd/go#hdr-Remote_import_paths

I don’t write go myself but man did they get a lot of big decisions right. I’d be totally open to writing go in future but I have java so don’t really need it. I envy go’s build and deploy stories though.

siebenmann2• 2 years ago

The obvious advantage of using domain names and in general URLs as the package names is that the Go project doesn't have to run a registry for package names. Running a registry is both a technical and especially a political challenge, as you must deal with contention over names, people trying to recover access to their names, and so on. By using URLs, the Go project was able to avoid touching all of those issues; instead they're the problem of domain name registries, code hosting providers, and so on.

tomphoolery• 2 years ago

As someone who used Go since before v1, I can say that Go's approach to dependency management was basically "we don't want to touch it until we know we have the right idea". At the time, dependency management solutions were pretty primitive to what we have today...there was no caching, and if the central server where you got your dependencies was down, you were shit out of luck. So this was really not a bad decision for the language maintainers if they wanted to keep their sanity and not subject their user base to "legacy on arrival" software. By depending on URLs and using a `$GOPATH`, this problem was solved from a localized perspective...as a developer, I could vendor in packages, not check them into Git, and compile the program. It worked.

However, the biggest problem with depending on packages in Go back then was that it was difficult to communicate to other people on your project the exact version of the dependency that you wanted them to install. For projects in which you were the only developer, this wasn't an issue...but as soon as you started to use Go in a team setting, it became a real blocker toward getting your work done. Go developers needed _some_ way to communicate what version of each package they wanted to install, and a bunch of solutions popped up to help with that. But they were all still bound by the `$GOPATH` constraint and such.

Although it took a lot longer than many predicted, I'm still pretty happy with how Go approached dependency management at the end of the day. Generally, all I have to do is import a dependency from a known URL and my editor/Go modules will take care of the installation work. This is way better than the JS world, in which if I want to just sketch something out I have to actually install the dependencies I need otherwise TypeScript will yell at me. With Go, it all seems to happen automatically.

richbell• 2 years ago

At the time, dependency management solutions were pretty primitive to what we have today...

What has fundamentally changed since then, in your opinion?

User23• 2 years ago

Go’s initial development priorities were ordered for Google’s needs. The described problems weren’t major issues at Google because they store or at least used to store all of their code in a single repo.

azornathogron• 2 years ago

Not only that, but Google doesn't use language-provided build systems anyway because it's important (to enable many other useful dev tools and practices) that everything is built using blaze (the thing externally released as bazel).

So really, Go's decisions about modules, vendoring, source layout, and the "go" command itself are all kind of irrelevant within Google. (The underlying Go compiler is used of course)

p_l• 2 years ago

Golang's approach in the days of GOPATH fit very well with its native environment, which was Plan 9, where mounting remote sources as /n/... wasn't unusual.

Waterluvian• 2 years ago

The one issue I had with Go when poking at it years ago was that it seemed to have fairly strict opinions on directory structure. It felt like if your project wants to use Go it had better be about Go. It was immediately in conflict with the other code I had going on in the library and where on my disk I wanted to put things.

MereInterest2• 2 years ago

Not only that, but those extremely strict opinions on directory structure extend to your entire home directory.

As an example, I wanted to install "gopls", the language server for go. There was a command that could install the dev tools, but it also generated a $HOME/sdk. With such a generic name in my home directory, there's not a chance that in 6 months time, I'd remember what it was for, why it was there, and whether it is safe to delete. No amount of changing $GOROOT could convince it to write into a different directory, nor was the go language server available through any other installer.

evilduck• 2 years ago

Containerizing every Go project helped me mitigate Go's opinions on my home directory but needing to do so in the first place was irritating.

tapirl• 2 years ago

Go has almost no requirements on project directory structure. Could you elaborate more?

fsmv• 2 years ago

This has been fixed with the newer modules system. You do have to name the folders the same as your packages but you can put the folders anywhere you want now.

haiku2077• 2 years ago

That naming requirement is only a requirement _within_ the source code. The folder containing go.mod can have any name.

cesarb• 2 years ago

The whole thing is fairly similar to a materialized Python virtual environment.

The feeling I got about GOPATH, back when I first encountered it, was not that it should be used like a Python virtual environment, where you'd have a separate one for each of your projects. What I understood back then was that you were supposed to set up GOPATH only once for your user, in your shell's startup scripts, and all of your projects would have to live within that directory, in the rigid structure the Go tools dictated. The default for GOPATH added later was just to avoid confusing newbies who aren't used to adding environment variables to their shell startup scripts.

darioush• 2 years ago

The current go dependency management is lacking:

- If you want to make a change in one of your dependencies, you have to fork the repository, and in the forked repository you have to textually rename the package to the new name. This makes for abysmal maintenance and unneeded merge conflicts if you want to maintain parity with upstream code.

- There is the go replace directive, but this does not transitively apply to dependencies of the module that declares the go replace directive.

- If your patch gets into the upstream repository, now you have to undo the forking (again via a large textual rename).

- If a dependency of your code and your code both depend on the same package, you are forced to take one version per binary that gets compiled. This is just plain absurd and leads to situations where you cannot bump the dependencies independently. If you have a tree of these dependencies, you must update each dependency in the order that respects the dependency tree. This sort of defeats the purpose of specifying and locking the dependency version.

- Overall, go was designed for a mono-repo in a company (Google) that does not version their software (everything runs at tip), and it shows in any type of effort that attempts to re-use software in non-trivial fashion, with distributed development that happens at different rates in different repositories.

sunshinekitty• 2 years ago

The last point is simply not true, notably Google does not use the go build or dependency tooling, instead using blaze. Blaze handles the aforementioned issues as well.

Go's old $GOPATH story for development and dependencies

Related Stories