Indie game Haunts unable to compile its Go code

http://forums.thedailywtf.com/forums/t/27755.aspx

by pavel_lishin • 13 years ago

178 193 13 years ago

forums.thedailywtf.com

krakensden• 13 years ago

It's a good point. I've had it as a worry in the back of my mind for a while now, the 'best' solution for now seems to be linking to your own personal forks of projects you depend on.

revscat• 13 years ago

That's a pretty heavy solution.

Svip2• 13 years ago

Not for a video game.

potatolicious• 13 years ago

Sure, but it's also a common solution outside of Go. I write iPhone apps and we have all of our major dependencies forked, because at some point you're going to butt up against the limits of what your lib does. Even just redirecting a git submodule at a later point is a minor brainfuck - so nowadays we just instinctively fork as soon as we decide to pull it in as a dependency.

What I really want to know is why they chose Go in the first place. Go isn't a game development language and it has little to no support, documentation, or just plain old previous community experience in the gaming context. This feels like the classic software development of putting the technology ahead of the product - using a tool because it's cool, rather than using it because it's the best thing for the job.

It's all about shipping, people. Use whatever you need to you can ship what you want, when you want to, at the quality and reliability you desire.

Svip2• 13 years ago

Isn't that usually what people do in developments? They obtain a copy or fork of a library, and only update it at their discretion? This seems like an oversight by the developers rather than Go.

I realise that since Go supports this, it is encouraging to simply do that, but it isn't wise in the long term.

For instance, I've had some issues with Go-SDL, because one of the forks became unreliable. I decided eventually to simply fork another of the forks myself to overcome all these issues, and simply update my fork with commits to the main project whenever they came around.

bradhe• 13 years ago

I totally agree that packaging and dependency management is Go's weak spot, and I've experienced similar issues. That said, I wouldn't blame this on golang per-se. I have run in to the same issues in Ruby, although versioning is a bit better.

Whenever I see a fast-moving dependency I fork it and use my fork, then integrate upstream changes when appropriate. If you have to do that for every damn dependency I can see that being horrible, though...

MetaCosm6• 13 years ago

Why? I can't imagine building a serious product and being dependent on a github repo I don't personally control. It is terror inducing to even consider any real project that would be... even if you are based on a tag or a hash, the author can destroy or rewrite it.

numbsafari• 13 years ago

Agreed. Also, if your dependency graph starts to get unmanageable, it's usually a sign you've made a mistake somewhere (likely by relying on something else that's poorly designed).

If you keep your system simple and don't rely on too many "kitchen sink" frameworks, then keeping your own source tree of your dependencies isn't that much different from maintaining a local repository of the same code in compiled form (with the added benefit of having the original source available).

collinvandyck76• 13 years ago

Go didn't screw anything over. It sounds like poor version control practices. Although the Go team eschews versioning of dependencies, they do so mainly because they are in control of said dependencies. If you're building software that relies on untrusted source code, do yourself a favor and make a copy of it that is known to work well and use that copy in your build environment.

MetaCosm6• 13 years ago

Wait, are you saying that embedding a reference to some other fast moving master could be dangerous? No way!

csense• 13 years ago

> poor version control practices

I agree that, if this problem had been foreseen during development, better version control could have helped.

> Go didn't screw anything over

This is where I disagree. Having a language feature to fetch dependencies from mutable online sources means that you're practically asking for programs to break unpredictably based on external changes. The local cache makes it worse, because every machine will have a slightly different version of the library, and bugs won't be reproducible.

VikingCoder4• 13 years ago

By your argument, no one should use the Google cache of jquery. Or any of these: https://developers.google.com/speed/libraries/devguide#Libra...

Do I understand you correctly?

Lewisham• 13 years ago

> Having a language feature to fetch dependencies from mutable online sources means that you're practically asking for programs to break unpredictably based on external changes.

Go devs always said that Go is designed to solve Google problems in Googley ways. All Google code builds from HEAD, at least it did when I was last there. The reasoning for this is that breaking changes can be discovered ASAP, instead of depending on code several versions back that might be much more painful to upgrade should the need arise.

This is another case of the Google way having a sane reasoning, but perhaps does not work in other cases. I'd personally prefer some versioning system, but I can see why they've done it the way they did.

coolsunglasses3• 13 years ago

>Go didn't screw anything over.

Erm, yes it did. The civilized world (Java, Ruby, Python, Clojure, Scala, Haskell, OCaml) has version numbers in their dependency management. Albeit out-of-band from the source files (pom.xml, gemfile, requirements.txt, project.clj, sbt, .cabal, OPAM version pinning) but it does work.

Hell with Clojure and Leiningen (the standard choice for dependency mgmt) even the language version is a per project dependency ala:

[org.clojure/clojure "1.5.1"]

So you can still build jars that "just work" even if it's some legacy stuff that you haven't updated to the latest Clojure version yet.

That's not to say Clojure would've helped here - it's a game. And one that's aiming for better graphics - JVM isn't a good idea.

But that doesn't excuse Go's poor package management design that decided being facile and hid complexity of the real problems (changes breaking existing code) was more important than working builds.

The decision to make it facile, weak, and fragile was egregious and very out-of-character with the rest of their design decisions. I think Go can be faulted for failing at their own goals on this particular matter.

autarch2• 13 years ago

Actually, the civilized world has version numbers in the source too:

   use Moose 2.08;

That's valid Perl and will blow up if you have Moose 1.02 installed. Of course, it'd be even better if you could be more specific than just "2.08 or greater", but it's still useful as is.

wtbob7• 13 years ago

> Erm, yes it did. The civilized world (Java, Ruby, Python, Clojure, Scala, Haskell, OCaml) has version numbers in their dependency management.

Go has localised dependencies: you package your code with exactly the versions you need.

> So you can still build jars that "just work" even if it's some legacy stuff that you haven't updated to the latest Clojure version yet.

And with a Go project, one would have a complete system which 'just works,' complete with all dependencies, regardless of how old those dependencies actually are.

> But that doesn't excuse Go's poor package management design that decided being facile and hid complexity of the real problems (changes breaking existing code) was more important than working builds.

Go doesn't do what you think it does (probably not your fault: the article is misleading). It only pulls down updates if you tell it to. It sounds like the developer of the original code was doing the wrong thing, not using GOPATH the way it was designed, not checking his entire source tree—including dependencies—into version control. I could be wrong, of course: it's possible that he really did use it properly but discovered a misfeature I've not yet found.

jcromartie2• 13 years ago

No, really, the Go answer to version control of dependencies seems to be "clone the head of the master branch of a Github repo". Blaming someone for using the only thing that seems to be supported is a little harsh.

https://groups.google.com/forum/?fromgroups=#!topic/golang-n...

voidlogic2• 13 years ago

>>the Go answer to version control of dependencies seems to be "clone the head of the master branch of a Github repo"

Yes and it is only the fault of these devs that they did not do that. One of the first things I did when starting a job programming Go full time was go- woah, we need to clone our dependencies so we can manage upgrades. It was just common sense and easy.

buro9• 13 years ago

Or you could fork.

Or you could create a version of your src folder as a github checkout with submodules for each of the things you want to import.

I struggle to see what the issue is. I knew the first time I wrote an import in Go that this was possible, and that as the libraries for Go are relatively young that it is also likely.

My personal preference is just to keep track of changes fairly regularly and not allow much drift. But that's because up until Go1.1 I was following tip, which meant I was already in that habit.

Though it would be nice if ``go get`` could take a revision or branch/tag name.

MetaCosm6• 13 years ago

Yeah, this article made my head hurt. It can be summarized as

"Original author didn't localize dependencies, obviously caused problems."

sanderjd3• 13 years ago

Yep, and the comments can be summarized as

"I don't know anything about Go but I have incredibly strong negative opinions about it based entirely on this article. Oh also, I hate GitHub."

cpleppert• 13 years ago

I'm a little confused about how this would be an issue in the first place. I mean, surely all the changes are checked into a repository and the working code is on the programmers computer right??? So why doesn't he just share the 'working' code with all of the dependencies if he somehow didn't check in his local copies of the dependencies.

I'm a little suspicious when it is claimed that the original author can't get it to compile. That is just so strange and the simplest explanation was that it was never in a great state anyway.

carbocation2• 13 years ago

Go pulls down remote repos/libraries locally. If the original developer has any sort of backup of his system from the time the game was working, he will have all of the working code.

If the original developer does not have that but at least hasn't nuked his src directory, he will have all of the working code (but may need to, e.g., git bisect).

eric9703• 13 years ago

Yeah exactly. This shit is ridiculous.

landr0id2• 13 years ago

The dependency could have easily been avoided by forking the repo which you're using, and using that instead. Obviously the maintainer(s) of the original will be making changes, so wouldn't you want to make sure that if you're distributing the code, you're all on the same version?

> Why does this matter?

Maybe I'm not understanding this, but the dependency issue and the issues in this paragraph are not related. Issues like improper syntax? That's not related.

Sounds like bad planning as far as the dependency stuff goes. For everything else, sounds like a distribution issue.

eknkc• 13 years ago

Let's say I depend on github/A and github/A depends on github/B. If I fork A, that does not solve the problem. Now I need to fork B and modify import paths in A to resolve my fork. Wouldn't this be a mess in time?

I assume, instedad of forking, it might be possible to import a specific tag, or commit hash from a git repo. If it's possible, it'd solve the backward incompatible change problem but I guess the original author can always delete the github repo.

What is a good way to manage dependencies locally?

landr0id2• 13 years ago

I hadn't considered the A/B issue, and yeah that can get messy. Importing a specific commit hash to the repo would be great. As far as managing dependencies locally, I couldn't say since I haven't had much experience with Go, but there exists a package manager called "Go Nuts" (see: http://www.gonuts.io/). While obviously not widely used, it seems to fix this issue. It's almost like rubygems all over again though.

What I'd like to see is a site similar to this, with a similar tool. Instead of having to upload the package to the site though, you could add a `pkg.json` file (or something similar) with application data. Every time this is committed, the site would automatically check to see if the version has changed, and then index that commit hash as a new version.

This almost sounds fun...

ushi• 13 years ago

Hmm, they have issues with dependency handling and 3rd party libraries.

Go doesn't have a gem/pip/cpan/npm/... like ecosystem, instead you import git/mercurial repos directly. This is pretty nice for rapid development and to test things out, but not the right approach for complex long term projects like a game.

The original author of the game missed to include the dependencies into the project and now they are doomed. Put the libs you rely on in your version control, be it as a submodule.

I'm wondering if anybody is working on a package manager for Go libs... That would be great.

dkulchenko3• 13 years ago

> I'm wondering if anybody is working on a package manager for Go libs... That would be great.

@aleksi is: http://www.gonuts.io/

mpyne2• 13 years ago

I think everyone here saying "just fork your dependencies" is missing the idea that it's supposed to be an open-source game.

Managing DLL hell may be acceptable to ship a game, but it's not acceptable to openly develop software.

In this case it wasn't even intended to be open-source, which may be why the dev went the route he did, but doing so essentially closed the door on the later option of easily letting the community take over on code development.

There's a reason people codified semantic versioning. Projects like gstreamer take it even further and make the major version part of the library name, even allowing for co-installability of old and new (e.g. gstreamer-0.10 and gstreamer-1.0 can both be installed and used).

stcredzero2• 13 years ago

> Managing DLL hell may be acceptable to ship a game, but it's not acceptable to openly develop software.

Why not? Having something that compiles and runs is probably a bit of a boon to the development effort.

mpyne2• 13 years ago

Yes, that's my point.

It's not acceptable to merely "manage" dependency hell as the net result is still unmanageable at OSS scale. You have to essentially eliminate the interplay of "which exact version do I have" with your code, which is why things like semver and soversions (on .ld.so's) try to accomplish.

Being able to have an unknown user download your code and the listed dependencies and have a repeatable build is invaluable for development.

ominous_prime3• 13 years ago

I totally agree, as long as you can trust your dependencies.

Go's dependency management will even allow a library to have multiple versions depending on the version of Go being used to build (either a branch or tag matching the Go version). This isn't a major feature right now, since Go is backwards compatible within major versions, but could be much more important when there's both go1.x and go2.x in production.

The problem here arises when your remote library can't maintain a stable master, either because the maintainers don't care, don't know how, or the project is new and still in flux. The onus is then on you to incorporate that code into your project.

People need to think of this system as giving all dependency developers commit access to your project (which is nearly is). Would you just let the world check-in code willy-nilly, or are you going to review what's going in?

ChuckMcM3• 13 years ago

Wow, that's awkward.

One of the things I spent a bit of time on when Java was being developed was class management. We had an ongoing discussion about embedding versioning information in class files, creating signatures that could be checked at runtime, and just winging it. Bill Joy was convinced that it was possible to give the class loader the ability to statically analyze a class and determine whether or not it would work with the code that was trying to invoke it. This was when I was first exposed to the idea of creating a digest type 'signature' for a method/class invocation that would encode semantics.

The end result was 'just wing it' of course. This was expedient but didn't advance the state of the art :-)

Go's implementation seems a bit more dangerous. Not the least of which being drive by malware injection attacks on abandoned/poorly administered code repositories. It seems in principle to be no more or less dangerous than something like "apt-get install libcairo-dev ; make" but that would only be true if there was a trusted repository system.

Can someone from Go comment on the ability to create and manage trusted repositories? Should we look for a go 'distro' to build with?

carbocation2• 13 years ago

> It seems in principle to be no more or less dangerous than something like "apt-get install libcairo-dev ; make" but that would only be true if there was a trusted repository system.

Not that this is really a response one way or another, but given the way that Go currently works, you effectively place the same amount of trust in each individual library maintainer as you do with the collective trusted repository maintainers in other approaches.

If someone were to decide to start curating repositories and forking/maintaining them, people could choose to trust that individual. So the possibility for a trusted repository exists, but as far as I know nobody has fully stepped up to create such a thing at this point.

derleth• 13 years ago

> Bill Joy was convinced that it was possible to give the class loader the ability to statically analyze a class and determine whether or not it would work with the code that was trying to invoke it.

Here, 'work' must mean 'has the methods and other stuff the client code is looking for with the types the client code expects', except the Java type system isn't really expressive enough to do a very good job of that, especially if there are any generics involved (such that a lot of stuff has been hammered down to type Object). But Java didn't have generics that long ago, did it? Something like Eiffel, with design-by-contract and automated contract checking, would be able to make more use of the concept.

Pretty trivial, especially if you can rely on the compiler to do dead code elimination before checking to see what the client code is expecting. I don't know if javac (as opposed to the JIT) does dead code elimination, though.

ChuckMcM3• 13 years ago

The actual explicit definition of work was 'not throw a ClassCompatibility Exception' because the JVM was convinced that the method it was about to call was the same semantically as the method had been when the calling class was compiled.

The idea was to create a 'semantic message digest' (I don't know if such a thing exists or not) which could be computed at class load time and checked/validated at method invocation time. The canonical case for the talking point was you have a class Foo with methods doA, doB, and doC if you change doB incompatibly but the client only calls doA and doC the class is still 'compatible' (in a call contract context) with the class even though the class is 'different' in an incompatible way. Guy Steele and Bill came up with a pretty interesting list with things like methods added, methods missing, more parameters, fewer parameters, additional fields, fewer fields, additional constructors, Etc. All the ways you might mutate a Class and yet for a class of callers it would still work just fine.

If one can do this, another interesting feature is better capability type systems. Those systems have to deal with changes that may change the capability represented by a method even though it is still semantically compatible.

yepguy• 13 years ago

When you install a package with `apt-get`, that package can potentially change your system in any way it pleases. Running `go get` will download and compile code, but there is no security risk until you run or use that code without checking it first.

reidrac• 13 years ago

You're comparing things that are different. When you install a package with apt-get:

- it comes from a repository that has been signed by the distributor.

- the package has been tested and has a good level of integration with the system (ie different distributions may package things differently).

- depending on the distribution, there may be a commitment to API/ABI stability (ie. Debian backports security fixes most of the time).

That said... running "go get" is very convenient if you control the code repository where you're fetching the code, but it's not a good idea when using 3rd party software that is in active development.

EDIT: formatting

mineo• 13 years ago

If you don't trust your package manager & distribution supplied packages (read: your whole operating system), the security risk between an 'apt-get' and 'go get' is most likely not the most important thing to worry about.

lcampbell• 13 years ago

> Can someone from Go comment on the ability to create and manage trusted repositories?

You basically just need to throw up a HTTP-accessible HTML page with some <meta> tags that describe how to fetch the actual code with a variety of source control systems. Anyone can create their own trusted repositories.

[docs] http://golang.org/cmd/go/#hdr-Remote_import_path_syntax

ominous_prime3• 13 years ago

I'm not making any judgements on the project itself, or its dependencies, but I see a lot of people just use code off github (or butbucket, or launchpad) with no regard for the quality of said code. Just because it's on github doesn't mean it's good code, or the maintainer has any idea what they're doing. In general, if I can't trust a project to maintain a stable master branch (or a Go version tag) following best practice guidelines, I'm not going to trust that code in my project without a full review. In that case, I'd fork it (or clone it into another VCS) and work from there.

There are some issues with the way the Go tooling handles dependencies, but I think the minimalist way in which it does is very useful, and easy to build upon. Go uses a distributed dependency system, and you can't give it the same level of trust you give to centralized repositories, like one does with gems, Pypi, NPM, CPAN, and so on.

xtx23• 13 years ago

why would they use go, a language aimed at building scalable web services to build a game is puzzling...I am not sure if people who made that decision actually knows what they are doing. Many open source libs and system-level linux libraries can be installed from git or github repos... that is not news at all. Can't see why people are fighting over such things. O.O

jcromartie2• 13 years ago

Looking at their Kickstarter page, they really could have used anything.

etchalon2• 13 years ago

I don't understand the issue, or why it's such a show stopper.

If the original programmer is using git, he should have a complete history of his code. They know the date (or thereabout) when the last version of the code "worked". Revert down to the a commit around that date.

For each dependency, do the same. Pull the dependencies repo down, revert to the last commit that was around the date of the working version, and use a new fork of that repo to get things back to the working state.

One by one, move the dependencies forward, resolving bugs as you go.

up_and_up• 13 years ago

What about branching and then doing:

git reset --hard commit-hash

where commit-hash is the last known working commit.

Just do that with all the repos around the date it last worked. If there are alot of commits between the last working version and HEAD it is easier to do this then reverting each commit.

thepicard3• 13 years ago

I was also mystified as to how not a single one of them did this after alleged months of effort.

stdbrouw4• 13 years ago

Maybe they did and it didn't work? You're giving them very little credit.

etchalon2• 13 years ago

That wouldn't make any sense. Either the code worked at some point, or it didn't.

biot• 13 years ago

Another possibility is that the developer is using this as a convenient excuse to abandon a project where they bit off more than they could chew. $28K isn't a lot of money having already put in months of work and with potentially many months of work remaining.

The acid test will be if someone fixes their issue and says "Here you go, it's compiling again. Now just make sure you handle dependencies like this to keep everything working." Will the project be picked up again?

bluehex3• 13 years ago

Unless he lost his local copy he wouldn't even need to guess what commit around the date he was at in each repo. The info is stored in the ref-log of the local git repo, (as well as the dependencies' local repos).

So you could rewind all the repos to where they were at a particular date/time with: `git co HEAD@{2012-11-26 18:30:00}` in each repo. If you run that same command in the project repo, and the dependency repos you should have a snapshot of the developers local checkout at that time.

HEAD could be a branch name or whatever ref. You can specify the time like ref@{2.days.3.hours.25.minutes.ago} as well, and these time specifiers work wherever you can specify a revision for example `git log -n1 master@{yesterday}`

see: https://www.kernel.org/pub/software/scm/git/docs/gitrevision...

qznc• 13 years ago

The problem is dependency which changed with any of the git-tracked content changing.

My interpretation:

1. Original author writes a program using Lua. Author downloads the latest version of Lua into his system and includes and links it. Note that Lua is not checked into his version control system nor does he note the used version.

2. Lua upstream does a backwards incompatible change

3. Author extends his local (old) copy of Lua

Result: An external dev will find no combination of Lua version and program revision which compiles. The original author seems to have lost his local modified Lua version.

You can probably spot various mistakes an experience software developer would avoid. The interesting point is that Go seems to encourage Step 1 nonetheless?

EvanYou• 13 years ago

Seems to me lame excuses from the original developer who made horrible version control decisions and then abandoned ship.

TazeTSchnitzel• 13 years ago

It's not even a Go problem, you can be equally stupid by putting "package": "*" in npm's package.json instead of, say, "package": "1.2"

Udo2• 13 years ago

How is this Go's fault? Just because you can pull code automatically from a remote repository doesn't mean you always should. It seems to me that keeping the entirety of the code necessary to compile a project in a local build directory is a very good idea if you don't want to compile against moving targets.

nilsbunger• 13 years ago

or at least make sure a remote Github reference is for a specific checkout!

dkulchenko3• 13 years ago

IIRC, Go does not support importing anything but HEAD (yet, at least).

Udo2• 13 years ago

Exactly! But trusting a remote repo to contain an essential part required to build the project is just extremely short-sighted, no matter what language is being used.

mercurial6• 13 years ago

Github is not a substitute for a packaging system.

tiredofcareer6• 13 years ago

You are correct. Go's remote imports are dangerous for long-term project maintenance, but the feature is still useful for quick, throwaway projects.

Go badly needs two things:

1. A best practice that dictates that you never import from remote repositories in production, long-term code; the feature is fine for one-offs and experimentation, but the article summarizes only one way this style of work can lead you in to a maintenance world of pain. What happens if the repo you're importing from Github is deleted? What do you do for fresh clones? You're going to end up changing the URL anyway. I feel the Go community has kind of glossed over this (and I like Go).

2. An equivalent of CPAN or PyPI, which you could then import from in concert with a tool to manage those dependencies, a la:

    import (
        "cgan/video/graphics/opengl"
    )

This model works for CPAN, PyPI, and so on for a reason, and that reason is avoiding several of the dependency/merge hells that remote repos can create. CPAN provides Perl a lot, such as distributed testing in a variety of environments. I personally think such a thing is necessary for long-term maintenance of any software project that utilizes third-party libraries. This is one of Google's oversights in Go, because they have an (obviously) different take on third-party code. Here's a good case:

Developer A checks out the code clean. Five minutes later, developer B checks out the code clean. In both cases, your "go get" bootstrap script fetches two different commits, because in that five minutes, upstream committed a bug. Developer B cannot build or, worse, can build but has several tests fail for unknown reasons or, even worse, the program no longer functions properly. Developer A has none of those problems. In a world with a CPAN-like, developer B can see that he has 0.9.1 and developer A has 0.9.0, developer B can commit "foo/bar: =0.9.0" to the project's dependency file, then everybody else doesn't suffer the same fate. In the current world, you're either massaging your local fork to keep the new commit out, or any other troublesome, non-scalable approach to this.

Building large software projects against a repository never works. You need tested, versioned, cut releases to build against, not master HEAD. It only takes one bad upstream commit to entirely torpedo your build, and you've now completely removed the ability to qualify a new library version against the rest of your code base. Other people are suggesting "well, maintain your own forks," so you're basically moving merge hell from one place to another. I, personally, have better things to do with my time; I've seen (Java) apps with dozens of dependencies before, and keeping dozens of repositories remotely stable for a team of people will rapidly turn into multiple full-time jobs. Do you want to hire two maintenance monkeys[0] to constantly keep your build green by massaging upstream repositories, or do you want to hire two feature developers? Exactly.

I've started writing a CPAN-like for Go a couple times but I'm always held back by these threads:

https://groups.google.com/forum/?fromgroups=#!topic/golang-n...

The second one highlighting how difficult Go is to package as a language -- my personal opinion is treat Go just like C and distribute binary libraries in libpackage, then the source in libpackage-src. If one message in that thread is true and binaries refuse to compile with different version compilers, I'm troubled about Go long-term.

[0]: I'm not calling all build engineers maintenance monkeys. I'm saying the hypothetical job we just created is a monkey job. I love you, build engineers, you keep me green.

encoderer• 13 years ago

Hopefully the project maintainer will keep a go1 branch in their repo that only contains 'released' versions of a consistent API.

And in cases where that's not obviously the case, my habit is just to fork the repo and "go get" my own fork. I can then update it as needed.

Not saying these are optimal solutions, but they are solutions. I do personally wish I could "go get" a specific commit or tag.

MetaCosm6• 13 years ago

I agree 100% with your first point. It should be spelled out.

Your second point I am less sold on. Transitive dependencies (pkgA imports pkgB@v1 but my code need pkgB@v2 which is incompatible with v1) are the thing of nightmares in large systems development, which is what Go is designed for... that lack of versioned imports wasn't an oversight, it is a feature.

Centralized repos are centralized points of failure, and only as good as they are well managed. NPM versus CPAN if you will. Any serious project will localize dependencies, even if they are in CPAN, you never know when CPAN will be down or other unforeseen things might happen.

skybrian4• 13 years ago

You're right that syncing directly with the net is a problem. However, the problem is basically developer education. Go doesn't work like other languages and the consequences aren't well-documented. If you do it right, the process works fine; it's basically how Google works internally.

The basic idea is to commit everything under $GOPATH/src so that you never need to touch the net to reproduce your build. After running "go install" you should check in the third-party code locally, just like any other commit.

Then updating a new third-party library to sync with their trunk is like any other commit: run "go install", test your app, and then commit locally if it works. If it doesn't work, don't commit that version; either wait for them to fix it, sync to the last known good version, or patch it yourself.

If you aren't committing your dependencies then you're doing it wrong.

wereHamster• 13 years ago

Maintain your own forks of the libraries you need. Commit your changes there, and also submit upstream. Pull upstream changes back to your fork when you are ready. You have this problem with any language or tool that uses github directly as the registry.

npsimons2• 13 years ago

Maintain your own forks of the libraries you need.

Exactly this. This is just one reason I really like to "git clone --mirror" onto my server, then clone from that, and work from there.

azernik• 13 years ago

It sounds like it's even worse than that - the developer was maintaining a fork of the upstream projects, but didn't put it in version control. That is, he/she used Go's github import mechanism to get a copy of a project onto their own machine, and edited the locally cached copy. So of course, when other people tried to build, the libraries didn't match up.

The real issue here seems to be that this was a half-finished project, at a very early stage of development, and the developer probably never intended their local scratch copy to be used in production [http://www.joystiq.com/2012/10/24/haunts-anatomy-of-a-kickst...]

KirinDave• 13 years ago

From now on, remember this anecdote every time you curse at Maven for making dependency management a huge effing hassle unless you publish publicly on version drops to an official repo.

Go's sin here is being permissive. And it's true that clearly the game dev team just... didn't do things right. They were pretty much on a collision course with failure by managing their project's code the way they did. But Go allowed them to get much further along than they otherwise would have.

lnanek2• 13 years ago

They claim there is nothing about Go that makes it bad for games, but there certainly is. There are no large, mature game engines for it. If someone was hiring me to write a basic mobile game I could do with a big engine like Cocos2D, and said I couldn't use the engine, I'd probably multiply my time estimate by five.

I've written complex OpenGL ES games from the ground up before, and its a long slog, especially at the end where you you are trying to crank it up to 60 FPS and doing tons of work rewriting it to be high performance.

stcredzero2• 13 years ago

This indie project goofed up, so now they're (probably unintentionally) hurting Go and Kickstarter with more bad publicity. It's going to be information that's hard for the public to properly understand, so it will probably hurt that understanding as well.

eric9703• 13 years ago

This might sound harsh, but...

Maintaining software is hard.

Blaming your own incompetence on a library or an entire language is easy.

I've been guilty of this at times, but when you get down to it, it's your job as a software developer to understand how your software works and how to maintain it well. This doesn't just include the code you write; it includes your tools, third party libraries, and the internals of the language (VM, compiler, etc.). Its not easy, but it's part of the job.

On a more positive note, versioning packages in Go is already possible and is starting to gain momentum. Check out http://www.gonuts.io/

stdbrouw4• 13 years ago

> it's your job as a software developer to understand how your software works and how to maintain it well

And it's the job of language (and language ecosystem) developers to make this as straightforward as possible. It's perfectly possible to blame both the author and the language.

ricardobeat• 13 years ago

It's bewildering to see a project that "can't compile" after having $70k+ poured into it. This article sums up this clusterfuck: http://www.joystiq.com/2012/10/24/haunts-anatomy-of-a-kickst...

Back on topic, maybe Go should take a page from node's NPM: simple versioned tarballs with isolated dependencies.

djvu92• 13 years ago

IMO this is some sort of clueless scapegoating. Dependency could be an issue in software development but only for large projects. A project with $28k funding and probably 1 or 2 developers, seriously? And no details like lines of code, libraries being used, etc makes it hardly plausible.

cthackers2• 13 years ago

I agree. I believe is scapegoating too. The guy that wrote the mail had no clue about anything technical, and the other guys that help him explain, didn't really understood what the programmer did and perhaps wasn't even a Go dev. And it seems that the original developer just left and doesn't want anything to do with the game anymore. So...they blame it on the language and 3th party hobby libraries from github because they can't set up the project right. That shit happens on any language even with inhouse built libraries if you put unskilled people to to the job.

iand3• 13 years ago

I offered to help out on the Haunts game when they had their first round of problems. They had lost the developer who had chosen Go and so were left with no way to continue development. I wrote and offered my help and apparently so did a large number of other people because we were all sent a long questionnaire to fill in with our skills, experience and what we could offer the project. It just seemed like a big barrier to cross simply to offer my volunteer services, so I didn't bother completing it.

In retrospect they were probably being protective of their IPR, but erred on the side of being overprotective and pushing away a few folk that could have helped.

iand3• 13 years ago

It seems that Haunts uses Makefiles to wrap the Go tools so there is some non-standard complexity there. See https://github.com/losinggeneration/haunts/blob/type_fixes/M... which was linked from the 7 month old issue https://mrg-trac.sourcerepo.com/mrg_Haunts/ticket/34

eric9703• 13 years ago

I love how people jumped all over Go on that thread, without having any idea of what they are talking about.

Gotta love the Internets.

orangethirty2• 13 years ago

Question: Could we give Open Source git repos permalinks?

stdbrouw4• 13 years ago

What do you think the hashes are for? :-)

orangethirty2• 13 years ago

Well, I know. But seems people need to be told about it in a different manner. Not every product fails due to the product itself. Sometimes the failure is due to how it is referred to. If we had something like permalink this repo, then people who be moved to used that.

meshko2• 13 years ago

Wow, how is this Go's fault? Shouldn't it be obvious that one has to be very careful when patching upstream libraries and maintain a forked repository and/or submit patches upstream?

prodigal_erik2• 13 years ago

Did they have a fork with local changes? The way I read it, they just did

  import "github.com/go-gl/opengl/gl"

and then at some point the go-gl project (which they don't manage) made incompatible changes that broke their build. Presumably there is some version of go-gl they could build against (which happened to be cached on their developer's machine), but they don't know which one because there's no version or tag in that string.

meshko2• 13 years ago

But this is what software development is all about anyways... it's like complaining that they have to type a lot in order to get _anything_ working. My whining detector went off.

cthackers2• 13 years ago

I bet tomorrow morning half of people here will download and backup their 3th party libraries :). And perhaps even burn them on a CD (just to be safe)

iand3• 13 years ago

I used go get on the source code and the errors are caused by dependencies on 2 C libraries: opengl and lua. Basically the Go bindings for the C libraries are out of date and/or unmaintained.

Sample output:

    # github.com/runningwild/glop/gos
    /usr/bin/ld: cannot find -lglop
    collect2: error: ld returned 1 exit status
    # github.com/runningwild/opengl/gl
    gl.go:142: cannot use _Ctype_GLint(mapsize) (type C.GLint) as type C.GLsizei in function argument
    gl.go:147: cannot use _Ctype_GLint(mapsize) (type C.GLint) as type C.GLsizei in function argument
    gl.go:152: cannot use _Ctype_GLint(mapsize) (type C.GLint) as type C.GLsizei in function argument
    gl.go:158: cannot use _Ctype_GLenum(internalformat) (type C.GLenum) as type C.GLint in function argument
    gl.go:164: cannot use _Ctype_GLenum(internalformat) (type C.GLenum) as type C.GLint in function argument
    gl.go:170: cannot use _Ctype_GLenum(internalformat) (type C.GLenum) as type C.GLint in function argument



    /usr/bin/ld: Warning: size of symbol `luaX_tokens' changed from 8 in $WORK/github.com/xenith-studios/golua/_obj/lcode.o to 256 in $WORK/github.com/xenith-studios/golua/_obj/llex.o
    /usr/bin/ld: Warning: size of symbol `luaT_typenames' changed from 8 in $WORK/github.com/xenith-studios/golua/_obj/lapi.o to 88 in $WORK/github.com/xenith-studios/golua/_obj/ltm.o
    # github.com/xenith-studios/golua
    /usr/bin/ld: Warning: size of symbol `luaX_tokens' changed from 8 in $WORK/github.com/xenith-studios/golua/_obj/lcode.o to 256 in $WORK/github.com/xenith-studios/golua/_obj/llex.o
    /usr/bin/ld: Warning: size of symbol `luaT_typenames' changed from 8 in $WORK/github.com/xenith-studios/golua/_obj/lapi.o to 88 in $WORK/github.com/xenith-studios/golua/_obj/ltm.o

Seems like a familiar problem that every similar language encounters when using bindings to native code, not just a Go issue.

pygy_• 13 years ago

The Lua issue is weird. The bindings are maintained (last update one month ago), and the repo provides its own Lua source files (but the API of minor versions of Lua is stable, so it shouldn't matter).

They should file an issue on GitHub.

https://github.com/xenith-studios/golua/issues

djvu92• 13 years ago

FYI source code at https://github.com/mobrulesgames It seems to me the developers must have been working really hard and the failure is due to that the size of the project is way too large for the two person team. It is unavoidable and the dependency issue is just a symptom.

babuskov2• 13 years ago

+1. Although one might argue that failure is not really due to the size of the project, that's just the side effect. Failure was created when they decided to use Go without understanding how it works.

Starting to build something big with tools you are not yet very familiar with is always risky.

lunixbochs• 13 years ago

That GitHub page tells me they did clone all of the dependency repos six months when they were trying to recover it.

If the game was finished and working but does not compile anymore, that sounds like a trivial thing for a hardworking developer to fix in a pretty short amount of time.

I've seen much worse when porting games across platforms. This doesn't look even close to "beyond repair".

wtbob7• 13 years ago

I've only been writing Go for a short while, but my experience so far is that 'go get' only pulls down updated versions of dependencies if you tell it to, with -u.

The developer should have set up a proper Go tree for his project, prepended it to GOPATH, run 'go get DEPENDENCY1…' (which would have put the complete current source of that dependency into his src directory) and then checked his project tree into his VCS (adhering, of course, to the licenses of his dependencies). Anyone who downloaded his code would have gotten the compatible version of all dependencies from the VCS. Then when he felt it was time to upgrade the dependency, he should have committed his work (and branched in something like git), then run 'go get -u DEPENDENCY1…' to fetch the latest version, which—yes—would break his code's compilation. The next step would be to fix his code to work with the new version, then commit that.

It sounds like the guy was doing something developing his project in his main Go tree, not putting his dependencies into version control, blindly updating dependencies and so forth.

But possibly I missed something.

bencollier49• 13 years ago

From the Kickstarter trailer: "If we don't get this game out by Halloween, something will claim our souls".

hpcorona• 13 years ago

Wait, so, they were expecting to develop agains an X version, and then after release, when they release the source it will compile easily for all it's users with zero dependencies problems?

I don't see how bashing Go would help these guys out of the hole they are in, same would have happened with any other programming language in a project that uses fast-moving dependencies...

They were making a game, they could have just forked the projects they needed, and build from that, that's the way to do these kind of projects, you stick on a revision, and when there is an update on a dependency, the team should agree to make the upgrade changing all the required source code.

For what they say, the main developer had no idea that the repositories were changing a lot, and he never updated them, so everything was going thru smoothly... I don't see the need to bash Go, neither the main programmer, i just believe he needed more experience to handle the project correctly.

What i don't like is their attitude "nobody can fix this Go garbage code because Go is all wrong"...

codygman• 13 years ago

I'm willing to bet that any Go programmer worth their salt not tired of the project can get it running if it were simply build dependency version issues.

I'll take a stab at it this weekend (if I have time).

For the curious, here is their repo: https://github.com/runningwild/haunts

uvTwitch• 13 years ago

It's a poor workman who blames his tools.

frio2• 13 years ago

This sounds like a problem the community can resolve. Check http://golang.org/cmd/go/#hdr-Remote_import_path_syntax -- anyone can create a Go "repo" by using a <meta> tag in an HTML page.

I don't think it'd be difficult to create a CGAN/GoPI/Gobal/gojars/etc. that provided versioning.

bluehex3• 13 years ago

That's a good idea. The limitation being that your service can redirect go get to a different repository url, but go will still fetch that repository and check out what it thinks is the best branch for that version of go: one with the name of the current go version, or master. So you'd have to play some tricks to keep a full clone-able repo with master locked at some commit for each version of each library, rather than just one repo per library with tags. I guess there are some ways to do that where the clones share the objects so you aren't wasting space, though so not insurmountable.

frio2• 13 years ago

Space usage doesn't worry me too much. If a "package" is just a set of files, then storing them in some sort of compressed form (.tar.gz) and simply un-tarring them and making the API mimic a git repo with one revision should be fairly straightforward.

entropy_• 13 years ago

This is ridiculous, there's a correct way to handle this and it's called git submodules. Instead of using go get for non-trivial long-term projects you should just be adding the dependency as a submodule in your git repo and including the files directly. This way you can have your submodule pointing to a specific commit in the upstream repo, you can update it when you want to/need to and all this information is kept track of in version control.

Also, if you want even more control(ie, you want to modify the library you're using in some way), fork it on github and include your fork as a submodule. That way you can patch bugs easily(and send the upstream a pull request) without having to wait for upstream to fix things you know how to fix yourself.

bradjs• 13 years ago

I'm a Go developer using Go for commercial projects in production.

I can tell that the article and its comments is absolutely misleading. Seems that the developer(s) are trying to justify the lack of professionalism and blame language/tools/someone else. In the article I see total misunderstanding of how the Go ecosystem works, strange decisions, obvious failures during development and weird situations like 'original dev cannot compile the code anymore'.

I think an engineer should have a decent understanding of the tools/concepts/... BEFORE starting to use them. Not AFTER some problems occur.

p0nce• 13 years ago

Death march project fail, blame the tools.

bbayer• 13 years ago

I really struggle to understand what is going on here. Wrong usage of a feature isn't tools fault. I saw lots of projects that messed up things with gcc and Make. So can we consider gcc and Make are both broken. Developer should know pros and cons of feature that is about to be used.

Using an unofficial release or development version of a library is always risk. It is not language dependent. Things can always be broken if you don't follow release notes on an actively developed project.

smegel• 13 years ago

This is ridiculous. I have come across numerous projects written in half the languages under the sun that were impossible to build and/or run for various reasons, mainly because the original developer had so much stuff in his environment that it depended on. At least Go has urls to original source repos which is a whole lot better than mysterious binary blobs that noone knows what the hell they do let alone where they came from (but were most likely brittle chunks of Fortran77). And if you modify someone elses code and it gets out of sync...well that's a problem but its not Go's fault.

scrapcode• 13 years ago

Go fuck itself?

qompiler• 13 years ago

This project was haunted from the get-go.

Indie game Haunts unable to compile its Go code

Related Stories