GitLab Annex: Large Binaries with Git

https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/

by jeremiep • 11 years ago

47 24 11 years ago

about.gitlab.com

jobvandervoort3• 11 years ago

GitLab engineer here, let me know if you have any questions.

We're quite excited about GitLab Annex and curious to hear what people think about it.

michaelmior• 11 years ago

Duplicate of https://news.ycombinator.com/item?id=9063705

alimoeeny• 11 years ago

I am trying to understand why putting large binaries under source control is a good idea in the first place. Unless you have a way to make sense of the diffs this will not help right? and to to make sense of the diffs, you need to understand the structure/format of the binary at least after you have the diff, or maybe even to make a useful diff, right?

barkingcat• 11 years ago

Diffing is just one small part of version control. Keeping binary files in version control is the number one step away from messy file-name-based versioning.

Many design studios are forced to use project1.ver1.psd project1.ver2.psd project1.ver3.psd, etc and so on in order to version their files. Single psd files can be on the order of high hundreds of megabytes to low gigabytes for high resolution ready-for-press files.

Not being able to diff the files is not a problem from an organisational point of view. Of course, in an ideal world there would be diffing of large binaries in a way that makes sense, but thinking there's no use in versioning binaries is very short sighted.

jeremiep• 11 years ago

This exactly. We keep our game's asset in SVN at the moment only for the convenience of versioning. Artists are still locking files to ensure no conflicts are happening (and therefore no merges are needed).

It's much easier to naively do a repository checkout/update than manually detecting changes (or rolling your own solution using rsync or similar).

Especially when considering the game has over 100Gb of raw assets. SVN, Git or Perforce might not be the best tools for such a task but it works great.

jewel• 11 years ago

You're right that you can't get a diff of a large binary (or even a small one). It's still useful to know when the file changed and who changed it.

For example, imaging you work at a video production company where you have videos on each page of your website. (Let's also imagine that you're not using vimeo or youtube to host them.) You might make a branch of the home page with a different video, and you want to coordinate changes across pages.

git-annex is amazing for this use case, as you don't have to actually have a local copy of videos that you're no longer using.

noselasd• 11 years ago

Where would you put them (honest question) ? If you're housing a large part of assets, be it large 3D models, CAD models, music or the like that you're developing, you'd want some place to at least store, version, comment, track them as you're developing and changing that content.

rmc• 11 years ago

I keep my photos in git-annex. It means I have a backup of my photos. If I edit one, the old version is still there. I can "fork" the directories and do some experimental stuff, and switch back to the original version if I need to. Git annex, like git, uses content based hashes for names. The content is preserved. I can copy files to/from remote git annex repos. Easy backup and redundancy.

deathanatos2• 11 years ago

I've always belived that source control was for source. Source is any piece of data created by a human being, which includes things such as large binaries, such as psd files (mentioned in a sibling). This belief excludes, however, large binaries output by a build system: the build script is the source (and should get versioned), the output, not so much. (The output should likely get stored somewhere, however. We keep our build scripts in git, and binaries in S3, for example.)

DougN7• 11 years ago

It seems people disagree with me, but checking in the source, the binary DLLs, and debug symbols and tagging them all is fantastic. If we get a crash dump from the field, a quick checkout gives us exactly what we need to diagnose the problem. Everything in one place and perfectly synced.

I'm not even sure it's possible for a rebuild to produce exactly the same binary down to the byte (internal versions, timestamps, etc). And even if it could, it would require guaranteeing the build system hadn't been changed, patched, etc.

I can only assume people that don't commit their binaries don't have to look at crash dumps very often...?

flyingyeti• 11 years ago

What is the difference between GitLab Annex and git-annex?

https://git-annex.branchable.com/

Edit: Just found the relevant quote from the article

In GitLab 7.8 Enterprise Edition this problem is solved by integrating the awesome git-annex.

I guess the repeated branding of "GitLab Annex" just seems a bit strange to me.

jobvandervoort3• 11 years ago

GitLab Annex is git-annex integrated in GitLab. Meaning you have the authentication and protection that GitLab gives you with git-annex.

icebraining• 11 years ago

Joey Hess mentioned this in the git-annex devlog: http://git-annex.branchable.com/devblog/day_256__sqlite_conc...

bitwize• 11 years ago

Most grownup software shops use Perforce precisely because it can handle large repositories with large amounts of non-source non-textual content. Maybe this will bring Git in the same league with the big boys.

Ysx• 11 years ago

Your condescension must be an unending source of rich and gratifying relationships.

jobvandervoort3• 11 years ago

We believe source control should be everywhere and git isn't great yet to work with large binaries. Git(Lab) Annex makes things much better for people working with these kinds of repositories.

We'd love to get feedback on it.

Jare2• 11 years ago

As I mentioned in another thread, efficient storage of these files is necessary, but not sufficient. Exclusive locking is a requirement for working with editable (but not mergeable) binary files in a team environment. It's the reason most games studios use Perforce or SVN instead of git. Tracking locks using a separate system does not scale, it needs to be integrated and enforced by the VCS.

PlasticSCM has a hybrid approach that is worth studying.

tomjen3• 11 years ago

That is an incredible assholish comment that does utterly nothing to benefit this community. I am sorry to have to call you out on it, but could you please provide some constructive critism in the future?

farresito• 11 years ago

I don't think he was that negative. It's a well know issue the problem that git has with large files, and he was not disrespectful.

gumby• 11 years ago

Ignoring the snark of your comment: indeed, many people chose Perforce either because they have binaries that require revision control or because their trees are huge.

We have to suffer under Perforce for the first of those reasons (we have artwork and CAD files none of which are text). Sadly, P4 is pretty much the only game in town for a medium sized organization with binaries (someone large like Google can afford to replace it with an in house solution but that isn't worth it for most of us).

I tried annex a few years ago and it was also painful, but perhaps it's gotten better. Certainly I miss the power of git when using P4.

GitLab Annex: Large Binaries with Git

Related Stories