Storing large binary files in Git repositories (2015)

http://blog.deveo.com/storing-large-binary-files-in-git-repositories/

by pmoriarty • 10 years ago

54 23 10 years ago

blog.deveo.com

gcb02• 10 years ago

isn't it idiotic? if you can't get a textual diff, or if a textual diff is meaningless, what benefit do you even get having the file on git?

this sounds like a problem for teams blindly following orders from clueless managers.

mbel• 10 years ago

    > if you can't get a textual diff, or if a textual diff is meaningless, what benefit do you even get having the file on git?

A nice property of checking out whole repository using only single program. All assets are downloaded at once and always proper version of code comes with proper version of binary assets.

AndrewDucker• 10 years ago

Version control isn't about generating a textual-diff. It's about storing all of the necessary assets to be able to generate an application. Some of those assets are code, some are binary. And you want to keep them in sync, in some way.

Using multiple different systems to get myself "up to date" is a massive pain - I should be able to run one command and then have a wholly buildable set of files.

gcb02• 10 years ago

So, you keep your compiler source and binary in your project source tree?

yeah, you don't. :D

dominotw• 10 years ago

Git is not just textual diff. Branching, tagging, commit history are still useful without diff.

jkot• 10 years ago

I have GIT repo over my home folder for incremental backups.

jessaustin2• 10 years ago

I suspect many of the binary-in-git cases could be better handled with just a bit more work on the part of repo maintainers. There are many package managers that work well with binary files, for instance, and part of the build process could be calling out to those. Then you'd have a nice separation between assets that are generated from code and assets that are generated by some fiddly manual process.

mikeash2• 10 years ago

Git isn't usually for the assets generated from code, but for the code itself. Code is an asset generated by a fiddly manual process, and both code and other assets go into producing the final results, so is that really a point of separation?

pmoriarty• 10 years ago

You couldn't do a textual diff, but you could do a binary diff, and storing a bunch of binary deltas could save you a ton of space when you have lots of large, frequently changing binaries to store. Further, you'd have all the other benefits of version control, such as versioning, accountability, comments/logging, etc.

CoolGuySteve• 10 years ago

It may be sacrelege but you could store your binaries in hg and in hg use a git submodule for the git repos that were used to build those binaries.

This is what I'm doing after a disastrous excursion into git-fat.

But really, if git's maintainers don't want to store binaries, why shoehorn it in? Use the right tool for the job.

Mathiasdm• 10 years ago

For some added context about handling binaries in Mercurial, see the (in development) Mercurial book: http://hgbook.org/read/scaling.html#handle-large-binaries-wi...

ShookOnes• 10 years ago

I also went with the hg solution to enjoy it's core large-file support but ultimately switched to git-annex so I didn't have to support and learn each technology. git-lfs looks to be good but I'm still waiting a small, standalone production-ready server implementation.

exDM69• 10 years ago

Having a separate git repo for large files and shallow cloning it should be similar to having another repo in hg/svn for large files.

m0th87• 10 years ago

We made git-fit, which was inspired by git-media: http://github.com/dailymuse/git-fit

On the plus side, it's stupidly simple. On the down side, it's stupidly simple. The readme explains how it works.

bokchoi• 10 years ago

It would be nice if git would support large files natively rather than relying on these extensions.

joeyh• 10 years ago

This is a good article, but its git-annex information is getting slightly out of date. git-annex has a new repository format available which makes its interface work more like git-lfs. http://joeyh.name/blog/entry/git-annex_v6

Also, it's worth noting that git-lfs stores 2 copies of each large file on local disk. git-annex stores 1 copy. Which is a pretty big plus the article left out.

drdaeman• 10 years ago

Whoa, this looks great!

Sorry for possible off-topic, but I wonder if anyone knows a way to convert/upgrade older (created an year ago) git-annex repo to a newer v6 one?

sk3w• 10 years ago

https://git-annex.branchable.com/upgrades/#index1h2

I haven't tried this yet myself.

petertodd• 10 years ago

Have you considered using the copy-on-write features that some filesystems (btrfs) now support? For example, the lowly cp command now supports CoW via --reflink=always

Also, that ks for git-annex! I use it for hundreds of gigabytes of files, and find the one-copy-per-file support essential for how I use it.

mschuster91• 10 years ago

I have a couple of projects with huge binary assets under source control (movie files, 4K images, essentially the whole website). Deployment via jenkins and rsync, it's really fast after the initial checkout.

How? Well, the framework stores all assets inside /assets, which normally is a git submodule. Jenkins works pretty fine with this construction, and for developers there are a couple self written PHP shellscripts that execute a shallow clone outside the repository and then do a bit of rsync magic to sync it all.

Storing large binary files in Git repositories (2015)

Related Stories