How do game companies share massive files?

http://www.bbc.co.uk/news/business-25037653

by Libertatea • 13 years ago

34 44 13 years ago

bbc.co.uk

simias2• 13 years ago

So basically rsync?

Honestly synchronizing 50GB of contents across the globe doesn't sound that terribly novel or difficult to me, or the article makes a pretty bad job of explaining the difficulty.

RBerenguel• 13 years ago

I thought the same. Sending only the delta of the content has been known for a long while...

mryan• 13 years ago

That's because some of the complete game files were as large as 50GB

50GB is not the total size of the files needing to be transferred - some of the files were 50GB. Knowing the total size would have been interesting for the sake of discussion.

debian69• 13 years ago

Yeah thats bollox , sorry but no game has 50 gb files on launch , and what kind of idiotic company would send round a file they weren't using in launch code base ?

simias2• 13 years ago

Mmmh, that's not how I understood it. I've never worked in game development but 50GB seems a reasonable amount of space for a full game. Also TFA mentions that the issue is synchronizing with testing centres, I doubt they send the whole source files there, only the current debug build I assume (but really, even if it's the full material in 2013 it shouldn't be an issue unless they're using dialup).

But at any rate the size of the individual files is not what matters. The important part is the total size of the game files and the average change rate. I would be surprised if they weren't able to synchronize everything with a good internet connection and a bunch of rsync scripts.

golergka2• 13 years ago

Sounds a lot like git or other typical solution.

samnardoni• 13 years ago

git would be pretty bad in this situation. It sounds more like rsync.

golergka2• 13 years ago

Thanks, I'll look into that.

t0• 13 years ago

Sneakernets are still the fastest way to transfer data. http://en.wikipedia.org/wiki/Sneakernet

Piskvorrr• 13 years ago

Not necessarily the cheapest, though - express shipping a bunch of disks is an uncertain venture ("oops, we accidentally dropped it a few times, is that bad?") and "I need a return cross-Atlantic flight ticket NOW" is sort of expensive.

That said, I have heard of companies actually couriering semi-large volumes of sensitive data this way.

Zenst• 13 years ago

Also in some area's there is a legal aspect due to law of some contract that makes sending somebody with a briefcase with data an easier way to move forward without triggering some technicality in contract or law.

Indeed some countries and there data protection laws make things most interesting in legal ways to transfer data, even today. No standard on data protection World Wide and in itself a minefield, let alone other industries like medical research companies or those dealing with country wide firewalls.

notduncansmith• 13 years ago

Not gonna lie, when I saw "delta" I immediately assumed Git or some other source control solution. What boggles the mind is how these professional game developers recently discovered this technology. I didn't think Git was so exclusively beholden to the web development community.

AndrewDucker2• 13 years ago

You wouldn't use git for massive binary files that change on a regular basis. Particularly when you only need your testers to have the latest version of them, not the complete history (which is what git would give them).

unwind• 13 years ago

You are fantastically mistaken if you think game devs don't use source control. Heh.

Of course they do, and I'm ... let's say pretty sure that much of EA runs on http://www.perforce.com/.

Also weird that the article starts with a BF4 image, but then claims that BF4 was developed in Californa. I guess that hurts my national pride (go DICE!).

NickPollard3• 13 years ago

Git solves a different problem; sending large binary blobs around is not it. As another comment said, this is essentially RSync, not git.

I've been out of game development for a year, but unfortunately git is not being taken up much by large game developers. The main reason cited is actually git's handling of binary files, which leads to every checkout containing many duplicates of huge files. However this is possible to work around and the benefits of git still make it worth it.

js2• 13 years ago

That said, git does have a delta compression mechanism for reducing disk space and saving network bandwidth that is quite complex:

http://stackoverflow.com/questions/9478023/is-the-git-binary...

Negitivefrags2• 13 years ago

The binary file reason is big enough, but it's also the ability to check out a sub tree without checking out the entire Repo. The repo is 1TB and there really is no reason for your texture artist to be checking out the raw uncompressed audio files.

The other reason not to use git in game development is because it's incredibly hard to use, and more than half the game development team are not programmers and/or not technical.

Getting them to use good source control discipline is hard enough with TortoiseSVN. I shudder to think the insane mess we would have using git.

ethanhunt_• 13 years ago

I think a lot of game developers actually use perforce because it handles large files better. This is at least what I've gather from eavesdropping some of them (e.g., Jon Blow) on twitter.

NickPollard3• 13 years ago

Perforce is very popular in Games, I think 4/5 studios I've worked for used it.

I still vastly prefer git. Perforce does have some pluses, but it's mostly people just going with what they know.

7952• 13 years ago

When you look at how non-programming projects are managed you often see a lot of the same issues that source control addresses. Files (and information in peoples brains) become out of sync, or need to be locked.

Programmers always focus on the technology of source control, but the difference it makes is also social. It forces a particular workflow that allows tasks to be carried out concurrently. This workflow could be adopted without necessarily needing git.

boothead• 13 years ago

As someone said on twitter the other day:

  If "tech experts" writing news articles aren't really experts, what does that say about all the other experts..?

_sabe_• 13 years ago

How do game companies share massive files?

Related Stories