How Google's build system works

http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html

by skybrian • 15 years ago

113 18 15 years ago

google-engtools.blogspot.com

thurn• 15 years ago

It's a little annoying that every big tech company has solved this problem in house in their own way. Google, Amazon, Facebook, Apple, and Microsoft all have internal build systems that do basically the same things.

robfig3• 15 years ago

Are there any open source build systems that work similarly? I looked at basically everything out there, and none provided this functionality. So my company ended up rolling our own, based on the Google model.

The particular characteristic that I saw missing was package-level build files. All the open source guys really wanted you have one big build file that described how to build everything, rather than bunch of small ones that could be composed.

JoshTriplett2• 15 years ago

We're working on that for Apters (http://apters.com/). We have a simple language used to describe an individual package build, which acts as a module that you can import from another package that needs it as a build dependency. The whole thing uses content hashes for versioning, like Git, and caches any build you've already run once.

The whole thing is Open Source; we'll also have a hosted version as a service.

Can you tell me a bit more about what you need?

ChuckMcM• 15 years ago

This is the problem, no there are not any open source systems which can build systems like Google has. I think that is in part because there are no open source projects of the complexity and scale of what Google builds and thus those teams haven't run into the problems that Google did and had to come up with their own ways to work around them.

When I worked there Google had 20,000 employees. More than half of them were engineers, and probably at any given moment in time 5,000 unique users might type 'build' at their desktop after having checked in a change. That means 5,000 instances get build, 5,000 regression test suites run, following at least 5,000 different 'views' into the repository. (which sat one some Perforce servers that nearly glowed from all the work they were being asked to do).

There are few places where such scale is needed, perhaps a couple of dozen, and they all have competent engineers so they all come up with their own build system.

If open source were ever to address this particular challenge, it would have to be in the creation of some really solid interfaces which would allow a build system to emerge from the tools, rather than trying to explicitly create a build system from scratch.

evmar• 15 years ago

Chrome's build system, gyp (http://code.google.com/p/gyp/), is based on the google3 model at a high level.

The nodejs guy seems pretty happy with it: http://twitter.com/#!/ryah/status/100019851803885570 http://twitter.com/#!/ryah/status/99736138843029504

Meai• 15 years ago

I love Tup, it basically replaced Make for me. It tracks dependencies and files, automatically cleaning and updating everything even more precisely than Make, making it a lot faster and easier to use. https://github.com/gittup/tup

(yes, you can use Tupfiles in each directory, if that's what you mean by packages. As far as I know however, (and this is by design) there is no way to control the build. It will always build exactly what needs to be built to be up to date.)

aphexairlines• 15 years ago

Debian has http://www.debian.org/devel/buildd/ and SUSE created http://openbuildservice.org/ out of their distributed build service.

Lewisham• 15 years ago

I think that having an in-house system was partly driven by the need to fully utilize Google's proprietary/confidential computational infrastructure. My understanding is that every big company is different, and leveraging your CPUs to their fullest extent means there isn't a one-size-fits-all model.

The only way you could move towards a single model is if all the companies open-sourced their data centers like Facebook did, so you'd at least have a good idea of which infrastructures you need to run on.

DISCLAIMER: I work as an intern at Google in engtools (the group that manages the linked article), but have no idea about the design decisions that went into the build system.

ryannielsen• 15 years ago

Their build systems may do basically the same thing, but I bet they all do it dramatically differently and probably for good reason.

Every company you listed has different products, resources, deliverables and goals. Their build systems will have evolved over many years to take advantage of those resources and meet those needs, and that process will naturally lead to different solutions.

checoivan• 15 years ago

I asked myself the same question once. Having worked a bit on this before on msft I learned that builds, just like other dev tools, need constant improvement as complexity increases. Faster builds, customization and allowing scripting, parallel building, etc... It's one of those things taken for granted because it's always been there, like the F5(compile), but It's a tool which can be improved just like any other.

vl• 15 years ago

Microsoft's build system is a joke by any modern standard. It's a combination of a tool called 'build' (some version is available in DDK) and nmake. As the end result, since 'build' doesn't enforce dependencies (and as a result can't reliably trace them), they leak, and developers put manual "drains" into build files, reducing system parallelism. They try to upgrade manifest-based builds, but nothing came out of it as far as I know. Automatic incremental builds do not work either.

It took me 12 hours to build Windows in 2001, it's still takes approximately the same time to build Windows now (or so they tell me). No distributed build in sight. I think that across entire Microsoft few thousand man-years are wasted in last decade because of the build issues, and cost of lost opportunities is unmeasurable.

I'm happy blaze user now.

Game_Ender• 15 years ago

I need to read more before I really understand, but what was the problem with something like distcc, ccache and CMake? The two downsides I see to the distcc/CMake approach is slow dependency resolution (looking at lots of files instead of just packages), and duplicate coping of files by distcc to remote machines.

forgotusername• 15 years ago

This is quite a high level overview, so possibly underneath they're using distcc. This at least used to be true for ad-hoc builds on their corporate network in previous years.

slug• 15 years ago

Don't know if it still holds true, but they used distcc a few years ago: http://google-opensource.blogspot.com/2008/08/distccs-pump-m...

I also recommend ccache, but it might run into problems with the pump mode, although it works fine with regular distcc.

jdefarge• 15 years ago

Rumours has Blaze, Google's build tool the article describes, is being open sourced. At least this information was floating around early 2011.

How Google's build system works

Related Stories