Large-scale cluster management at Google with Borg

http://blog.acolyer.org/2015/05/07/large-scale-cluster-management-at-google-with-borg/

by mrry • 11 years ago

54 30 11 years ago

blog.acolyer.org

gtaylor• 11 years ago

I've been hesitant to spend a ton of time playing with Kubernetes yet, since it is still in such a state of flux (and the docs are woefully inadequate). With that said, if anyone can make something like this awesome, I'd put my money on Google.

Exciting times!

samkone3• 11 years ago

Try Mesos/Marathon in addition to Kubernetes. Been running Kubernetes on Mesos and it's just great.

throwaway1979• 11 years ago

Swarm isn't shabby either.

AdmiralAsshat• 11 years ago

Does anyone else feel like calling your massive infrastructure software "Borg" might be tempting fate? I mean, I'm not superstitious, but I would be weary of calling my satellite network "Skynet."

oesmith• 11 years ago

Skynet, like the UK military satellite comms network?

https://en.m.wikipedia.org/wiki/Skynet_(satellites)

_dark_matter_• 11 years ago

Can anyone explain how MapReduce is built on top of Borg, when Borg doesn't support scheduling with data locality? I'm having a hard time reconciling that. One of the principal use cases of the MapReduce framework is reducing data movement...

nrr2• 11 years ago

From the MapReduce paper: "The MapReduce master takes the location information of the input files into account and attempts to schedule a map task on a machine that contains a replica of the corresponding input data."

One of two possible designs immediately come to mind: (1) The MapReduce master interacts with Borg directly on behalf of the programmer and places a requirement on the task to run on or near any node that has a replica of the data required or (2) for each Borglet, there is an accompanying MapReduce agent running to accept RPCs from the MapReduce master.

GauntletWizard• 11 years ago

It's the former.

powera2• 11 years ago

The MR master handles it.

Mapreduce workers can request shards (work) that have data local to the host it is running on. For large mapreduces this basically works because both the data and the tasks are in most of the cluster, and for small mapreduces it doesn't matter.

thrownaway24243• 11 years ago

You seem to be suggesting that using local i/o to read inputs would be preferable to using remote i/o, but I don't think that's a supportable statement in a datacenter with a high-performance network.

wmf• 11 years ago

But it's a fact that original MapReduce/Borg used data locality (and networks weren't as fast 15 years ago).

nrr2• 11 years ago

Even with high-performance networks in place, you absolutely must budget your bandwidth between management communications, low-latency service communications, and batch job communications, amongst possibly even others. Having data locality is still a really, really big priority.

(Furthermore, building out separate networks is extremely cost-prohibitive once you've expanded out to a total footprint in the low-1e6 ballpark.)

Think about it this way: In recent history, we've added L3 cache to commodity processors because keeping data on-chip is still faster, in 2015, than going off-chip to RAM. The same analog holds when trying to fetch from local disk versus remote disk.

Brendan Gregg actually provides a really good time-stretched view of various operations in his book on performance tuning. Go give that book a read; there's some _really_ good stuff there.

samkone3• 11 years ago

to get an idea look at how Hadoop or spark run on Mesos. You push the data locality into tasks, executors and their caches.

dcsommer• 11 years ago

Does Kubernetes use chroot jails like Borg, or has it advanced to LXC containers? Nowadays LXC isolation seems strictly preferable.

jhspaybar• 11 years ago

LXC uses chroot + other things. chroot is essentially the foundation of any containerization tool.

zobzu• 11 years ago

Not really- chroot provides no isolation, namespaces do. choot is the visible portion (changes root.. ie changes /mnt/blah/bleh to /) but not "the foundation"

Things can in fact work isolated without chroot, too. Its just convenient.

exacube• 11 years ago

Note that chroot will provide containerization, but without isolation (processes can always fork out of their chroot jails). You still need things like pid namespaces etc.

jbeda• 11 years ago

Kubernetes uses Docker (or now rkt) to do the actual containing. The focus is on scheduling and managing lots of containers.

You may run O(10) containers on a single machine. When you run O(10k)+ containers in a cluster you need new tools to manage things.

wffurr• 11 years ago

Borg also uses cgroups, just like LXC.

From the paper: "We use a Linux chroot jail as the primary security isolation mechanism between multiple tasks on the same machine… all Borg tasks run inside a Linux cgroup-based resource container."

samkone3• 11 years ago

Remember that Borg is an old thing. At the time they probably started out with chroot jails, because cgroups weren't ready yet. But Kubernetes, Mesos use docker and cgroups(Mesos).

powera2• 11 years ago

cgroups were largely written by members of the Borg team.

mandeepj• 11 years ago

Can anyone throw some light on what google is doing with tens of thousands of machines in these clusters? Are they hosting crawlers on them?

coops• 11 years ago

- crawlers

- websearch

- google brain

- video transcoding for youtube

- appengine (think snapchat)

- gmail

Almost everything Google does runs on Borg.

WestCoastJustin• 11 years ago

They host pretty much everything -- Maps, Gmail, Search, etc. Highly suggest reading the Wired link below as it talks about this in detail [1], but if you want a more technical look, check out the Youtube video [2], or the Borg paper [3]. One really interesting development, is that Mesos + Marathon + Chronos + Docker, puts a very similar system into the hands of your average IT team today [4].

[1] http://www.wired.com/2013/03/google-borg-twitter-mesos/

[2] https://www.youtube.com/watch?v=0ZFMlO98Jkc

[3] https://research.google.com/pubs/archive/43438.pdf

[4] https://www.youtube.com/watch?v=hZNGST2vIds

dekhn• 11 years ago

When I ran Exacycle http://googleresearch.blogspot.com/2013/12/groundbreaking-si... the worker pool was just a low-priority (priority 0 in the paper) job like everything else, soaking up idle cycles. The command and control jobs were just regular borg jobs that managed the worker pools by assigning tasks to them.

So whatever else was going on, protein folding and drug docking and telescope simulation was soaking up whatever leftover cycles there were.

Large-scale cluster management at Google with Borg

Related Stories