What does it take to make Google work at scale?

https://docs.google.com/presentation/d/1OvJStE8aohGeI3y5BcYX8bBHwoHYCPu99A3KTTZElr0/

by mrry • 11 years ago

247 130 11 years ago

docs.google.com

halotrope• 11 years ago

Kind of ironic: "Wow, this file is really popular! Some tools might be unavailable until the crowd clears"

Incidentally, that's part of scalability: knowing when you can drop functionality and the user won't care. For Google Docs, it's highly unlikely that a file with 1000s of viewers is actively being edited by 1000s of users, and more likely the link just got shared on the Internet. So you can drop edit functionality for everyone but the original author and they won't care.

Another example - one that I've heard second-hand - is how Facebook gets consistency on the news feed. Apparently your own writes to Facebook are sent separately to a write-aside cache, and then the webserver merges them back in whenever you view a page. As a result, Facebook is always strongly-consistent when it comes to your comments (you'll never post something and then fail to see it show up), but it's only eventually-consistent with respect to other peoples' comments. But then, you won't know about or care about the latter, because how would you know that you're not seeing something they posted?

patio11• 11 years ago

Same at LinkedIn. (Source: presentation by their then-CEO at JavaOne, circa 2009.)

Rainymood• 11 years ago

That's really smart. Never noticed a thing. Awesome :)

ska3• 11 years ago

No. It would be ironic is if it stopped working at all with no notification.

As it is they are just demonstrating competence at scaling.

mason55• 11 years ago

This is actually a really good example of the CAP theorem. Because of the way Docs merges updates to a single document they had to either tie an editable document to a single machine or risk inconsistent updates. The choice was made to ensure consistency. If a doc gets too popular to be served off a single machine then it goes into read-only mode and gets distributed to multiple boxes. When the traffic dies down it can fall back to a single box and re-enter read/write mode.

davis_m2• 11 years ago

"NTP isn't accurate enough so we use a hardware clock assist that uses GPS and atomic clocks" - Google

I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale.

Every review I hear about working at Google makes me want to stay away, just like the recent conversations about working at Amazon, but crazy stuff like this always piques my interest.

ska3• 11 years ago

   I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale.

Time keeping is one of the things that is most often underestimated and screwed up at any scale, in my experience. Many a subtle bug turns out to be a poor assumption or misunderstanding about how clocks work. Or calendars.

jasonpeacock• 11 years ago

i.e. A lot of distributed systems problems can be simplified if you can trust your clocks.

zo1• 11 years ago

Can you elaborate on the need? What possible reason, short of a very contrived one, is there for having to keep a large number of machines' clocks in-sync?

And for that matter, why would anyone build any process/system/software that requires a distributed system's machines to all have their clocks in-sync. I am baffled.

mad44• 11 years ago

Hybrid Logical Clocks offer a simple and feasible alternative to Google's custom hardware based TrueTime.

http://muratbuffalo.blogspot.com/2014/07/hybrid-logical-cloc...

MattCruikshank3• 11 years ago

"Every review I hear about working at Google makes me want to stay away"

I loved working there. Sure, who your boss is has a huge impact on your happiness. I ended up with a boss I enjoyed working for.

Does that review make you want to stay away, too?

tikhonj• 11 years ago

It does for me, at least. I'd absolutely hate having my job satisfaction determined so much by a manager and team that I was assigned. (Ideally I wouldn't want it impacting much at all, but truly flat companies are still hard to find, despite the rhetoric.) It works for some people, but it won't work for me.

vidarh• 11 years ago

What makes me want to stay away, apart from the ridiculous recruitment process, is the insane staff turnover. No amount of positive reviews will overcome that for me.

robrenaud• 11 years ago

Working at Google is pretty cool.

Personally, I get paid pretty good money and have tons of resources to build models that serve 100s of millions of people.

drcross• 11 years ago

After 9 different interview rounds I've all but giving up on trying to get into there. Any advice on trying to get in?

SEJeff• 11 years ago

This isn't super surprising tbh, all of the finance industry has PTP timesources in most every datacenter. Different needs for slightly different industries.

munificent• 11 years ago

Every review I hear about working at Google makes me want to stay away

Google was Fortune Magazine's #1 best place to work for six years in a row. The reason you occasionally hear a story about how Google isn't perfect to work at is because those are news. In general, it's a fantastic workplace.

untog2• 11 years ago

Everyone is different - while I don't doubt that Google has fantastic perks, I can't think of many things that I would want to work on that aren't the moonshot crazy experimental projects (and, let's face it, most people won't be working on them). And working on a giant campus that's isolated from the outside world... no thank you.

But like I said, we're all very different.

davis_m2• 11 years ago

Reading it back, my comment was harsher than I intended, especially with the reference to Amazon, but most of my aversion to working at a place like Google is just the shear size as well as where most of the offices are located.

I grew up in a tiny midwest town and I love it here. I would not enjoy living on either one of the coasts.

Many of my classmates in college couldn't wait to get out of the midwest. I have friends at Google, Amazon, Microsoft, and other large names in the tech industry, but more often than not, when I hear them talk about their jobs, even when they are talking about them positively, I am glad I stayed in the midwest because that is what fits me the best.

martincmartin• 11 years ago

A good article about this is "There is No Now" on ACM:

https://queue.acm.org/detail.cfm?id=2745385

mozumder• 11 years ago

I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale.

Timekeeping is such a basic problem that Albert Einstein invented the theory of relativity because of it.

tajano• 11 years ago

"something as simple as keeping time"

Keeping time has been a difficult problem for... well, as long as we've tried keeping time. For an interesting historical account on past difficulties, you might check out the book "Longitude" [1].

[1] http://www.amazon.com/Longitude-Genius-Greatest-Scientific-P...

mmoya• 11 years ago

I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale.

Check this talk: https://archive.fosdem.org/2015/schedule/event/ntimed_ntpd_r... . It's more difficult than it appear.

ohitsdom• 11 years ago

I understood some of these terms.

nickpsecurity5• 11 years ago

Nice overview. I particularly like how they visualized BigTable and MapReduce. Will make it easier for some to understand. For those liking Spanner, Google has exceeded that with their awesome F1 RDBMS (built on Spanner/GFS) below:

https://static.googleusercontent.com/media/research.google.c...

Can't wait for someone to clone this. Come to think of it, has anyone cloned Spanner? I recall some calling GPS a hack but I thought it was engineering brilliance. Spanner was some good work, too.

sargun2• 11 years ago

The people who are working to it are the Cockroach Labs folks: http://www.cockroachlabs.com/

nickpsecurity5• 11 years ago

The Wired article says they're not fully replicating Spanner, esp is time tricks. However, they are replicating a lot of its other capabilities. So, that's still a great tech in development. Thanks for the link. :)

Still gotta wait a while for a full Spanner replacement with GPS tricks and all. Or something even better hopefully.

bpicolo• 11 years ago

Also https://www.treode.com/

magicalist• 11 years ago

previous discussion on work from the same author: https://news.ycombinator.com/item?id=9652528

david_shaw• 11 years ago

Wow, this file is really popular! Some tools might be unavailable until the crowd clears.

Hopefully the irony isn't lost on the readers of this document :)

DannyBee2• 11 years ago

What irony?

To me, this isn't irony at all. It shows that google can't defeat the algorithmic complexity of things like consensus and shared state editing, and so gracefully degrades functionality.

That shows a very good understanding of scaling - realizing any solution you choose has limits and tradeoffs, and thinking about and handling those limit cases sanely ahead-of-time, rather than waiting for it to fall over and hoping for the best.

munin• 11 years ago

I got an error message saying the document wasn't available at all and to come back after the "crowd clears" which is much more ironic... after I refreshed a bunch it became available...

pjungwir2• 11 years ago

I think it has more to do with preventing concurrent writes stepping on each other. Google Docs must be the number one example of real-world systems doing Operational Transformation [1]. It's not like HN is bring down Google's servers.

Incidentally OT is another example of the value of in-sync clocks, for those asking about that elsewhere in this thread.

[1] https://en.wikipedia.org/wiki/Operational_transformation

Thaxll• 11 years ago

The thing that might be scary working at Google is that you work with tech than no-one else uses.

obeattie• 11 years ago

I'm not sure that Google thinks of that as a problem. In fact, I'd argue a lot of the tech that many companies use nowadays exists because Google pioneered it.

nostrademons3• 11 years ago

I think that he's talking about the employee perspective, not the company perspective. Google's tech stack is mostly proprietary, which means you can't take your tech skills with you to a new employer.

...And I haven't found it to be a major problem, being an ex-Googler a little over a year out. Tech skills are easy to pick up on your own. My current startup is based on Node.js, a native Android client, and AWS; the one before it was Django & Heroku. Haven't tried looking for jobs yet - I made enough at Google to not have to worry about that for a while - but I occasionally get in-bound interest from big-name, fast-growing startups. Most clueful hiring managers look for experience with problems, not with solutions, and Google lets you face problems that the rest of the industry won't deal with for a while.

threeseed2• 11 years ago

A lot of technologies ? No not even close. More like 3.

GFS, MapReduce, BigTable were the key Google inventions that became the sparks that ignited the modern day big data/analytics revolution.

rco8786• 11 years ago

A lot of times that tech no one uses today becomes the industry standard tomorrow.

It's scary being at the bleeding edge.

segmondy• 11 years ago

There are plenty of other companies that also have tech that Google doesn't have or use too. Google has all these amazing tech, but I can assure you working at Google doesn't mean you will ever get to use it directly.

threeseed2• 11 years ago

Exactly. Facebook and Yahoo have contributed equally as much to the technology landscape as Google has.

And unlike Google they actually make it available to the public and then follow it up with supporting the projects.

jrockway2• 11 years ago

The one thing that worries me about leaving Google is how much time I'd have to spend building hacked-together imitations of the infrastructure I'm used to using. It's not because I don't know how other things work, it's because I know how these things work and I like them better.

At least the build system was recently open-sourced, so I won't have to build that from scratch. But things like Borg and D are both elegant and easy to use, and I would hate to have to go back and care about deploying software or configuring RAID arrays again. Unfun and uninteresting. Totally solved at Google for the kinds of problems I work on.

suls• 11 years ago

For me - from the outside - the build-system and the trunk-based development is the single most important factor for an organization that scales. Yet I wonder why it is so hard to convince my co-workers .. any anecdotes about how it was started at Google and later at Twitter?

jordz• 11 years ago

It's presentations like this that give me a that spark to go out and build crazy-shit. Sometimes you HAVE to reinvent the wheel because wheels just don't work to move objects the size of the planet the wheel is bound to. I'm not a fan of re-inventing the wheel by any means but it's this kind of stuff that makes me love going to work every day and basing ideas off presentations like this!

i336_2• 11 years ago

Well, it isn't every day you get to redefine π :D

i336_2• 11 years ago

Are these slides attached to a video presentation or something similar? I'd really like to see the talk this accompanies.

blablablksheep• 11 years ago

There is no mention of SPARK like in-memory compute engine(s) Did i miss it in my read ? thanks in advance

peterwwillis• 11 years ago

Keep in mind this is really "How does Google (and other big companies) do it", not "what does it take", because what it actually takes is much simpler than custom hardware and custom software. This was all pretty accessible 15 years ago using available software; the trick was putting it together right.

In general, the maxim of "cache everything, everywhere" is the cheapest way to gain both speed and availability. Of course you need to have results already to cache them, which is where all that big data processing comes into play, but there's still way more optimization going on here than is necessary to get results like this. You could make a tradeoff and give people newly-processed results more slowly and still give them mostly what they want, and not have to maintain so much custom software+hardware infrastructure.

Sometimes I feel like Batty from Blade Runner when I think about how well old crappy software used to work. "I've seen things you people wouldn't believe. Attack ships on fire off the shoulder of Orion. I watched mod_perl apps serve 50 thousand dynamic requests per second with nothing but Apache and MySQL. All those moments will be lost in time... like tears in rain..."

yzh• 11 years ago

This may seem irrelevant, but as a GPU computing researcher, I'm still disappointed that GPU still hasn't made its way to be the first citizen at data centers. I know world of HPC looks much different, but I wonder when GPU can be vastly used in data centers, and will NVIDIA's Pascal architecture make a huge impact on this?

ploxiln• 11 years ago

I don't think GPUs will be big with "web services" datacenters anytime soon.

As the slides mentioned, the difference between this web-services stuff and HPC is that HPC has a much higher compute/data ratio. Said another way, this web-services stuff is all about moving data, collating it a bit, and less about intensive mathematical processing of it.

Most of the boxes in the diagrams are datastores. Block stores, object stores, columnar stores, caches. I hate to say it but... big data. Not (as) big compute.

xyzzyz2• 11 years ago

They are. With as much neural networks stuff as Google does, it's pretty obvious tons of GPU-hours must be in use.

http://www.wired.com/2013/05/gpus-in-the-data-center/

faragon• 11 years ago

Wow. That's amazing. Anyone knows how they do a backup/restore of a GFS distributed file system? (in case they do backup of some sort)

dejv• 11 years ago

My guess is that they just copy the blobs and then send it to other machines/datacenters.

jcrites• 11 years ago

I don't have any special knowledge of GFS, but one purpose of distributed file systems of that sort is generally to avoid the need for additional backup by replicating the data many times as part of its routine operation. The data storage layer itself replicates data many times, across multiple data centers, and continually verifies the replicas' integrity and re-replicates as needed.

The GFS paper describes replication in a bit more detail:

Users can specify different replication levels for different parts of the file namespace. The default is three. The master clones existing replicas as needed to keep each chunk fully replicated as chunkservers go offline or detect corrupted replicas through checksum verification

Some other reasons you wouldn't backup a distributed file system (not saying there are no reasons ever [1]):

(1) Difficult to add another layer of backup without impacting performance unpredictably at backup time. It's more predictable to implement much of the replication synchronously within the request (while optionally some replicas to catch up out-of-band) (2) Files are differently important - some may warrant a greater degree of redundancy than others. The file system can understand this and take advantage of it; a separate backup system on top of the file system probably can't. (3) A standard backup/restore process often implies downtime during recovery. One goal of distributed systems is to avoid downtime by handling faults transparently. They continuously repair themselves. See: recovery-oriented computing. (4) A backup and restore process that's in any way intrusive on the operation of the system will not be easy to test on an ongoing basis the way that failure recovery will be tested constantly within the distributed file system. (In a big server fleets, drives will fail all the time, giving you no end of opportunities to exercise your recovery process.)

[1] One reason might be a defense against "unknown unknown" faults in the file system itself that cause it to irrecoverably lose track of data.

xyzzyz2• 11 years ago

Apart from other people mentioning that distributed file systems have replication as a part of the architecture, Google also uses tape[1][2].

[1] http://www.tested.com/tech/1926-why-google-uses-tape-to-back... [2] - http://www.theregister.co.uk/2013/12/29/a_year_of_tape_tittl...

vayarajesh2• 11 years ago

Is it possible to view the video or the presentation given in which the speaker talks more in detail explaining the slideS?

ms705• 11 years ago

Hi, author here.

The talk for which I put the slide deck together was given at a summer school and unfortunately not recorded, but if there is sufficient interest, I might tape a re-run and upload it. (Though unlikely to have time in the next month, so it might be a while.)

In the meantime, the original papers (listed in the bibliographies at http://malteschwarzkopf.de/research/assets/google-stack.pdf and http://malteschwarzkopf.de/research/assets/facebook-stack.pd...) have a lot more detail than my (very condensed) slides.

vayarajesh2• 11 years ago

Great! thanks for the share

badtoyx• 11 years ago

Please DO re-run the talk and share it with us. It would be quite valuable for a variety of audience. Thanks in advance!

cronopios• 11 years ago

Wow, that would be awesome!

I hope you do it!

whitenoice2• 11 years ago

Thank you for sharing this! and the consolidated papers list. Would be great if you can record and upload your next re-run!

whitenoice2• 11 years ago

Do young (new to google) googlers get to work on these projects?

novaleaf• 11 years ago

Wish there was audio / text that went along with these slides :(

What does it take to make Google work at scale?

Related Stories