The Friendship That Made Google Huge

https://www.newyorker.com/magazine/2018/12/10/the-friendship-that-made-google-huge

by adrianhon • 8 years ago

532 101 8 years ago

jdpedrie• 8 years ago

Site's down for me. Archive link: http://archive.is/w4oWL

bigtones• 8 years ago

This is a wonderfully told story of two very senior engineers who developed a lifelong friendship and wrote some of the core of the Google search engine I use countless times per day - I really enjoyed this piece.

VikingCoder• 8 years ago

"Jeff Dean once shifted a bit so hard it ended up on another computer."

modeless• 8 years ago

"Jeff Dean puts his pants on one leg at a time, but if he had more than two legs, you'd see that his approach is actually O(log n)."

jpatokal• 8 years ago

Sounds like he discovered the Rowhammer exploit?

shafte• 8 years ago

I'm so glad this this article (unlike so many other representations of Jeff and Sanjay's work) does not overlook Sanjay's contributions.

I don't know what it is; maybe Sanjay is a weird (read: non-English) name, maybe it's that computing culture's obsession with lone hackers leaves no room for a partnership like Jeff and Sanjay's. Anyway, kudos to the New Yorker for not falling into that trap.

Itaxpica• 8 years ago

I think a big part of it is something the article touches on: Jeff is an extrovert, Sanjay is an introvert. Jeff is the much more visible of the two; the one who people are more likely to have heard speak personally or to have dealt with. It sounds like this is how they both prefer it, but it definitely means that Sanjay's contributions get passed over in the common discourse. I was also glad to see the article avoid that trap.

ugh123• 8 years ago

You mean like Steve and Woz? j/k :)

nlh• 8 years ago

Ditto on the sentiment. I’m not a Googler but I’ve worked closely with a few, and Jeff was revered accordingly (along with the Chuck Norris-esque jokes). I’d honestly never heard Sanjay mentioned (and didn’t know who he was) until I read this article. I’m glad he was given his due and I’m glad to have learned about his contributions.

AceJohnny2• 8 years ago

(along with the Chuck Norris-esque jokes)

Funny thing, that was started by kentonv [0], who worked on open-sourcing protobufs [1], then CapnProto [2], then (with Jade Wang) Sandstorm [3]. He lurks around here :)

[0] https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe

[1] https://github.com/protocolbuffers/protobuf

[2] https://capnproto.org

[3] https://sandstorm.io

joatmon-snoo• 8 years ago

So much!

As a college student, it was so weird learning about Jeff Dean's meme status, and it wasn't until after joining Google and coming across things like the original protobuf design doc that I realized how significant Sanjay's contributions are.

Jyaif• 8 years ago

“In terms of sexiness, compilers are pretty much as boring as it gets,”

That bit surprised me. For me it doesn't get much sexier than compilers.

ineedasername4• 8 years ago

Yeah. Maybe that's the outside view, or view from the surface level folks concerned with the next shiny framework, but I think the best engineers are always at least a little enamored with deeper nuts and bolts.

AlexCoventry2• 8 years ago

I think he meant the business side isn't going to give a damn about a compiler.

amelius• 8 years ago

Writing a modern concurrent garbage collector is as complicated as writing a compiler, to a great degree because of their "real time" constraints and concurrency.

But to call them sexy is another thing ...

strikelaserclaw2• 8 years ago

these are the kind of articles i like reading, well written, interesting subject and good content length. Too many times i read bold headlines on HN, "Why X is bringing america down", then go to read it and just when they get to the crux of the matter i find the article has ended smh.

woodhull• 8 years ago

The New Yorker does this very consistently every week at an extraordinary level of quality.

They will even mail you a copy in exchange for a subscription. It's magical.

strikelaserclaw2• 8 years ago

i will look into it

forgot-my-pw• 8 years ago

Only $6 for 12 weeks for print + digital. Subscribed!

fspacef• 8 years ago

Good stuff. Every sport has great partnerships (Jordon - Pippen, Brady - Gronk) and I always wondered if such a thing would even be possible in SE as I've felt overly self conscious and inefficient while pair programming.

Though what really makes this my favorite long read of the year is how it follows their story over the course of Google's rise and the subtle juxtapositions between early Google of "doors across sawhorses" and today with "Level 0-11". Given all the recent news with engineer walkouts and protests it makes you realize that Google have probably done the best job of handling the scaling challenges of being a unicorn and making the necessary cultural adjustments as they grew in size.

warent• 8 years ago

Don't forget Sam and Frodo

ineedasername4• 8 years ago

Yeah, teams are good but every once in a while you need that lone crazy Tom Bombadil genius to come along and save your butt from disaster.

gowld• 8 years ago

http://cycle-gap.blogspot.com/2007/09/extreme-pair-programmi...

Extreme Pair Programming - Guy Steele and Richard Stallman

xapata4• 8 years ago

long read of the year

Are books extra-long?

NelsonMinar2• 8 years ago

What a lovely profile. I worked at Google early enough that I got to know Jeff and Sanjay, although never worked directly with them (to my regret). Every aspect of this article rings true to me, the way they are, the way they work together. I always admired how incredibly collaborative they were. They never competed with each other. Or really with anyone else, either, they just quietly did excellent work and helped others do their work too.

glenrivard• 8 years ago

Great article. Glad to see Sanjay get the credit he deserves.

skybrian3• 8 years ago

I wonder if more couldn't be done to encourage pair programming and collaboration in general? What are some other great partnerships?

Daktest• 8 years ago

Would highly recommend checking out `The Undoing Project` by Michael Lewis. It's a biography (of sorts) about Amos Tversky and Daniel Kahneman that has excellent stories about their fruitful collaboration in behavioral economics.

enneff• 8 years ago

If you want to see an example of Sanjay's elegant code, take a look at his Go stream package: https://github.com/ghemawat/stream

stdplaceholder3• 8 years ago

Sanjay's "experimental" directory in Google's repo -- a place where users can commit unreviewed code that are just ideas, or park some code that isn't ripe yet -- is full of gems.

puzzle2• 8 years ago

Holy moly, a mention of Bogdan! He was a legend of his own. Some of the most consequential changes, like elevating your service to higher QoS levels, which something like e.g. Gmail would require, were gated on his approval and his only. Someone made him a SW Easter egg, but the story already on the Internet gets some basic facts wrong.

wsxedcrfv• 8 years ago

Do tell! There have got to be a few good ones. He interviewed me for his current company.

puzzle2• 8 years ago

The internal Easter egg was basically a random fortune, which cycled through his greatest "hits" (#1: "No"). You really had to know where to look.

lowercased2• 8 years ago

“I don’t know why more people don’t do it,” Sanjay said, of programming with a partner.

=======

Maybe because most employers won't pay for that. Two people? For one level of output? You'd have to prove yourself a level 11 googler before most places would give you the luxury of working as a long-term pair.

skybrian3• 8 years ago

I don't think that's it. It's typical to have a team of programmers working on a large project, and many managers are willing to try things.

However, pair programming does require a lot of commitment to make it work. I've only seen it work on a team where it was an XP project to start with, so everyone hired was already willing to give it a solid try. Your average team isn't self-selected in this way and won't have the same level of enthusiasm for a new management initiative.

NelsonMinar2• 8 years ago

I think from the very beginning it was understood they were producing more than one level of output together.

jogjayr2• 8 years ago

Pivotal Labs has had across-the-board pair programming for years[1]. Although the folks there are undoubtedly talented, I'm fairly certain most aren't at Jeff and Sanjay's level.

I don't know why companies don't do it, but it's not cost alone. Here are some of my hypotheses:

1. It seems like it would be more expensive - which might deter companies from thinking about it

2. If you're not already doing it, it's hard to convince all or even a large fraction of your engineers to do it. It's a completely alien way of working to most people

3. It's not obvious to employees who haven't done it that it's clearly better. And it might not be - I honestly don't know

4. For 100% pairing, you need 2 programmers' schedules to line up exactly, 5 days/week - so goodbye flextime, work from home, remote working and all the other arrangements that people structure their lives around

5. It must be exhausting to sit with, and talk to, another person constantly your entire working day

1. https://pivotal.io/careers

macintux• 8 years ago

I’ve paired a bit before, and #5 is definitely true. I can only handle 2-3 hours of it.

groby_b2• 8 years ago

There've been studies on the "more expensive" part, and IIRC it's about a 15% premium in terms of development time - there've been a few studies. Here's one I could turn up quickly: https://collaboration.csc.ncsu.edu/laurie/Papers/XPSardinia....

Simultaneously, you get a bunch of benefits. (Most importantly, there's no bus factor of 1 for anything)

wpietri3• 8 years ago

Regarding #5, you get used to it. It's intense at first, but after a while, it's just normal. One secret is to take actual breaks. And another is to not over-work. When pairing, there's none of that stuff that might stretch out your day. E.g., checking Hacker News. It's solid work time, and after 8 hours of that I'm definitely done for the day.

lazyasciiart• 8 years ago

Thats definitely not the only reason. I've been on teams where managers were keen on pair programming but only a couple people ever did it.

tareqak• 8 years ago

After reading this article, I'm surprised that Google doesn't encourage pair programming / developing developer relationships at the level that Jeff and Sanjay have here. One might even argue that the manner in which interviewing and hiring is currently done in the tech industry in general might actively work against building the sorts of relationships where the output is greater than the sum of the inputs.

Edit: added a missing "the".

why_only_15• 8 years ago

Their specific choices seem hard to replicate. Very strong friends, both very strong programmers, both willing to work at the same company. I'm sure that parts of their successes can be applied elsewhere, but from the article a lot of it is about having a strong relationship.

sdenton4• 8 years ago

See also Andrew and Brad: https://m.youtube.com/watch?v=yG-UaBJXZ80

skybrian3• 8 years ago

In a way, Google does encourage it by requiring all code to be reviewed before check in. If you're pair programming, you already have a reviewer and can check in right away. One-person projects can be more frustrating due to the delay.

It doesn't seem to be enough to encourage widespread pair programming, though.

jogjayr2• 8 years ago

Every company I've worked at as a software engineer, even my first where we emailed patches to each other like cavepeople (and no we were not using git) has had code reviews before check in. It's not at all the same thing as pair programming though.

stirner• 8 years ago

Most of the fragile insights that laid the foundation of a new vision emerged not when the whole group was together, and not when members worked alone, but when they collaborated and responded to one another in pairs

The article gives several examples of this dynamic, but one that came to mind for me was how Monty Python members created sketches for Flying Circus:

Writing started at 9 am and finished at 5 pm. Typically, Cleese and Chapman worked as one pair isolated from the others, as did Jones and Palin, while Idle wrote alone. After a few days, they would join together with Gilliam, critique their scripts, and exchange ideas. Their approach to writing was democratic. If the majority found an idea humorous, it was included in the show.

https://en.m.wikipedia.org/wiki/Monty_Python

jamestimmins• 8 years ago

Anyone know where folks like Peter Norvig and Urs Hölzle fit on the L1-L11 scale? I assumed that all of the early employees/now legends were at the top level, so I was surprised to see the article state otherwise.

stdplaceholder3• 8 years ago

Urs doesn't need a level on the individual contributor scale; he's a Senior Vice President.

joshuamorton• 8 years ago

Urs however does have one, he's listed as a Google fellow (L10) if you go looking.

As a general comment, to compare, a director is L8 equivalent, and VP L9, (but on the management ladder) although at VP+ it gets tricky because there are subdivisions.

Also worth noting that the management ladder goes up more. But the total number of people who are >L11 is probably <11, so it's not like there's any data on how exactly that's broken up.

Shebanator• 8 years ago

Peter is a director, and Urs is a senior VP. As such, they are now both on the management ladder. I think the article was discussing only people who were on the eng ladder.

pk2200• 8 years ago

I can't remember what Senior VP equates to on the tech ladder, but in terms of his overall impact on the company, Urs is definitely an 11.

warunsl• 8 years ago

Was surprised to hear the audio version of the story has a different title - "Binary stars". I think the editorial board went with a more click friendly title.

panopticon• 8 years ago

That was also the title of the print version. The New Yorker tends to re-title online "reprints" with less vague/terse/artsy titles.

timdellinger• 8 years ago

Does anyone have a reference for "In 1966, researchers at the System Development Corporation discovered that the best programmers were more than ten times as effective as the worst." ?

I'm seeing the year 1968 at http://thecomputerboys.com/?tag=black-art#fn-156-1

Is there a 1966 reference out there somewhere? Or perhaps the actual study was done in '66 and didn't make it into press until '68? I didn't pull the original paper.

(The fact checkers at the New Yorker are typically very sharp.)

edit: I pulled the original paper (10.1145/362851.362858). The studies were done in 1966!

Interesting to see the genesis of the myth of the 10x programmer.

azhenley• 8 years ago

It is awesome that you were able to find this. This very well may be the genesis!

For anyone else, that DOI points to: https://dl.acm.org/citation.cfm?doid=362851.362858 "Exploratory experimental studies comparing online and offline programming performance" from CACM 1968.

akhilcacharya2• 8 years ago

I'm not so sure it's a "Myth" when Jeff and Sanjay are out and about. Honestly, I'm beginning to think it's inherent and can't be taught.

thwy12321• 8 years ago

"Interesting to see the genesis of the myth of the 10x programmer."

How could it be a myth? Ever see a professional sports player compete against a non professional player? Why wouldnt this dynamic exist in programming?

superhuzza• 8 years ago

Maybe I'm unimaginative, but in sports where performance is objectively comparable, I can't think of any where professionals 10x amateurs.

Amateur marathons are about 4 hours, professional are about 2 hours. Amateur deadlifts are about 500lb, professional are about 1000lb. Amateur long jump might be 6m, professional almost 9m.

dlederle• 8 years ago

I emailed the author and he pointed me towards this copy of it: https://apps.dtic.mil/dtic/tr/fulltext/u2/645438.pdf

I haven't read through it yet, but it appears to be a more extensive whitepaper version than the CACM version.

bertmuthalaly• 8 years ago

I was so delighted by this piece and then I got to the byline - James Somers strikes again!

You might know his writing from “You’re Probably Using the Wrong Dictionary” or his amazing article about New York’s subway system told through the lens of the countdown clock: http://www.theatlantic.com/technology/archive/2015/11/why-do...

Highly recommend all his work: http://jsomers.net/

misiti3780• 8 years ago

seems to be a awesome engineer too.

sjg007• 8 years ago

This article will queue up a slew of startups that pair program.. perhaps a good thing!

partycoder• 8 years ago

It is refreshing to see an article celebrating collaboration.

sytelus• 8 years ago

Interesting tidbits:

To solve problems at scale, paradoxically, you have to know the smallest details -Alan Eustace

Today, Google’s engineers exist in a Great Chain of Being that begins at Level 1. At the bottom are the I.T. support staff. Level 2s are fresh out of college; Level 3s often have master’s degrees. Getting to Level 4 takes several years, or a Ph.D. Most progression stops at Level 5. Level 6 engineers—the top ten per cent—are so capable that they could be said to be the reason a project succeeds; Level 7s are Level 6s with a long track record. Principal Engineers, the Level 8s, are associated with a major product or piece of infrastructure. Distinguished Engineers, the Level 9s, are spoken of with reverence. To become a Google Fellow, a Level 10, is to win an honor that will follow you for life. Google Fellows are usually the world’s leading experts in their fields. Jeff and Sanjay are Google Senior Fellows—the company’s first and only Level 11s.

akhilcacharya2• 8 years ago

Level 2s are fresh out of college; Level 3s often have master’s degrees. Getting to Level 4 takes several years, or a Ph.D.

I..don't think this is still accurate. New grad SWEs are L3's.

habosa• 8 years ago

Yeah it's mostly accurate but drop level 1. Lowest I've seen for technical employee of any kind is 2. New grad is 3. 4 is about 2 years later. 5 is the point where you have "no expectation if advancement" and is the most common level for mid-career people.

sandGorgon• 8 years ago

Does pair programming only work with friends?

wpietri3• 8 years ago

Nah. You can do it with anybody. I've been part of long-lived teams that pair. But there we rotated pairing partners once or twice a day, so that over the course of the week you'd pair with everybody.

I think that's much more sustainable for most. Pairing with the same person every day is sort of like sharing a snowbound cabin with them. There are very few people I get along with so well that they wouldn't grate after a few weeks.

AlexCoventry2• 8 years ago

Enemies are the best to pair with.

the_arun• 8 years ago

There was an earlier thread in HN on them here - https://news.ycombinator.com/item?id=16744353

continuations2• 8 years ago

The world’s most robust computer systems... used special hardware that could tolerate single bit-flips. But Google... bought cheaper computers that lacked that feature.
Together, Jeff and Sanjay wrote code to compensate for the offending machines.

I'm guessing the article is referring to Google buying servers with non-ECC RAM.

How do you write code to compensate for non-ECC RAM?

mav3rick• 8 years ago

By pushing this complexity up the stack and also having redundancy in storage. What's not correct can be an error and result in retries from the client that may be routed to good storage.

continuations2• 8 years ago

But how do you even tell something's not correct? For example in the article they had to manually examine the index and find out some words were missing. How do you detect that not on the hardware level but up on the software level?

quicklime• 8 years ago

When a car goes around a turn, more ground must be covered by the outside wheels; likewise, the outer edge of a spinning hard disk moves faster than the inner one. Google had moved the most frequently accessed data to the outside, so that bits could flow faster under the read-head, but had left the inner half empty; Jeff and Sanjay used the space to store preprocessed data for common search queries

So Google was a company that was too cheap to buy ECC RAM, but willing to let half of their hard disks go to waste. I'm struggling to understand the logic behind this; can anyone explain?

xapata4• 8 years ago

RAM is expensive. Disk is cheap.

hn_throwaway_99• 8 years ago

Wow, this article is pretty amazing to me. I have a joke (err, kind of a joke) that I like to say, "It's never a cosmic ray." What I mean by that is that when you are debugging and getting incredibly frustrated, right when you feel that you want to give up because you can't find the bug, that you shouldn't blame "cosmic rays" because 99.999% of the time it's a logic problem.

Here was a case where it actually was cosmic rays (at least, not a logic problem), and these guys found it. I also agree with the title and intro of the article - I think it's very possible that if these 2 exact people hadn't been at Google at the time than Google's path could have been very different. Makes me also appreciate how much luck is involved in any successful indeavor. Not luck that they found the bug (that was hard work), but luck that these 2 guys found themselves working at Google right when Google needed them.

true_tuna• 8 years ago

It’s never a cosmic ray but it’s often bad ram. You don’t have to inspect the binary to diagnose bad ram. The system would be behaving erratically in other ways due to regular memory corruption. It’s an amazing story but by no means the only way it could have played out.

nyc1112• 8 years ago

On Sanjay’s monitor, a thick column of 1s and 0s appeared, each row representing an indexed word. Sanjay pointed: a digit that should have been a 0 was a 1.

I was curious how he can see "a digit that should have been a 0 was a 1" in a "thick column of 1s and 0s"? Can you give a specific example. I'm not questioning that he did but I'd like to know a specific example.

stdplaceholder3• 8 years ago

For example the high bit in blocks of ASCII text will always be zero. In a binary dump it would appear as a column of zeros.

nyc1112• 8 years ago

Ok. So you don't need to actually "read" and understand the binary code like you are reading a programming language. You look for out of place numbers.

techs• 8 years ago

Great article! Two powerful minds working on analyzing and solving on same thread of thought. Kudos to the writer, very well presented.

hi41• 8 years ago

It is such a beautiful story of friendship. Even married couple don’t this level of match in mind. Only a few a luckily like that. Their partnership has resulted monumental contributions.

pumaNpl• 8 years ago

Awesome article

gt2• 8 years ago

Serious question - is this an ad? For the company, showing they have a solution to new growth problems which will keep them a growth company? (custom chip, architected for AI)

And how they solved the problem before with this team?

The Friendship That Made Google Huge

Related Stories