Google Engineering Management Mistakes

https://docs.google.com/present/view?id=ajfcdscjzbvp_976rt9cfw

by auxbuss • 16 years ago

296 135 16 years ago

docs.google.com

plinkplonk3• 16 years ago

"Stack ranking system was harmful:"

whoa! I didn't know Google practiced stack ranking. There is a lot of unnecessary stress generated at Microsoft due to this[1]. I would have thought the Google folks would be wise enough to avoid it.

As for ".... SVP response: “If you wanted to get promoted for these non-engineering tasks, move into management"

heh! How big company ish is that? The best way to join Google these days seems to be as part of an acquired startup. Then you are at worst running out any vesting period while enjoying those free massages.

[1] http://minimsft.blogspot.com/2005/07/microsoft-stack-ranking...

tptacek5• 16 years ago

I had to look up "stack ranking" just now, so, for the benefit of the class:

Stack ranking means that if you have a group of 20 people, during each performance review cycle the group manager is expected to rank them from 1 to 20, and the company distributes bonuses and promotions accordingly.

That just seems completely asinine.

plinkplonk3• 16 years ago

"Stack ranking means that if you have a group of 20 people, during each performance review cycle the group manager is expected to rank them from 1 to 20, and the company distributes bonuses and promotions accordingly.

That just seems completely asinine."

Worse, there is often an associated ideal distribution ("the curve") with fixed ratios for "exceeds expectations", "below expectations" and so on and managers are under pressure to conform. If you have an all superstar team, a certain number must still be in the bottom most bucket, and the bottommost 5-10 % is often let go or put on a watchlist (I am not saying this is how Google does it).

Over a few years this gets very political as you can imagine. The (bad) middle managers love this part of their jobs.

matwood2• 16 years ago

I once worked at company that employed a lot of hourly wage type people. One exec wanted to stack rank each group of wage employees each month and fire the bottom 25%. I tried to explain to him the negative morale effect that would lead to lower performance across the board, the fact any employes who were decent would just leave on their own, and there is a significant training cost for each new employee that he was ignoring. But nope, he wanted me to write the system anyway.

Thankfully someone higher up with a bit of sense got wind of the plan and nixed it. It wasn't too much later that the original exec was canned.

BarkMore2• 16 years ago

Stack ranking means just that: the employees are ranked. What the company does with the ranking can vary. The uses of a stack rank can be beneficial or harmful.

At one company where I worked, the rank was used to identify employees for promotion. If an employee consistently ranks above employees at a higher level, then the employee should probably be promoted.

The company also merged the ranks for smaller groups into groups of one hundred or more. This resulted in more consistent reviews and promotions across the company.

I consider these uses of stack rank to be beneficial.

The company also did something harmful with the stack rank. The company graded on a curve and used the stack rank for the grading. Even if everybody in a pool is doing great work, somebody needs to be given a bad review.

btilly2• 16 years ago

There are many, many ways to set up a stack rank system. People that I know who have worked at both Microsoft and Google far prefer Google's system. That is not to say that they find Google's system perfect. Just significantly preferable to Microsoft's version.

As a Google employee I am not at liberty to discuss what the differences are.

sriramk2• 16 years ago

That's interesting. I'm guessing these are Microsoft employees who quit before the perf review system here at Microsoft was basically changed from scratch 3 years ago?

aaronbrethorst• 16 years ago

I was thinking the same thing ('oh man, i bet a Microsoftie could have created the same deck back in the mid/late 1990s...')

piaw11• 16 years ago

The massages aren't free. They're subsidized at $20/hour.

tptacek5• 16 years ago

Wow, this is one of the coolest things HN has found in awhile.

It's remarkable how, time and time again, formal incentive compensation schemes are shown to cause more harm than benefit. I started paying attention when Joel Spolsky called it out 6-7 years ago, and then read Peopleware (an excellent book) which reiterates the point (and calls it "teamicide"), and then read the Harvard Business Review article ("Why Incentive Plans Cannot Work").

Why would a company as smart as Google forge ahead with schemes like this in the face of all available evidence that it's a bad idea?

anonymous246• 16 years ago

> Why would a company as smart as Google forge ahead with schemes like this in the face of all available evidence that it's a bad idea?

My guess: because it can be algorithmized.

neworbit• 16 years ago

But clearly not well

mattyb2• 16 years ago

Spolsky's article:

http://www.joelonsoftware.com/articles/fog0000000070.html

mrcharles• 16 years ago

One of my coworkers has one of those Ship It awards. It's... pretty sad.

shimon• 16 years ago

This presentation comes out against some specific incentives, but doesn't state a judgment on formal incentive schemes in general. From what I've heard, these schemes are quite alive and well at Google. So perhaps they have found a way to make them work, or perhaps they just take them for granted at this point.

In either case, there must be something about the incentive structure that is serving Google well.

piaw11• 16 years ago

Depends on how you feel about the recent spate of defections to smaller companies.

iampims• 16 years ago

Regarding the Incentive system actively promotes bad behavior on slide 7, Dan Pink gave a very interesting talk on Motivation at TED — with surprising results:

http://www.ted.com/talks/dan_pink_on_motivation.html

Splines• 16 years ago

TED appears to be down. Here's the same (I think) video on youtube: http://www.youtube.com/watch?v=rrkrvAUbU9Y

gcheong• 16 years ago

Most likely because they didn't bother to actually look at the available evidence.

tkahnoski• 16 years ago

While we're quoting Joel Spolsky: http://www.joelonsoftware.com/articles/Ladder.html

I wonder if Fog Creek has had the similar frustrations with engineering ladders that Piaw mentions. Although given the distinct difference in company size, it might not be a valid comparison.

cma• 16 years ago

Watch "The Wire"; it is a whole show about incentive structures.

loewenskind2• 16 years ago

Incentives are one of those things that people just want to be true (like e.g. "one true soul mate") and do everything they can to reject any and all evidence that contradicts this belief.

mattyb2• 16 years ago

Context: http://piaw.blogspot.com/2010/10/facebook-and-google.html

kenjackson13• 16 years ago

Curious, why does he end the presentation saying that a tech ladder is bad? Is the idea that they should have a management ladder, but all engineers are at the same level? That sounds like a recipe for disaster.

plinkplonk3• 16 years ago

"Curious, why does he end the presentation saying that a tech ladder is bad?"

From the blog post which provides some context to the presentation (http://piaw.blogspot.com/2010/10/facebook-and-google.html),

"Many consider Google's #1 mistake to be having a tech ladder in the first place. I covered that topic in my post on Promotion Systems (http://piaw.blogspot.com/2010/04/promotion-systems.html) , so did not feel the need to belabor the point."

A snippet from the promotion systems post

"What if I tried to design a promotion system to piss off as many employees as possible? What characteristics would it have?

- No pleasant surprises. In other words, you can only be disappointed if you didn't get a promotion, you can't be pleasantly surprised by a promotion.

- Create unhappiness by dependence on scarce resources. In other words, gate promotions based on scarce resources so that even people who would otherwise be qualified could become disgruntled through no fault of their own.

-Eliminate accountability from people who make the promotion decisions (e.g., through a committee). That way, promotion decisions can seem arbitrary.

-Ensure that promotions are competitive races between all qualified candidates. This ensures that people who manipulate that packet in such a way as to have the best looking packets will win over people who are trying to get feedback and improve, which is supposedly the point behind all these feedback systems.

When I looked at Google's promotion system through this lens, I was very impressed. It seemed as though the system was designed to create disgruntled employees out of people who might otherwise be perfectly happy. And note that the ex-Googlers I interviewed were people who benefited from the system, as in: they were very well respected and were hence ranked highly. They were people who couldn't care less what their ranking was, yet they disliked the system anyway, purely because they thought that having a ranking system actually had a deleterious effect on the exchange of ideas."

There is a follow up post (http://piaw.blogspot.com/2010/05/promotion-systems-redux.htm...)

A snippet

"My spies at Google tell me that my essay on promotion systems drew a reaction on a Google mailing list that I didn't read even back when I was employed at Google.

I did get a few interesting responses. I won't post them here, but I'll address what a few points:

Someone.....pointed out that the real compensation at Google was much more dependent on raises, stock and bonuses, so the promotion system didn't matter. But that's not true at all! First of all, there was a level-based salary cap, and people did hit them. So at some point, you have to get promoted if you want your salary to keep going up. Bonuses were based on salaries, so those don't go up if your salary doesn't go up. Secondly, even if it was true that options and bonuses was all that mattered, then you just validated the claim that eliminating the engineering ladder wouldn't hurt productivity at all!"

I've found Piaw's posts to be very worth reading, even if you don't end up agreeing with him. Many gems on his blog.

kenjackson13• 16 years ago

Interesting, as historically the problem has been quite the opposite. If you go back not that far most devs simply had a flat position in the org. There were no Distiguished Engineers or even Architect titles. You were a programmer.

Of course people started to get upset because the only way they could advance in the company was to go into management. A corporate VP can make $250k/year but a programmer never could. Maybe if you single handedly implemented Google you'd get a $50k bonus that year, but your base salary is still $80k/year. And maybe you get a 15% pay raise for your great contribution.

And when everyone is a "programmer" it is more apparent than even now that everyone is a replaceable cog.

It sounds like the Google promotion system is just bad. Since technical ladders don't have human resource bottlenecks (you can't make everyone the dev manager, but you can promote everyone to Level 4 SWE) you really should have very clear guidelines as to what constitutes a promotion.

When I've managed it looks like this: "This is where you're at today. To get to the next level, this is what you need to do. With your help we've put together this plan for your next 3-6 months. You'll finish up A, make sure B gets deployed. C gets checked-in with QA sign off. Do this w/o ticking anyone off or being a general jerk and that looks like Level XYZ".

Every week we talk about his/her progress against the plan. There's no pleasant surprises, but there's also no big disappointment. If you're not getting a promotion we've known about it for a while, and we're usually both not happy because I've made it clear that something is not going well.

Notice, there's no competition against other employees or any other such race. With that said, there are cases where there is little or no money associated with the promotion, but surprisingly this is often not a concern.

dpritchett• 16 years ago

Career ladders are wishful thinking and an easy way to undervalue contributions. They imply that professional development largely occurs along a single axis rather than multiple ones.

Who's worth more, Peter Norvig or Marissa Mayer? What about an overachieving SWE I vs an mailing-it-in SWE III?

kenjackson13• 16 years ago

Who's worth more, Peter Norvig or Marissa Mayer?

Who's paid more? Who has the better title?

Who's worth more a very strong CTO of Google or an overachieving SWE I? Presumably with title comes responsibility befitting the title.

clr• 16 years ago

The worst part about career ladders at Microsoft was that it put people who were great engineers but terrible, terrible managers in positions of authority, because there was only one advancement path. When I left,they were just starting to create a career ladder for individual achievers, because they had finally realized this. I don't know if it actually ever happened.

pyrhho2• 16 years ago

Additionally, poorly structured career ladders can to lead to "Promotion Until Incompetence", where people are repeatedly promoted until they are unable to warrant another promotion. Thus, everyone is stuck at a level of mild incompetence.

rdouble• 16 years ago

If you just want to do interesting work, an issue with career ladders is that little interesting work filters down to the people at the bottom of the ladder. This becomes a catch 22 for rookie employees. It becomes difficult to level up, because you're never working on anything that will wow your reviewers. In fact, the reward for excelling at boring but necessary tasks is often being stuck on those tasks forever. Thus begins the cycle of churn and lowering of hiring standards.

nostrademons3• 16 years ago

There's a very curious effect at Google that the most interesting work goes to interns. The least interesting goes to heroic senior engineers who've taken it upon themselves to clean up Google's codebase.

This is as it should be, I think. Actually, I've noticed an almost direct correlation between the quality of a Tech Lead/Manager and his willingness to do the boring stuff and pass off the interesting, challenging tasks to his subordinates.

sruffell• 16 years ago

"biggest mistake is still having a tech ladder"?

What specifically is this referring to? A separate career progression for techs vs managers? I thought that was desirable so you don't need to move your best engineers out of engineering positions to promote them. I fear I'm missing something.

mmt• 16 years ago

If I understand correctly, it's suggesting that having formal levels, regardless of whether managers do or not, is what's undesirable.

I infer the alternative to be flat, such that there's only difference in compensation, with no difference in formal title.

Perhaps I'm biased, since I very much believe you can call me whatever you want, as long as I'm suitably paid. This function-over-form attitude is why I prefer startups.

bmj2• 16 years ago

Perhaps I'm biased, since I very much believe you can call me whatever you want, as long as I'm suitably paid. This function-over-form attitude is why I prefer startups.

Then I'm biased, too. I don't work for a start-up (100+ employees, with only seven of us doing software development), but we really don't have "career tracks." There are two management positions in our group (VP and lead engineer), so unless one of them leaves the company, no one is moving up, nor do titles change, even with raises. And this suits me perfectly fine.

martincmartin• 16 years ago

What is a "Peer bonus structure"? I appears in #3 Recognition, "Peer bonus structure was very well done, but not widely used inside engineering."

shadowmatter• 16 years ago

You can give a peer bonus of $200 to any individual with their manager's approval (which is very easy to get). They encourage giving peer bonuses for when someone goes the extra mile on something, but very few engineers initiate giving a peer bonus. After tax, it's about $100, hence the "I can poop $100" bit.

_cbsz• 16 years ago

Is "close to 50%" the going marginal tax rate in general? I calculate ~30% federal + ~10% California + possibly ~6% social security & Medicare to get that number; did I miss anything? (current grad student with a marginal rate a lot closer to 20%, haven't had the joy of full-on taxes yet)

al_biglan• 16 years ago

100% confirms my fear that this is a group of incredibly smart, hyper intelligent people who don't understand that specializing in a field (like, say... Management) isn't something you can "dabble" in and do well. My experience is that most "managers" in Google (when I interviewed, I met a few Directors, Managers and Tech Leads) are 99% technically focused. Google swung the pendulum too far the other way. Engineering should be in charge of the company, but rely on Managers to do management things, not have Engineers dabble in it (with the best of intentions) I would think if Google applied the same hiring practices to hire in hyper intelligent people with a management focus and could integrate them into the teams correctly, this would be a huge win. (I haven't interviewed there since '07, so things may have changed since then)

brown9-22• 16 years ago

I wouldn't spend too much time looking down your nose at Google's mistakes here, as piaw cautions that they're still a really great company to work for:

-At 1500 people, Google was more agile and less political than most 200 person startups!

- At any given size, Google was the best company in its class to work for.

- At its current size, Google is still more desirable an employer for engineers than Oracle, Cisco, Microsoft, Adobe, Apple, and many other well known names.

I think you'd find many similar stories at other software companies if you applied as focused of a lens at them.

kenjackson13• 16 years ago

This also reminds me of a paper I had submitted in the past. Randomly promoting people is more effective than using merit:

http://arxiv.org/abs/0907.0455

VladRussian3• 16 years ago

do you know of any research on randomly electing people in the government offices? I believe it would also be much more effective.

kenjackson13• 16 years ago

In the US we've been doing this for years.

BenSS• 16 years ago

This is great, but I think EVERY tech company struggles with finding the right balance here. Where I currently work, they also fail each and every one of these.

piaw11• 16 years ago

That's not true. I think the traditional management systems do a good job of not setting bad incentives for tech leads.

paradox95• 16 years ago

Any large company is going to have those same mistakes. There is really no way around it. Especially the part about only the big sexy project got recognition and not the little grungy tasks. In a company of 20,000, unless you have a 1:5 manager to employee ration at all levels you can't avoid that. If these are Google's biggest mistakes then I would happily go to work for them.

piaw11• 16 years ago

I think that it is possible to build a culture where the grunchy unsexy work becomes much less un-thanked. Microsoft at its best did it: Raymond Chen was much celebrated for his heroic efforts at preserving backwards compatibility in Windows. In fact, when the Raymond Chen camp lost its political battle, Microsoft got Windows Vista, which arguably cost it billions. I think the stakes are very high in exactly this sort of political battle, and it is worth while for top engineers to spend their time fighting for precisely this sort of culture shift.

LiveTheDream2• 16 years ago

"The sum of money just appeared in my bank account, but was so insulting low that I felt devalued. If a manager had just talked to me about how much my work was appreciated, it would have been better than money."

This is true at a certain threshold. Appreciation is immeasurably better than an demoralizing, insultingly-sized bonus. However on the other end, if you go above and beyond normal expectations and save or earn the company millions of dollars, a "thank you" from your boss on the company newsletter is not going to cut it.

gruseom2• 16 years ago

I would challenge this. Let's replace your trivializing phrase "a 'thank you' from your boss on the company newsletter" with just recognition, which obviously can take many forms. We have evidence, cited in this thread, that recognition is generally more satisfying and motivating than financial rewards are (at least those produced by formal incentive plans). What evidence do you have that this effect only holds "at a certain threshold"? Or that it weakens as one's work becomes more important?

It's important to resist one's intuitions when faced with a counterintuitive finding, which this is. Most of us have been conditioned to think of "never mind the touchy-feely, show me the money" as hard-nosed realism. That it turns out not to be is immensely interesting.

oconnor0• 16 years ago

I think part of it is that if you've just saved your company a million dollars getting a bonus of a few thousand seems unfair & like your company's being greedy. Yes, I absolutely want to be recognized for the good work I do, but it would also be nice to get monetary incentives commensurate with the work I've done.

LiveTheDream2• 16 years ago

I left "a certain threshold" undefined on purpose, because it varies from person to person. The point is that the reward must be meaningful to the person being rewarded, otherwise it is much worse than no reward; the person will become cynical, demoralized, and unmotivated.

The main problem is conflicting valuations of the work in question, which holds true whether the the payment is money or recognition. Who is to define how much recognition is satisfactory? Take the Microsoft "ship it" award[1] as an example of a formal recognition plan, which failed.

My evidence, like the evidence in this thread and the presentation, is just personal experience.

[1] http://www.joelonsoftware.com/articles/fog0000000070.html

mturmon2• 16 years ago

What type of award is valued is very person-dependent.

Some people thrive on non-cash awards (like medals and award ceremonies), some like recognition from above, others care more for money.

It's very easy (even expected) for a large organization to guess wrong in a particular case. So that anecdote is not really data.

nostrademons3• 16 years ago

I have, along with the relatively small teams I've worked with, saved/earned Google at least several hundred million dollars. Possibly over a billion. Certainly more than I can expect to earn over my lifetime.

What size bonus is appropriate for that?

I don't view it in those terms. I get paid reasonably well, but certainly not millions of dollars. And that's because the employment agreement I signed when I was hired is essentially an apportionment of risk. It basically says "I will give my best efforts for the company in exchange for $X/year, and in exchange, the company takes on the risk that those efforts will have zero or negative effect."

After all, I don't have to give back my salary when the I cost Google millions of dollars, which has happened on more than a few occasions. Hell, I've done virtually nothing useful since last May (I've done a whole lot, it's just that none of it has turned out to be useful).

I've played the startup game, and for the roughly 16 months that I was working on my own startup, I earned precisely $0. I worked harder then than I do now. But basically none of the risks I took paid off. That's the nature of risk: you win some, you lose some. It's not really fair when you collect huge paydays when you win and small paydays when you lose (though this seems to be the compensation scheme upon which the financial industry is founded).

jfb3• 16 years ago

It's funny how you could take "Google" out of that slide and put "Apple" in it and it would be almost word-for-word applicable. Or maybe it's not funny.

Too, I recognize that this author of this presentation spent time to make it public, and sharing this sort of information is always welcome, so moaning about the format might seem a little rich. But in the glorious tradition of the internet, I'ma gonna bitch about it anyway. I really really hate this new fad of posting these Powerpoint-style slide decks. I would always prefer to read prose, even inexpert prose, than <next> through yet another comically inept link on scribd or google docs.

Moaning off. Thanks again to the author.

piaw11• 16 years ago

Sorry, the slide deck was there as a presentation to Facebook's VP of engineering. I wasn't about to transcribe my own talk.

jfb3• 16 years ago

Like I said, I appreciate that you took the time. I was just being J. Random Internet Asshole.

(But I really do hate slide decks.)

afterburner2• 16 years ago

Honestly, this sounds like the problems at any largish engineering company. At least the nice thing about Google is that they're openly discussing it, and sort of scientifically at that. At other companies, you just resign yourself to politics being more than 50% of the equation, and the company slowly goes to hell in between increasingly rare and brief spurts of innovation.

piaw11• 16 years ago

I don't think Google openly discusses it. At least, certainly if there was such discussion I wasn't privvy to it. I only felt free to post these thoughts post Google.

gaius2• 16 years ago

biggest mistake is still having a "tech ladder"

What does that mean - there there shouldn't be an engineering career track, the only promotions are into management? (To avoid conflict of team leads being evaluated as individual contributors)

piaw11• 16 years ago

See the contextual blog post above and below this comment.

silverlake• 16 years ago

There is no perfect review and compensation scheme. People will figure out how to game the system. Perhaps the best scheme is to randomly change it every few years so people can't game the system permanently.

piaw11• 16 years ago

I've often thought about that. I think it would be a worthwhile experiment.

benzheren• 16 years ago

There is no perfect system and it is always very difficult to find the right incentives for everyone in the company.

othermaciej2• 16 years ago

He says offhand that Google is a better place for engineers to work than Apple, but as far as I can tell, Apple makes none of these five mistakes.

piaw11• 16 years ago

I know several Apple and ex-Apple engineers. Let's just say that the environment isn't even comparable. Google wins hands down. One of my friends at Apple had to file for an information request so that she could debug a problem that spanned multiple layers of abstraction. A Google engineer faced with that kind of inanity would just quit.

othermaciej2• 16 years ago

I've also heard complaints from Google and ex-Google people that wouldn't come up at Apple. Examples: - Launching a hard-to-understand product and providing it with scant marketing support, so it promptly goes nowhere (when Apple launches a product we are committed to making it a success). - Central hiring leads to o way of knowing what kind of project you will end up working on, and high risk that it will be infrastructure grunt work which is not respected by the culture.

Does internal secrecy suck more or less than those kinds of things? I dunno. I would not be so bold as to make a claim either way without firsthand experience of both environments.

brown9-22• 16 years ago

Certainly over the course of Apple's decades they've made many managing mistakes and it would be trivial for an insider to make a list of any five of them, without ordering of "the top five" or anything like this.

It's impossible to be around for so long and to employ many people and not make any mistakes. No company is perfect. Coming away from this presentation thinking "pffft Company X is so much better because they don't do this" is a huge mistake - if you aren't making mistakes, you aren't trying.

brlewis• 16 years ago

I'd like to hear Paul Buchheit's take on this. He was thinking a lot about company culture issues at FriendFeed, and now at Facebook.

piaw11• 16 years ago

Paul "liked" the blog post on Facebook. :-)

leelin• 16 years ago

Wall Street solves this problem quite efficiently! The annual bonus is really the same as the performance review, so they simply tell everyone "you did a great job this year, here are your compensation numbers."

That way, as a rank and file employee you don't really know whether your "review" was above, at, or below average. First order, because you'd have to ask your peers about their compensation, which most people consider rude (at least in the US). Second order, because the bonus is supposed to be an aggregate of your group and individual performance (plus your seniority), so it's usually impossible to find a suitable cohort for comparison.

sdizdar• 16 years ago

I think the problem is much more general. It seems like the underlying problem is that technology companies have idea to divide people into two categories: manager and individual contributors (developers). And then rank them into levels.

This might work for non-tech companies, but I think that is wrong approach for tech companies. One of the key problems is that big money is in "management career path", thus very smart young people which are unable or not willing to go into management will leave to start their own companies.

For example, if you look around you, you see that your fellow colleges and managers are quite different. Some of them are "mentors" (people who like to mentor), "intrapreneur" (people with weird ideas), "fire-fighters" (people who are good in fire mode), "coordinators", etc.

I'm not sure what is the solution. Maybe something like mash up structure where person who is deciding on ones bonus and raise is not his/her manager.

sriramk2• 16 years ago

I actually think this is one of the places Microsoft has done very well in over the years. I don't think used to be the case a decade ago though but the IC career path is as lucrative/supported now as the management path. One can choose to be a Anders Hejlsberg/Dave Cutler as much as one chooses to be a Steven Sinofsky.

Google Engineering Management Mistakes

Related Stories