Google Hasn’t Changed Their Interview Questions

http://blog.geekli.st/post/53477786490/sorry-folks-google-hasnt-changed-their-interview

by gaylemcd • 13 years ago

51 69 13 years ago

blog.geekli.st

seunosewa• 13 years ago

Today I learned that Google's brain teasers are internally known as "estimation or market sizing questions"!

gaylemcd12• 13 years ago

Asking "how would you estimate the number of cars sold in a year?" is not a brainteaser. It's an estimation / market sizing question. You can deduce a reasonable estimate from the population size and other assumptions.

A brainteaser is something like "A man pushed his car to a hotel and lost his fortune. What happened?" That's a brainteaser -- and nothing Google would ask.

Zikes• 13 years ago

He's playing Monopoly!

So, do I get the job?

RogerL• 13 years ago

Is there anyone that have read one of the readily available interview books that can't answer one of those estimation questions? I'm sure the answer to that question is 'yes', but I am also sure that their utter inability to prepare for an interview or reason on their feet will be discovered in the interview by other means.

proksoup3• 13 years ago

"How would you estimate the number of cars sold in a year?"

This boggles my mind. Where? For what purpose? Is the expectation that the candidate googles that question?

I would google lots of different things until I found some answers (car manufacturers sales reports from car magazines, news articles that may have the answer, literally for the phrase "number of cars sold in a year" etc, etc).

If you want me to talk through it, it's just mind numbing making asinine assumptions. Market sizing? Obvious assumptions? What?

I can't understand how anyone can say with a straight face that that question is straight forward and not ambiguous.

The ambiguity is what makes these google interview questions brain teasers, to me at least.

But that's probably the last item in a list of reasons why Google wouldn't hire me :)

ChuckMcM2• 13 years ago

I got asked the cars sold in a year in California question. I answered by saying I would look for a car that was new two years ago, and one that was new this year, and then difference the license plate numbers and divide by 2.

Took the interviewer a while but they finally figured it out.

ubercore3• 13 years ago

My phone interview with Google had something I would probably describe as a brain teaser (which is probably why I didn't get a follow up interview). It was something along the lines of explaining how an insect in a bottle could jump out if the bottle was frictionless. Something like that. Anyway, I don't remember specifics well enough to make a strong statement, but I would argue that this isn't an absolute in either direction -- I bet a certain class of question is used less, but some that border on brainteaser are still used.

sharkweek• 13 years ago

How did you answer that question? -- I honestly don't understand what possibly could be drawn from asking that.

Deductive reasoning? Logical thought pattern? Creative thinking? Seems like there are a lot of better ways to find out if someone is capable of such things

makerops• 13 years ago

"Quit wasting my fucking time google-interviewer."

"Correct. You're hired."

diminoten2• 13 years ago

Using a swimming technique to push off the air the bug could either gain momentum by using the bottle's sloped bottom as a kind of half-pipe, or start moving in a circular path until sufficient speed were built up that the bug would begin moving up the bottle's side.

Alternatively, the bug could fly out if it's that kind of bug. Or it could knock the bottle over by rocking it.

Clearly I'm Google material because of my "bug in frictionless jar" degree.

ubercore3• 13 years ago

I fully admit there might have been a detail I'm neglecting that made it more clearly useful, but I do know for a fact it didn't fall in the "market estimation" bin, either. At least not fully or obviously.

gaylemcd12• 13 years ago

I would agree with you that that's a brainteaser -- or at least a stupid question that shouldn't be asked.

However, brainteasers are indeed banned and they always have been. The problem is that what you and I call a brainteaser isn't what everyone else calls one. Thus, a brainteaser could still be asked, despite the ban.

ubercore3• 13 years ago

Agreed, and that's fine. I think this is a clearer statement of what I was trying to express with my initial post :)

spankalee• 13 years ago

You actually got asked that question? That shouldn't have happened, I'm sorry.

phamilton• 13 years ago

I was asked how I would roll out a kernel upgrade to a datacenter on the Moon. I thought that was a little silly (though I get that it was an easy way to provide strict constraints on a problem).

vsla• 13 years ago

I aced the teaser back in college a s still didn't get the internship... What does that tell you?

diminoten2• 13 years ago

That you didn't ace the teaser!

Or that there are other parts to a Google interview besides the teaser that can determine your employability at Google.

gaylemcd12• 13 years ago

Yup. Exactly.

morpher4• 13 years ago

I personally think that "estimation" questions (sometimes called "Fermi problems" since Enrico Fermi was famous for asking them) are a great way of gauging an individuals thought process.

They aren't about getting the right answer, but rather about seeing how one breaks down a seemingly difficult question into simpler pieces and determines what can be reasonably estimated. They really don't fall into the category of "brain teasers" which involve some sort of trick.

However, the statement that software developers are not asked these types of question is not universally true, nor should it be. In and SDE interview, I was asked an estimation question that was both interesting and relevant to the position (although I signed an NDA, so I'm not going to give specifics).

In my case, there were several pieces of required information that I realized I couldn't reasonably estimate on the spot, so I gave a description of how I could ballpark them from quick measurements.

yelnatz2• 13 years ago

How many golf balls can you fit in a bus?

How many plumbers in New York?

jareds• 13 years ago

You can fit two goulf balls in a bus and there are two plumbers in New York.

morpher4• 13 years ago

As I said, relevant to an SDE position at Google. Those are not.

gaylemcd12• 13 years ago

Try something like: I have a 100 billion webpages and wish to build a hashtable-like structure that maps from each word to a list of the documents that contains that that word. How many computers do I need to hold this hashtable?

fecak• 13 years ago

I'm a little surprised that all the comments thus far and the article itself never used the term "Fermi Problems". Whether or not someone calls them a brainteaser is up to them I guess. I too think of brainteasers as trick questions with some gotcha answer that you normally wouldn't think of based on how the question was phrased. Fermi problems aren't in that category for me either.

asnyder2• 13 years ago

Sounds like she's still perpetuating the problem:

Software Engineering interviews will focus on your standard coding, algorithm, and system design questions...

Why are algorithm questions still being asked in a high-pressure environment? Very few people actually work on algorithms once hired and never in my experience has it actually been a good indicator of actual development competency. As DHH states: http://37signals.com/svn/posts/3071-why-we-dont-hire-program..., unless you're hiring someone to code algorithms it's not useful.

As a Director at AppNexus I've done my best to reverse this trend by asking what I consider competency questions. Such as "On a scale of 1-10, 1 being novice, 10 being creator of said technology, how would you rate yourself?" Then based on this answer I'll ask a question at that level. I find that most people screened don't actually know the basic fundamentals of the technologies they list.

After you've gotten the basics down you can then get into system-design, or thinking questions. In the case code analysis is necessary I think it's much better to present a sub-optimal pre-written function and ask the candidate what the function does, and whether and how it can be improved. In this way I know whether they understand code, and whether they're competent enough to improve.

The faster we move away from these algorithmic questions the better.

morpher4• 13 years ago

How do you account for the Dunning-Kruger[1] effect when asking your "How do you rate yourself?" question? I would guess that answers in the range "6 or 7" could span a really wide range of actual knowledge. Does that agree with your experience?

[1]http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

pisarzp3• 13 years ago

I understand that he just asks real question on that level to calibrate your answer. It's just faster to start from some reference point.

drivebyacct24• 13 years ago

Ask for code. TALK about algorithms at a high level. I have an aptitude for reading people, but my day two impressions of coding with a new person tend to lead to valid assumptions about them months later.

gaylemcd12• 13 years ago

I don't see how I'm perpetuating the problem. I'm just explaining what Google actually does and what they don't do.

gaylemcd12• 13 years ago

"Such as "On a scale of 1-10, 1 being novice, 10 being creator of said technology, how would you rate yourself?" Then based on this answer I'll ask a question at that level. I find that most people screened don't actually know the basic fundamentals of the technologies they list."

See, and I hate questions like that.

If you're going to ask candidates to rate themselves in a technology, don't ask for a number. What I think a 7 is might be different from what you think it is. Just ask the candidate how comfortable they feel in a language. Words work better than numbers.

Also, you should know that many candidates are really scared by someone asking them to rate themselves on a scale from 1 to 10. You're kicking off an interview with a degree of intimidation that's probably unnecessary.

You're right though that many people don't know the fundamentals of the technologies they list. This doesn't mean that they're bad engineers; there's just a lot of confusion around at what point you should list a technology on your resume.

Remember that a lot of companies aren't particularly looking for competency in a particular language. Thus, drilling into the specifics of a technology doesn't assess much for them.

sigsergv• 13 years ago

I'm always answering “7” or “4”.

nostrademons• 13 years ago

I've found that algorithms are one of those things that doesn't come up often in your daily work, but when you need them, you really need them, and otherwise you'll end up delivering a substandard product. It's very much like Joel Spolsky article on hitting the high notes:

http://www.joelonsoftware.com/articles/HighNotes.html

I've found that in my work, a tricky algorithmic problem comes up on average about once a month. It's a tiny fraction of the total work I do - but if I couldn't solve algorithmic problems, then there are whole projects that I simply couldn't tackle. We'd either do something substandard for users, or we'd throw bodies at the problem (which still happens a lot, unfortunately), or they'd need to bring in someone else to do my job.

Moreover, I've found that there's a cultural shift that happens when enough people are familiar with algorithm design within a company. A lot of other places I've worked are basically "Make the user fill out a form, dump it in a database, format it for display." When a significant fraction of your employees are capable of hitting the high notes, this becomes "Make the user interact with your system in the manner they're most comfortable with, extract meaning out of their ordinary behavior, compute interesting results, and show it to them." The complexity moves from the user to the software, and as a result, users would rather use your software. There are whole subsystems within Google Search - synonyms, spelling, refinements, authorship, snippets, ranking, translate, voice search, etc. - that would never have happened if there weren't a critical mass of people willing to dive into difficult problems.

nakedrobot2• 13 years ago

I had dinner with 4 googlers, one of whom was very proud to ask the Prisoner's Dilemma to lots of interviewees.

Sorry but this article is false.

pisarzp3• 13 years ago

How is Prisoner's Dilemma a brainteaser?

gohrt3• 13 years ago

The prisoner's dilemma is a basic game theory concept (with a lot of interesting math inside it). It isn't a brainteaser and there isn't even an "answer"

gaylemcd12• 13 years ago

No, it's not. Brainteasers are banned and have been forever (or at least since 2005). However, everyone has a different definition of what a brainteaser is. You think the Prisoner's Dilemma is a brainteaser, but other people (as you see in the comments below) do not.

raldi3• 13 years ago

For anyone wondering what relevance an estimation question could possibly have in real life, imagine it's 2004 and you're about to launch GMail. How many hard drives are you going to need?

vampirechicken• 13 years ago

all of them.

drivebyacct24• 13 years ago

That's a complicated question with a complicated, nuanced, business-driven answer.

raldi3• 13 years ago

And yet someone had to answer it.

codex• 13 years ago

Yes, they did--just not recently.

According to Laszlo Bock, senior vice president of people operations at Google, "We found that brainteasers are a complete waste of time." How did they determine that without asking candidates any brainteasers? Clearly, they did at some point--probably before the author's direct experience.

This article is really just a means to promote the author's books and prevent readers of the original article from presuming that that her books are obsolete now (they're not).

gaylemcd12• 13 years ago

(1) I was at Google when the study was done. It wasn't before my direct experience.

(2) I've confirmed with a bunch of people currently at Google that, indeed, nothing has changed. This study was done 5+ years ago.

(3) My books don't focus on brainteasers. Thus, if people believe these changes at Google are real, then this would actually make my books seem more relevant, not less.

(4) If brainteasers are banned, this doesn't mean that no one has even asked them. Some people break the rules (because they're unaware of them or because they don't feel that a particular question is a brainteaser). Thus, Google could, theoretically, study how effective brainteasers are even while they are banned.

(5) What Laszlo is saying is provably incorrect. He's saying estimation questions are brainteasers and that they no longer ask such questions. This is false. If he wants to define these questions as brainteasers, he's welcome to do that. However, he would then be wrong about Google continuing to ask brainteasers. By his definition, Google absolutely does ask these "brainteasers" frequently.

(6) What I really suspect is going on is that he misremembered the study (a reasonable thing to conclude, given #5). It was, after all, done 5+ years ago. I don't think the study ever actually looked at brainteasers. I read the results, and I don't remember anything about brainteasers. (They did look at interview scores and job review scores though.)

(7) Huh? It's "just" a means to promote my books? That's a huge leap. My books aren't even mentioned anywhere except for in my bio. You could argue that it indirectly promotes my books, but then basically everyone ever writing anything is promoting their stuff. And, even so, you couldn't say that it's just to promote their stuff. This article is a means to counter a lot of the myths around Google hiring practices. I don't like candidates walking into interviews misinformed.

delasher• 13 years ago

Thanks for writing this, Cracking the Coding Interview has been one of my go-to books these last few weeks. A must read for any new grad, and the coding problems / answers are very insightful.

johnobrien1010• 13 years ago

I don't know about Google in particular but I've always been bothered by the market sizing question.

In part, it's because it seems to have come out of consulting, where a common task seems to be to size a market for which there is no commonly available data. So, understanding how someone might go about doing that seems reasonable on its surface.

But in fact, I've met too many consultants who seem inclined to build "castle in the air", spinning market details about theoretical markets without properly defining either the product or service or the buyer of that product or service. And since they are for markets for which there is by definition no solid data (since that is why the consultant was asked to size it in the first place), their estimate is never validated (or if it is, it is so long after they have left that they never hear about.)

At the same time, no one ever seems to mind if the answer to such questions is off by a country mile when the actually size of the market is known, so long as the thought process was rigorous and shows the right kind of logical decomposition of the problem. But this justification for the value of the question also bothers me, as it seems to be testing more whether you can come up with convincing-sounding bullshit than whether you can correctly estimate a given value. This, again, may be an accurate measure for whether someone can become a consultant, but I never liked the notion that we should judge people on how well they can spin bullshit.

Finally, the question seems to also be gauging the interviewees willingness to "play along" with what an interviewer is asking. The questions are often on seeming random topics (piano tuners in New York or gas stations in Wisconsin), and are often not directly related to the domain. In the real real world, you would actually probably find a list of such things on the internet, or conduct some basic research that can be done to answer such a question. Or the correct response might be something like, "We can make a rough estimate, but without more solid data than a few random facts which we've rubbed together to come up with a market size, maybe we shouldn't be pursuing this market". In which case, the contrived example serves to allow the interviewee to demonstrate that they are the type of person who will enthusiastically pursue whatever random intellectual exercise they have been assigned by the interviewer, so long as there is a chance of getting the interviewers approval.

All of the things that the question tests for, then, seem to not be characteristics you would actually want in someone if they were to answer the question in the real world. In the real world, you might do a back of the envelope calculation, sure, but you would also do a lot of research on the internet, conduct a lot of interviews and surveys, and/or conduct evaluations of competitors to understand and size a market.

morpher4• 13 years ago

I would argue that if one is unable to do the back of the envelope estimate, then they would have a difficult time doing the rigorous calculation / determination. The main difference between the two is the quality of the numbers that go into them, right?

I agree though, that overconfidence in one's ballpark estimate is not to be desired. But, that's more useful information gained from the question, not less.

autarch• 13 years ago

Personally, I'd find these questions a lot more palatable if they'd change the format a little. Instead of asking "how many gas stations are there in Manhattan" why not ask "what steps would you take to estimate how many gas stations there are in Manhattan"?

The latter makes it clear that this isn't some sort of brain-teaser, it's question about process. You could also phrase it as "what information would you need to estimate how many gas stations there are in Manhattan?" This in particular would be a good question for people who will doing a lot of work that involves some speculation, such as long-term product roadmaps, long-term expansion planning, looking at new markets, etc.

BTW, I've been to Manhattan and noticed that there are remarkably few gas stations there for a city of its size, so whatever estimate you come up with will probably be very, very wrong.

mdisraeli2• 13 years ago

Changing the wording of a question can dramatically change the outcome, and which skills are tested.

"how many gas stations are there in Manhattan" actually makes sense as a question for a venture capitalist firm employee expected to make investment decisions, as the leap from "statement" to "this is something I need to estimate" is absolutely key to the role, and hence something you'd want to test for. I'm sure others can think of much better examples of this kind of thing.

Rather than massive re-wording of the question, you can instead change the framing of the question. The most scary aspect of such questions is that they might come out of the blue, or immediately after a number of completely different questions. Mental inertia will then dictate you stall massively and find the question completely confusing.

Imagine instead that the interviewer stated "We are going to ask some questions that will test for reasoning and deductive skills that you will use on a regular basis in the role". This is similar to changing the wording, but more like real situations. You know you have a meeting with clients (so you're prepared), but they'll ask questions in a difficult and something obtuse manner.

gaylemcd12• 13 years ago

"BTW, I've been to Manhattan and noticed that there are remarkably few gas stations there for a city of its size, so whatever estimate you come up with will probably be very, very wrong."

Yes, and that's precisely the sort of thing you're supposed to take into account. Again, you're looking to get in the right ballpark. They're not looking for precision.

Asking "what steps would you take" wouldn't be quite the same question. They want you to actually take those steps to come up with a number.

Ultimately, I'm not necessarily defending estimation questions. I'm just saying that they are, in fact, asked because interviewers don't consider them to be brainteasers.

rajksarkar• 13 years ago

Market estimation questions are primarily asked in non-engineering interviews at Google like PM, PMM etc. I think market estimation question is a very good way to test how a person thinks. There are no right or wrong answers, mostly the candidates are being tested on their thinking abilities in ambiguous situations, which happens in your real job all the time. For example, you are trying to launch a product in a new market and you have to estimate sales/marketing dollars/headcount the 1st year - how do you go about it? Are you asking the right questions?

jerrya• 13 years ago

I was definitely asked a "cutting the cake" brain teaser question when I interviewed. Cut a cake fairly into X pieces using Y slices.

And it was a reasonable question that may or may not have knocked me out of the job, as long as by reasonable you think it's "fair" to serve some people the bottom half of a cake and other people the top half of a cake.

But I of course don't know if that's what knocked me out.

mdisraeli2• 13 years ago

If you were interviewing for a software engineering position, that question makes sense - that reads to me like an algorithm problem (albeit a three-dimensional one). There's also a couple of other roles for which it would make sense.

The big problem with that question is "cake". Mappings change our entire perception of a scenario (The mass-grave-filling problem of Tetris being a classic example). Cakes have tops, bottoms, fillings, decoration, and so on. This makes a perfectly reasonable question into something much more nasty

gaylemcd12• 13 years ago

Yep. I don't like questions like that, mainly because -- whether it's really an algorithm question or not -- candidates will feel it's a brainteasers. Given that there are more than enough questions that won't be perceived as brainteasers, there's no reason to ask one that might be.

This is the same issue I have with the egg drop problem. It can be logically deduced and so it's sort of a fair question for a software engineer. But, given the abundance of more relevant algorithm questions, there's just no reason to ask it.

Ultimately what it comes down to is this: brainteasers are banned and have always been (or at least for a very, very long time). However, since everyone defines brainteasers a little differently, you could still get a question that you feel is a brainteaser.

Google Hasn’t Changed Their Interview Questions

Related Stories