Cargo Cults and Interview Questions

http://raganwald.posterous.com/bob-and-carol-and-ted-and-alice

by raganwald • 14 years ago

229 158 14 years ago

raganwald.posterous.com

kenrikm• 14 years ago

Great example and a great follow up to "I don't hire unlucky people" As always a pleasure to read your blog.

romey• 14 years ago

How do you strike a fair balance between ridiculous interview questions (such as the example here, write code interfacing with some library from memory) and actually finding out a candidate's chops (assuming they don't have an open source history / github account). I think requiring candidates to have contributed to open source to be an unacceptable solution (although I can certainly see lending some weight to candidates with such credentials)

Anecdotally: We had a guy in for interviews a few months ago. In the phone interview prep by HR, it was mentioned that there would be some written “tests,” and the guy was extremely resistant. We had him in anyway and he did very poorly. The test was pretty simple: some stuff about magic methods in php, a fizzbuzz type question, and a few jquery selector questions (it was for a front- and back-end position), etc. Were we to granular with the questions? Did we fall into the same trap as Ted here?

aidenn0• 14 years ago

1) You do what Bob did; have them write code and then ask why they did it that way.

2) Were the written questions on paper or a whiteboard? If so, you should at least give them the option of doing it on a computer with a screen editor (you know the way that almost everyone has written code since the 80s). We have a few computers with just about every editor known to man (well win32 and *nix only, but we could probably scrounge up a powerbook if we needed to) installed on them along with gobs of syntax-highlighting themes so that candidates can feel comfortable when writing code.

Ultimately some good programmers have trouble writing code on paper, and most good programmers have trouble writing code on a whiteboard (though they should at least be able to sketch out an algorithm graphically or with pseudo code as that's a useful communication tool).

qdog2• 14 years ago

I agree with #2 a lot, although when I start interviewing not sure that would make the first couple of interviews go smoothly. Usually takes me 3 or 4 interviews to get warmed up to the type of problems people like to solve in whiteboard interviews.

When I just randomly interview a single time, I feel like a dolt, do pretty miserably. After a few interviews all the whiteboarding starts to take effect and I do much, much better.

When I interview people I usually explain a problem I'm working on and see if they can think through it, then ask a lot of questions about what they've learned from their past failures. People that can go into detail about something they failed at and learned from usually impress me, as I suspect they won't make the same mistakes over and over. Also, if you haven't failed spectacularly with something very hard and you've been a software dev for 10+ years, you might not be doing it right.

Obviously, I've been a dev for 10+ years, and I can't pass whiteboard tests very easily, so I suppose maybe I'm not doing it right, either...

bithive1232• 14 years ago

Simple tests (what is this jquery doing? why is this ruby script failing?) are great as bogofilters as long as you actually use jquery and ruby on the job.

I interviewed somewhere a few months ago and had a former programmer turned product manager grill me about b-trees (this was a Ruby on Rails position). I was not surprised to hear later that he was the one who gave me the final thumbs-down, I assume he thought I wasn't worth it because I didn't work at the same level of abstraction that he did when he was a coder.

spitfire• 14 years ago

He was seeing if you had a depth of understanding. I do that too.

I want to see that people are aware of what's going on below them. Not that it's just "magic" to some keyboard basher. ruby on rails attracts an extreme number of keyboard bashers, so they get extra scrutiny.

codeonfire2• 14 years ago

It makes it worse that you just know that guy stayed up the night before studying b-trees for hours just to show you up.

EvilTerran• 14 years ago

One option would be to ask them to bring in some code they've written, and then talk through it with them in the interview. Even simple stuff like greasemonkey scripts could be pretty informative as to the candidate's level.

I'd be very wary of asking questions about potentially-obscure corners of tools (I don't know how obscure magic methods are in PHP) -- in a real work environment, you could look it up in a moment, but you usually can't in an interview. You could easily trip up a programmer who thrives with the references to hand, but isn't yet confident enough with the tools not to struggle without the references.

jseims• 14 years ago

If I were to ask 2 questions, I would:

1) Ask something simple, like FizzBuzz. You'd be shocked at how many people talk a good talk but can't actually write basic code.

2) Ask an open-ended question, like "how would you design a multi-player chess game". That leads to a discussion that lets you assess all sorts of traits, the most important of which is communication style.

philwelch2• 14 years ago

So "ability to write basic code" and "communication style" are your top two priorities?

richardlblair2• 14 years ago

I think you were fair. jQuery selectors are pretty easy stuff, and you can pretty much guess at the jQuery interface and probably be right.

This is coming from a python guy who does very little javascript.

If he did get some of the methods wrong you could always ask the candidate what he thought the method did. If he knew there is a method to accomplish a specific task and he just got the name of it wrong that I'd say he is good to go. He can always google that kind of thing on the job.

horsehead4• 14 years ago

I had never heard of the fizzbuzz question, so I googled it. This is what I found: http://www.codinghorror.com/blog/2007/02/why-cant-programmer...

..... I sincerely hope he's kidding. Having less than 8 months experience with a language (AT ALL!) I can write code out on paper that works. IS this 95%+ figure correct ? :S It's come up a couple times in google results, but I dont see where people are getting the number.

sp332• 14 years ago

If you write it on paper, are you sure it's correct?

xsmasher4• 14 years ago

Spot on. Asking questions about esoterica (defined as any language feature that's >0 levels above their current level of knowledge!) is useless.

I'd rather get a candidate to write some (relatively simple) code using what they do know about the language, and then we can dive into the code and see if they really understand what it does, and what happens if I poke this or prod that. I don't even care what language they use - although if they pick one I don't know it may be harder for them because I'm going to ask pointed questions about how parameters and equality work in that language.

The interview "question" is just a starting point for a conversation. Trick questions and minutia don't serve that purpose at all.

astrodust• 14 years ago

Exactly. It's not if they know the answer that matters, but if they can determine the answer. Poor interviewers only test knowledge, not ability.

stcredzero2• 14 years ago

Poor interviewers only test knowledge, not ability.

Right. There are lots of very capable people who have optimized by offloading a lot of information from their heads because they can get it from their tools. These are very different from cargo-cult programmers who are only guessing about which completion in their IDE they should use. However, the two types will be unable to answer many of the same questions when away from their IDE.

DannoHung2• 14 years ago

I don't think esoterica is entirely useless in an interviewing context. I've interviewed a few people for language centric positions before and one thing I liked to ask was, "What feature or piece of this language do you find interesting or frustrating?" Someone who's worked a lot with a particular technology is bound to have something meaningful to say. I suppose they could just not be particularly thoughtful about their tools, but I like working with thoughtful people.

pgeorgi• 14 years ago

The problem is with requiring knowledge about a certain set of "esoterica".

Your question is fine: it's open-ended, the interviewees can answer whatever they feel like (and you can assess their knowledge of the language to some degree), and ideally you get a nice little discussion going, that also covers other topics.

On the other hand "What's up with that explicit keyword and constructors in c++?" is a pop quiz questions.

trustfundbaby• 14 years ago

I agree ... as long as you don't use said esoterica as an immediate qualifier/disqualifier, it can give you an interesting insight the depth of knowledge a candidate has about a particular language. That being said, there are people who know crazy minutae but get into a team and simply can't turn out code at the rate you need them too, for whatever reason. Definitely have to watch for that and pay more attention to how they actually construct and write code.

dalke2• 14 years ago

I've found that I don't think about languages that way. I think of things more like "I want to achieve goal X; what are the interesting or frustrating aspects with respect to that goal?"

For example, someone asked me once about the warts in Python. I started thinking about complaints I've heard others say, like Python 2.x's "print >>fileobj" syntax. I understand the reasons for and against it, but I nearly only use print statements for debugging, so it's not one that I want to bring up. Or there's the lack of block syntax, which Smalltalk/Ruby developers love, but Python has different ways to handle most of the underlying requirements.

So there I'm thinking about the different aspect of Python, trying to figure out which ones I want to bring up, but I hadn't fully considered the other issues with the language, in order to figure out which is the most relevant, but then, I don't know how to rank the relevancy because the question was structured without any rank preference.

Now if you had asked "... when used for systems programming ..." or "... as a programming language for scientists ..." or "... compared with Perl ..." then I would find it much easier to give the thoughtful answer you're looking for.

I had the same problem a couple of years back at our monthly Python meetings. We did a code refactoring exercise: given code, make it better. I had a really tough time because I couldn't figure out what "better" meant. The exercise code worked, although it mixed business logic, database interface, and presentation into a single file. But without an idea of the direction the code was supposed to go for the future, I couldn't figure out why anyone would want to modify working code.

dclowd99012• 14 years ago

As someone sitting idly in a rather crowded Sunnyvale hotel lobby awaiting my own interview, I'm prepared for a Bob, but I'm afraid I'll get a Ted.

I'm not a formally schooled developer. I don't know a heap about traditional computer science. I don't know all the sorting algorithms off hand (but could throw one together with a little time and reference). I'm not even fluent in all the jargon, vernacular and glossary terms that devs bat around.

I don't know if I'm the best for this job, but I know only 3 short years ago, I was doing graphic design and 3 years before that, I was trying to make good on my degree (and dream) of journalism.

None of us choose how we learn, but I think the best of us just know how to.

richardlblair2• 14 years ago

Good luck! I hope you end up with a Bob over a Ted.

wunderland• 14 years ago

This story was poorly written. I had to re-read it a few times just to understand who was speaking when.

qdog2• 14 years ago

Had a random interview scheduled with Amazon a couple weeks ago when they were in town, they didn't even have regular whiteboards because of some mixup. So, they put these pieces of plastic on the windows that was adhesive on one side, worst whiteboarding experience so far. I couldn't concentrate because of all the crap going on outside, poor interviewer probably wondered how I tied my shoes in the morning.

Hope you have better luck ;)

sethg• 14 years ago

I was in one interview at a local startup where there was a whiteboard in the room, but none of the engineers who interviewed me asked me to touch it; they were satisfied to ask questions of the form “what do you know about Java Foobar Technologies?” After an hour or two, I could feel the whiteboard... looming... blank... silently warning me not to take this job.

blhack• 14 years ago

Don't worry, you'll be fine :)

Can you write code? Yes? Well then you probably can fit the role of somebody who is hired to write code.

Nobody on the planet is expecting you to waltz in the first day and grok their entire stack and their entire codebase; expect the first couple of months to be just you getting your head around what you'll eventually be doing.

Don't sweat it, you'll be fine :)

slurgfest2• 14 years ago

This isn't correct in my experience. Interviewers do typically insist on more than simply being able to write code and it is common for them to try to hire someone who groks as much of their stack and codebase as possible from day one.

edit: but maybe it is not the same in Silicon Valley?

seldo• 14 years ago

I agree with the others: you'll be fine. And if not, feel free to apply to our company :-)

horsehead4• 14 years ago

These are the same things I worry about in learning to code on my own -- Will I know the "right" terminology to "prove" to an interviewer I'm capable of doing X task in Y time?

A funny coincidence, i did journalism too -- you're not missing out on ANYTHING (I'm trying to escape the career myself, even contemplating school).

xsmasher4• 14 years ago

That's part of the benefit of a CS degree - you get a common vocabulary for common problems and techniques.

TimGebhardt• 14 years ago

> I'm not a formally schooled developer. I don't know a <strong>heap</strong> about traditional computer science.

Well you sure do talk the talk. ;)

Ba-dum---Crash!

dclowd99012• 14 years ago

;) As I wrote it, I was tempted to kick myself.

srdev• 14 years ago

I don't know how it is with most interviewers, but for interviews that I do: If you don't know a term, just ask. I'm more interested in whether you're familiar with the concept and can apply it than whether you know the appropriate term. Chances are that you're familiar with the concept already and are just not familiar with the name.

If you're in the interview room, I'm not so concerned about book learning and knowing facts, but whether you have the correct amount of knowledge to do the job and the ability to apply it.

robyn• 14 years ago

The best piece of technical interview advice I've ever gotten: It's much more impressive to NOT know something, ask about it, and then be able to understand it and apply it to your answer during the course of the interview, than it is to just already know it.

thebigshane• 14 years ago

  You can know the name of a bird in all the languages of 
  the world, but when you're finished, you'll know 
  absolutely nothing whatever about the bird... So let's 
  look at the bird and see what it's doing — that's what 
  counts. I learned very early the difference between 
  knowing the name of something and knowing something.

- Richard Feynman https://en.wikiquote.org/wiki/Richard_Feynman and https://www.youtube.com/watch?v=05WS0WN7zMQ

RedwoodCity• 14 years ago

If they don't hire you because you don't know the most recent buzz words you probably don't want to work for them. A good job especially in your cases is one where you will learn and feel challenged. If you really knew everything they wanted the job would get boring.

albertsun• 14 years ago

Hey! If it doesn't go well or you change your mind you can always come back to journalism now. We are all desperate for people who can write good code and most of us aren't formally schooled either.

nickbauman• 14 years ago

The symbol of the thing is not the thing: people (should) care about the thing. You don't need a degree to get your head around algorithms. If you can write a simple binary search, for example (arguably the most fundamental algorithm), that's a plus. Therefore a good interviewer will get you to write binary search (the thing) on a whiteboard without saying "binary search" (the symbol of the thing) and see if you can reason your way through it. They will be much more interested in your thought process than whether your syntax is correct.

nextstep• 14 years ago

So we're doing this thread again? Stock interview questions don't test real knowledge, they test knowledge of stock interview questions.

In my experience, the most accurate interview processes is to ask a candidate to solve a short (but non-trivial) problem in the language of their choosing (or pseudocode). Ideally, candidates could be interviewed by a few people independently to ensure an accurate sample of that interviewee's knowledge. This isn't always possible (due to time and other constraints) but this process better guarantees that a smart candidate isn't tripped up by a very unusual question, or that a weak candidate doesn't memorize common interview answers.

j_baker• 14 years ago

I think the purpose of this post was to examine why people decide to ask these kinds of questions.

slurgfest2• 14 years ago

Surely the most accurate would be to ask them to solve a problem closely tied to the sort of job they will be doing (doing variations on FizzBuzz under time pressure does give information, but that information may not be entirely relevant to the job).

And the objective should be to establish a level of capability which is acceptable for business purposes, not to try to optimize on intelligence.

Be truthful with yourself about whether you really want the smartest guy possible. The smartest guy sees the boss's (or company's) bullshit and is more likely to say something about it. This makes trouble. The smartest guy may be a "bipolar" type. The smartest guy, with the mind that needs meat to chew, is more liable to implement things in a clever way than one which average joes will be able to maintain and extend easily. You probably don't want the smartest guy because he probably isn't as good on some other dimensions.

What I see in this industry is that we project intelligence onto the people with traits which satisfy our prejudices (typically people with high class/status). Then we filter on this perceived-intelligence with the rationale that we want a company with people who are as smart as possible. The actual effect is at least as much to favor people with high social status and self-promotion ability and physical attractiveness as it is to favor smart people.

Hiring processes are irrational not because nobody can ever work out the right processes, but because we work backwards from unexamined pictures of "the right candidate" to the processes which will select that candidate.

lusr• 14 years ago

This is pretty much what I've always done - describe the sort of problems they're likely to encounter in the role they're being hired for, perhaps mentioning a few challenges we've had (and possibly already solved), and ask them to explain how they would solve the problem. It's pretty difficult to bullshit your way out of answering a real problem.

Other than adding in some rudimentary skills tests (say, FizzBuzz + walkthrough writing the code with them), and checking if they've developed anything out of their own initiative, I can't see why you'd bother asking anything else.

ryanmolden2• 14 years ago

Perhaps my terminology is off, but as a minor nitpicking point I have never heard of this type of constructor refered to as a copy constructor. A copy constructor is a constructor for type Tthat takes an existing T and initializes the new T to have the same underlying state. So something like

  class Foo
  {
  public:
    Foo(const Foo& other) 
    {
      //copy state from other Foo here  
    }
  };

What this example is talking about is a normal constructor that can be (perhaps unintentionaly) used in the context of an implicit conversion, so something like

  class Foo
  {
  public:
    Foo(int heyThere)
    {
    }
  };

This allows the C++ compiler to create an automatic conversion (in some instances) from int -> Foo simply by inserting a call to this constructor. So it can cause code like this to compile even if it was not intended

  void HandleTheFoo(const Foo& theFoo)
  {
    //Do something here
  }

  void SomeMethod()
  {
    Foo intended; //assume there is a default ctor available even though I didn't show it above

    int whoops = 5;

    HandleTheFoo(whoops);
  }

Marking the constructor explicit prevents the compiler from using it in an implicit conversion context.

raganwald4• 14 years ago

I’ve heard it that implicit conversion is a specialization of copy construction. So you can still use explicit Foo (const Foo& other) if you wish to force all construction of Foos to use the new keyword and not to be passed by value. But I’m certainly willing to accept the suggestion that the two are distinct, in which case my understanding is that the explicit keyword blocks implicit conversion (as described in the story and in your example) and I understand that it also blocks implicit copy construction.

ryanmolden2• 14 years ago

Ah, that makes sense. Though technically you don't have to use the new keyword unless you want a heap allocated instance, something like

  HandleTheFoo(Foo(5));

or even

  HandleTheFoo(Foo(otherFoo));

is perfectly cromulent with the constructor being marked explicit. But that is just me engaging in terminological quibbling, I get what you meant/said in spirit and don't disagree that people asking questions they don't have a deep understanding of themselves can only lead to fail.

evincarofautumn• 14 years ago

Right. The term is “conversion constructor”, one that can be called with a single argument and is not declared “explicit”.

raganwald4• 14 years ago

Thanks! “Given enough eyeballs, all essay problems are shallow."

geebee• 14 years ago

This was an interesting story. Sounds like an astute interviewer, the kind who can really get at the core of technical knowledge. And of course, people have every right to set whatever standard they like in their hiring.

But every time I read about someone like Alice, it does cause me to reflect on the "hiring troubles" companies are experiencing, the "shortage" of top candidates.

She has contributed to open source, shown the ability to learn and write small apps in new languages. She can code, definitely, but she shows some weakness... NO HIRE. Not only no hire, she's a faker, a cargo cult-er, she "pulled the wool over your eyes." She's not just unqualified (by high standards, in my opinion), she's insincere.

Please, understand that I'm not saying that hiring standards shouldn't be high, or that weak team members don't do more damage than good. I get all these things. But the bar is high enough that it should be utterly unsurprising that there's a shortage of "qualified" people, and that a lot of people (who might have become good programmers) bail out, get forced out, or want nothing to do with it after their first taste.

I'm ok with these high standards, but I think we celebrate it a little too much.

EDIT: raganwald, jbooth, peter...

thanks for replying rather than just downmodding - unfortunately (for me), yeah, you're right, I misread this.

I'd like to take it down, but once someone has responded, I leave it up with an EDIT...

peter_l_downs• 14 years ago

Maybe I'm misinterpreting both the original post and this response, but it seems that you think Alice is the "cargo cult-er". I interpreted raganwald's post as critical of _Ted_, criticizing _him_ as a "cargo cult-er".

raganwald4• 14 years ago

I may have done a poor job of conveying why the story is related to Cargo Cult religions.

The term cargo cult programmer may also apply when an unskilled or novice computer programmer (or one not experienced with the problem at hand) copies some program code from one place and pastes it into another place, with little or no understanding of how the code works, or whether it is required in its new position.

https://en.wikipedia.org/wiki/Cargo_cult_programming

I don’t know if this term can be applied to Alice, but I am pretty certain that the story describes Ted blindly copying the surface form of something Bob did in an interview with Carol into Ted’s interview with Alice with little or no understanding of how Bob's conversation worked, or whether it was required in Ted’s conversation with Alice.

Alex3917• 14 years ago

The message was clear, some people just see what they want to though.

jbooth• 14 years ago

I think the article was agreeing with you. The cargo culter was the one who saw someone ask an improvised question about implicit and turns it into a litmus test.

noduerme6• 14 years ago

The point, as always, is that intelligent people find other intelligent people to hire by asking questions that aren't uniform or one-size-fits-all, or which only require regurgitation of facts, but rather by probing how and why a candidate thinks a certain way.

This should be obvious by now. Standard CV reviews, colleges and references say nothing about how efficiently a candidate is able to think through new problems or how elegant their solutions will be. Nor does that mean that formal training and real-world skill exist in an inverse relationship as a lot of self-taught coders (myself included) like to pretend it does. They simply aren't co-related, at least not as far as we can tell from most leading posts about hiring on HN.

So if the goal is efficiency, why not just hire people based on their IQ? What else is there that qualifies people for positions where their bosses 2 levels up have no idea how they're actually executing the work that's put in front of them, and rely completely on them to come to the best logical solutions to new problems creatively, quickly and every single time? The only other standard you might want to look for is trustworthiness, but you can't measure that in an interview.

Anyway, it doesn't matter if someone with 140+ IQ has ever coded before. It's probably worth paying to train them. So let's just cut to the chase. You need people with brains and integrity, and the rest of it is just huff and puff nonsense from HR people who don't have a clue, and couldn't do the job their hirees do.

npc• 14 years ago

firstly, there's legal issues with IQ tests in job interviews in the US at least

secondly, IQ testing covers a number of areas, many of which are not necessarily related to the kind of thinking needed for programming. i think most of the verbal stuff in IQ tests wouldn't relate that much

thirdly, even if someone aces the IQ test, there's still no indication that they would actually enjoy programming if they've never done it before. It's kind of pointless to train someone to code if they're just going to get sick of it and quit in 6 months.

and fourth, an IQ test doesn't say anything about a person's ability to work with other people or prioritize things properly. an IQ test provides no protection against ending up with a prima donna who obsesses over brace style at the expense of getting useful things done.

really, i think the best way to find out if someone can code is to look at something they've coded

noduerme6• 14 years ago

Is it actually illegal to hire in the US based on IQ? I mean how is that discriminatory more than any other test for employment? Race, religion, political affiliation, disability are obviously all irrelevant in an IQ test... so in a way, isn't it more equitable?

You make a good point about primadonnas and other potentially toxic personality types with high IQs, but you can screen that too. And I mean, I don't care about IQ. To me, pretty much anybody with talent should be able to come in with something awesome and say "look what I did", and if it's great, then who cares? Hire them. Obviously this isn't an argument for creating a new narrow-minded structure to replace the current one; I'm just talking about prioritizing along the essential lines instead of dancing around the real question. Isn't someone who codes and who can give you a fibonacci sequence in realtime more valuable than someone who codes, but can't? I can't do it, but I'd hire that guy.

JoeAltmaier6• 14 years ago

Ok, you brought it up. IQ tests test one thing - can you do an IQ test. Scholars love them, write them, and they filter for ... yes! Scholarship!

A human mind is multidimensional. The idea you can find one number to measure it is downright ridiculous. You wouldn't even buy pants based on just one number, say waist size. You'd get home, they'd be too long or too short. Obviously!

But folks buy into the IQ cargo cult big time. Totally baffles me.

jcrites• 14 years ago

> I think most of the verbal stuff in IQ tests wouldn't relate that much

Most software engineering jobs involve communicating with people. Customers, other engineers, other teams, leaders, etc. I would hazard to say that most effective senior engineers will spend at least as much time communicating with people as they will spend on individual technical tasks, if not more.

Communication skills are vital. If you cannot effectively persuade a team of people that your approach or point of view is correct, then your views, no matter their objective merit, will not impact the organization. Being an effective engineer requires having good ideas and also influencing other people to understand and adopt them.

A person can be an effective journeyman individual contributor engineer without strong communication skills, but to become a master with wide impact requires it.

msutherl• 14 years ago

My IQ is lower than it should be because I have what you might call "ADD". The final IQ score is an average of scores from a few different tests. I score high on some tests and average (which is actually well below average for college graduates) on others. The resulting score, then, hides my strengths and weaknesses.

Suffice it to say that I would have a hard time getting a job based on my IQ score because I can't do repetitive arithmetic operations as fast as other people.

JoeAltmaier6• 14 years ago

Yes, my family has ADD in spades. Actually a help in programming jobs - the focus part is great. The intuitive jumping around can be just the thing when chasing a bug.

6 ADD kids from a farming community. One Entrepreneur :), an environmental researcher, a Nursing research specialist, two software directors (Case pumps, Intel) and one IT head.

moron• 14 years ago

Have you taken an IQ test? I have. It didn't strike me as a way of finding out much about a person, except perhaps whether they are able to do difficult arithmetic quickly. Also, people with lots of raw intelligence often find ways to misapply that intelligence in awesomely terrible ways. Smart in the little way, dumb in the big way.

mgkimsal• 14 years ago

I took a couple a few years ago, and the results roughly matched the test results from 30 years earlier. They weren't exactly the same tests, so the raw score numbers weren't the same, but the percentile number hadn't budged a bit one way or the other.

One of the tests had a lot of memorization - given a list of 9 numbers, repeat them backwards. Or 9 letters. Or a combination. Getting read "q9dh73b1wm6" and then having someone expect to hear you say back "6mw1b37hd9q" was quite nerve-wracking. Another test was "say as many names as you can in 30 seconds" (or something like that). Some of the tests appeared really odd, but the test-giver explained the thinking behind some of the questions later, and it was interesting.

aswanson3• 14 years ago

You don't even need an explicit test. Just ask for their SAT and divide by 10. If post-1995, divide by 10 then multiply by 2/3. I still wouldn't hire purely on IQ though. An intelligent adult with no coding skill isn't that way for no reason. You cannot make someone into something they haven't much interest in being.

jaredsohn• 14 years ago

Your math for comparing SAT scores is off (what is the point of dividing everyone's score by ten?) There have been two main changes in scoring: in 1995 they recentered the scores which in effect increased many peoples' score on the order of 50 points or so (one can convert old scores here: http://2-bit.com/misc/satcalc.html). In 2005, they added a writing section so now your score is out of 2400 instead of 1600. Also, I think you'd want to compare peoples scores based on the current scale which would mean adjusting scores prior to 1995 so that they are recentered and either multiplying by 3/2 for scores before 2005 or discarding the writing portion for everyone.

But many people don't even take the SAT. At least when I was in high school (and likely still today), midwest universities tended to prefer the ACT. Thus, you're essentially looking for people who hit some percentile on national tests, similar to what one would do when applying for MENSA.

Also, as you and others have stated, this isn't really the best way to evaluate people for programming jobs.

lolilives• 14 years ago

I can see the appeal in using SAT tests as they are pretty standardized but I don't think SAT scores would be a good indicator of whether or not you should hire someone for a couple reasons.

1. SAT scores can be greatly inflated simply by taking an SAT class. 2. The SATs are pretty bias towards native English speakers (as 2/3 of is reading and writing and the 1/3 that is "math" has a lot of reading comprehension required) 3. SAT scores aren't really a good indicator of the prospect employee's work habits or ability to play nice with others, both of which are pretty important points to consider when hiring someone.

Well. I suppose points 1 and 3 also apply to IQ tests.

bane• 14 years ago

What about people (like myself) who never took the SAT?

xsmasher4• 14 years ago

IQ is a terrible metric because "raw" intelligence is only one skill required to be a developer. It discounts creativity, people skills, personality, drive, domain knowledge, style, and learning speed.

If I got a candidate with a 140 IQ who was unpleasant, derisive, and had only coded in Pascal, I'd send them packing. It speaks to their drive, personality, and people skills.

A winning team is not built on intelligence alone.

codeonfire2• 14 years ago

Unfortunately, unless you also have a similar iq, the candidate will naturally seem unpleasant to you. Anyone would agree that they would not like to be around someone with an 80 iq. however for the same reason, namely differing values, attitudes, and thought processes those same people will not enjoy working with a 140 iq coworker.

option_greek• 14 years ago

Not really related but I wonder about this all the time. Most of well known companies (Microsoft, Amazon, Yahoo, Google, Facebook etc) have a lot of data structures and sorting algorithm related problems as part of their interviews. They might require you to explain the algorithm or write code for a given data structure problem. Do these companies use data structures like binary trees (with all the variations) and sorting algorithms a lot ?

I'm asking this because, the companies I work for are mostly tier-2 companies which hardly use any of this stuff and generally get away by using proper libraries. So when ever I have to prepare for attending interviews of the tier-1 companies, I'm forced to re-learn most of this stuff - because knowledge fades when you don't use it.

potatolicious• 14 years ago

I used to work for Amazon, and used to interview a lot while I was there.

The answer is: it varies, but generally no. There are some roles where you're going to be spinning mad data structures & algorithms day in and day out... but for the most part you write code that isn't overly dissimilar from what's written everywhere else.

This is my own observation and hypothesis: "tier 1" companies have very algorithm and data structures heavy interviews because their signal to noise ratio is horrific. Being household names, you have tons of people applying who can't even Fizzbuzz. In this environment the natural reaction is to check, first and foremost, for CS fundamentals, because you can't assume that it exists and is just rusty.

I now work in startup-land, where hiring is so much easier, since people who find their way to your doorstep are overall much more qualified by default.

I'll admit, when I interviewed for Amazon, I always hit with algorithms and data structures questions first, because by my own unscientific estimates, at least 50% of applicants at the phone interview stage do not even understand what a tree is, much less how they might go about crawling one or representing it in data. I'd reckon about 10-15% cannot pass Fizzbuzz.

glenntzke• 14 years ago

No decent company should allow a single interviewer to shut down a candidate. I am a regular interviewer (phone and in-person) and I've got a few pet problems that I throw at all candidates who pass the simple introduction. I would never can someone on and 1 type of question, but I think it's fair to say that missing 2 or more conceptual questions will pretty much end the interview for me. That's why we have at least 2 2-person teams interview in-house -- if they passed the phone screen and make it in our doors they deserve it.

jchonphoenix• 14 years ago

I generally believe interview questions aren't a terrible way to screen people. There are, however, stupid questions and good ones. Ones that test how people think or how good they are at designing/accomplishing tasks are good. Ones that test stupid syntactic knowledge or language specific things are dumb.

strlen• 14 years ago

I entirely agree with raganwald's point point: language trivia is useless. I'd go further and say if someone knows actually knows C, and has a grasp of using OO in a way that's not painful for others working with them (prefer composition over inheritance, etc...), they can learn C++ on the job. If they know all C++ trivia, but lack a strong knowledge of either C, or good OO, they can do harm to a code base.

However, this isn't to distract from importance of using explicit and copy assignment. Here's a public service announcement: really, if your C++ class has a constructor that takes one argument, please use explicit.

Additionally, please disable copy constructor and assignment operator, unless you actually implement them yourself. Here's a free macro for you, courtesy of Google's C++ style guide:

  #define DISALLOW_COPY_AND_ASSIGN(TypeName) \
    TypeName(const TypeName&);               \
    void operator=(const TypeName&)

  class MyClass {

  private:
    DISALLOW_COPY_AND_ASSIGN(MyClass);
  };

I will thank you for this.

stickfigure• 14 years ago

Errr... if you concluded that "language trivia is useless" was the point, I think you fell into the actual trap the essay was trying to warn against. For example, broad knowledge of language trivia would be a positive attribute when hiring someone to design a language.

The point, I think, is that asking interview questions without understanding why the questions are appropriate is worse than useless - it's actively destructive to the end goal of hiring good people.

You can generalize this Cargo Cult behavior to any field - blindly carrying out activity without understanding deeper principles is likely to produce undesirable results.

syncsynchalt• 14 years ago

boost has an elegant solution that I like, inherit from boost::noncopyable.

  class Foo : boost::noncopyable
  {
    // ...
  };

SomeCallMeTim4• 14 years ago

I doubt anyone knows all C++ trivia, and that if someone even knew most of it, they could be that clueless about OO design.

I work on libraries more than apps. When I want to hire someone to work with me on libraries, you better believe I'm going to look for a C++ expert. And the explicit keyword is crucial in a lot of cases -- sometimes it can mean the difference between code compiling or not, but even worse is the bug where a parameter is missing and an int is converted to be some class it shouldn't be.

Using existing libraries and just assembling bits into apps? Sure, learning on the job can be OK (though honestly I wouldn't trust C++ written by anyone using it for less than a year full time). Though even then if you don't understand C++ well you're likely to shoot yourself in the foot with the language.

But there are some times that I really, really want someone to know the explicit keyword, at a minimum. Your example of DISALLOW_COPY_AND_ASSIGN is also good to know -- and it's likely in the Google style guide so that people don't have to be experts.

But I'd rather have people know why it's important and how it works than people who are just trusting to the compiler and luck to make everything work right.

I've seen C programmers clear a class that contained a std::string with memset(&foo,0,sizeof(foo)), which happened to work on one implementation but failed badly on another. I've seen people take the raw pointer out of a smart pointer and attempt to manage it themselves -- including trying to wrap ANOTHER smart pointer container around it later. I've seen people fight with a bug for weeks because they didn't bother to initialize all their class members (fixed that bug in about 10 minutes on code I'd never seen before).

The list goes on, and the list of "coding standards" we created to try to enable people who weren't good enough at C++ to work effectively with our C++ library got to be so large that no one could remember it all, rendering it useless. [1]

So yes, if I'm going to have someone work on C++ for me, I either want them to have at least a solid year of real experience, or I'm going to want to code review every line before it gets committed to main. Maybe I'll still hire them, but I've seen so many examples of people misusing C++ (and then having to fix their bugs later) that I've been more than a little burned by it.

Even more esoteric than explicit is the mutable keyword. Major bonus points for anyone I'm interviewing to know what that one means and when it's used. But no major negative if they don't know -- it has very few compelling use cases, after all, so I can imagine people not remembering it.

[1] https://developer.playfirst.com/standards

kragen6• 14 years ago

Huh, your comment makes me wonder if I should look for contract work doing C++ development. I haven't done it seriously since the last millennium, and basically I've learned to program since then [0], but I know all the stuff you're talking about cold. (Except mutable, which took me a few seconds to remember, and then I looked it up to be sure.)

It's just that I feel like higher-level languages like Ruby, Python, or JS are so much more productive for most things, most of the time, despite things like Rachelbythebay's interesting set of posts about hacking together stuff in C++ [1].

But, aside from things like that, I guess there are some domains where C++ is a viable option and Lua or whatever still isn't. And it sounds to me like the average quality of C++ has gone up since I was doing it, to the point that most people have read Alexandrescu and never call delete explicitly, so it might not be quite so painful?

Should I be taking advantage of being one of the Fat Guys Who Know C++? [2]

[0] http://lists.canonical.org/pipermail/kragen-tol/2007-March/0... [1]: http://rachelbythebay.com/w/2012/01/14/blort/ [2]: http://medriscoll.com/post/9117396231/the-guild-of-silicon-v...

msg2• 14 years ago

I agree with the story. To interview, you don't turn little details into big questions. You turn big questions into little details.

Interviews are lossy compression of a career, experience, and brain shape, over a noisy channel of whiteboards and human speech. The best you can hope is that your interviewer will recognize the larger pattern.

BUT. There is limited time in an interview, so we try to scry nuances like a soothsayer with a pile of sheep guts. Every detail gets magnified.

Examples:

"You chose to return an error code instead of throwing an exception." Is this an optimization, or have you spent a lot of time on system programming?

"I see a couple of fenceposting errors." Is it nerves or do you not have the right shape of the problem in your head?

"You passed some bookkeeping information in a recursive call that should take care of itself." Do you not understand recursion, or are you misremembering something cool about accumulators and tail recursion?

"You chose to use inheritance instead of composition." Have you ever had to extend or maintain such a design, or did you read about it in a book?

"You used template metaprogramming." Are you a misunderstood genius or do you delight in writing unmaintainable constructs from the dark corners of C++ into production?

"You chose to use Perl but don't know what a hash is." Do you have any questions for me?

ben1040• 14 years ago

>"You chose to use Perl but don't know what a hash is." Do you have any questions for me?

This reminds me of someone who once told me they knew C and were proficient at it, but they just "didn't get pointers and I don't use them."

JoeAltmaier6• 14 years ago

Scoff if you like, but I hired that guy. He's a fellow at Adobe now. They bought his startup.

We interviewed a Busuness school Finance senior to help on our COMSTOCK satellite feed parsing project.'Do you know C++?" we asked. "Yes" he lied, then "Can I come in over the weekend to get started?" How enthusiastic! Sure!

Turns out he spent the weekend studying code, learned to program C++. I didn't see him for a few weeks, then consulting on his code over something I mentioned "This would be better allocated". "Yeah, I wondered what the pointer stuff was about". I was astonished he'd come so far on guts and smarts. I explained malloc, pointers, he was off again.

adrianhoward2• 14 years ago

This fable reminded me of my favourite, unfortunately all to real, cargo cult experience.

I gave a client's boss a copy of DeMarco & Lister's "Peopleware" to read. I was younger and more innocent at the time and thought it would persuade him to change some of his incredibly damaging management practices. He'd produced an incredibly adversarial fear-based working environment that was killing productivity.

For those who've read the book (and if you haven't you should... now... seriously....) this guy was the poster-boy for the Teamicide chapter.

He came back to me after the weekend telling me how much he loved the book. Could really see how it could help the organisation. Would be making changes this week. I walked away feeling smug.

What was the lesson he drew from the book? Of all the lessons that he could learn?

In one chapter there is a brief section on how to spot jelled teams where they mention signs like going out for a drink together after work.

So he decided to take the company bowling on Friday nights.

Attendance compulsory.

Correlation. Causation. So easy to confuse.

(At this point I'm less smug.)

Let us say the result of mandating adversarial teams working fifty hours weeks socialise around a competitive game where alcohol was available was not ideal. During week 1 there was shouting. Week 2 resulted in a fight. There wasn't a week 3.

On the bright side it was the final straw the caused some folk to resign :-)

jbrains• 14 years ago

Did this manager spot the causation between compulsory bowling and folks resigning?

MartinCron3• 14 years ago

It sounds like the manager would see the causation between bowling and resigning and just force a different compulsory activity.

adrianhoward2• 14 years ago

I'm unsure. For some reason he no longer talked to me :-)

minikomi• 14 years ago

How about.. The interviewee gets to know what languages, libraries, components of the stack you are using a week in advance. They then can implement whatever they like using those technologies. Then, they present what they made to a technical member who can ask them questions which would show they have developed familiarity with at least some parts of what they use. The more specific the position, the more in depth the questions.. The interviewee can also ask questions back to the interviewer from a more level position.

lawnchair_larry2• 14 years ago

In this field, this will probably get you lower quality applicants or kids right out of school. High quality applicants have recruiters banging down their doors, and there are always several other options on the table that won't waste a week of their time.

Zaheer• 14 years ago

Hey everyone, I just created a new service that sends you a new technical interview question every other day! Check it out at: www.InterTechTion.com

eshvk• 14 years ago

Some perennial favorite "interview questions" of mine:

1. Tell me the difference between final, finally and finalize in Java. (Of course it was completely irrelevant to the discussion that my resume at that time never claimed that I knew Java; The only mention of Java was a library that I had used as a black box)

2. This is a fairly new one but most software engineers must have seen equivalents over the years :).

Rate yourself on a scale of 1-10 on the following:

-- Machine Learning

-- Convex Optimization

-- Linear Algebra

You know what was really disappointing? The second question was asked to me first by the recruiter (whom I don't blame for asking me the question) and then again by an engineer (after we had just concluded a long discussion on my projects touching upon those topics).

Cargo Cults and Interview Questions

Related Stories