Ask HN: coding challenges as part of interviews: yes or no?

https://news.ycombinator.com/item?id=203377

by rtc • 18 years ago

17 57 18 years ago

news.ycombinator.com

just curious what the group thinks of coding quizzes / challenges that are often part of job interviews for programmers. how predictive are they of a coder's future performance? are they appropriate only for entry level positions (when someone has less industry experience), or are they suitable for any level? are they more likely to pique your interest or to come across as cliche?

jamess• 18 years ago

Trick questions, or questions with some kind of gottcha have no place in interviews, However, simple, straightforward programming exercises are essential. I've seen people who claim to have X years of industry experience who can talk a good game, but literally can't write a switch statement without reference to the internet.

I also like to ask people to peer review some code with both obvious and more subtle errors in it. That's a pretty important skill to me too. Anyway, ask them to code some fairly simple algorithm and then critique their own work. If there are several ways to solve the same problem, some more efficient than others, all the better.

ericb2• 18 years ago

Hmm... I know > 10 languages. I'm not sure I could remember the exact syntax in several of those languages on the spot. Is it case, is it switch, etc. If you do this, your test is on rote memorization.

Give them a novel problem and have them solve it with a real computer and in tact syntax lookup. Then see what and how they did it and go over their reasoning. I'll hire the guy with the wisdom over the guy who can spit out syntax trivia flawlessly. Those are different skill-sets that do not predict each other.

astrec3• 18 years ago

Agreed. My first job they gave me a copy of Deitel & Deitel's "C How to program" along with the test. You actually lost points if you didn't open it!

sanswork2• 18 years ago

If you don't know the basic features of a language you don't know that language anymore. You maybe once knew it really well, and you still know the general syntax, but you aren't a insert language programmer anymore.

I do agree though that testing specific language skills is mostly pointless. The only time I've been given a programming test as part of an interview I was given a basic language spec that filled a couple pages then a book of problems to solve in the new language. Thats the way you should test programming skill in my opinion.

paulhart• 18 years ago

I agree wholeheartedly with the above. Gotcha questions are nice if you value having a sense of superiority over your candidates. Seeing how someone approaches a simple coding exercise tells you a lot about their techniques.

Two examples stand out:

1) A guy who had more Java certifications than had eaten hot dinners couldn't complete the simple exercise after three or four hours of effort.

2) Guy initially attempted to hit Google up for examples (why not!), and on finding that the net connection was disabled for that user account, wrote the best solution to the exercise I've received.

Overall, my interviewing process has had two stages. First, talk to the candidate and get a sense of what kind of person they are, and if they'll fit in with the culture. Second, if they seem reasonable, give them the test and see what their code is like. It's not foolproof, but it does weed out most of the weaker candidates.

bmj• 18 years ago

This is exactly how I was hired at my current employer. The VP of product dev interviewed me, and asked very general tech questions regarding work I've done. Then I was hammered by my now-direct manager. I didn't have any resources, but his questions often evaluated my thought processes as well as my coding skillz.

I should add that I thought I bombed the technical interview. So much so that I wanted to ask the VP if he had the right guy when he called with the offer.

dangoldin2• 18 years ago

Some questions do let you guage how the person responds to stress and unknown environments - that may be helpful.

ericb2• 18 years ago

Good point. Have you seen Swordfish? You can also evaluate how they respond to modern management practices.

astrec3• 18 years ago

Programming challenges sort the wheat from the chaff - in particular those who cannot code at all, but beyond that I've rarely found them predictive of future performance. Much more important that you hire a good fit for your company. An unhappy or toxic coder can bring a startup to its knees - hire slow, fire fast and all that.

goofygrin• 18 years ago

This is true. I aced the coding portion, but after 6 months, I was definitely not a good fit in the culture of the last company I worked for.

aggieben• 18 years ago

Agree with jamess, but additionally, I think interviewers that don't ask coding questions might be an indication of something negative at the company. Basic coding ability has to be one of those checkbox questions that determines your eligibility for the position, and if the interviewer doesn't ask anything technocal, it makes me wonder what kind of dumbasses I might end up working with.

dsplaisted• 18 years ago

Indeed, whether candidates write code during their interview is one of the items on the Joel Test (http://www.joelonsoftware.com/articles/fog0000000043.html).

icey• 18 years ago

I like them as long as pseudo-code is acceptable. I don't really care how good someone's memory is with regards to language syntax; I care about whether or not they can solve problems and think them through.

jzell• 18 years ago

If you have the sense from whiteboard discussions that the candidate understands computer science (like gets recursion) but you want to make sure they can code, why not choose a subset of a problem that you (your company) needs to solve?

gaius• 18 years ago

Good way to get some free consulting ;-)

scott_s• 18 years ago

I don't think he means open questions, but things related to the problem.

I interviewed once with a company that OCRed medical documents. One guy held up a sheet of paper and asked "How much memory does this take up?" which was a deliberately open-ended question. I was initially stumped, but then he started feeding me domain knowledge and assumptions, such as common DPI values and expected fidelity. Then I started coming up with estimates.

I don't think he expected me to have answer right off. I think he wanted to see how much I could figure out on my own if given hints.

thwarted• 18 years ago

I do this, and provide a subset of REAL input data and perhaps a sample of the expected output. This ends up showing capabilities for more than just minor singleton operations, and shows that someone can use the atomic units of programming (loops, recursion, functions, etc) to actually produce a result. The chosen programming task is one someone in the office recently had to tackle themselves, so there is a working benchmark to compare against. I give it to them ahead of time, before an in-person interview, and say tell them I'm not necessarily expecting a working program (however, that would be nice), but come to the interview having considered the task and be prepared to talk about it, or talk about their specific solution/implementation. I find this also helps communicate the kinds of tasks the company deals with, and maybe a little of the company culture.

cconstantine• 18 years ago

/agree with most of the people here. Coding questions have a purpose.

You need to consider what you are trying to determine by asking any question. Really, what is it about the candidate you want to know? After pinning down precisely what you want to know then tailor a question to figure this out with as little peripheral stuff as possible.

If you want to know if someone can generate some code in their preferred language on the spot, fine. Pick a very simple problem, like 'Write a program to display numbers in the fibinocci sequence', and explain the fib seq to them if they don't know it. I wouldn't even count points off if they make any kind of mistake that a compiler would catch as long as they know how to fix it. Have them do it on the white board.

This kind of Q isn't designed to figure out how much they know of 10 different languages. It's designed to see if they can put some code down in their preferred language and have it do something useful.

The biggest constraint in an interview isn't the honesty of the candidate, or even their knowledge; it's the N hour time limit. Your goal as the interviewer is to squeeze as much information about who the candidate really is in the time allowed.

jarnold• 18 years ago

I think taking some time to sit down and _read_ some code is one of the best and least used techniques during an interview. I think it's more of an indicator for job performance than writing some green-field code. After all, on the job, more time is spent reading and understanding code than writing code with no preexisting context.

eru6• 18 years ago

You may use the solution your last candidate came up with as an example.

gcv• 18 years ago

A trivial coding problem is essential. I've seen people with 10 years of experience in XML#Java.NETEnterprise2012 who don't understand conditionals, loops, or boolean expressions.

For phone screens, I find Steve Yegge's guide a useful starting point: http://steve.yegge.googlepages.com/five-essential-phone-scre...

I'm not a fan of asking candidates to reverse a string or to implement atoi or a binary search, because those have corner cases which, under interview pressure, might be too much to ask for. I just ask for an implementation of a gcd (greatest common divisor) function, in any language. I always explain the problem in detail. Only one guy (a PhD in physics) ever gave me Euclid's algorithm (section 1.1 in Knuth's AOCP Volume 1), so I don't expect it. I do expect a trivial loop, and most candidates take 15-20 minutes to come up with it, and need lots of hints.

As far as I'm concerned, anyone who can't come up with a trivial loop (or recursive or tail recursive function) for something like this shouldn't be programming for a living. Maybe "developing enterprise solutions," but never programming.

raganwald3• 18 years ago

"As far as I'm concerned, anyone who can't come up with a trivial loop (or recursive or tail recursive function) for something like this shouldn't be programming for a living. Maybe "developing enterprise solutions," but never programming."

Priceless, If you ever put this on a cafepress mug or tee shirt, you can count on selling me a dozen or so as office gifts. Thanks for the laugh.

eru6• 18 years ago

We use the (generalized) Euclidean Algorithm quite often in proofs. But - I study math.

raganwald3• 18 years ago

My experience:

Successfully pasing a code challenge is not a good predictor of future success. However, failing to pass a code challenge is a very good predictor of future failure. Furthermore, making the test harder does not make it predict future success well.

Therefore, I belong to the "FizzBuzz" school: Make the test ridiculously simple. It should take no more than fifteen minutes to complete. If the person gets it obviously right, throw the result away and move on to more important quetsions. Do not try to deduce a lot of malarky about their programming style or ability from an obviously contrived problem.

For example, writing:

a = 5; b = 10

Does not mean they don't know Ruby well, and writing:

a, b = 5, 10

Does not mean they are a Ruby expert. And writing:

fsquare = lambda { |x| x * x }

Does not mean they are a closet Bipolar Lisp Programmer. If the program works, it's a pass, move on. If there is a syntax error or some such, who cares, move on.

But if they struggle... You need to investigate the reason for their difficulty with a simple problem.

edw519• 18 years ago

Necessary but not sufficient.

I always give a fairly simple problem, pencil and paper, a soda, and leave them alone for 15 to 30 minutes. Before I leave the room, I make sure they clearly understand what's expected.

What they do isn't important. Our discussion about what they did is. I don't care what syntax or mannerisms they use; I do care if they understand what was needed and what they did well enough to explain what they did and why they did it.

Almost every time some good follow-up questions and segues ensue. What about this? What about that? I've learned more about people in the discussion after the problem than from anything else.

jaydub• 18 years ago

If I were interviewing someone, I would be more likely to test someone's general problem solving skills, questions that illustrate how a candidate thinks and approaches a challenge. This of course becomes more subjective, but can really help identify creative talent. Who knows, you might even learn something too. This is probably a big part of the puzzle type questions that companies like Microsoft and Google seem to favor.

eru6• 18 years ago

From the employees point of view: I would have no problem to do a coding test as part of an interview.

sfpeter2• 18 years ago

I like to believe that I can get a pretty good understanding of someone's technical abilities by asking thorough questions about his previous experience, and maybe ask them to explain a couple of basic patterns or programming concepts to me. I remember interviewing a senior java developer who claimed 5+ years of experience and had no idea what a Singleton was. What made it worse was that even after explaining it to her, she was unable to figure out how to implement one. Developers asked the strangest questions when interviewing other developers. I worked with a guy once who liked to ask his interviewees what the decimal value of 0xFE is. One candidate didn't know the answer so my coworker didn't think he could possibly be a good software engineer. I admit that (having done assembly coding in school) it seems odd that any software engineer wouldn't be able to figure out what FE is. But I still thought it was a bad way of testing technical skills. Would you rule out a candidate who doesn't know 0xFE ?

icky2• 18 years ago

> Would you rule out a candidate who doesn't know 0xFE ?

If they didn't know it after a moment's thought, I'd cut them some slack; If they couldn't calculate it, with pencil and paper, in 10 minutes (and that's giving them plenty of extra time for interview-nervousness), then I'd be seriously concerned.

andrewcooke• 18 years ago

i read through some answers here and i nodded in agreement with raganwald saying that this only worked as a filter on failure.

but then i read this and i thought "would i really want to give a job to someone who needs a pen and paper to subtract 1 from 255?"

i'm not sure what the lesson is here - perhaps that programming tests are dangerous because it is very easy to over-interpret the results, even when you start with the best of intentions?

[edited for grammar]

comatose_kid• 18 years ago

Depends on the context. If this was a question posed for an embedded s/w eng position, I would not be inclined to hire.

sfpeter2• 18 years ago

I also worked for a company once who had several enthousiastic , seemingly bright people with somewhat limited experience come in for interviews, but they didn't make the cut because they didn't pass some coding challenges. The guy who did get hired went to a good school and was a good coder, but was way too arrogant, difficult to work with and repeatedly refused to do what his manager asked him to work on. And how did that guy interview other candidates? He screened them by their education, and when they would come in he would give them one programming challenge without asking any other questions. Nobody seemed to realize that there were big problems with the company's hiring strategies and that developers themselves were largely to blame for that.

arupchak• 18 years ago

I've always found that asking a reasonable numerical solving question where you can easily figure out the answer in your head is a good starting point to gauge programming/algorithm ability. If someone knows the solution they are trying to get at, they can focus on the program itself, as opposed to worrying about solving the problem and coming up with the algorithm. Ideally there are a few ways to solve the problem, brute-force methods or more efficient ways. It can be very difficult to test a programmer's ability in an interview, you need to focus more on seeing how the candidate thinks, how they approach problems, and whether they can admit when they do not know something.

jkkramer• 18 years ago

I use a simple coding challenge as a weeder question. It requires basic use of recursion (without explicitly saying so) -- no math, no big-O, no gotchas, just writing a simple recursive function to process a small tree. It takes decent coders a few minutes to complete. Less-than-decent coders really struggle with it.

This is only a wheat/chaff separator, though. If possible, I like to see code from actual projects they've worked on to get a feel for the bigger picture (tip: open source your personal projects and obsess a bit over the code, so you have something to show off). Plus there's all the other non-code aspects to the interview.

tonystubblebine• 18 years ago

This reminds me of a group of main frame programmers I knew who had been forced to become Perl web developers. About a month later they were asked to hire another Perl programmer and one of their interview questions was what's the difference between 'chomp' and 'chop' (chomp removes the final character of a string if it's a newline, chop does it regardless).

They were shocked that the interviewee didn't know the answer because they considered it basic knowledge, "it's in the first chapter for gosh sake." The first chapter in this case covered differences between Perl 4 and Perl 5.

In Perl 5, nobody uses 'chop', they all use 'chomp.' Many experienced Perl programmers might even have forgotten that 'chop' existed.

And that's my major problem with gotcha questions, they tend to be pretty far from measuring if a person can actually write productive code.

mlazur• 18 years ago

Writing code is artwork. You can ask the artist about the specifics of his tools, but you will never grasp his artistic ability until you see the completed artwork. I just ask some general questions about the tools. The tools for code writing are: starting, constants, variables, addressing, comparing, branching and ending.

neovive• 18 years ago

I think it depends on the position and responsibilities the person will have once hired. If the programmer will be coding directly from specs without any room for "interpretation" then a coding challenge may be very applicable. If you are looking for a programmer that also understands your business and will be required to make decisions and think on their feet, the value of a coding challenge diminishes.

It also depends on the time frame of the position. If you are looking to hire a programmer on a per project basis, then a solid understanding of the development language is critical. However, if you are hiring an developer for the long term, it's better to hire a smart and agile person that can contribute to the overall business as well -- which is hard to measure from a coding challenge.

ig1• 18 years ago

They're a necessity. For the simple reason that too many people lie on their CV, no matter how good their CV looks you have no-idea how much of it is accurate.

Obviously there are cases where you wouldn't use it, but generally they only apply when you know someone's capable of coding anyway (i.e. due to an indisputable reputation, experience working with them, etc.).

However as a predictive measure it's fairly limited in that it's a binary indicator. If someone fails then there's a fair chance they're a terrible developer, if someone passes there's a fair chance they're not a terrible developer.

Ask HN: coding challenges as part of interviews: yes or no?

Related Stories