Ask HN: What do you think is the ideal programming interview process?

https://news.ycombinator.com/item?id=9696856

by mckee1 • 11 years ago

28 21 11 years ago

news.ycombinator.com

After the recent thread on Google rejecting the creator of Homebrew. If you could design the process from scratch, what would it involve?

soham• 11 years ago

Interviewing is not a standardized test. On the spectrum of human interactions, it's closer to a date than it's to a class test.

i.e. by definition, there isn't a perfect interview process. It's different for different companies, teams and people. Everyone hires people who "fit" their thinking. Case in point: Despite Google having <1% pass rate, nearly everyone who applies gets a job somewhere.

It also follows, that the "ideal" process, even if one exists, is going to be a combination of techniques. And even then, it's not going to be perfectly correlated with job performance. Simply because there are so many human elements in it and (figuratively speaking), human is the antonym of ideal.

You can take your time to really evaluate an engineer different ways, if you are hiring slow, say 1 person a month or so. But the moment you hit scale (say 15 engineers a quarter), it's very hard to do better than what we do today. The process today is the path of least resistance, which is what humans will take when left with a deadline and quota to hire.

(About me: Founder of http://InterviewKickstart.com)

speedyapoc2• 11 years ago

After a basic call to make sure that the culture fit is ok and to learn some background about someone, I believe the best way is to assign a small (4-10 hour), realistic, paid (market rate) task for the interviewee to work on.

I don't think that programming on a whiteboard is a good way to evaluate how someone works, and many of the problems don't reflect the real world. Interviews without programming can also favour someone with good charisma and ignore programming skills. Instead, by providing a realistic task for someone to work on, a company can better gauge how a person works, what solution they come to, etc. The interviewee is also at home and in a comfortable environment versus one with time pressure.

I've had two contracting jobs that evaluated this way, and both of the teams were nothing but joy to work with. In the first one, I was commissioned for 10 hours to evaluate an existing codebase, refactor a little bit, give input, etc. Their CTO agreed with my feedback and I was hired. In the other job, I was commissioned to layout their mobile application's view controller and navigation architecture. Once again, my solution was reasonable and I was hired.

Neither of these tasks are unnecessarily algorithmic, tricky, etc. They're simply real world problems and in both cases, even if I wasn't a good fit, the company would have (hopefully) benefitted in some way or another.

carlosdp• 11 years ago

I actually don't like the take home problems solution. If I'm currently working somewhere, I really don't want to work on some interview task after hours. I'd rather interview in person, talk about what I've worked on before, go into specifics about my experience, and have the employer trust that the fact I've worked at the places on my resume and am able to explain programming methods as enough to evaluate my competency.

I don't know of any other industry (other than show business) that requires people to audition for a job, especially not engineering fields.

speedyapoc2• 11 years ago

I've never minded it since my experiences have always involved pay at market rate. I'd be much more wary about doing a take home task for free.

brianwawok• 11 years ago

Maybe offer a choice?

Some people hate homework. Same hate whiteboard. Offer the person their choice?

lewisl9029• 11 years ago

You'd still need to take a day off for Google to grill you on algorithms in person.

I don't see how taking a day off to work on interview problems independently is any different from that perspective.

What the latter offers that the former lacks is the opportunity to work on it after hours without affecting your day job at all.

nate510• 11 years ago

tl;dr: Essentially agreeing with soham.

I don't believe that there is any ideal process. A lot of it comes down to having good hiring managers that know how to evaluate a candidate based on their alignment to the company goals and the role they are being hired for.

Reid Hoffman wrote a great piece recently calling for a change in the company/employee relationship from "join our family" to "help us while improving your skills". He said that a standard interview question at LinkedIn is (paraphrasing) "What would be your ideal role after you leave LinkedIn?" The point of the question is to A) identify candidates who are motivated to grow their talents, and B) communicate to the candidate that it's OK to imagine quitting someday.

That's an example of what LinkedIn believes works, but it might not be appropriate for another company, say that wants to hire programmers into more of a stable career pipeline. Similarly, some shops might be highly specialized in certain roles/technologies, making deep-dive technical interviews more valuable than at a company that needs bright generalists to solve poorly defined problems. There can be no one-size-fits-all solution.

One thing that's worked well for me is to have candidates meet a significant number of the peers they will be working with. Another is to ask a quick "weeder" tech question and then an open-ended architecture design question (i.e. Describe how you would build the Facebook activity feed). Checking references is also valuable. But these are just ideas and they may not make sense (or be possible) in every situation.

steven2012• 11 years ago

Have a deep conversation about technical topics, have more conversations over lunch. Try to see if the person is a good fit with the team. Check references.

Then hire this person.

If they can't code, if they are too slow, if they wont be productive at the end of one month, then give them 2 months severance and let them go.

sheepmullet2• 11 years ago

Spot on!

Maybe 1 in 10 will pass the initial filter and yet be a crap developer/bad fit. You can quickly spot these people in the first month and kindly show them the door.

The cost works out to be <2 weeks salary per hire. And as an added benefit you wont pass over good devs.

The 2 months serverance means the employee has time to find another job, and makes firing them a LOT easier.

philjr• 11 years ago

I'm just curious. In the entire lifetime of checking references, has anyone ever not hired someone based solely on a reference check of someone who was given to them by the candidate?

gargarplex• 11 years ago

I have never received severance from an employer or known personally someone who has (USA). How common of an occurrence is this?

sheepmullet2• 11 years ago

Very few will if you are fired, especially in the first month. In reality it is win/win situation for the employer to offer a decent severance.

Without it bad fits and bad developers stick around for years or you get the reputation as a "fire fast" company and a lot of good people don't want to work with you.

staacyc• 11 years ago

Interviewing and hiring is an expensive process so a one size fits all model would be ideal. There are a handful of variables to consider, like company stage, who's hiring, etc. At Grok + Banter, an early stage startup, my co-founder and I have vetted a number of potential developers who were either someone we know or they were recommended to us. Keegan sticks to conversations about tech while I try to uncover goals and agendas to see if we can work together to align individual goals and company goals. We look for people who challenge our thinking in constructive ways (rather than demeaning ways). When we speak with someone who seems adept at solving real world technical problems as well as someone who is clear and forthcoming with regards to their goals, we work together to create a 30 day contract that will either end employment or continue employment with an extended contract after 30 days. What I have found is that with in 2 weeks, give or take a few days, Keegan and I both know if the person is a good fit. With in a year, we've ended three contracts. Interestingly, all parties look great on paper and are able to talk (white board) their way through problem solving, all three are highly intelligent, unfortunately, all three lacked the ability to execute. We've also interviewed and hired people that lack an ability to demonstrate their defining attribute(s). This attribute inevitably starts to appear around week two- three. If you can determine whether someone will execute or not as well as draw out defining attributes in an interview, you may be well on your way to a one size fits all model.

MichaelCrawford2• 11 years ago

I would find it far easier to persist in my job search, as I expect would many others, were my interviewers to avoid needless cruelty.

"I'm sorry but we are unable to offer you a position" while unpleasant to hear, is at least what I expect when I don't get a job.

Just last week I was given a 24-hour challenge test. As I had to take an early morning call from a potential client, I chose to get some sleep rather than staying up all night to work. I did complete the challenge, with beautiful C++ code and a couple dozen unit tests.

Unfortunately they had a test that I did not, so my code tripped an assertion. The CTO emailed me to say "According to the rules of our process, the conversation ends here".

I'm dead certain he did not even look at my source; one of his engineers ran the test. Most coders don't know what assertions even are - expect most HN members do, but not coders in general - and it is uncommon for C++ coders to write exception safe code.

This company has been advertising the position for months. Why was it so important to stick to "the rules of our process"? Even if it was so important, did he really have to say "the conversation ends here"?

I don't have a problem with whiteboard tests but this was actually the very first time I've so much as attempted a challenge question outside of the interview.

I'm going to bill them for my time; if they don't pay I'll take them to small claims court.

lgieron• 11 years ago

I'm going to bill them for my time; if they don't pay I'll take them to small claims court.

After you're done, please please do a writeup of how it went.

MichaelCrawford2• 11 years ago

Your Wish Is My Command.

andrewstuart• 11 years ago

No company gets hiring right. Accepting that is step one to rethinking hiring.

audieleon• 11 years ago

First, do a phone screen, not more than 1/2 hour of questions around the tech, team fit, and motivations. Involve a few people from the team the person will work with. This phone screen should weed out 80-90% of candidates. If you have a doubt about them on the phone, that's a no.

If they pass the phone interview, invite them back and tell them to be prepared to code during the interview.

Start your actual interview with code test like FizzBuzz or something similar. Have them do this in front of you and one or two other competent developers. Many candidates will outright fail this type of test. If they do, politely end the process there.

The rest of the interview should be a few panel type interviews with the team and both technical and team fit questions. Include non developers if you can (testers, marketing, ops, etc.). Leave empty spaces in the conversation for the candidate to fill. They more they talk the more they will reveal. If you start getting a preponderance of thumbs down, politely end the process immediately. Don't waste time.

Take everyone's feedback seriously, and arrange to get it quickly from everyone. Preferably you would have a thumbs up/thumbs down from everyone immediately. (Know your team well enough to know who gives only thumbs down...) Look for and pay attention to biases within the group. As the manager, sometimes you have to make a call that the team might not like. Be prepared and courageous enough to do so. (For instance, hiring a process-oriented qualified woman candidate might not sit well with your team of male cowboy coders, but it might make the team better...)

Typically, if the candidate has gotten this far, then I take them and the team to lunch to get them in a more informal setting. As long as the candidate doesn't say something stupid or otherwise fall on their face during lunch, I'm ready to hire them.

If you are in the position to do so, have the offer ready, and make the offer right then and there. If you find the right person, ACT. Cancel remaining interviews with your apologies.

Hope this helps. Good luck.

ljk• 11 years ago

If you are in the position to do so, have the offer ready, and make the offer right then and there.

only if more hiring managers did that...

They more they talk the more they will reveal*

The* more they talk?

NathanKP• 11 years ago

I've been pretty involved in the hiring process at the startup I currently work at. Our process works like this:

1) Phone screen of roughly 30 minutes.

2) Coding challenge. There are a couple different ones you can work on, such as exploring and scraping data from an HTTP REST API programmatically, building it into a report and submitting it to a different endpoint, or slightly more complex one that involves reading raw socket data to download a video stream, validate it's checksums, and extract it. This is an offline challenge that candidates can do on their own time, using whatever tools, docs they want. But the point is that candidate's choose which one they want to work on, giving them some choice to pick the easy thing or the harder challenge, and send it to us as a sample of their coding style.

3) Assuming the challenge code they sent us wasn't god awful they can usually get an on site interview. During this interview they sit with a few engineers they would be working with and each engineer has their own type of approach to interviewing. Personally I have developed the following test that I find works quite well:

https://github.com/nathanpeck/wordladder/blob/master/ladder....

One advantage of a code exercise like the one above is that you can go as deep as you wish depending on the candidate's skills. When interviewing a very junior candidate you can focus on their understanding of how to organize code, refactor it, and their knowledge of the basic JS toolkit.

For someone more advanced you can get down into details like "Is this a breadth first or depth first algorithm?" or "How can this be optimized to make it more memory efficient?"

zerr• 11 years ago

Assuming we're talking about mid-career professionals. The fact that someone is invited to the interview means that you liked CV. Now the task is to ensure that CV is truthful - by discussing previous projects, experience...

After this discussion, if the interviewer is unable to tell whether the candidate is lying or not about his past experience - this means he shouldn't be interviewing in the first place.

And there is one thing to remember - false negatives, i.e. skipping over good candidates, is very expensive. Big G might ignore this fluctuation, but it is much more crucial for those wannabe-be-big-G startups who mimic even how interviews are held.

Ask HN: What do you think is the ideal programming interview process?

Related Stories