The Programming Interview Experiment

https://sourcegraph.com/blog/programming-interview

by matm • 12 years ago

39 51 12 years ago

mathattack• 12 years ago

Doing real code is very important. I'm curious if the OP has had any instances of catching cheating. I've seen instances of this with work products in other areas.

jasonpeacock5• 12 years ago

"Bring the source code—we’ll ask to take a look at it."

Seriously?? What actual company do you know that would allow its IP to walk out the door? And possibly to a competitor?

Additionally, projects are usually the work of a team, and are heavily iterated upon. "your contribution" is usually hundreds of little changes building upon other changes, not "this is the application I wrote".

beliu• 12 years ago

This is a good point, and one I forgot to address.

We would never ask to see proprietary source code. Hopefully, a candidate would have a side project or open source work they're willing to share. If not, we'd definitely work around this. For team projects, we'd ask them to describe their contribution. So far, this hasn't been a big problem, but we're still iterating and making the process better!

usea2• 12 years ago

The article is definitely not clear about this. In fact it says "bonus points if it's open source" implying that you expect to see proprietary work. It was a bit of a red flag for me while reading.

benched2• 12 years ago

I think it's assuming you've done something substantial on your own time, or worked on open source. I don't know exactly how common either of those is, but I'd guess pretty common.

Bahamut2• 12 years ago

I had an interview once where I did not have to write a line of code - it felt very refreshing to have my capabilities respected, although I just somehow lucked out it turned out.

jasonpeacock5• 12 years ago

I've been burned too many times by people who can talk the talk, but cannot walk the walk.

Design, data structures, development methodologies, etc, they knew it all. But ask them to code a simple method (find the intersection of two lists) and they take an hour to deliver a barely-working, sub-standard solution.

Trust but verify.

fishtoaster3• 12 years ago

Interesting- I have the opposite reaction. I would (and have) refused to work for a company that doesn't ask me to code in an interview. It may be nice for me, but it means they probably didn't ask my potential coworkers to code either, and I don't want to work on a team picked like that.

FLUX-YOU• 12 years ago

Yep. Fear of failing a technical interview keeps me from applying to places and keeps me learning new stuff, but I wouldn't want a job that I could talk myself into.

Bahamut2• 12 years ago

Well, my GitHub is public (and was probably convincing enough), and everyone but me at the company was asked to code during the interviews - there was evidence on the TV from someone who coded for an interview that morning even. I was found from a local AngularJS meetup, and someone at the company was trying to gently prod me towards interviewing with them over the course of months...which wasn't so much of an interview process as seeing whether I was interested, since they clearly wanted me for my expertise.

benched2• 12 years ago

I didn't have to write a line of code during interviews for my current job. It felt like they looked at my resume and concluded it wouldn't add anything. I did, however, had to write code to solve a non-trivial, job-related problem, as a pre-interview screening step. If I hadn't had to write any code at all, I think I might have felt a little uncomfortable. Like, is everybody really sure about this?

fishtoaster3• 12 years ago

To start, I think your interviewing practices are significantly better than the stock whiteboard-based version (and your criticisms thereof are spot-on).

That said, some concerns:

Leading with the source graph challenge seems a bit much. My experience has been that it’s necessary to have at least some filter for the unwashed masses before requiring that much work of someone. At my last company, we had a 30-minute non-technical talk with a founder (later replaced by our well-trained recruiter/office-manager). This helped filter out the people who weren’t what we were looking for at a really high level (e.g., an ops engineer applying for a dev role, a really junior person applying when all we have room for right now is a senior one, a dev who only knows pascal, etc) before making them invest the time required for an in-depth take-home test.

Also, what’s with the 24-hour limit? It seems too long to prevent cheating, but short enough to be annoying. For similar challenges, I’ve given people as long as they want (up to a few weeks), but asked for a blow-by-blow writeup of how it went and how long it took them. We may have gotten some liars, but we never had someone arrive on-site who claimed it took them 2 hours when, based on their skill, it probably took them much longer. People seemed reasonably honest about it, and the convenience seemed to be appreciated.

Asking to see source code for a significant project is tricky. I know good developers who wouldn’t have any code to show for that because everything significant they do they got paid for, and can’t in god conscience show to someone else. For example, I worked pretty long hours at my first job and built some interesting stuff, but I didn’t have much time for side-projects because of it. A friend of mine works normal hours now, but does contracting on the side. Plenty of good code, but nothing he could show to someone else. That said, asking to see some source is very high-signal, so it may be worth the tradeoff of only selecting people with significant free side projects.

That said, it’s still a way better process than teasers and whiteboard algorithms. If I were looking for a job, I’d definitely apply. :)

pavel_lishin• 12 years ago

some filter for the unwashed masses

I have a feeling that an unwashed boson wouldn't be able to complete the challenge at all.

jasonpeacock5• 12 years ago

But what if the ops engineer can pass the coding challenge? He maybe trying to change roles.

Or the dev that only knows pascal - if they can pass the challenge using pascal then I'd think they could easily learn whatever coding language your team uses. Learning a language is easier than learning coding.

That's what I like about coding, and about these blind challenges - it's purely skill based. If they want to apply and can pass the test, they they're worth talking to. If the wrong people are applying (and passing) then your job description and challenge are wrong and need to be calibrated.

fishtoaster3• 12 years ago

It is possible that someone with little dev experience could complete our challenge. We were mostly trying to avoid wasting time: theirs to complete the challenge and ours to review it, if we thought it was extremely unlikely they’d complete it.

However, when I referenced the hypothetical ops guy, I was referring more to a misalignment of goals: someone looking for a job where they’d be doing more devops stuff vs our need for a dedicated developer.

Regarding the hypothetical pascal developer, it really depends on your need. I’ve been in the position to hire bright people who can learn our tool set on the job, and that’s great. I’ve also been in the position where we can’t afford a month or two while a new dev learns our language an framework before they become productive. When in the latter position, it makes sense to weed people you’re not interested in out early.

I would love it if a well-written job description would prevent unqualified people from applying, but my experience has been that it just doesn’t. That’s why some people use fizz buzz (http://www.codinghorror.com/blog/2007/02/why-cant-programmer...). That’s why we used a short phone screen.

mamcx• 12 years ago

This part is the key, to me:

The second thing is the strong focus on topics taken from >classroom algorithms and the lack of concern about >practical programming ability. A candidate might somehow >remember that the way to implement an optimal string >suffix matching function is to use a suffix tree, but >where is the question that gauges the ability to create, >test, and deploy a complete application in a reasonable >amount of time? Which skill is more important to you?

gtz60• 12 years ago

I thought that was a an unfair comparison. One is a small piece of trivial knowledge, the other is a huge amorphous amount of voodoo and experience. But I agree that practical knowledge trumps theory.

danbruc2• 12 years ago

Not in a single interview ever have I been asked about the runtime of an algorithm. On the other hand it is mentioned in almost every article about interviews for programming jobs. How many companies do actually ask such questions? If your future job is mainly building standard business applications, web sites or whatever, no one cares about theoretical runtimes. If your future job is somewhere in research, game, database or operating system development or some other fancy stuff, no one will care to ask for trivial things like asymptotic runtimes. Personally I think most of the time people - interviewers and article-about-interviews writers - just mention Landau notation because they think it makes them look smart without realizing how trivial it actually is.

jasonpeacock5• 12 years ago

All those companies should be care about algorithmic runtime, it's part of performance and scalability. If you're not thinking about that then you're not writing production software.

I ask about runtime at every interview (part of "coding, data structures, and algorithms"). It is most definitely not a trivial concept - how do I define the correct solution to a problem when there are multiple approaches?

The requirements for a correct solution (even simple coding problems) include style, correctness, performance, and scalability.

danbruc2• 12 years ago

I meant that Landau notation is a trivial concept, not finding a good algorithm to solve a given problem or determining the runtime of a given algorithm.

chrismcb• 12 years ago

If your future job is mainly building standard business applications, web sites or whatever you better be caring about theoretical runtimes. I ask about runtime whenever I interview someone. It is important, if you ever deal with more than a few things at a time. Exponential goes up pretty quickly.

fsk4• 12 years ago

What about candidates who don't have several hours to spend on a programming assignment? Everyone seems to want one nowadays, and I don't have time to do one for everyone who asks.

I've done a bunch of programming tests, without progressing to the interview stage after doing them, and now I usually refuse.

georgemcbay• 12 years ago

Agreed.

While I do think that there is plenty wrong with traditional programmer interviews and that new approaches are needed, it isn't really a scalable solution on the candidate's side to spend hours to days on challenges for individual companies if everyone is asking for one. It becomes sort of like everyone wanting their own app store, or their own always-running polling service running on a desktop or mobile or whatever. If there are just a couple of people doing that, no big deal, but once more than a couple of entities start doing it, the system falls apart. If you're Google, Apple, Github, or some other prestige employer you can probably get away with this system even despite the scalability issue; otherwise good luck with it.

As a practical compromise, companies should be willing to waive the specific challenge if the candidate already has their own existing shareable source code they can point to, thus allowing a candidate to reuse the same example as a portfolio piece with multiple companies.

However, few companies are likely to do this since the same basic ego problem that breaks traditional programmer interviews comes into play ("Well we do things very different here, we're very special, so we must make the candidate jump through hoops X, Y and Z").

fsk4• 12 years ago

What makes it even more insulting is that, I do the assignment, and still no interview.

If you give a programming assignment, then you should at least have the decency to give an on-site interview to everyone who gives a solution that compiles and runs and gives the correct answer. That's why I usually refuse to do them now.

Not only am I spending a couple hours on someone I never met, the conversion rate is so low that I'm convinced it's a waste of time.

Also it's insulting. You're asking me to waste a couple hours of time for the privilege of maybe getting a chance to talk to you? When "maybe" turns out to be "almost never", I've really soured on the pre-interview programming test.

owenjones• 12 years ago

Whenever a post like this comes up I'm always curious why so much of developer interviews are focused an algorithmic performance. Is it because it can be mathematically proven while so much of what otherwise makes "good code" is subjective?

donretag2• 12 years ago

At this point in my career (professional developer since the late '90s), I am extremely picky about who I work for. The interview is a two way street, where the prospective employee also gets to ask questions to the potential employer.

In the described workflow, the candidate must spend several hours before talking to anyone in depth at a company. This scenario would simply not work for me. I will not spend hours of my time for a company that I potentially might not want to work for.

jasonpeacock5• 12 years ago

Would your rather that they have you come on-site and perform the coding challenge for half the day, then continue their normal interview process for the other half the day?

Or worse, go through a normal 5-6hr interview loop doing random questions on the whiteboard?

I think their approach is much more flexible and approachable. The overall time commitment is the same either way.

donretag2• 12 years ago

You can achieve a mutual understanding during the initial technical (not HR) phone screen. This approach eliminates the candidate's ability to ask questions before a multi hour commitment. The traditional phone screen is important. Most questions cannot be answered by HR.

I have a much better understanding a company after the question portion of a half hour phone screen than I can get from searching the web for information about a company.

at-fates-hands• 12 years ago

I'm completely the same way after getting burned early on in my career. I once did 5 interviews and several back-to-back code challenges for one company who in the end said, "Meh, we don't think your javascript skills are strong enough." I got this in a what amounted to a form email.

You want code challenges? Check out my GitHub account and pull some of my repos and check my code. THEN call me if you still want to interview me. I'm at a point now where I've amassed enough code samples (apps, games, plugins, etc) that you should be able to determine if my coding skills are up to what you're looking for.

RunForFun2• 12 years ago

I also agree with this. When I interview them, my goal is to interview the client, and their goal is to interview me. If I am faced with too many impediments to reaching my goal, I will choose not to continue the interviewing process. Employers need to realize that the process is indeed a two way street and that they need to convince us that we should work with them as much as we need to convince them why they should want to work with us.

bduerst• 12 years ago

As someone who completed grad school and went through many interviews not too longer ago - I can tell you that this type of process is a much, much more enjoyable than the typical phone + whiteboarding interviews.

I think that some companies become so intensely focused on hiring the best candidate that they sometimes lose touch with the fact that the candidates are human too. Interviewees are rational and trying to find a job that has a best fit. By taking the time to give them a project to work on, it allows them to get a taste while demonstrating that you have respect for them.

My favorite interview was where a company gave me a sample data set, and basically said "Do anything with this in python". I ultimately didn't get that job (not enough experience), but I remember that company and the guy who I went over my creation with, and now respect them more than the dozen other companies who made me feel like I was on a conveyor belt.

fatman2• 12 years ago

What would happen if an interviewee 'flipped the script' - gave the interviewer a coding challenge? Maybe under the guise of assessing the skill level of his potential co-workers? Is there any way an interviewer takes the bait?

fatman2• 12 years ago

Or, what if the interviewer explicitly challenged the interviewee to come up with an interview coding challenge? And perhaps even worked the challenge, with the interviewee hinting, helping, and evaluating? Maybe it would be a good way to demonstrate complete command of a subject, or the ability to think on one's feet, or even to evaluate the candidate as a teacher/leader of younger developers?

nanook• 12 years ago

This is a great post and I agree with everything you say and I am glad you guys are doing this.

I just wish you had more information about your company online (Angellist/Crunchbase etc). I think few people would be willing to put in 3-4 hours on a coding challenge unless they had a good idea about your funding, size etc.

I might have got a coding challenge from you a few weeks back but skipped it because I didn't have enough information. Ah well.. might do it this weekend.

Hope this helps!

bri3d• 12 years ago

Why ask to see source for a project above and beyond the work-sample test you've already administered?

I think work samples are a wonderful way to screen candidates (really, the best way) but to presume that all qualified candidates will have an additional sample of code (to the scale of a "significant project," even) available upfront for you without requiring an additional commitment seems disingenuous.

You end up applying an implicit filter for "developers who feel confident and comfortable contributing to OSS and/or possess copious free time for a personal project," which probably isn't really what you were trying to select for and tends to unfairly filter out certain types of candidate.

akanet• 12 years ago

Anyone interested in doing more complicated questions during the phone screen should check out https://coderpad.io - you get a live REPL or compiler (depending on whichever language you're using), and can run code instantly from your browser.

The best part is the environment is shared between you and your candidate, so you can both see the code being written and execute, live.

OhHeyItsE• 12 years ago

When I get asked things like: “Write a function that finds the longest positive-sum subsequence in an int array and runs in O(n) time.”

I really wonder if those places ever find people that can do this. I never get offers from the places that ask this. I always get offers from the ones that have a similar process to the one in the article. I don't think I've ever even met someone who could answer these questions on-the-spot, with a marker on a whiteboard.

usea2• 12 years ago

It mostly filters for people who spend a lot of time preparing for interviews, practicing that problem or similar ones, from books of problems known to be asked by interviewers. In a more general sense it filters for people who want a job badly enough to do the goofy dance that many companies expect of candidates, and are savvy enough to know the dance exists.

sukuriant• 12 years ago

Problems like that are the distilled versions of problems you'll often see at places like TopCoder where you have an hour and a half to solve 3 of them. They're looking for skilled TopCoder competitors, basically.

fsk4• 12 years ago

“Write a function that finds the longest positive-sum subsequence in an int array and runs in O(n) time.”

That's a horrible interview question. If you know the trick (saw the question before), it's easy. If you don't know the trick, it isn't reasonable to expect someone to figure it out in 5-15 minutes.

That's a brainteaser question disguised as an algorithms question.

d3gamer• 12 years ago

But there isn't really a "trick" here, though. The point of these questions is to figure out if the candidate has enough CS fundamentals to construct a reasonable solution to a problem that they've never seen before.

qianyilong• 12 years ago

I would add that the respect for candidate time is critical. I applied to one place and they requested a similiar coding project but this was just before the christmas holiday and they gave me a deadline of christmas day which just struck me as a little arrogant to think that I am really going to be doing their challenge on christmas.

bitcrusher• 12 years ago

It's a total nit and pet-peeve for me, but one does not "Architect" code. You architects domains.

You "design" code.

iggym001• 12 years ago

This interview process assumes candidates don't have anything else to do. I think they should pay the candidate for time spent looking at their code and making suggestions for improvements

RunForFun2• 12 years ago

In 12 years and more than 10 clients I have yet to run into any developer so incompetent technically as not to be able to perform his/her job effectively. However, what I do run into are individuals who cannot conduct themselves appropriately, those who cannot work in teams, and those who overlook delivering the value for their client/employer that they were hired to deliver.

All of these individuals can code, no question. However, they are impediments to accomplishing the business goals that my teams are tasked with solving.

I find it interesting that everyone I know who has been terminated or laid off has been done so for reasons not related to technical ability, however technical ability is the majority of what's interviewed for.

I think it is much more important to hire people who can work with others and deliver the value desired, but yet this is not what most employers interview for. In my opinion, if I can get a credible recommendation that a candidate is technically competent, then I don't think spending an extra minute on technical questions is important. I would rather be sure they they are competent in all other aspects of the role.

lauradhamilton• 12 years ago

Asking candidates to write actual code for your real app could be legally problematic.

Legally, the candidate will have the copyright to any code he writes unless there is a "work done for hire" agreement in place.

The Programming Interview Experiment

Related Stories