A Method I’ve Used to Eliminate Bad Tech Hires

https://mattermark.com/the-one-method-ive-used-to-eliminate-bad-tech-hires/

by nickfrost • 10 years ago

1214 499 10 years ago

raoulw• 10 years ago

Trials are how we do things at Automattic. Not only do we get a feel for the candidate, but the candidate gets a feel for us too.

Kluny2• 10 years ago

I've actually got a technical interview from Automattic sitting in my inbox right now, that I'm planning to work on tonight! Any tips? Is it possible/encouraged to write back and ask questions during the test?

dep_b• 10 years ago

The thing I liked most about their hiring process that it seemed to be done by permanently overworked people that didn't bother to give me any kind of answer after I worked all that hours and pushed a working solution.

fowlerpower• 10 years ago

I think this is a great system that reduces false positives, significantly. It helps candidates shine and it helps employers get good engineers.

I would not rely solely on this as no process is perfect. However, I do weigh this part of the process the highest for my hiring.

happytrails• 10 years ago

We do a similar thing at our office. Phone interview to make sure basics are known and then a "homework" assignment. This weeds out a lot of folks who can't cut it. Then the interview itself we can get the social aspect out of the way.

Sometimes a candidate is a good coder but a horrible communicator or human :|

stove• 10 years ago

Does anyone have experience with this style of interview from the candidate side of the equation?

I want to move to this style of hiring; however, I fear that because it's against the grain I may miss out on good candidates. Is that unfounded?

peller• 10 years ago

Well, if somebody legit wants to work for you, I suspect they'll be as curious about working with your team as you are curious about working with them. It's really a win-win situation for both sides.

cldellow• 10 years ago

We do a variant of this, although we don't currently pay. Perhaps we should. IMO, it skews towards people who are diamonds in the rough (junior, or non-traditional background) or experienced people who are burnt out on traditional whiteboard interviews.

It hurts the size of our funnel, but I think the results we get are pretty good.

ubernostrum4• 10 years ago

I do from both sides, albeit without being paid to do the exercise. Take-home exercises are great.

When I was looking for a job last year, I liked that it was something simple and low-pressure (compared to typical coding-interview practice) which approximated how programmers actually work. It was OK to look something up in the documentation if I didn't have an API memorized off the top of my head. It was OK to change my mind partway through and realize I could do it more cleanly another way. It was OK to pause for fifteen minutes and get something to drink.

Now that I'm on the other side of it, I like that it's a more natural thing. I like that I'm not having to loom over someone or try to fill awkward silence while they live-code. I like that I can read their solution before I go into the interview and have some prepared comments or questions to get discussion going. I like that it emphasizes the collaborative nature of programming in a team.

Good candidates are beginning to revolt against traditional coding interviews. Take-home exercises are one of the alternatives that people actually seem to like on both sides of the interviewing equation. So you're not going to miss out on good people by switching your coding interview to a take-home; if anything, you're more likely to start attracting them.

StavrosK4• 10 years ago

I want to second that, with the caveat that, if the exercise is unpaid, I am much less inclined to do it (mainly because I am too busy with paid work to take on more, especially unpaid).

I've interviewed with companies that gave all kinds of exercises. I have no problem with take-home exercises, if they're implemented like the article suggests. A few times I've gotten "do this exercise and we can maybe pay you a bit for your time", followed by them never paying and me never asking. If you decide to pay someone for exercising, tell them how much beforehand, and pay them as soon as they hand it in.

The best interviewing experiences have been with paid onsites for companies, even if I didn't end up getting hired in the end (I got to meet the team, work with them, they took me out to dinner, etc, it was nice).

philipkglass2• 10 years ago

I got my first software job (at a startup) by doing well on an unpaid take-home challenge that I was able to spend a weekend on. A few years later I missed a job opportunity with another company by failing an optimization challenge question that had a much shorter time limit; I started formulating it as a graph search and by the time I realized that was a dead end approach my 3 hours were up. I definitely prefer having a weekend as opposed to a few hours. (Maybe I would have been a bad hire for the 3-hour-challenge position, if a typical work day really involved solving problems like that in a few hours, but I suspect it wasn't representative.)

When I was first looking for a software job I was a researcher who'd written a lot of his own tools but I didn't yet have an open source portfolio or commercial development experience. Solving a challenge was a great way to break the chicken-and-egg dilemma of needing experience to get hired and needing to be hired to get experience.

If I had to switch jobs again I'd also love to find employers screening with weekend challenges instead of live coding. Paying me to complete challenges would just be icing on the cake. I've done well at every company I've worked at, but asking me to write heapsort or whatever while you're peering over my shoulder and the clock is ticking is extremely stressful and unrepresentative.

I'm one of the people screening hires at my current job and I don't want to subject candidates to stressful live coding. But neither would I want to hire someone who can't write decent code and challenges look like a good way to prevent that. Surprisingly, even some people who include a github link in their CV are sharing code that's quite poor. (Like turning O(1) operations in an inner loop into O(N) because they picked the wrong collection type, in a language that offers perfectly suited collections in the standard library.) Maybe they think that nobody will actually read it, and having a github account at all will mark them as savvy?

wccrawford• 10 years ago

I did a weekend exercise for a company once. I didn't care for it. I felt like I was wasting my time when I could be doing something I wanted to do instead, but I wanted a new job and they seemed decent.

However, before they had time to review it and get back to me, another company interviewed me and offered me the job and I took it.

From the other side of that, we now give these assignments when hiring. It has virtually eliminated the bad candidates, and we've hired a few good junior programmers for a change.

However, neither of those 2 situations paid anything. They were throw-away projects that the company couldn't actually use to avoid any legal issues. If we were going for experienced programmers, I think I'd recommend the payment route. But junior programmers? We got too many applicants to send it to all of them, and so we'd end up pre-filtering them like we used to do, which makes the process less valuable.

epage• 10 years ago

I've only interviewed at one place that used homework. They had two homework assignments with follow up phone interviews to talk about it. The next round was going to fly me out for an in-person pair programming interview. We then parted ways because my circumstances changed and I wasn't going to be able to work in one of their set locations.

I might be more ok with it if it was meant to replace the in-person interview but otherwise it took up significantly more of my time. It was nice that they gave me 2-3 $100 amazon gift cards but, like other posters, I'd rather have my time than extra compensation.

The first assignment was supposed to take 4 hours but took me over 6. The second was supposed to be a whole day thing but I could only put in about 4 hours.

It was useful to gain some insight into their culture. I'm still mixed about what I think of it. The short takeway is that they are continually learning and updating their practices (good) but they were koolaide drinkers (bad) and throwing around buzzwords from specific individuals' blogs as if that was everything without showing a deep understanding of those buzzwords (causing them to completely miss that I exercised those same principles, even if I had no clue what they were talking about).

The longer version. They originally were going to turn me down for not using enough <buzzword>. They were also disappointed that I didn't use <buzzword>. Originally, they weren't even going to explain more than that but I pushed on it to see what I could do better and through the conversation they warmed back up to me. Later I looked up their buzzwords.

The first was SOLID which appears to have limited use in the industry. First, they couldn't even break up which part of SOLID I was violating but instead were throwing the acronym of acronyms around like that was all that needed to be said. When I dug into their feedback more, I found out it was my main function that "wasn't SOLID enough" which did argument parsing and delegated the rest of the logic to other code that was so well designed that they had to throw out the main follow up question.

I literally could only find one software development subcommunity that used the second buzzword and that was mostly limited to one consultant's blog. They also showed a lack of depth in it because I was actually using it, just at a different layer in the stack, solving the same problem they thought it would address.

hslee16• 10 years ago

Having candidates work on small projects simulates the communication required to complete actual/mock work projects. This will definitely foster better rapport between people involved.

Personally, I'd rather be given a weekend to work on a smaller project rather than be tested on algorithms that are available by common libraries of most programming languages.

illicium• 10 years ago

Or I can have the candidate code with me for an hour during the interview, and save $100. They get to ask me questions about the problem directly, and I can see where they are struggling, and how they debug (trial and error? Google/SO?)

Kluny2• 10 years ago

It sounds like you would get a lot less out of this method than you would from the one described in the article.

ahh• 10 years ago

This system is interesting but has a crucial flaw: I legally can't take part. I have a job now, which claims ownership over any tech-related IP I create. There are processes for getting exceptions, but a) you won't get them for actual job-related things b) I'm not asking Google for an exception so I can interview somewhere else.

This is true for nearly anyone with a standard SV job. So if you're willing to restrict your pool to the currently unemployed, this can work, but I think you'd rather not do that.

mattnewton• 10 years ago

Playing devil's advocate here: how will this come back to bite you if you do it on your own time and equipment? How would the current employer know to care? Not saying you should require your candidates to break the law. Also, though IANAL, California does have laws that give employees the right to their own IP on their own time/equipment when it doesn't use work IP https://news.ycombinator.com/item?id=2208056)

lisper• 10 years ago

Yeah, this. A little discretion goes a long way.

RandomOpinion5• 10 years ago

This system is interesting but has a crucial flaw: I legally can't take part. I have a job now, which claims ownership over any tech-related IP I create.

By that interpretation, you can never legally take part in a whiteboard coding session during an interview either since the code you whiteboarded also belongs to your current employer.

There is such a thing as being too literal.

kasey_junk4• 10 years ago

Anti-moonlighting agreements, where they are legal, are common and definetly apply in this situation without requiring extra literal interpretation.

femto• 10 years ago

The problem would be the payment. That turns it into a paid gig and leaves a paper trail. A solution might be to allow the candidate to forgo payment in lieu of the interviewer making a donation (in the interviewer's name) to a charity of the candidate's choosing, so the candidate never sees any compensation.

serge2k• 10 years ago

Or the pool of people who will say to hell with it and just do the thing anyway.

I'm probably in that pool. Really, I'm not creating anything of massive value by doing a little example project. If my employer wants to sue me over it they can go ahead. I will probably lose, but they will have a rep for suing people for taking interviews.

literallycancer• 10 years ago

Are clauses like that common in the US? Do they hold up in court?

neuronexmachina• 10 years ago

Pretty sure it's not legal in California: http://www.leginfo.ca.gov/cgi-bin/displaycode?section=lab&gr...

sheeshkebab• 10 years ago

non-competes are common - but often can't be enforced by employers (and in some states are just ignored/illegal). blanket ip ownership not so common - but big shops sometimes require them, when they can get away with them.

Some people negotiate over such things or have them struck out from whatever HR paperwork. Consultants definitely have such crap taken out from contracts and replace with more straight forward and standard non-disclosure agreements (or charge a lot more - clients will pay if that stuff is important).

In other words, employer/client can put whatever crap they want and you need to always read and understand contracts you sign, as an employee or contractor.

chubot• 10 years ago

Do you think Google would sue you over 2 hours of coding? Technically, they might be able to, but anyone can sue you for anything.

If you think they would, you're probably overvaluing what you can accomplish in 2 hours, and you're underestimating the cost of a lawsuit to them. I can't imagine a situation in which they would actually pay any of their legal staff to even look at something like this, given all the rest of the things the lawyers have to do there.

I worked for Google for 11 years and wouldn't have hesitated to take a paid interview as long as it was all done in good faith (i.e. for the purposes of getting another job).

Google is trying to hire good engineers and retain them with interesting work and good compensation/perks, not shackle them there by legal means.

bigiain• 10 years ago

I dunno about Google, but I've got a friend who's legal advice was "Stop working on your phd immediately" when his lawyer read the new employment contract we was being asked to sign when his company got acquired by Oracle. It took months for him to get an extremely limited exemption to the standard contract to be allowed to publish his thesis according to his phd requirements...

countingteeth3• 10 years ago

Are we reading the same article? The one I read said:

DON’T use a real problem because of tribe knowledge needed to fix.

I do not see how this is different from a whiteboard problem, literally speaking. It's a "problem", not a "product", and presumably the solution is throw-away, or at least not used for commercial purposes.

If the author was talking about tasking potential hires with a small, real consulting task, then that would be an issue, whether it was paid or unpaid (and by industry norms, using applicants to do unpaid work is extremely unprofessional, though not unheard of).

deckar01• 10 years ago

[Ask HN] Does my company have IP rights to the stuff I do in my spare time?

California ... has a law on the books that generally prohibits employers, on public policy grounds, from making claims to IP generated by employees working on their own time and using their own resources.

https://news.ycombinator.com/item?id=2208056

maccard2• 10 years ago

If you're being sued by your employer for breaching a moonlighting agreement, you've already lost. It doesn't matter whether you win or not.

eveningcoffee• 10 years ago

Interesting. I probably could not work in Google then if this is something that is not negotiable to a reasonable level before signing the contract. I consider myself a free man.

But I am afraid that they just slipped it in and you were too worried to ask it taken out. This is definitely something one should look in the contracts and ask to be redacted (IANAL).

peteretep12• 10 years ago

    > I legally can't take part. I have a job now, which
    > claims ownership over any tech-related IP I create
    > ... This is true for nearly anyone with a standard
    > SV job.

For the majority of (admittedly non-SV) employment contracts I've read, most will limit it to work that is linked to your employment only. In the few cases as a developer and recruiter where that's not been the case, I've had no trouble at all getting the company to sign-off on a clause that limits it to that. Legal is much much much more worried that you'll rip off actual company IP or technologies that you've built in the workplace than that you'll discover cold fusion in your garage and they won't get their cut.

metaphor• 10 years ago

> I have a job now, which claims ownership over any tech-related IP I create.

Is there an omitted qualifier to this, e.g. ...while on company time; for IP with sufficient correlation to your primary role?

I'm having a difficult time swallowing the legal enforceability of "all-day-everyday-any-tech-related-IP" outside of military personnel subject to the UCMJ. Would that suggest that you technically couldn't contribute to an open source project?

davidgay• 10 years ago

Yes.

walshemj6• 10 years ago

Yes but can you afford the legal representation to prove it is not related.

sethammons• 10 years ago

I have to get exemptions for any open source contributions or personal projects. My employer gives those out readily, but I'm expected to follow the process.

arthurdenture• 10 years ago

"I legally can't take part" is overstating the problem. It's not illegal for you to take part. What do you care if Google can claim ownership over the IP you create? And why would Google care to do so? The IP is for a throwaway problem; it's worthless.

(xoogler, with some knowledge of this area)

nunez2• 10 years ago

Just do it (but not on their time or hardware lol); their process for finding these things out is REALLY slow unless your thing goes really public

Also, fuck Google and their ridiculously limiting anti-entrepreneurial IP agreement

DenisM• 10 years ago

If your current employer wants to take ownership of that micro project let them have it. Why do you even care?

cloudjacker2• 10 years ago

A standard SV job .... in CALIFORNIA .... must restrict those agreements to things done on company time, on company equipment or suffer the whole employment agreement being invalid, severance clause or not!

Use that paid time off, or "flex" hours.

And regarding the paper trail, payments under $600 need not be reported to the IRS. 1099's only get filed after you have been paid over $600. Ask for cash.

erikb3• 10 years ago

Look, every time you do something that may piss of someone, like having a job and also applying to other jobs, you need to bend one law or the other. Like you are probably not allowed to take sick days for a job application appointment at another company, but you can't take all your holidays either.

There are laws that are really there to be followed correctly, like the ones about theft, murder, etc. But there are laws that basically just enable people who you piss off to get back at you. And the latter ones are a risk you must take when you want to improve your life.

ilaksh• 10 years ago

Then people should start crossing that crap out in contracts before they sign up.

umanwizard4• 10 years ago

The IP stuff is a red herring. Anti-moonlighting agreements are more likely to bite you here.

lacampbell• 10 years ago

I wish more companies would give example problems to solve in your own time. I did one for a fairly large Australian company, and they rejected it. By the time they got round to letting me know (very long turn around time) I had already found other employment. But still, their reasons for rejecting my solution - the output was not formatted to their liking, and they didn't seem to know what an anonymous function was - was hugely beneficial to me. I would not have been a good fit at all.

redwards510• 10 years ago

I don't mind homework, but companies need to start specifying their salary range up front. I've been having a lot of time wasted lately by getting through Tech Interview #1 or Homework #1, only to find out their max salary is laughable for San Francisco.

"You should just find that out upfront" I hear you saying. Well that is not always possible. Recruiters either don't know the range, or lie about not knowing, or you have to submit a big packet of info (cover letter, history, resume, homework) before you can even talk to a human about salary.

ryandrake2• 10 years ago

This is true, and one of the worst inefficiencies you encounter as a candidate (and probably as an employer, I don't know): There's so much song and dance that has to go on before you get to the meat of things: Compensation. Companies could easily solve this by posting clear salary ranges, but most just don't.

Another thing that companies could do better is think really hard about "required" and "desired" skills, and be truthful. Lots of companies list things as required which they're probably willing to bend on for the right candidate. All this does is scare off potential candidates (odd when you're complaining about that mythical shortage of engineers). On the other hand, recent events have encouraged me to not be afraid to apply for jobs that I'm unqualified for :) so maybe it's not such a big deal.

jtwebman• 10 years ago

I agree, I always ask what the salary range is and if they are not willing to give it I tell them I am not willing to go any father in the interview process. They say then if they really want to interview you.

ubernostrum4• 10 years ago

In addition to salary range up-front: find out up-front if they're actually hiring you, or want to do the "contract for a while, then maybe we'll hire you for real" thing.

I had to say no late in the process to several companies which offered me the contracting-to-maybe-hire dance.

peteretep12• 10 years ago

Recruiter here with a history of working as a developer

    > but companies need to start specifying their salary
    > range up front

The costs outweigh the benefits for employers (encouraging existing employees to ask for more money, losing candidates who will accept significantly less but wouldn't apply at a lower range, etc) -- so don't hold your breath.

    > I've been having a lot of time wasted lately by
    > getting through Tech Interview #1 or Homework #1, only
    > to find out their max salary is laughable for San
    > Francisco.

State it upfront when you submit your application, rather than "find that out":

"I am interested in roles paying above my current salary of $xxx,xxx only, and am looking for an improvement on that to reflect my increased experience and the risk of moving roles"

    > Recruiters either don't know the range

Yeah they do.

    > or lie about not knowing

"Could you tell me about offers you've previously made for this client?" They may also be lying about actually being authorized to recruit for the role(!!!) so consider asking them for a copy of the technical specification. If it's a larger company, ask what the salary banding and official job title for the range is, then head to Glassdoor.

    > you have to submit a big packet of info (cover letter,
    > history, resume, homework) before you can even talk to
    > a human about salary

I had a hunch that roles like this weren't worth applying for when I was a developer. As a recruiter, this has been almost completely confirmed.

nixme• 10 years ago

Have you tried hired.com ?

arielweisberg2• 10 years ago

100% This. I just wasted a bunch of time extracting an offer from one of AmaGooFaceFlix. Supposedly a top SV company with good compensation.

Got an awful offer to relocate to an area where 1/4 to 1/2 the house costs twice as much and I would no longer be able to work from home. I would also lose a significant portion of my life commuting. The offer was comparable to what I made working remote from Boston.

I don't mean comparable in the sense that they tacked on a CA bump to handle the cost. Comparable as in same money, but I would have to live in CA.

Ended up working remote for more somewhere else. Everyone's time wasted. I don't get it at all. It's not like they didn't know it was low.

dgudkov2• 10 years ago

Alternatively, you can specify your expectations up front. It doesn't have to be an exact number, just a hint -- e.g. "No talk if the salary isn't well north of 100K".

user59944617• 10 years ago

Most companies pay poorly.

If you are in the higher range of salary, you gotta find out which local employers/industries are worth talking to and limit your application to them. There probably aren't many.

ryuker16• 10 years ago

Being a freelancer in all areas of tech, I've offered this to every employer and 100% refused or ignored it.

Closet thing I've been offered ironically was rather tough sys administration tasks for no pay for a few gigs (5 hours of work) or asked to solve an obscure ntp issue on a Linux server they couldn't figure out.

For programming jobs? Never.

Jgrubb• 10 years ago

I was actually hoping for a post on how to cull bad hires after they'd already happened. Any advice there? Asking for a friend...

GuiA• 10 years ago

Set up a proper culture of mentoring and training in your org to turn people who are a little too green or inexperienced into effective team members.

Not as catchy as "fire fast", but infinitely more sustainable.

ryandrake2• 10 years ago

My guess would be that most "bad hires" are either insufficiently skilled or insufficient motivated. For the former, you could provide training if you at least see potential. The latter shouldn't be too difficult to figure out.

ohyoutravel• 10 years ago

Is there a list of people who hire/give interviews based on take home problems?

pauljaworski• 10 years ago

This is literally the only way I've gotten job offers. I'm a very competent programmer, but when I'm asked to solve some silly puzzle with someone staring over my shoulder, I freeze under pressure and can not think at all.

Luckily not everyone thinks solving leetcode problems is a good way to hire.

um304• 10 years ago

Exactly my case. My startup is ending and I'm looking for work. Wonder if there's a way to look up companies that hire by giving assignments.

Vekz• 10 years ago

Your next startup could be a job board that specializes in this. I'd subscribe

zzzcpan2• 10 years ago

Be persuasive, ask for an assignment. Everything is negotiable.

Rapzid• 10 years ago

If the company is large enough there may be interview process reviews on glass door.

usloth_wandows• 10 years ago

If you can't code under pressure, you aren't a very competent programmer. Real world projects come with high pressure scenarios for software engineers. Code monkeys may be able to get away with low pressure, minimally impactful projects.

67726e2• 10 years ago

I can't code with people staring at me, talking to me, asking me questions. If you can more power to you, but if that reflects the nature of the job then they can take that one and shove it. It simply doesn't reflect reality.

TheSpiceIsLife• 10 years ago

It's not like software development is the same as being a fighter pilot.

Code now or die isn't usually something most people face in their day job.

GrinningFool2• 10 years ago

Upvoting you to help prevent your comment from reaching oblivion - I don't agree with you, but I think what you said is a common belief.

The thing is, it's a vastly different kind of pressure. When I am coding as part of a job - by myself or with others - there is no inner monologue in the corner of my brain doing something like this:

"Could do that or... no damn, that's O(n^2), damn I'm taking too long that one looks bored, if I instead put those keys in a hash... no, I'm sure there's a better way, running out of time though, is he looking at his phone now? Okay, what was the profile of merge sort again? ... oh, shit! She felt like she had to offer me a guiding hint, I'm doing it wrong!"

And it never stops going. It picks apart every choice I make, and by five minutes into the interview I come across as a sub-articulate mess.

In no other high-pressure situation does this happen to me. Hell, this past summer I actually spoke at a conference and I didn't have the kind of trouble I do in an interview.

Whiteboarding, paired coding, participating in architecture design - these things come easily. In the context of an interview, though, it goes out the window.

taneq• 10 years ago

You're getting hammered for this, but for some jobs you're absolutely right. If you can't handle being put on the spot for a simple interview question, how are you going to go when a production line is down and it's costing your client $100k/hr, and the foreman's standing looking over your shoulder wanting to know when it'll be working?

Not all jobs give you all the time in the world on a test server without interruptions.

vehementi• 10 years ago

How do these opinions even survive? Yikes

Shorel• 10 years ago

I can code under pressure.

But I will not work in a company where that's the norm.

I value my health more than the potential money from that stressful job.

nolite2• 10 years ago

I've been in situations together with clients deploying a product, them presenting it for their managers, orpreparing for a big demonstration, and something goes wrong due to some unforeseen environmental factor you couldn't have imagined or predicted, and breaks in a way you don't expect, despite the weeks of testing you've done.

You've got your main client staring over your shoulder, arms crossed saying fix it, and every two minutes asking "is it fixed yet?"

You damn well better be able to code under pressure... And like a freaking ninja

aianus• 10 years ago

Most tech jobs aren't like that. If the job you're hiring for is one of the few that involves high-pressure coding in front of strangers, then by all means test for that.

keithnz• 10 years ago

if that's common then it's a reasonable criteria for the position, sure. That hardly ever comes up in my work environments, so I'd be happy with an engineer who is awesome 99.999% of the time when they aren't in this position.

Noseshine2• 10 years ago

I can present to audiences pretty easily, I actually enjoy it. I also don't mind at all having a client watch what I'm doing. And yet, while I have no trouble doing interviews - having worked as a freelancer that would have been pretty bad - the situations are not comparable even for me. I loathe interviews. Fortunately a lot of clients, and I've had several major companies, had people hiring me without any of the usual stuff based on a gut feeling, I had to answer silly irrelevant technical questions only twice. And once I even failed "explain a right join" - something surely everybody would be expected to know, and of course I do, but at that moment I blanked. I go the job anyway but it goes to show that even to me interviews are a very different and greater kind of stress than anything normal work can throw at me. Even if it looks the same, it isn't, not at all! The brain knows context.

An interview situation is much more adversarial: They are looking out for reasons not to hire you, you are competing with others for the job and only one can get it! No such consideration on the job, unless the work environment is completely and ridiculously broken. When are you ever in a work situation where several people work on competing solutions and everybody else but the person who made the winning one is fired? At work you are working together, and to solve a problem, not to week you out of the pool.

JustSomeNobody• 10 years ago

This isn't the same kind of pressure, though. In this case you are familiar with the people, problem, code, etc.

edblarney2• 10 years ago

"You damn well better be able to code under pressure... And like a freaking ninja"

Being able to 'code under the pressure of a deadline' is very different from 'solving a tricky problem with someone staring right at you'

cgio• 10 years ago

In consulting you get in similar situations. In those cases it is not a "be able to code under pressure..." thing. Rather, a "know your code well" one. I have also been in a few demos where everything fails. As the manager sitting next to the client with your arms crossed, you have to know how to defuse the situation without having someone sweating their soul on a keyboard. Been on both sides.

greglindahl4• 10 years ago

This is not a good way to hire people. I can do all those things, but I'm not normal. Some engineers do a lot better under stress with a system that they're very familiar with. Others do better under stress when they're working with people they know well. And still others are great engineers but aren't the person to be in front of the client under stress.

Congratulations that you can do all these things -- but unless 100% of the engineers in your company need to be like you, you might not want to use your list as hiring criteria.

developer22• 10 years ago

> despite the weeks of testing you've done

I have never in 12 professional years been given weeks to test anything. On only a couple of occasions have I been given 3-4 days to extensively test a project with multiple months of development time behind it.

Clearly you're talking about the project where the developer(s) told you very explicitly that it could not be done in less than 3 months. So you just assign the Jira ticket a 3 week deadline anyway. And then flip out when a half-assed project fails to ship. Welcome to the world of software development.

xor1• 10 years ago

clients

The type of situation you're describing is is why I avoid working for any sort of shop or agency.

OpenDrapery• 10 years ago

This is why the demo should just be a pre-recorded video that you can narrate live. Seriously.

donretag• 10 years ago

I very much like the paid aspect to this approach. Not that I need the extra money, I am already financially compensated enough by my current employment, but it reduces the chances for the interviewing company to flake out after such a programming assignment. By offering some form of compensation, they are investing in the candidate and will probably not ignore the candidate after this stage of the interview. My assumptions at least.

daxfohl4• 10 years ago

Excellent insight. I just finished a weekend project for a company (and really the onus was more on my wife who had to take the kids by herself all weekend), and all I got back was "nope".

jbrozena22• 10 years ago

Question to all - would you consider paying for access to a platform that organizes the interview process Amir described? It sounds useful to me but would like to hear other opinions.

ronilan• 10 years ago

Any employer should be required, by law, to pay candidates interviewed, an hourly wage equal to that of an employee in same position.

It will be good for candidates, good for employers and good for the economy.

omribahumi• 10 years ago

Do you mean pay them for the interview time? People will start interviewing for a living.

a13n• 10 years ago

I find this pretty reasonable if you're interviewing with a handful of companies but I remember doing 20+ interviews in 2-3 weeks in university and would simply not have time for that.

countingteeth3• 10 years ago

I'm all for pounding the pavement if your network doesn't line you up, but 20+ in 2-3 weeks suggests a very different assembly-line approach. I'm having difficulty imagining how it would be logistically possible for me to line up 20-30 interviews in 2-3 weeks where I had predetermined that they were all places I wanted to work and could make my case. I would hope for a batting average and negotiating leverage where if I got in the door at a lot fewer places I would find something that worked.

Then again, everyone is different.

Gustomaximus2• 10 years ago

Payment option is really good points. With take home problems, there is risk their brilliant buddy helps them out, and teaches them about the solution to help them talk about it. For this I feel its best to do any tests onsite.

jon-wood5• 10 years ago

Is this really a thing, or are you just optimising for a problem that could potentially happen? Maybe I'm just being naive, but I'd like to think if someone did that I'd quite quickly realise either at interview or on hiring them that they're not actually capable of doing the job by themselves.

Gustomaximus2• 10 years ago

I'd quite quickly realise either at interview or on hiring them that they're not actually capable of doing the job by themselves

Not sure. The whole idea of the test is to get around people that talk well vs do well.

And assuming the friend told them about how/why they did something the interviewee can talk about this solution as if they did it. How would you know? See I'm assuming the 'friend' adds value to the solution to make it amazing, and the person who is interviewing still has average knowledge so its not like they are completely winging it.

are you just optimising for a problem that could potentially happen?

Absolutely. You should optimise for problems that could happen if they have reasonable likelihood and will become significant issues if they do happen. Not going to rule out all issues. Thought this seems a likely risk over a reasonable number of candidates.

eranation• 10 years ago

This is great, except the over the weekend part. I have a wife and kids, I rather be with them than solve (even paid) challenges. I rather take half a day off, solve it right there in the environment and team that will potential work with you. I would encourage candidates to also reach out with questions during the challenge and see how they communicate ideas and react to suggestions. Other than that, completely agree.

kohanz• 10 years ago

Completely agree. I love the idea, except the weekend part. Not only do I not want to do your coding puzzles over the weekend, it will also make me think that weekend work is a thing at your company.

greglindahl4• 10 years ago

Are you imaging an employer too dumb to allow you to suggest an alternative format that fits your schedule? If they say "no", you've learned enough to not want to work for them.

pitaj• 10 years ago

It says that you should give them the option. If you don't want to do that, just finish it the day of the interview.

jon-wood5• 10 years ago

I'll second that. When interviewing for jobs recently the sheer volume of tasks being handed out to prove I can program was a real problem - I work five days a week, my wife works Saturdays and I look after our son. That leaves an hour here or there available in the evenings (when I'll do a bad job because its 10:30pm), or taking a chunk of time out of the only real time we get together as a family.

If you apply for say 4 jobs and they all want 2 hour work samples you've just taken on an entire extra work day to apply for jobs. I can't even just take a day off work and batch them because of the way applying for jobs works.

Honestly, I don't know the right answer. From the other side of the table I like this approach because it lets me assess how someone thinks about solving problems and writing code, but its just reinforcing a culture where only young people without any real responsibilities outside of work can get into the industry.

mcv4• 10 years ago

The advantage of being paid for this is that you should be able to afford taking time off from a paid job.

mcv4• 10 years ago

I think it makes sense to discuss which schedule works best for the applicant. Maybe they prefer to do it during the week, maybe they prefer to do it right there and then.

paxy• 10 years ago

This is one of those "aha" solutions that people keep mentioning, but completely ignore how it would work in a non-ideal world.

• What 2 hour project can you assign that would let the candidate go into enough depth for you to judge their code, design decisions, architecture skills?

• If you give them more time (e.g. a weekend like in the article), is it still financially viable to the company, especially at $100/hr??

• How do you filter out the candidates that aren't serious about the job but just want to make a quick buck?

• What about all the candidates out there who you can't legally pay (e.g. workers on a visa)?

This works when you are a small company that needs to pick 2 out of 6 total applicants, but otherwise completely unfeasible.

ubernostrum4• 10 years ago

What 2 hour project can you assign that would let the candidate go into enough depth for you to judge their code, design decisions, architecture skills?

Where I work, our take-home is a simplified version of a task that's common enough (in our field) that we built tooling for it. It's generic, it tests understanding of a few things, and we've gone through three iterations of our own tooling so we know there are multiple viable approaches.

As for judging decisions, that's why the interview that goes with it is a collaborative code review. You treat it the same as if you were reviewing code a co-worker had written; ask questions, discuss tradeoffs, etc. etc.

mahyarm4• 10 years ago

Being on visa makes this payment thing harder unfortunately. Maybe the charity idea might work for those people.

philipkglass2• 10 years ago

I read the money/difficulty calibration as "pick a problem that you think would take someone about 2 hours to implement and offer $200 for completion." Not "offer $100 per hour and then cough up $1600 when the following Monday the candidate swears that they spent two full days on it."

mjevans2• 10 years ago

A key point of the specification for the solution is that the produced work //is not// a product. You are using the exercise as a test.

Maybe it would be better to explicitly state that the results of the work are to be released under CC-BY?

ohitsdom• 10 years ago

Exactly. Their example:

"Create a single page app that lets me enter movies in my home movie collection, store them in offline storage and search through them. I want to search by Genre, Title & Actors."

Significantly more than 2 hours, unless you are looking for a quick hack and for the candidate to repeatedly explain "I would have done it this other way if I had more time..."

silverbax884• 10 years ago

I completely agree. I saw that and thought - um, that's not a 2 hour job unless you want a massively hacked together app. I might be able to do something modestly decent over a weekend, but 2 hours? Nope. You have to have ways of inputting/verifying data, tagging genres, etc. I've done a lot of similar code but that's why I know it's basic, but not quick. It would let me know that the job wasn't going to last because of the lack of knowledge in estimating software development.

ebrenes3• 10 years ago

That's only if you're expecting a candidate to provide a perfect solution; and don't make it clear that the time limit should inform their design and not just measure their speed of implementation.

Part of software development is tailoring a solution to the scope of the problem, and this includes taking into account the time and effort available to come up with a solution. Someone who looks at that problem statement and designs something that will take a week has missed a very important requirement.

You could reasonably scale up any problem to take an arbitrarily long amount of time, by adding onerous requirements (must have 7 9's of reliability). What I've found more rare and precious is the developer who can limit the scope so that items that are not up for negotiation like time (2 hours) & resources (1 developer) are respected.

I fully expect a developer to state their assumptions up front, so that I can judge their design based on the stated assumptions and not some unstated imaginary assumptions. It also lets me see how or if they prioritize.

closeparen2• 10 years ago

• What 2 hour project can you assign that would let the candidate go into enough depth for you to judge their code, design decisions, architecture skills?

What whiteboard interview would?

susam• 10 years ago

What 2 hour project can you assign that would let the candidate go into enough depth for you to judge their code, design decisions, architecture skills?

Let me answer that with an example. In my previous job, I had to implement a simple username/password based authentication API in the first interview. The problem was explained to me. A two page document with a written specification of the problem was also provided to me. After I read the document, I was asked how long I would need to solve the problem. I asked for 40 minutes. Then the interviewer left me alone in the room with a laptop. After 40 minutes, the interviewer returned and went over the entire code I had written, discussed the API design decisions I had taken such as the corner cases that were handled or not handled, exceptions that were thrown, how empty or null usernames or passwords were handled, what error messages were generated, how concurrency was handled, what performance problems could occur if thousands of requests invoke the API per second per thread, how the password was secured, etc.

After I got the job, I learnt that my colleagues had also gone through a similar coding interview with different problems of similar size and scope. Some people asked for 30 minutes while some asked for 90 minutes. Therefore, I believe that it is possible to have a tiny 30 minute to 90 minute project that lets a candidate demonstrate his or her software engineering skills.

I felt this method was very effective because it evaluated a couple of key skills that were important to do the work that we were doing there: time and effort estimation, API design, writing clean and correct code, robust error handling, and design decisions from performance and security perspective.

I didn't get paid for that 40 minutes but it was part of their half a day onsite interview process, so I did not have to spend any extra time apart from the time I had already scheduled for visiting their office. Also, they did provide me free lunch, snacks and drinks, so I didn't complain. I wish more employers did this.

clifanatic4• 10 years ago

What 2 hour project can you assign that would let the candidate go into enough depth for you to judge their code, design decisions, architecture skills?

Well, the movie project suggested in the article was a pretty reasonable one.

silverbax884• 10 years ago

If you write that in 2 hours, honestly I can't see that being code I'd hire someone for. You'd have to hack together garbage code and you'd look like someone who threw things together without thinking.

linkregister• 10 years ago

Since this would be a 1099 contractor relationship and not an employee relationship, would there be any issue for Mattermark in this case? All they need is a TIN for their 1099 filing; it's up to the visa holder to know whether they are permitted to do contracting work.

user59944617• 10 years ago

What 2 hour project can you assign that would let the candidate go into enough depth for you to judge their code, design decisions, architecture skills?

Write a program that reads a text file, and output each unique word with the number of times it appeared.

The program must be done in a programming language among: C, C++, C#, Go, Java, PHP, Python, Ruby.

umanwizard4• 10 years ago

foreach word: ++(wordmap[word])

print wordmap

This is on the level of a pretty simple first phone screen problem.

draw_down2• 10 years ago

Remember: if you do something that completely eliminates bad hires, it probably eliminated some good ones too. Of course, lots of people wave this away by saying bad hires are more expensive than bad rejections. OK- but how much more expensive? And does this practice make sense in light of that figure?

Not to say you shouldn't do this but it's very important to remember a lot of these kind of things when it comes to recruiting involve tradeoffs and you rarely get an unalloyed good.

sargun• 10 years ago

How do you avoid people outsourcing their projects?

teach• 10 years ago

I'm a high school teacher, so my experience isn't directly relevant but I've certainly encountered my fair share of cheating.

The student that wrote the code can always explain not only how it works but why they made certain decisions along the way.

The student clueless enough to copy someone else's code usually can't even explain its function, much less explain tradeoffs.

erikb3• 10 years ago

If they need to present it on Monday to the team you can basically exclude that. If they do that they can't understand it well enough to really present it.

zenincognito• 10 years ago

I am afraid this is just another marketing for a paid product. There are a lot of assumptions being stated as a fact.

Since the candidate is getting paid, the candidate treats it as a legit consulting session and therefore gives their best effort in solving the problem.

If a candidate has applied for a job then it is in his interest to treat that application as his career depends on it. Best effort is wagered not on the fact that he is paid but on the premise that his life will become better by getting this job.

It gives the candidate a real life sense of how you interact with people on the team.

Where is this coming from ? This is just another assumption the author is making.

The biggest issue I have seen with tech hires is that they can become very defensive over their solution. I will purposely challenge their solution to see how they react.

You are asking us to be non-judgemental in the paragraphs above and then being judgemental on the candidate becoming defensive.

Pay them immediately on Monday

Why ?

minitech• 10 years ago

> It gives the candidate a real life sense of how you interact with people on the team.
Where is this coming from ? This is just another assumption the author is making.

It’s coming from the fact that you’re giving someone a project and discussing the result with them.

> The biggest issue I have seen with tech hires is that they can become very defensive over their solution. I will purposely challenge their solution to see how they react.
You are asking us to be non-judgemental in the paragraphs above and then being judgemental on the candidate becoming defensive.

That’s not being “judgemental”. It’s just using regular judgement – evaluating a person’s willingness to listen to criticism. The earlier points were about hearing rationale for choices without previous criticism.

> Pay them immediately on Monday
Why ?

It shows that you’re well-organized and keep your word.

I think most of your arguments were coloured by the mention of a paid product.

zachd1_618• 10 years ago

This. 100% this.

As many others are saying, I find that this is the most comfortable way for me as an interviewee to demonstrate that I do in fact know what I'm talking about. Put me in front of a room full of people and a whiteboard and throw something at me, and I'll freeze. Freezing in that situation and being a bad engineer are wholly separate things.

Anecdotally, when I get one of these problems, I spent 20 minutes reading the prompt, looking at resources and making an outline of my code. Then I step away for the afternoon and don't think about it. Let the subconscious internalize the problem for a while and when I come back, I've got a much better mindset to approach it. Solving new problems on the spot is just a skill I don't have. I prefer to think that this is what makes you valuable as an employee and engineer, solving a problem by the end of the day. Not while under scrutiny. Personally, my resume and experience don't get me far interviewing by the big companies, but on the occasions that I get take home code assignments, I tend to get pleasantly surprised and pleased reviewers and it takes me to the final rounds (where I face the dreaded white board and actually-there-is-a-correct-answer situationals).

My two cents, this just really resonated with me. Does anybody else happen to solve problems this way?

throwaway4job• 10 years ago

This isn't a viable approach for much of the hiring pool, as a majority of working engineers have contracts which explicitly forbid work for contract for other employers.

As such, if you hire any of those, you're hiring somebody with a proven willingness to ignore their contract, which is a strong anti-pattern.

zaroth4• 10 years ago

IANAL, but that's not what this is.

"Work for contract for other employers" could literally mean I can't help my neighbor mow his lawn in exchange for a beer. In this case what it really means is a non-trivial amount of work done for meaningful profit.

If I fix a friends computer in exchange for a nice meal, I am using my relevant skills to perform work, but I'm not actually violating a contract forbidding outside work in a way that could ever possibly result in damages or firing for cause.

For a potential employee to be so rule-stricken as to concern themselves with performing a coding interview, that would be the potential red flag for me. So this could actually be another positive outcome of this approach.

sillysaurus3• 10 years ago

It's possible to not pay them. Most candidates will do this for free, as it's far better than traditional ineffective hiring practices.

The thing is, there are always reasons why this method isn't viable. It's not till you try it that you notice huge gains.

wpietri• 10 years ago

Will they? I've seen contracts saying you can't work for competitors, but I'm not sure I've ever seen one that forbids all other employment.

In California, it's illegal to prevent employees from moonlighting: https://californiaemploymentlaw.foxrothschild.com/2015/03/ar...

elchief• 10 years ago

Or a $200 donation to their fave charity

inopinatus• 10 years ago

IANAL but I took business law 101, which taught me that courts almost universally lean strongly towards the employee when considering restraint clauses.

Moreover, if you have a current employer that is sufficiently disrespectful of their employees that they'd even consider enforcing an anti-moonlighting clause in this circumstance, with the preposterous assertion that it qualifies as employment - run away now.

The law is not a programming language.

moron4hire3• 10 years ago

> a majority of working engineers have contracts which explicitly forbid work for contract for other employers.

Well that was very, very dumb of them. Maybe we engineers ain't so smart after all.

quicklime• 10 years ago

On top of this, many candidates have visa-related restrictions that would prevent them from doing this legally.

dimino2• 10 years ago

I don't agree that it's a string anti-pattern whatsoever. It's a clear signal that the person can interpret the meaning of a rule and adhere to it in a way that makes sense for the person who made the rule as well as the contractor.

majormajor4• 10 years ago

I don't understand the paid part at all. If I'm working full time, $200 isn't going to swing me one way or the other on deciding if I want to do your homework assignment. And if you come up with a ten-hour problem, then we're talking more of a chunk of cash, but you're also asking me to burn a lot of free time.

I'd much rather travel and meet you in person and do stuff there, but if you don't have the budget for it, we could figure other stuff out (shared docs, skype, whatever, I've done a few of these on both ends with some luck).

"Since the candidate is getting paid" seems like far weaker motivation than "since the candidate wants the job you're offering," which is what's going to determine my level of effort.

The "not a real problem" thing is key, though. I've tried testing out a few different "this is one of our real problem" things but there's almost always more hidden business rules or assumptions in there than you think.

elcct2• 10 years ago

I think person who is either very well compensated for that or with slave mentality would sign such a contract. Either way I don't think those people are a target group of this post.

st3v3r5• 10 years ago

Non-competes are worthless clauses that should never have existed in the first place. I am perfectly ok with someone ignoring that blatantly anti-employee clause.

blazespin• 10 years ago

This works, but make sure they spend the 2 hours in your office and not 12 hours at home solving a 2 hour problem..

The other problem is that when you pay them you generate all sorts of nightmaris legal liabilities.

davidbanham• 10 years ago

Absolutely agree with the sentiment here. I have used this technique for a couple of years now, except that I kept the task to 20 minutes (and didn't pay people, since it's a short amount of time).

To save dragging people into an office for that time period, I set up a custom git server that marks the timestamps of their pulls and pushes. The instructions for the task are contained in the README of the repo, so as soon as they pull it they've started.

https://github.com/Prismatik/codescreen

It's critical to me that the limit is adhered to so that over time I can compare candidate responses on a level playing field.

Udo• 10 years ago

This works, but make sure they spend the 2 hours in your office and not 12 hours at home solving a 2 hour problem..

This goes against the core message of the article. The candidate is supposed to think about a problem on their own, and propose a well-reflected solution on Monday. That's pretty much the opposite of "sit down in this noisy corner in an unknown environment and give us something in two hours".

The other problem is that when you pay them you generate all sorts of nightmaris legal liabilities.

If hiring a freelancer to do a little thing tends to generate all sorts of nightmarish legal liabilities, your company is either a government contractor or it has even bigger structural problems than hiring.

JauntyHatAngle• 10 years ago

This works, but make sure they spend the 2 hours in your office and not 12 hours at home solving a 2 hour problem..

Did you read the article?

That's what the article author is saying not to do, they prefer to see a well worked solution after you define a particular technology, showing they are adaptable and have thought it through properly.

Limiting them to your office, timeboxed, advantages people who spit out lines of code quickly, which is definitely not the right fit for a huge amount of coding positions, especially when you are emphasising logical choices and design decisions.

deepGem4• 10 years ago

What's the legal liability ? It's not illegal to do a small project for someone and get paid for it. If the interviewee is on a H1B or some visa like that then I can see some issues.

jobu• 10 years ago

The other problem is that when you pay them you generate all sorts of nightmarish legal liabilities.

I was wondering about the tax and legal implications around paying someone for an interview challenge as well.

The overall idea seems fantastic, but paying or receiving money for an interview coding challenge is a little weird.

jpalomaki• 10 years ago

2 hour deadline and forcing the task to be done at the office turns it to a different kind test. This becomes more like how do you perform under stress and how well can you propose things out of your head. These are valuable skills as well, but I think having the weekend to spend with the problem more accurately models the actual development work.

ubernostrum4• 10 years ago

This works, but make sure they spend the 2 hours in your office and not 12 hours at home solving a 2 hour problem..

Yes. And to make sure they don't cheat on it, maybe don't let them use a computer or the internet. A whiteboard ought to be enough. And just to be really sure, make them explain what they're doing in real time as they do it.

That will do quite well, and offer a wonderful alternative to the standard, awful, whiteboard coding interview.

stillworks6• 10 years ago

I don't care about getting paid. I don't care if I am doing it at home or in the interviewer's premises.

Don't let me connect to internet. I'll also leave my phone with you just to make sure I don't have access to the internet. I don't mind taking the time away from current employment to go through the process.

BUT PLEASE LEAVE ME ALONE THOUGH WHILE I AM AT IT. FOR GOD'S SAKE DON"T SIT NEXT TO ME AND TALK TO ME OR EXPECT ME TO TALK TO YOU WHILE I AM SOLVING THE PROBLEM !!!

Let's discuss the solution or the non-solution AFTER I have spent some alone time with the problem. I don't mind getting rejected AFTER I have made a serious attempt at it.

It is simply beyond me why people think it's all right for a complete stranger to sit next to a programmer while he is trying to solve a problem under the significant cognitive load of a new environment, new people possibly a laptop/keyboard he has never used.

This did sound like a rant :-/ but what I wanted to say was that the process in the article seems fair enough. Interviewees should be given the alone time to solve the problem.

Most people are decent and sociable enough that once they are one board and are comfortable and familiar with the rest of the team they will be able to function and be reasonable in pair programming envoirnment or other technical discussions as well. BUT not when they are in an interview. No matter how it is sugar coated, interview is where a programmer is getting JUDGED and that by default leads to pressure.

blowski8• 10 years ago

Sounds like I'm completely the opposite of you! I'm fine with someone sitting next to me - I love pair programming.

But cutting off internet access means no StackOverflow, no online documentation. The test should be very similar to normal working conditions.

hexagonc• 10 years ago

Basically, you just wrote nearly (exactly?) what I wanted to write. I'm good at algorithm problems. I LIKE algorithm problems, but having someone watching me on a whiteboard in a high pressure situation diminishes my abilities. At a minimum, it is a distraction. If it is an easy or familiar problem then it usually doesn't matter and we can discuss the solution and I might even want to talk about it. But if it is a tricky problem -- and a lot of tech companies intentionally give problems that are tricky or vague or superficially easy -- then I need to use my natural process for systematically and rigorously tackling tricky problems. I'll still probably finish it in a half hour and we can still talk about the solution and discuss improvements but let me work on it alone first.

One thing I've found from technical interviews is that, due to the pressure to speak and communicate and "show my thoughts", I'm more susceptible to falling prey to trick and vague questions that I would otherwise treat more carefully. This is consistent with scientific results on the way people respond the pressure situations[1]. They tend to fall back on what is familiar instead of trying approaches that may be better. Working alone, there is no fear of judgement if I need to try an unusual strategy[2] that may not work or if I need to backtrack. Let me dabble and doodle on my own sheet of paper and iterate rapidly over whatever comes to mind freely.

Furthermore, turning problem solving into a theatrical event unavoidably increases the cognitive load on an interviewee simply due to the fact that you have to make more decisions. You have the technical decisions like data structures and algorithms which are normal and expected, but on top of that, there are all these non-engineering decisions that you have to make because they want you to talk about how you're solving the problem in real time. Should you mention some fancy algorithm or approach that you are considering? You must choose your words carefully since anything you say can be used against you.

I'm not claiming that everyone works better alone or even that I don't like working with others. I just think that the interviewee should have the option of working on the problem alone for up to 30 minutes. As suggested by parent, we can discuss the solution or non-solution after I've become familiar and comfortable with the problem. One of the worst feelings is when I fail to solve a problem during an interview, only to have the solution come easily when I'm able to tackle the problem in a pressure-free environment. Expecting some kind of portal into someone's mind while they solve an unfamiliar and maybe tricky problem strikes me as overindulgent and voyeuristic. It is a flaw in the current technical interview process and the only way to mitigate its effects is to just get more comfortable interviewing.

[1] https://www.psychologytoday.com/blog/choke/201106/flocking-t...

[2] - For tough problems, tend to use my own idiosyncratic variant of Z-Notation. I have a very high success rate solving algorithm puzzles when I use it but it is easy to forgot to use it under pressure of an interview.

erikbye• 10 years ago

If you can't solve a new problem "on the spot", or at least quickly, then one is at least not suited for SRE work.

z3t42• 10 years ago

There's a big difference acting on instinct and heuristics, versus actually thinking witch is very slow.

SamUK96• 10 years ago

actually-there-is-a-correct-answer situationals

Ugh. The worst step of interview process. Remember that guy who wrote a post about Google's hiring process, where they outsourced the asking of technical questions to untrained call-centre staff.

"Standard way to allocate memory in C?"

"malloc"

"Incorrect. Actually there's this cool library that does it for you with additional checks. Sorry, you have failed the interview"

"Wut?"

I've always found that those kind of interview processes put far too much stress on being correct to their "ideals" rather than be consistent with your own and your team's. I.e. I don't care if you name your column "domain_id", but for christ's sake name the other column "account_id".

mathw• 10 years ago

FWIW, my experience of Google's hiring process wasn't like that at all. Sure, it was hard, and they did do whiteboarding problems in the interviews (all seven of them), which is in its own way terrible, but it was all done with people who really did know the stuff they were asking me about.

I didn't get the job, but it was an interesting experience. Certainly selects for a certain kind of engineer, and would not be my preference for how to hire people if I was setting up my own operation.

expertentipp• 10 years ago

Remember that guy who wrote a post about Google's hiring process, where they outsourced the asking of technical questions to untrained call-centre staff.

Hey, I had this experience with Amazon when applying for a software related position based in Europe. For the entry screening I got a call from Indian phone number and a person with Indian accent was clearly going through a list of various quirks an features of a programming language. Wasn't so bad though... the second round with MBA product managers was much worse:)

SonicSoul• 10 years ago

i do love this idea but not sure if doing it remotely is a good idea. We do this in house because I am not sure how to be certain the candidate did not get help. I mean i believe people are generally honest but there will be cases when they are not, and I'd rather not find out 3 months after hiring them.

Also there are many indicators along the way, i.e. which parts the candidate got stuck at, did it take them 1 or 3 hrs to get to the same point (i.e. basic wiring of DB calls). I think all that would be lost if they did it on their own time.

Lord_Zero• 10 years ago

I am not sure how to be certain the candidate did not get help

Pressing them hard for information on Monday will expose this.

sidlls• 10 years ago

Do employees at the place you work at never ask their colleagues for help?

naterr• 10 years ago

I'm a subconscious problem solver also, I do it on purpose alot of the time, I can solve some things immediately but most of the time I get it later in the day when I go back to it or let the answer come to me.

md81544• 10 years ago

Yes, me too. I often find far better solutions to problems after backing off and deliberately not thinking about the issue.

hinkley• 10 years ago

We tend to think in a one size fits all manner and in a team environment I find that counterproductive or even destructive.

Every team needs a couple of people who can solve problems on the spot. They're going to fix your production issues. You need a couple of people who want to know why all the time. Someone with a big picture view. Someone with good aesthetics. Someone schedule driven, and a process nerd.

But they don't have to be the same couple of people. Diversity is more than fine, it's essential to avoid echo chamber syndrome. Some of the worst projects I've worked on had little to no dissent in the ranks. The code was self consistent but entirely wrong (bad info architecture, slow, brittle, impossible to learn after the fact).

But everyone tries to hire people that think exactly like they do. If we weren't so bad at determining that we'd probably never make anything interesting at all.

Florin_Andrei• 10 years ago

I spent 20 minutes reading the prompt, looking at resources and making an outline of my code. Then I step away for the afternoon and don't think about it. Let the subconscious internalize the problem for a while and when I come back, I've got a much better mindset to approach it.

That's definitely a thing, and I believe it applies to some extent to everyone.

But here's the other side of the coin - the ability to operate efficiently when the world is burning. I've been in the industry for a while, and this sort of thing definitely happens. But it's very hard to simulate in an interview. You don't know who the adrenalin junkies are on your team until PagerDuty lights up glowing red across the board.

The only indication I've seen is the correlation with a preference for risk-taking activities, such as sky diving, etc. But it's quite circumstantial. And there's definitely also the true reverse of it: some of the car racers and bull fighters out there are not good team players in times of peace.

Well, if at least a few key players can handle pressure gracefully, I guess you'll be fine, since it's a skill rarely used. You don't need everyone to be able to switch to catastrophe mode at a moment's notice.

mahyarm4• 10 years ago

This is the coding exercise/homework interview.

A bunch of place do this in some form or another, usually without paying the person, and as part of a series of interviews on an interview day.

I myself don't like homework questions, mostly because I still have to do an onsite, and I only have so much time to do these things. They are usually much longer than 2 hours too. If you communicate upfront that it should only take 2 hours and would remove the need for an all day onsite, then I would definitely be pretty excited. It would probably let you interview a segment of employed engineers who don't want to sacrifice a vacation day to do interviews.

I really like coding exercises as a general interview type although, I feel like it's the most important & real part of an interview.

The paid version doesn't work for people on a visa or are worried about IP assignment agreements, but I like the charity idea brought up by another person to solve this.

verbify• 10 years ago

I agree, maybe I'm a poor programmer, but 2 hours seems incredibly unrealistic for:

a) Creating a frontend form to submit movies b) Creating the database to store movies c) The 'search to find the movies' feature

Maybe I could do it in two hours, but it'd be pretty bad, and there's no point showcasing poor quality code.

My favourite was when they asked me to produce examples of things I had coded. Like any other creative industry, a portfolio is normal.

rnovak2• 10 years ago

Do you not use frameworks or anything?

CoC (Convention Over Configuration) solves most of the problem for you, since it's basically CRUD, and that is boilerplate stuff in pretty much any language.

mahyarm4• 10 years ago

We only expect you to make what is possible in 2 hours, and we've seen a bunch of similar exercises. It doesn't have to be poor quality, just basic. Usually the UX isn't expected to look nice at all either.

physcab• 10 years ago

One thing I enjoyed about consulting is the opportunity to work with so many companies that gave me a glimpse into their work culture. There have been companies where the experience I had interviewing completely shielded the shitshow underneath. And other companies where the interview process was really boring and routine, but the people inside were dynamic and amazing to work with.

I like this approach of merging consulting and interviewing because it gets interviewer and interviewee closer to the source of truth about how a real work relationship will exist.

bbcbasic• 10 years ago

For me time is money is less important than time time. 100 hours aggregated over interviewing for a few shops would yield $10k which is nice but not extremely useful, whereas those 100 hours taken out of family time is a real pain in the butt.

The solution to this I feel is a bit of a prisoner dilemma because the entity that can make this easier is the company the candidate is a planning to leave (!). If they have flexible working, make it easy to take vacation on short notice etc. then that ironically helps.

teen• 10 years ago

Coding on a whiteboard works. There are just a lot of people who can't code, and complain & make excuses. I'm so tired of these clickbait posts.

kasey_junk4• 10 years ago

How do you know?

lsiebert• 10 years ago

Wow, so you want me to work weekends is what you are telling me.

I suppose it's nice that this is paid. A lot of companies give me HW problems... and frankly it doesn't scale.

peteretep12• 10 years ago

Looks reasonable from my experience. Don't think I'd offer money for up to two hours coding test, but I've certainly offered money for something that took a whole day before.

honkhonkpants• 10 years ago

No mention of the tax implications? Paying every candidate sounds like a wonderful way to increase paperwork.

yoloswagins• 10 years ago

Collecting a W-9 takes 2 minutes, and is only required for payments above $600.

pmiller2• 10 years ago

There's an inexplicably [dead] comment in reply to your comment here. I'm reposting it so more people can see it.

@throwaway4job, you appear to be hellbanned for no reason I can discern. :(

throwaway4job 8 minutes ago [dead] [-]
This isn't a viable approach for much of the hiring pool, as a majority of working engineers have contracts which explicitly forbid work for contract for other employers. As such, if you hire any of those, you're hiring somebody with a proven willingness to ignore their contract, which is a strong anti-pattern.

duskwuff• 10 years ago

I've vouched for the comment, making it-- undead? I suppose.

geofft• 10 years ago

My guess to reasoning is that they called a startup founder a sociopath who would kill people to make a profit.

That said, that was a completely defensible and evidence-based thing to say in context (the founder threw a fit about regulators who wanted to know what the chances are that people would die), so unless HN is cracking down on the casual use of "sociopath" by people who aren't making professional medical diagnoses (which, to be fair, I do find objectionable, but not to the point of hellbanning), it seems like it was for no sensible reason.

dang2• 10 years ago

It isn't always possible to figure out from the public data why an account is banned.

When a banned account's comment gets vouched for and rescued, we look at the account history and unban it if we see that we banned it unnecessarily (we try not to do that but it happens) or if the commenter has reformed. Alternatively, there may be a good reason for the account to stay banned, but if it posts good comments, HN users will vouch for and unkill those. This system has proven to work well, even better than we hoped when we introduced it.

dang2• 10 years ago

There's an inexplicably [dead] comment [...] I'm reposting it

Please don't do this. We created vouching for just this case and it works much better. To vouch for a [dead] comment (assuming your account has > 30 karma), click on its timestamp to go to its page, then click 'vouch' at the top.

Besides being distractingly off-topic, making a big deal out of [dead]ness confuses the thread when the comment does get vouched for and rescued, as typically happens for good but [dead] comments. For example, in this case the replies to throwaway4job's comment are now split across multiple places in the thread. We'll move them.

We detached this subthread from https://news.ycombinator.com/item?id=12916816 and marked it off-topic.

zaroth4• 10 years ago

The 2 hour version of this is what I've always done in actual interviews, except where we peer develop the solution together the whole time, and I let the candidate drive and lead as much as they want to.

The take home version might be useful beyond that, but I've found that solving some fun toy problem like, "let's build a little app for X" in a couple hours you absolutely know if the candidate can code and have a good feel for what it would be like to work with them.

zzzcpan2• 10 years ago

No, yours is different, basically an exam, that requires a lot of preparation, good sleep, etc. to perform well. Very few would be able to do it.

zaroth4• 10 years ago

I think it's different from an exam because the candidate is working on their own laptop, with their own tools, with full access to Google, StackOverflow, etc.

If for some reason they don't have a portable dev environment I guess we could provide it but that does hamper the process somewhat because it's not really "in their element" anymore.

I don't know how you really prepare for such an interview, other than actually having the portable dev environment and being well rested.

The only thing we use the whiteboard for is understanding the problem statement and sketching out the steps to creating the solution, as in, things a whiteboard is actually meant to be used for.

ryanSrich• 10 years ago

Great post. In summary:

Pay people to complete a real life project and then bring them in to talk about it.

Why is this so hard for most tech companies to understand?

nunez2• 10 years ago

My current employer (ThoughtWorks) interviews like this. I had a week to do the problem they gave me (it was unpaid).

It was the best interview I've ever done. I rode a huge wave of emotions the entire time and learned A TON.

Technical minutiae interviews like The one I did at Google really suck

phillro• 10 years ago

I know its only a small fee, but legally wouldn't this require issuing a 1099?

eseehausen• 10 years ago

It's under the limit ($600): https://www.irs.gov/businesses/small-businesses-self-employe...

But even if it wasn't, a company of any significant size (enough to pay even 1 person's salary, say) should have a low effort and cheap way of tracking contractor payments and issuing 1099s.

halfear• 10 years ago

Something that we experimented with was pointing candidates towards one of our public repos and asking them to make a meaningful PR to be discussed at the interview. The PR would be discarded afterwards irrespective of the interview outcome.

tdeck• 10 years ago

Love the idea of paying people. That's great. However, consider the perspective of someone who already has a job in the industry. If every company did this, I wouldn't be able to interview at more than one per week. Not only that, but the Friday and Monday onsites mean I miss work on TWO separate days. This would be a non-starter for me if I had any decent pool of interest from other companies, although it's great for people trying to break into the industry.

Another problem, depending on your philosophy. Specifying the exact technologies someone has to use means that you've closed yourself off to generalists, unless they have time to learn a whole new stack that weekend. Not that it's rocket science to do so, but that turns a two hour project for me in say Angular into an eight hour project in your favored stack. I can always learn your stack on the job, as any good engineer can.

extortionist• 10 years ago

Agreed with this. I wouldn't take a 4-day weekend to do interviews and a homework assignment unless I was desperate or already out of a job.

BigJono• 10 years ago

I feel like it's in the best interest of the employer to leave the choice of tools fairly open and see what the potential employee elects to use. For the example given in the article (a basic web application), I'd be very interested in whether the interviewee decides to go with a heavy stack, does it completely in vanilla JS, or something in-between.

oddlyaromatic• 10 years ago

I wonder about this. I'm pretty new to js frameworks and I see the advantage when things need to scale or get very complex, and I get that companies use these things. But for a simple assignment like this, is it better to use vanilla js and maybe handlebars, or "show off" that you can implement all the routes and whatnot in Angular, even though that's overkill? Or maybe the interviewee's justification of their choice teaches you that they understand the trade offs.

ohwello• 10 years ago

Yeah, it makes sense to me as a sort of moneyball approach, to cheaply hire skilled candidates who are not as good at the standard evaluation process, are likely to have few competing offers, and who are willing to do extra work on nights/weekends.

Nacraile• 10 years ago

From the perspective of someone who already has a job in the industry, I'm really not going to want to do more than one interview per week anyway.

This format also doesn't appear to want full days: it should be an hour or two Friday afternoon, plus a half of Monday. Two part-days isn't really that worse than a full day off (Mileage may obviously vary; particularly the mileage to get to the interview). The initial interview could also very easily be done remotely.

Even if it is a bit higher-impact, I would still prefer it simply for the greater quality of experience, and that's before throwing in the payment.

avodonosov• 10 years ago

How to invent an interesting problem which can be solved in 2 hours, which can not be found in the Internet and would allow candidate to demonstrate something?

And you need to produce such problems relatively often, right?

jon-wood5• 10 years ago

Ideally you only produce one good one, which allows you to benchmark answers against previous ones. The other important thing is that you should do the task as well - that both ensures it is actually doable in the allotted time, and gives you an initial benchmark of what you're hoping for.

datashovel• 10 years ago

I won't promise anything, but have been contemplating an idea for a relatively long blog post about exactly this.

Having lived for the last several years in a very rural area of the midwest I would say that not only do you need the right mechanisms in place to sort through / vet candidates, but you also need to be willing to constantly venture outside your comfort zone. Why are you always recruiting locally? Why aren't you willing to look in all locations throughout the country? To be frank, I would guess that almost never do you find a "diamond in the rough" outside of large metropolitan areas, but my personal experience is that as soon as I work with a team in a metropolitan area I realize I'm almost always substantially ahead of the game in terms of best practices / capacity, etc. I would probably argue that I'm an exception to the rule, but (not to be political) the recent US election leads me to believe that the "norm" is not the "norm" any more. People are thinking outside the box and we (as recruiters / employers) need to realize this and not instantly discount a person just because of where they live (btw I'm pretty strongly Dem living in a mostly Rep world).

rnovak2• 10 years ago

Having been on the hiring side of this, I'll note that it does require a bit of start-up cost (i.e. work) on the employer's part, if you want to standardize the process (i.e. to make sure you're giving the same exercise to every candidate, the expectations are clear and consistent, etc).

For example, I set up the project for my team in 7 different languages (Ruby, C, Python, Perl, C++, Java, Go), with the appropriate project structure, etc, for each. This took considerable work on my part, but the end result is that I can judge the output based on whatever the candidate is most comfortable in.

Granted, that's all assuming my skills in each language doesn't completely suck (hopefully).

ceronman2• 10 years ago

This is a great way of hiring, but unfortunately it doesn't fit in all scenarios.

First, this model alone doesn't scale. Big companies interview hundreds of candidates every week. It's not possible to design and evaluate a weekend project for each one of them. This doesn't mean that they can't have projects, but early filters are needed. That leads to the classic phone and F2F interviews to rule out those who can't code. If you freeze in one of those because of the pressure, you're out.

Second, for simple developer positions, such weekend projects might be very good. But if you are trying to hire a senior developer who is going to solve problems at big scale, then the ability to create a simple movie catalog doesn't tell you anything. And there are no big scale weekend projects. You have to ask about algorithms, data structures and other CS topics. These things are important!

Bottom line, this is a nice hiring technique suitable for small companies at a small scale. But it might not be that useful for big companies.

StreamBright2• 10 years ago

Sorry but this is simply not true. After a while we decided to do exactly the thing that the article describes as our preliminary interviewing process at Amazon. We also had extensive onsite interviews, however we did not ask candidates to code at the whiteboard. Btw, when you work for such companies it is expected that you spend 10% of your time on hiring. Getting hiring right is one of the most important things that you need to do at the scale of Amazon, Google and all of these large companies. Having a single bad hire can bring down the performance of an entire team.

pakitan• 10 years ago

I'm curious if you have any statistics about the different interview styles/processes. Obviously it's hard to figure out the false negatives (good engineers you didn't hire) but how about false positives? Also, is homework assignment now a part of the standard interview process at Amazon or is this just your team/department?

pgl3• 10 years ago

It's not possible to design and evaluate a weekend project for each one of them

Can't you just design and evaluate one project that they can all do?

burger_moon• 10 years ago

That will only work briefly because of glassdoor interview posts. However that might only mean tweaking it slightly every week or two.

ceronman2• 10 years ago

Even if you design only one project. You still have to evaluate them, have a discussion, etc. That takes time. It doesn't scale to hundreds of interviews per week.

rguzman• 10 years ago

You are right about the scalability of the process, but not about the rest. It seems possible to design weekend projects that would be way better for evaluating senior engineers than asking them to do CS brain-teasers on whiteboards. CS topics are important, but the typical interview is not great at identifying who is good at using CS to solve real production problems. It seems like the typical interview doesn't have many false positives but a lot of false negatives.

The common approach for big companies is to declare that false positives are way worse than false negatives, but if the problem is that you can't find enough developers to fill your roles, it may be better to have false positives coupled with a trial period or something similar.

fredophile2• 10 years ago

One important thing to realize with paying people for code they write as applicants excludes a large number of potential hires. People on work visas are not allowed to moonlight. I wouldn't risk ending up on the wrong side of ICE just for an interview.

throw_away_777• 10 years ago

Well then those people can refuse the pay and still take the interview.

fredophile2• 10 years ago

No they can't. If they did they would be doing volunteer work for a position that is usually paid and this also violates US immigration law (at least for the visas I'm aware of).

shams93• 10 years ago

Paying people to do demo work is revolutionary, I love this idea. I can't tell you how many hundreds of hours I had to put in over the last 25 years doing these "evaluation tasks" where essentially the company is getting me to put in 20 hours of work for $0. Then there's the places that use puzzles from CS courses to exclude anyone who hasn't been through CS school. Practical experience is often treated as worthless by our industry. They don't make a lawyer who has been practicing law for 25 years take some bullshit test to get a project. but in our industry the more experienced you are the more the industry diisrespects you.

ma5ter• 10 years ago

The company I applied for last week did similar thing for a devops position - They asked me to deploy an open source web application which is less popular and almost 0 documentation available on internet.

morgante3• 10 years ago

This interviewing method suffers from the tragedy of the commons. It works great when almost nobody is doing it because if most of your interviews are traditional it's easy to squeeze in 1-2 homework assignments a week.

However, it would be totally unworkable if most companies were to adopt it. Try finishing a dozen homework programs during a typical job search.

RandallBrown• 10 years ago

If it's supposed to be 2 hours of work, then it wouldn't be too big of an issue.

Even a simple phone screen is going to eat up 2 hours of my day, much less the full day interviews that most big tech companies do.

morgante3• 10 years ago

In practice, I rarely see assignments that are actually 2 hours. Even this article suggests giving candidates the weekend.

moconnor• 10 years ago

If they're all paying $100/hr you don't need an actual job, you can just perform a dozen interview questions a week and down the offers.

morgante3• 10 years ago

If you want to be a freelancer, you can do a lot better than $100/hr and spending 2+ hours to source each (small) gig.

nchuhoai• 10 years ago

I really like this, and have been trying to implement this at my company. One thing we are struggling with is to find the right challenge/project, in my case for the frontend. Two issues:

1. What is a reasonably sized project? I.e. often times two hours is not enough to produce something of substance/complexity, yet asking the applicant to do more becomes prohibitive. 2. I see a bunch of people asking candidates to do greenfield applications, while virtually all work most future workers will do is refactor and maintain. The latter however requires additional time investment to get to know an existing system/codebase.

How are you guys handling those issues, and trade-off time complexity against insight value?

stutsmansoft• 10 years ago

I'm a huge fan of the "demo project" interview. It really allows me to shine where whiteboard gameshows don't.

I don't need or expect payment, either. If I'm legitimately interested in the company, I am happy to spend a couple of hours showing them my stuff. Hell I'm proud to do it. I do enjoy programming, after all.

cloudjacker2• 10 years ago

Pretty good but rule #4 is totally totally opinion without any consideration about why someone would be defensive.

So given that Amir of June is the only person doing this quirky interview style (where you actually pay them for their time), the interviewer still has absolutely no idea of this psychological gimmick Amir has baked into his process. Right at the very end!

Amir Yasin admits that he is elevating them into a contracting gig status, and contractors ARE THE EXPERTS! They have to be the expert! They have to make up stuff on the fly to defend their implementation, "its a feature".

But for Amir's warped interview process, of which the candidate is being bussed around on a literal auction block at other companies before having the luxury of getting his groundbreaking interview style, he still has this one random assessment.

Look, Amir is the one that elevated his tiny sample size of engineers into gospel. There likely was a literal 2 people that formed his and his company's entire opinion about Rule #4.

So I think it is worthwhile to point that out.

eloff• 10 years ago

I wish I could filter jobs by this criteria. Can we do this in the who's hiring thread? I'm sick of the interview process and massive unpaid work assignments cough Compose cough only to find out after investing thirty unpaid hours that they can't offer US market rates to a Canadian for 100% remote job. Oh and they also want to own anything you do in your free time.

Sorry for ranting, I'm just pissed off at how an industry that brags about how they pamper their employees can treat them like garbage during the interview process.

deepGem4• 10 years ago

DuckDuckGo follows this process. They in fact do a paid 3 month assignment. May be that's taking it to the extreme and perhaps works very well for them. If I were looking for a job and this process was followed, I'd definitely be game. I will most likely be following a similar procedure while hiring as well. Just come to the office hang out with us for a day - we'll fly you in, put you in a nice hotel + pay you a per diem. All we'll do is work through a problem, not through a series of daunting hypothetical interview questions or some algorithm nonsense that you'll never use in your job.

zerr• 10 years ago

The only issue with DuckDuckGo job requirements - they require you to be an existing/active user of their product...

kctess5• 10 years ago

In some sense many companies do this via 3 month internships. They're basically just low cost future employee vetting opportunities. Of course, they are generally limited to college students.

ranveeraggarwal• 10 years ago

Is someone out there actually using this method? How do I get an interview?

yexponential• 10 years ago

How do you find companies that follow this or a variant over the traditional process. I've been interviewing at companies over the past couple of years and I've only participated at 1 take home task (startup) and know about 1 other company (on my list) that follows that.

I, like others stated above, tend to perform much better in my own space and time than under eyes on a whiteboard..

gigatexal• 10 years ago

Anyone know of places that interview like this? I'd love a shot to interview with a company that does this.

grandalf• 10 years ago

I've tried to do this before but CEOs are typically not on board paying for throwaway code. It does make total sense for all the reasons the post mentions.

jlebrech• 10 years ago

the downside to this is that potential hire may have a few interviews lined up and you might not be the first, so the "best guy" might get an offer while you're still undecided or looking for someone better. It's a good strategy if you're not yet desperate for another developer, but if you're more established and desperate for more than one dev you could hire a few and later let go of a few (contract to perm is more friendly and you can hire them again as a later point). in a way the presented solution is a micro-contract to perm.

lifeisstillgood• 10 years ago

But ... both sides have to really really want this.

This is the end of the process final wash

And so you probably won't do it with those candidates that fail your normal process. If you are biased against old, black, women you won't give them this test

It reduces your false positive rate. But won't ever help your false negative

shandor5• 10 years ago

This seems like a pretty popular take on the interview process here on HN. However, there's one pretty glaring flaw that I've never seen discussed: if the interviewee can take the problem home, what prevents them setting up a hackathon with their friends at their place, and making the interview problem a team effort?

A couple of knowledgeable friends can do wonders, and if you also prep with them a little you can probably answer all relevant questions and beyond about the code on the Monday on-site interview.

Sure, that's cheating. And even if it wasn't, I would never do it for the wrong expectations it would set upon me if I actually got the job. But the ease of "embellishing" one's CV always comes up in these threads/posts, and while I wouldn't do that either, I kind of fail to see how this home project approach solves that problem.

Anyone able to ELI5?

kybernetikos• 10 years ago

We have an extension to the take home done paired on site that usually involves significant refactoring. If you don't have a clear understanding of the code you're sunk. Plus the presentation should weed out the most egregious cheats. I also strongly suspect that developers that can't solve the task well will mainly have friends that also can't solve the task well. If someone had skilled developers invested in them doing well I suspect they are most likely a skilled developer.

shandor5• 10 years ago

We have an extension to the take home done paired on site that usually involves significant refactoring

That sounds actually brilliant. While some might argue that this comes back to the "whiteboarding under pressure", honestly if you have already written the code from scratch you definitely should be in a much better position than with the usual "implement a red-black tree insert, you have 15 minutes" whiteboarding tasks.

deepGem4• 10 years ago

One view of this behaviour is cheating, another view is hustle. I mean, if you could arrange a hackathon to solve an interview question - you are hired my friend :)

shandor5• 10 years ago

Sure I would, I have the $X/h to divide between them for helping me ;)

Seriously though, I guessed that many companies probably must have this kind of laid-back attitude on the matter, since it would be stupid to presume they are ignorant of my original point. Still, I think it might be a little unfair to the people going by the rules-as-written, which somewhat diminishes the consensus of awesomeness of this approach.

That is, if the interview process is not actively trying to choose for hustle... :)

maccard2• 10 years ago

Do you expect this guy to organise a hackathon to solve every problem you give him? And what if that havkathon now involves the people at your company working on his problems as well as their own?

z3t42• 10 years ago

I would also hire the dude on the spot, and probably his/her friends too.

JabavuAdams2• 10 years ago

Depends whether you're hiring for a management role, or an individual contributor role.

pgl3• 10 years ago

No interview method is foolproof. If someone goes to those lengths, you'd find out soon enough that they're not actually competent enough to do their job after hiring them.

But I'd consider that fairly unlikely. Plus, starting from a position where you don't even trust someone enough to behave like a mature human being isn't a great way to begin a professional relationship.

cm2187• 10 years ago

...or it is being ressourcefull!

dksidana• 10 years ago

I have solved and implemented workaround of this. Rather than only one home assignment of 2 hours, I give one assignment for home. Next assignment, which is enhancement of first assignment, is given when they appear of face 2 face interview.

This makes sure that if someone has not solved home assignment with other's help.

Peroni• 10 years ago

This is the primary reason that you shouldn't make hiring decisions based on the test/problem results alone.

The interview problem should only serve to be a conversation catalyst. If they got some help with the problem, outsourced the problem or simply googled prepackaged solutions to the problem then they are going to have a really hard time talking through the context of how they solved the problem.

What you're testing for here is communication skills and, if possible, a really basic datapoint that may indicate their technical ability. The latter is very subjective and incredibly difficult to test for but the former is arguably a lot more important if they are going to be working as a part of a team.

jaymzcampbell• 10 years ago

Totally agree, conversation catalyst is a great way to phrase it too. This is very much the approach I've taken having had very mixed outcomes from trying to do just one thing or the other. I find using a simple, 100% self contained task that should take a few hours and then using that to kick start discussion has provided the best way for us to hire and screen people. We've found using a task that is very simple to get a working solution but provides a lot of scope for edge cases or growth to be effective. It allows the candidate to not blow a whole weekend on something for the sake of it but gives us obvious directions to take the conversation.

Communication skills seem to be held in much lower regard despite them being the key thing for effective day to day work. I can manage / handle working with someone who needs more time than a "rockstar" or hasn't memorized TAOCP if they'are able to communicate effectively. Outside of very specific roles, non-trivial software development is a team effort. It's usually extremely obvious within 15 minutes if a candidate genuinely gets what they've coded or is winging it.

kriro4• 10 years ago

Compared to "traditional interviews". It's probably well worth taking the risk of occasionally hiring someone like this. I'd say it's less likely to occur than someone who has more or less memorized some interview question book algorithms and happens to get asked the right ones.

matwood2• 10 years ago

if the interviewee can take the problem home, what prevents them setting up a hackathon with their friends at their place, and making the interview problem a team effort?

Nothing, and that's fine. In fact we encourage candidates to use all resources on our take home interview test. I would probably be impressed if someone setup a hackathon to solve a rather simple take home problem.

A take home problem is simply another signal, and a jumping off point for discussions. If someone used some novel approach we'll discuss it and find out if they actually understand what they did.

The real surprising part is how many take home problems we have received that either do not compile, do not pass the test data we give, or look like the instructions were completely ignored. It ends up as a low effort FizzBuzz on our side.

shandor5• 10 years ago

we encourage candidates to use all resources on our take home interview test

This sounds great. In my response to another comment I was worried if accepting-but-not-encouraging this kind of behavior would put the interviewees on uneven ground. But if you tell them in advance that basically anything goes, it's fair for all parties.

The real surprising part is how many take home problems we have received that either do not compile, do not pass the test data we give, or look like the instructions were completely ignored.

Sounds horrible. Though I guess that's only a good thing. Seems like the assignment is doing its job in those cases pretty well :)

umanwizard4• 10 years ago

Really, any resources? What if they just pay someone to do the whole thing? Are you really confident you could suss this out in the onsite discussion?

clifanatic4• 10 years ago

setting up a hackathon with their friends at their place, and making the interview problem a team effort?

Cool, can they do that for all of their work from now on? Heck, I'll hire all of them if they're that cohesive a team.

MorePowerToYou2• 10 years ago

ELI5: you discuss the solution with the person, too. If they can't accurately describe it, they fail. They either have poor communication skills, don't understand what they wrote, or didn't write what they submitted. Those are all easy fails.

galacticpony• 10 years ago

Quote: "I may say please use functional reactive principals in JS to solve this problem, but the decision of using Kefir, Bacon, or RX is left up to them."

That sounds like something a bad tech hire would say.

dschn• 10 years ago

All you need to look for is passion. Worked for me.

amgin3• 10 years ago

No company is going to pay someone for an interview project assignment when they know they could easily make candidates jump through hoops for free.

a_imho• 10 years ago

Problem is, many companies are only playing interviews.

When you want to hire people who gets the job done it is no harder than:

1. Please tell me about a project you have done, a sw book you read, a hard bug you solved, really anything you like about computing etc.

2.a (This guy seems smart) Let's discuss compensation and you can start your probation period asap.

2.b (This guy is not smart nor passionate) Thank you for coming in, but we don't want to hire you.

edit: I used this method with great success, imo programming exercises are just an indirection that only adds noise and loosely correlates with on the job performance anyway

arpa• 10 years ago

Exactly. Also you have no idea what kind of cultural fit you're having if you only rely on technical skill. And besides, why do you want me to solve $problem? Take a look at my portfolio and public code - it's telling enough.

huehehue• 10 years ago

My employers don't want me open-sourcing the work I do for them, so this only works if you open-source your side projects.

I see a lot of comments here about how they would never waste their time on an exercise like this, but a small project for the purpose of an interview can probably be done in less time than any of your public projects.

deepaksurti2• 10 years ago

> They refuse to do the project because “someone will hire me without it”.

This is wrong in certain cases. The candidate may have significant contribution to an open source project or may have shipped an open source project as an owner.

Why should that candidate not refuse? You already have the code in front of you. If you as an employer cannot take the time out to go through that and evaluate what you would otherwise with your take home assignment, then something is not right.

This one size fits all rigid process won't work. Given a process, the employers must learn to be flexible, all the while expecting employees to do so, is insane. So certain qualified candidates saying 'someone else will hire me' is not arrogance, but fact.

codeulike2• 10 years ago

But the method proposed is only 20% about code. It's mostly about seeing how someone approaches a piece of work and discusses a solution in a meeting. You can't get that from browsing GitHub.

deepaksurti2• 10 years ago

Agreed. The Github code is a substitute for the take home assignment. If his GH code is good enough as per the employer standards, invite him in office to explain the work he did on that project.

Now employers don't want to do that most likely I assume is because no one wants to go through the pain of figuring out the details of GH code. OTOH, the employer is expecting the candidate to understand the problem assigned to him and come up with a solution!

jon-wood5• 10 years ago

You can from browsing GitHub, reading the code, and then asking how the code came to be as it is. Talk through why they went with the particular structure they chose, talk about the research which resulted in the UI, ask why they chose functional testing but not unit testing, think of a new feature that would be interesting and talk over how they'd approach adding it.

hydandata• 10 years ago

I have offered exactly this to multiple companies who interviewed me, but so far none of them was willing to devote time to do it. I guess industry is just too used to the established practice, and "hey, if Google is doing it why not us?" thinking. It is not like I declined to do their normal interviewing process either, I offered this /after/ their process was finished. It is the best thing to do for both sides. And this was where they were trying to recruit me, not when I applied.

f.e. I really like my current job, so I would benefit from getting a better look at your workflow and interact with the team to decide if I really want to work for you.

Hopefully the situation will improve in the near future, but I am so taken aback with common practice that I currently see no other way but to start my own company, I just cannot imagine working for such stubborn and close minded people. And for darn sure when I hire, this will be the process.

kinkdr• 10 years ago

The Triplebyte folks are doing something very similar. But I got the impression their clients don't understand the idea yet.

zwischenzug• 10 years ago

Why not have those two hours + discussion in your office and call it an interview?

yAnonymous• 10 years ago

I will purposely challenge their solution to see how they react.

Exactly like Google, who tell applicants that their solution is wrong when it isn't. Most people complain about how bad their application process is and that interviewers don't know anything. Nope, they just weed out smart asses like you that are a pain to work with.

throwaway_3452• 10 years ago

I had my first ever whiteboard interview recently. I severely underestimated how difficult it is to write code as compared to a terminal. At the end of it I came out feeling deflated and rather angry as the same piece of work on a machine would have been trivial.

edpichler• 10 years ago

Digest: pay for a task before deciding to contract.

On the article, Amir wrote: "They refuse to do the project because “someone will hire me without it”.

On book "Smart and Gets Things Done: Joel Spolsky's Concise Guide to Finding the Best Technical Talent" Joel said that the "the best developers usually have lots of proposals, so they don't spend time doing tests for interviews".

I think Joel and Amir are right.

StavrosK4• 10 years ago

I came here just to make the opposite point. They aren't making you do a test, they're hiring you for a project. It's fine if you're too busy even for paid work, but why would you turn it down otherwise?

I think Joel means "unpaid tests". I routinely turn interviews with unpaid projects down, because each interview process gets a number of hours of my time for free (depending on how much I want to work for that company) and unpaid tests/projects usually way way overrun that time.

Or maybe that's what you said and I misunderstood?

RandomOpinion5• 10 years ago

#11 of the Spolsky test is "Do new candidates write code during their interview?". I think Joel would agree that a take-home test would fall under that.

http://www.joelonsoftware.com/articles/fog0000000043.html

tummybug• 10 years ago

As someone who is looking for a new role at the moment and absolutely hates the standard interview process I would definitely apply for a role at any company that had this process.

samlittlewood• 10 years ago

One possible source of tasks for this process is the bug backlog from open source projects.

gregjor• 10 years ago

I'm glad to hear this works for you. But I think the process emphasizes technical skills over what really matters when hiring: how the person works with the team and within the business.

Giving someone a toy project unrelated to the business tells you nothing about how they will perform in the real environment. Letting the candidate take the work home to work on it in isolation tells you nothing about how they will work with your team, or within the company.

In his study "People and methodologies in software development" Alistair Cockburn found that “People’s characteristics, which vary from person to person and even from moment to moment, form a first-order driver of the team’s behavior and results." (http://alistair.cockburn.us/People+and+methodologies+in+soft...). Other studies have come to similar conclusions.

In a technical job, actual skills and experience are necessary but not sufficient for success. Programmers fail in their jobs because they can't work with or fit in with the team, and the larger organization. Any competent programmer can learn a new language, framework, or technique like TDD in a week or two. But people who don't pay attention to the business environment and requirements, and can't work with their team, group, etc. will fail, often costing a lot of money.

Presumably you have weeded out the unqualified before offering to pay them to work for a few hours, but this seems like offering an unnecessary incentive to learn very little about the candidate's likely on-the-job performance.

I don't know any scientific way to measure how someone will fit in, and I don't think anyone else does either. That's why having several people from different parts of the organization interview is the norm -- the interviewers compare notes and reach a consensus because everyone understands they are going mostly by their subconscious (gut) feelings.

I spend time describing the business requirements and then ask candidates how they might solve an actual business problem. I don't expect a complete or deep answer; what I'm after is finding out if the candidate can think about a business problem at all. Examples: Our inventory system frequently doesn't match what's in the warehouse. We get a lot of people abandoning their cart at checkout. We have customers gaming our promotion codes.

Testing for things like "functional reactive principles" is an example of ignoring the business environment and favoring technical trivia. No business ever had the problem "We need 2,000 lines of Ruby code by next week," or "We need a system to organize a small movie library, you have two hours." What I find is that far too many programmers don't even hear business requirements and can't do basic analysis or think of the right questions to ask. They want to start writing code. Having expensive people writing code that doesn't address business requirements is costly, and very common. If a candidate isn't asking questions about the business and the team that's a big red flag for me. Programmers who just want to be left alone to do their own thing are easy to find but they aren't necessarily a good fit, or productive.

I also spend time getting as many people in the organization as I can to chat with the candidate, so we can compare our impressions. That is just recognizing human nature -- we all tend to form opinions of other peole within a few seconds, and then look for things that reinforce our impression. Of course I want to see evidence of actual technical skill, but I can do that with a few actual programming problems -- even something on the level of FizzBuzz will tell you if the person can solve a problem in code. And I want to see how they solve the problem, and what kinds of questions they ask.

If someone is uncomfortable working alongside another programmer, or solving problems on a whiteboard, or talking to a group, I get it, but those are all necessary skills too. Just like learning a new framework, a motivated person can learn to work in a group, listen and ask questions, speak to a room. They may be a genius when left alone in a corner not interacting with anyone but they probably aren't going to be a good fit in most organizations. I often ask candidates to tell me how they would go about hiring someone for the same job they are interviewing for -- suppose we have two positions open and you get to hire a peer. That can tell me a lot about what they value in a co-worker.

IloveHackerN84• 10 years ago

This should be the best recruiting process adopted worldwide. Unfortunately, at least in Berlin, companies go for stupid quiz and questions they even don't know by themselves but that they've read from books.

vog• 10 years ago

Interestingly, this is exactly what you do when you start to work together with a freelancer. First, you give them a small, isolated task. If that works out well, give them more tasks.

msimpson2• 10 years ago

We use a similar system which involves:

1. A phone interview to validate the applicants resume.

2. An in-person interview where the applicant is given an hour to complete level 10 of Blockly Maze (https://blockly-games.appspot.com/maze?level=10) after being briefed on the wall follower rule (if they are unaware).

3. Completion of an outside example project in the applicants strongest language, for which they are not paid to avoid non-compete clauses.

This usually is enough to evaluate whether the applicant is embellishing on their resume, capable of analytical thought in a completely alien language, and their ability to deliver a deployable project given specific requirements.

sundvor2• 10 years ago

That sounds like a great way to do it; if I were an applicant, I would have liked process. (I am currently looking for jobs, and do have a recent experience of an on the spot test which I just hated).

I thought I'd try your step #2: I have no idea of the wall follower rule, and no prior experience in maze solving, but did bring it home in 18 minutes.

That included figuring out this particular Blockly UI (have admittedly seen similar UI with Dash and Dot, only at a much simpler level of doing non-branching, sample tutorials with my 6 year old boy), trying to design a simple program, reaching some dead ends / observing effects, and then realising what needed to be fixed.

I did use all the blocks for 17 lines of JS though, so perhaps better ways to do it, but thanks for that interesting little challenge at any rate. :-)

I'm guessing really smart people would have it done in just a few minutes.. Also monitoring son's cough (night time in Australia) to make sure it didn't get worse, for a bit of distraction.

(Edit: Brought it down to 1 spare block / 16 lines having a 2nd look after writing this comment.)

tthbalazs• 10 years ago

Hey!

You might be interested in a solution that uses less blocks (i would be). I might have been lucky, took me 2-3 minutes to find this.

https://blockly-games.appspot.com/maze?level=10#taj4p4

Does anyone know of a better one?

Edit: just as i posted this i saw that you dont even need the first "Move Forward", so it is 10 lines total with 4 blocks left

JabavuAdams2• 10 years ago

The Blockly maze thing is interesting, but you should realize that you're implicitly biasing against people with poor spatial skills.

My ex-wife is a concert-level pianist who can remember texts (words) after only hearing them once. She's also, astonishingly bad at spatial tasks (gender stereotypes aside).

So unless you're hiring for computer graphics, or visualization or some-such, which has a clear spatial component, you're implicitly biasing against less-spatial people.

msimpson2• 10 years ago

We are aware of this bias as our positions and projects often require such skills. I simply cannot hire someone who is unable to comprehend array sorting, vector math, etc.

etimberg• 10 years ago

I tried the blocky level. Got it in 17 lines of JS in about 10 minutes. :)

Quarrelsome• 10 years ago

This is what you do when you're looking for people on eLance/upwork/etc. I did this on my last start up and it worked amazingly. Its more expensive than getting lucky but cheaper than getting unlucky and allows you to find amazing people to work with.

EnderMB• 10 years ago

The place I used to work had actually tried something like this, but it completely fell apart, and I imagine when I tell this story you'll realise why.

One guy came to us from a recruiter highly-recommended. He had a few years experience, but he was widely respected online via his community contributions. He was also out of a job due to redundancies. It was enough to get him an interview, and he could talk well, so my bosses got the dev manager to set up some basic problems on a client website to solve.

He seemed like a nice guy, and he had a lot of questions, which was a good thing. He had a nice chat with some guys over lunch, telling them that he'd been unemployed for several months, and he was down to his last savings. He was able to fix the problem purely through looking on some forums for the answer, and that answer helped him fix the problem. The dev manager was happy, and the PM's were happy. The devs that worked with him were brought into a room to see his solution, and everyone in there said he should be hired. We had a new dev!

He was instantly given a project from the backlog, which was billed out, and would take around six months. He clearly knew what he was doing, so he was left to his own devices. To cut a long story short, the code he was writing was absolutely shocking, and he'd engineered a solution you'd need a PhD to understand. He was also finishing tasks really quickly, but they were bounced right back to him because his code was full of bugs. He stayed with the company for a while longer, but was ultimately let go. That project went over budget by six figures. It turned out that he was let go from his last two places for similar reasons. He was a junior developer that believed himself to be senior, and as such he disregarded any advice from anyone but managers. He had claimed open source contributions, but he was widely considered because he'd made thousands of posts on an open-source project forum, and due to their community contribution rules these posts deemed him a candidate for MVP. Despite thousands of posts, he'd never really solved a problem for anyone.

I looked through the code he had written during his trial, and it was largely the same mess, but obviously smaller and self-contained. It wasn't an optimal solution, and he'd written 500 lines of code for what should be a 20 liner. The dev manager used to be a programmer, but hadn't written code in a while, and didn't see a huge problem with it. When the rest of the devs were asked about it, they all said that they thought his code was messy/crazy, but the managers wanted him, and the guy really needed a job, so they said yes.

I'm not saying that this solution isn't a good one. If you can do it, I'd argue that it's the best way for a typical company to hire a developer. What I would add is the following:

1. Management should not publicly offer their opinion on a candidate before technical advice is given. If the managers weren't praising him, our devs might have had the confidence to say no.

2. Any technical test should be fully QA'd before a "Yes/No" is given. You're not just looking for someone that can write code. You're looking for someone with domain knowledge of your tool-set that can offer solutions to problems. If you're shown a bunch of code with little context. Funny enough, the code that he submitted was actually fixed by another dev on the following Monday.

3. Make sure they're not working on something that actually needs to be delivered in the next week. This guy was brought into the office on a weekday because he was unemployed, but it turns out most of the pressure for this was from a PM who said that the problem he needed to work on was actually required in that sprint.

doppel• 10 years ago

I think one flaw is that everything happens over the weekend meaning the developer has Friday to understand everything and then be ready for Monday. Personally, I don't know half the issues I might run into when I first start a new project, and if they expect me to just "make assumptions" the solution I end up with could be miles from what the wish for (both in direction and size). If it was Monday to Friday (or Monday to Monday), there's a chance for me to ask questions, get smarter and iterate on the solution - same as I would if I was actually working.

mkoryak4• 10 years ago

I a single non tech question that is a very good heuristic for hire/no hire.

What is your favorite 4 letter word.

I usually ask at the beginning of interview and expect an answer at the end.

Answers like fuck and shit === no hire.

The other ones tell you a lot about the person

erikb3• 10 years ago

"four" So, what does it tell you?

mkoryak4• 10 years ago

nothing :)

adtac• 10 years ago

I'm sorry, but how does this actually help you decide whether to hire or not hire a person?

For example, if I reply "hire", would that impact my application positively or negatively?

mkoryak4• 10 years ago

the question is kind of a joke. If you reply 'hire' it would tell me that you are ok with answering a joke question and not flipping out like some other replies did.

If you get all pissed off that I asked you such a question, perhaps I should not hire you since you will get pissed off at silly things that happen at work.

draw_down2• 10 years ago

Good grief. We are real idiots on this hiring stuff.

wayn32• 10 years ago

The only honest answer to this question is "i have no favourite four letter word".

but you cant say that because idiot hiring managers who ask idiot questions would insist that you do.

sorry, but if you ask shit questions, dont expect non-shit candidates.

zimpenfish• 10 years ago

Another honest answer would be "I thought I was applying for a development job, not a place in kindergarten. Good day, sir."

user59944617• 10 years ago

I felt like "What?!" and then I realized that's the expected answer :D

perseusprime11• 10 years ago

This is good for testing for skills but does not help with team fit and behavioral issues down the road. I hired a person a while ago who provided with a clean solution to the problem with good separation of concerns and unit testing. I hired this person but after few months this person starts yelling at other engineers, won't complete the tasks. Ultimately, I had to let this person go which was very unfortunate given the quality of code. The headline is don't miss out on testing for team fit and other cultural values of your team.

CSMastermind2• 10 years ago

There's no chance I would consider a job that required me to do this.

clifanatic4• 10 years ago

I wouldn't say no chance, but I definitely wouldn't go through this for some no-name "startup". It would have to be a high-profile position at a pretty exciting company.

lolsal• 10 years ago

Why not? I find your perspective interesting because I much prefer this style of interview.

CSMastermind2• 10 years ago

I'm busy. I both work a lot and have a life outside of work.

If a company were to give me a take home assignment I'd see it as them not valuing my time.

When you do a traditional interview the company has skin in the game. They have to "spend" the same amount of development time as I do in the interview. That person interviewing me could be working, if they're interviewing me instead, that's the company signaling that they think I'm worth the time. You don't have that in a take-home problem. It's asymmetrical. I'm investing time on this and they aren't. That's not how I want to start a business relationship.

Then there's the marketplace. I have a lot of options for employment. I'm confident that if I quit my job tomorrow I could have another one, just as good, by Thanksgiving. The job interview goes both ways. I'm interviewing you as well. I want to talk to your current developers. I want to get a sense of your office culture. And I don't want to complete a take-home assignment before I get to do that.

Finally the people I like to work with are like me. They're busy, driven, and good at what they do. I can't imagine they would work for a place that used take-home work for interviews. So even if I did go through with it and get the job; what type of people am I going to be working with?

elcct2• 10 years ago

This approach is much better than something like solving another puzzle in Hackerrank. It also working both ways, because by the quality of assignment you can see through what kind of company you are really applying to. I personally find most of the time such assignments to be a good thing and enjoyed doing them, but also had one case where company asked me to do self-contained module that solves one of the problems they had and on the follow up interview they asked me to refactor it so that it could be embedded into their product. Which I did and they liked it and then decided not to proceed. I could sense they didn't actually need a new developer, but needed someone to extend their product for free.

dominotw• 10 years ago

Sounds great. Let's add this on to our 50 step interview process that we copied from google/apple/MS - Tech Hiring managers everywhere.

andy_ppp• 10 years ago

I liked the article but I'd rather people just come in and do some work rather than fitting things in (2 days in the office). I suppose with American holidays this isn't possible to take a couple of days but in Europe we get 25 days. I like the idea of paying for people's time too.

Probation is also important; I've used it when I don't like companies and I think companies should be willing to say to people it's not going to work out.

Three months is enough to know; people can look amazing on month 1 and lose their way. It is also important to never underestimate a manager who actually thinks through what they want from their staff and asks them to do it :-)

jkingsbery2• 10 years ago

My weekends are just as busy as my weeks. I have a wife and two daughters that I do stuff with on weekends, and other family members I like to see. This process seems to presume that I somehow have "free-time" on weekends.

darkstar999• 10 years ago

If you don't have time to scout out new job opportunities, then you are comfortable enough to not need a new job. A 2 hour project like the article suggests is reasonable.

shoefly• 10 years ago

I use this process and it works great. We give potential hires a 1-4 hour project to do at home, they bring it into our office to defend. We don't pay them to create the project. We don't look at education. Some of our top coders are self-taught.

mi100hael• 10 years ago

    For example, I may say please use functional reactive
    principals in JS to solve this problem, but the decision
    of using Kefir, Bacon, or RX is left up to them.

These days I'm utterly unable to tell whether someone talking about JS is being serious or if it's parody.

duderific2• 10 years ago

No, this is real...however, using reactive principals in JS is a bit esoteric and definitely not something that needs to be covered in an interview (IMHO). At my company when it comes to JS interviews we are more interested in proper separation of concerns, understanding scope, using proper data structures...you know, the basics.

laxk• 10 years ago

It would be nice to collect here the list of companies who is doing interviews in that way.

mcv4• 10 years ago

In 2004 I got a job at a company that used basically this method, except I didn't get paid for it. I think I got the assignment on Monday, and had to present the results on Friday, so it wasn't over the weekend. Though I could be wrong on that. I could ask the CTO questions (it was a small company), and on Friday I presented it for the other devs, who asked questions, and later voted by email on whether I should be hired. I was, and it struck me as the best hiring process I'd ever seen.

silverbax884• 10 years ago

I think the current interview process fails but...this idea of seeing if people can code a small project isn't much better - it might be worse. It's a lot different giving someone a tiny project to knock out in a weekend and working with them for weeks during crunch time, or getting estimates, or just if they are a good workmate. You're basing an entire hiring decision on whether the person can write some queries and a page?

I mean, I like the idea of proving what I can do, but software development - as in most jobs - are far more than technical ability.

wallunit• 10 years ago

I like that approach, and we tried doing something like that in the past, but here is why it didn't work for us, unfortunately:

We work remotely, so is our hiring process. While it would be possible to give a programming task to a candidate, we won't have any control over whether they did it themselves, or how much time they actually spend on it. Well, the former is less of a problem, once you discuss the solution you see pretty quickly whether they did it themselves and understand how their solution works. However, candidates that want the job will try hard to do as best as possible, so some candidates that you give a 2h task to, tend to spend much more time on it. Others might not have as much time to spend on the task, since they usually still have a full-time job. So the results are not comparable.

Besides that, I find it quite difficult to find a task that can be done in 2h, but gives a sufficient insight of their skills, however you cannot expect somebody to take on a more elaborate task, whether paid or not, because most candidates are in a full-time job when they apply.

Also, this might be different in the US, but here in Germany, if the candidate isn't a freelancer/contractor at the moment, but permanently hired by an other company (this is normal), there is no way to pay them officially.

Tistel• 10 years ago

I did CS and have been a professional programmer for more than 15 years on pretty much every platform under the sun. I am very introverted. I still panic during the whiteboard phase of the interview about 75% of the time. Like go blank on simple stuff. Some problems need collaboration but I find I can only really think clearly when alone. I have been debating taking one of those public speaking classes because I probably need to transition to more of management role (love programming, but, my hair is starting to turn grey and I hear we kick everyone out of the club when the grey happens). The public nervousness has not been very helpful in my life. I wonder what it's for? Why have that feature? Maybe it's a bug.

duderific2• 10 years ago

(love programming, but, my hair is starting to turn grey and I hear we kick everyone out of the club when the grey happens).

Please, no. There are plenty of companies that respect experience and the wisdom that comes with age. It may be challenging to find them, but don't go management just because "you're getting too old." If you love programming, keep doing it, and find a place to work that suits you. Signed, a gray-haired 47 year old programmer.

titanomachy• 10 years ago

If you are very introverted, then some of the companies that do collaborative whiteboard interviews would probably be a poor fit for you. The reason they interview that way is because they are collaborative work environments, or at least they like to see themselves that way. Extreme introverts, even highly skilled ones, might not fit in.

However, if you're experienced and skilled there will always be work available. You'll probably do OK. Overcoming your social anxiety would probably open some doors, though, and not just in management positions.

geebee2• 10 years ago

I'm coming late to this. I've been through two take-home projects. Both of them resulted in a no-hire, and each has refined what I'm willing to do and what I'm not willing to do.

The first one was a pretty egregious experience, honestly. I never spoke with anyone other than a recruiter. The first was a 2 hour Java test. The next was a take-home that I was supposed to spend 5-7 hours on. I did it and sent it in. I knew it wasn't great, but I put a hard stop at 7 hours. It took almost a month for me to hear back, from the recruiter, "we've decided not to continue with your application at this time..."

For a while, I said never again. However, after refusing (and passing on what might have been good opportunities), I did another. Circumstances were very different. I had two extensive interviews with developers prior to the take-home, and I was interested in the job. Furthermore, they did reply with feedback, though it was through a recruiter.

The feedback was mixed, some things were good, others were bad. Unfortunately, though, it was a hard stop. I didn't get and presumably will never get a chance to defend my choices. Although I understand the company probably doesn't want to open up a debate, they've moved on, it rankles that someone got to say those things about my code and I never got a chance to reply.

So, am I at the point where I'd never do it again? Almost. I think I would insist on an opportunity to defend my code base, at least one back and forth, before calling it quits.

It sounds like this company (mattermark) does this, which is a positive thing. You do get a chance to argue (though maybe they do cut some people off when the project is terrible, and who knows, maybe the people who rejected mine (no connection to mattermark) were just taking it a little easy on me, but were genuinely not interested after seeing it).

I don't actually care about getting paid for the project time, if it's a good opportunity. I have to spend as much time re-studying my data structures and algorithms book for whiteboard interviews anyway, and I actually my learn something from the take-home.

I'll probably talk to a buddy I consider to be a really good programmer who works in this area, and see what his take is on my project, asking him to be pretty brutally honest about his assessment.

ebrenes3• 10 years ago

I'd probably follow up on talking with your buddy. It's often invaluable to get an outsider's feedback on how others look at our code.

From what you typed, it seems like there wasn't a lot of communication prior to doing the coding exercise. I'm also not aware if you had channels to ask questions or discuss technical design options. If available, those are often "part of the test", to see if you make use of such resources.

When I'm reviewing code in such scenarios I'm often looking for other indicators aside from the code's correctness. Things like: naming conventions, structure of the code, patterns used, how over-engineered it is. Just as important is seeing if the candidate picked up on less explicit instructions or hints in the coding exercise. Finally, I look for code that follows the conventional patterns and standards for that programming language; I expect a candidate to discuss with me the deviation from those standards prior to handing in the code.

Anyways, I wouldn't completely give up. This might be an opportunity to grow and learn, but you'll never know unless you follow through on asking your buddy. Likewise, I've seen people here on HN volunteer to review code and provide feedback.

geebee2• 10 years ago

Thanks for your reply.

Your instincts are good here - one cited problem was indeed the departure from standard conventions. I had my reasons, not saying they were good ones, but I didn't get a chance to explain them either.

I'd like to, but it appears I won't get that chance. I gotta say, that is the part of this all I don't really like. It was vastly better than my only previous experience with a take-home project, where I didn't hear back for a month and got a one-line brush off from a recruiter. In this case, I am certain that they looked over the code reasonably carefully - they did provide a short bit of feedback, though it was delivered by a recruiter, and there was no back and forth here. I'm ok with not getting a job, but I would have liked an actual follow-up where we reviewed the code after that level of effort and time invested.

Lastly, the job was fairly senior, so it may have been a high bar, and that wasn't the only problem with the code. Then again, how much are you really supposed to work on a sample project?

Overall? I'd be hesitant to do this again. I might ask in advance what the policy is on a review. Maybe a 30 minute conversation is a requirement for me to do this. Still mulling it over.

MorePowerToYou2• 10 years ago

This method is great. It's worked for me when seeking new jobs.

Regarding the "algorithms trivia via whiteboard" style interviews: I think this type of interview is, unfortunately, the best large companies can do given their constraints. These constraints include hiring lots of new graduates, interviewing pipelines that include technically illiterate recruiters and engineers who lack the executive function to accurately assess candidates' viability. Lastly, the people involved in hiring at large companies are clueless about what the new hire will actually work on. It seems so obvious that tailoring the interviewing process to the actual fucking work that the person will be doing is a good approach. Once again, the contraints in a large company prevent this from happening in general.

Joof• 10 years ago

I never really know how much time to spend on take-home problems. I could spend a lot of time on it and it should be better -- possibly demonstrate more of my skills -- but often that ends in it being over-engineered and ultimately bad or otherwise unrepresentative.

Getting paid could provide the ideal constraints for the problem at hand.

amai• 10 years ago

Why give them an artificial task? Let your candidates bring their own pet projects or let them show their portfolio of projects and discuss about it. Designers, Artists, Photographers always have a portfolio of their work. Why shouldn't programmers have one, too. And no programmer is shy when talking about her/his own projects. See also https://blog.codinghorror.com/a-programmers-portfolio/

voycey• 10 years ago

I am 100000% behind this - I point blank refuse to do code tests because they are asinine and I probably have the exact same one in a private GH repository from one of the million other unimaginative tech hirers.

You want me to prove my worth? Fine, you can pay me for it and get something useful out of it at the end, as a senior developer I really shouldn't have to "Write a palindrome detector" to prove that I can code, I have a degree, we did that shit repeatedly!

Nice to see other people are coming around to this way of thinking

tulika_47• 10 years ago

The concept of paid interview is pretty innovative and interesting to judge the technical skills of a candidate as compared to the traditional method of interview . But before implementing such method the company infrastructure, availability of resources, mentality of the candidates in the market etc should also be considered as a factor to understand whether the company can afford and establish such method or not.

To implement any new method , system infrastructure is very important. According to me paid interview is not that much beneficial for a mid-level company as it will be for some MNC or IT-giants because the number of openings matters. The article suggests to bring a group together on Monday to discuss the solution to a particular problem given. In that case the number of opening & scheduling has to be huge in order to arrange such group for the G.D which is not possible in a mid level company where you have only 1-2 openings at a time and the number of candidates who apply to take the interview is also very limited i.e within 5-8 in a week out of which only 2-3 appear for the interview in a week.

According to the Indian mentality Paying against just appearing for the interview might not be that fruitful in absorbing the right talents in the company . If such concept of interview is out in the market people will take this as a scope of earning rather than joining. We might not get genuine candidates who actually wants to join the company. Candidates will just appear only to earn that money and will leave. Here what we can do is we can only provide the money to the candidates who gets selected and then joins.

Here we are allowing them to write the problem down and take them home. Here comes the question that what if the candidate is taking someone else's help to solve the problem and how much honesty he is maintaining to solve the problem. If the candidate is appearing for the interview in the office premise such factors won't be an issue as he will be monitored constantly.

If we are following such process of interview then we constantly need a team who needs to change the question for every session of interview as the question is going out therefore we will always be in requirement of new set of problems/question which is again tough for a mid-level company to maintain due to resource crisis and that to only for 2-3 openings at a time.

We are providing the candidates constant supervision and guidance to address the question, also we are providing them no time limitations and if they clear the test a group is there to judge his perception and the result instantly on a one to one basis which is less time taking and prompt as compared to the process suggested in the article.

kevando• 10 years ago

Wow. I did this! And you're right it worked great! However, the candidate was a student with no experience so I did't pay him. I said here is the git repo to http://www.wookietranslator.com now go make some changes. 2 weeks later he figured out git, python, js and I was very impressed! Kudos.

danielrm26• 10 years ago

The reason this works is because it uses Evolution Thinking vs. Design Thinking to find good people, which we need more of in tech.

https://danielmiessler.com/blog/why-you-should-hire-using-ev...

foobiekr• 10 years ago

$200 is cheap compared to the benefit of basically eliminating the ability of candidates to field multiple offers.

Which I mean sure has nothing to do with why this idea is being marketed so hard.

stevefeinstein• 10 years ago

So you only hire experienced developers? Where's your commitment to paying it forward? Where will we get the next generation of experienced coders if you don't give a few less than perfect candidates a chance? I guess if you get yours then screw everyone else, right? Or am I being too harsh and bitter?

When did we stop smart people who may not have fit precisely into the perfect little niche we've carved for them but showed a good ethic, and an ability to get the job done and learn new things as technologies evolve? We're really screwing up this industry in favor of getting it done fast, today.

umanwizard4• 10 years ago

Companies aren't charities. Their goal is to make money, not help anyone out.

darrenshrwd• 10 years ago

Thanks for this, makes total sense and I'll remember it. Can I ask what you use for your blog, I like the look and feel and the share bar at the bottom and would like to re-use it if I can.

daxfohl4• 10 years ago

Neat, but it still measures only technical aptitude. Anyone can look impressive playing with a fun project over a weekend. But how do they do at the grind when the important tasks may be much more mundane? In my experience this is what separates the good hires from the bad.

A Method I’ve Used to Eliminate Bad Tech Hires

Related Stories