In Defence of the Technical Interview

https://blog.plan99.net/in-defence-of-the-technical-interview-966f54a58927

by telotortium • 7 years ago

191 294 7 years ago

blog.plan99.net

Also describes the interview process this company uses:

Firstly, how do we hire at R3? We’re recruiting for technical roles around the world as part of building an open source Bitcoin-inspired decentralised database. The interview process for developers consists of, firstly, a piece of code that you’re asked to send back a quick code review of (this is meant to take about 5 minutes), then an 30–60 minute long video chat+screenshare in which you may join from home and some coding is done in an editor and language of your choice*, then finally an invitation on site to meet the team and talk to senior management. The code test sometimes also includes design and ‘talky’ questions, depending on the nature of the candidate and precise job role: it’s not just coding. We’ve found this process to be pretty accurate whilst still being quite lightweight (compared to some hiring processes at least!).

Waterluvian• 7 years ago

All I would need added to that style of interview is something along the lines of:

"And if you have to Google things during the video call, go right ahead. But try to talk through your thought process."

So that I don't go into the interview feeling super nervous about the moment where I get to say, "I'm googling the bidict library because I forget the API. But I want to use a bidict because..."

mike_hearn5• 7 years ago

(I wrote the article)

We allow candidates to use Google if they need to during the interview. However, the interviewer is free to draw conclusions from that. If a candidate is routinely searching for things that someone using the language would use every day, that's going to count against them. If they need to look up something that is only rarely used, nobody will care.

There can be occasional disagreement about what's reasonable to look up. A surprising number of developers don't know how to read from files, and I've encountered a few that didn't know files are random access. Some people think that's a problem, others think it's expected. I tend to see ability to use files as a proxy for general experience, which for R3's projects matters a lot (we aren't writing generic database-backed web apps, there's a significant R&D component to it).

tptacek9• 7 years ago

This post describes an unusually good technical interview program, in which most of the tech-out is apparently done at home, the interview is prefaced with a short but useful work-sample code review test, standardized questions are used, and candidates are allowed to use their preferred tools and languages.

That's commendable, but I think the argument is still faulty.

As I understand it, the premise of the argument for interviews and against work-sample testing is that candidates won't perform work-sample challenges. I've interviewed almost as many candidates as this person has (and screened out far more based on challenges) and that simply hasn't been my experience. More importantly, the argument is incomplete. Stipulate for a moment that the "best" developers won't take work-sample tests (I'll come back to that)... and?

The point of work-sample tests is twofold: to stop being beholden to subjective impressions of how well candidates can perform the work you need them to do, and to provide a framework for tech qualification that's straightforward to iterate on. If your work-sample tests are sound and you're getting enough candidates to saturate your headcount requirements after setting a bar you're comfortable with, why obsess over who exactly you're hiring? The industry is full of developers with glittering resumes and lists of past achievements who are dead weight on teams†. Why assume the developers with the best marketing are the ones who will perform best on your team? Why not instead just spend the time to figure out what exactly a good performer is, and then directly test (and iterate on testing) for that?

Regardless, the belief that strong candidates won't do work-sample tests is pervasive. And for good reason: work-sample testing at most companies seems cargo-culted: "take-home tests" (that often follow tech-out interviews) that can only rule candidates out and never rule them in. I agree: that's bad. But the solution is straightforward:

* (1) Allot a budget to the entire technical qualification process, including all interviews and challenges, (2) try to make as much of the process objectively scored based on a rubric your team has committed to writing, and (3) ideally move as much of the process "offline" (so the candidate can perform it at home and without you watching) as possible.

* Make the work-sample challenges as determinative as possible. Candidates who do well on our work-sample challenges are presumed technically qualified; 1:1 interviews that follow it are largely pro-forma. At Matasano / NCC Group (while I was there) the bar for a technical interviewer to overrule the output of the work sample challenges was quite high.

Regardless of whether you use work-sample challenges or scripted, structured interviews, a bit of low-hanging fruit that nobody seems to pluck is: relentlessly keep candidates informed about what to expect in your process. Since about 2010 we've been buying books (expensive books!) for candidates and providing study guides to go with them; we also provide fairly detailed descriptions of what our challenges will be. We're even moving to a system now where we offer candidates a practice version of some of our challenges (if I could fake up the entire AWS API, I'd do practice challenges for all our challenges).

But most employers seem determined to make their tech-out process as much of a gotcha game as possible, and tell candidates almost nothing about the actual hard problems they'll be expected to deal with.

This post uses a framing that says whatever tech companies do to qualify candidates must be rational; after all, if they're wrong, they're raising their own costs. I respectfully disagree: engineering teams do not, as a general rule, optimize their processes for efficiency or shareholder value. Many teams literally use interviews as a hazing process (one they often exempt "in-network" hires from). Still more delegate interviewing out to the entire engineering team and exhibit little or no interest in establishing rigor or rolling back bias. But engineers are as biased as the rest of humans, and most of hiring today is not in fact rational.

Also: sorting algorithms make dumb interview questions.

† This is in part due to a dynamics in our industry that make it easy to "gradually fail upwards": most companies can't fire fast enough after making a bad hiring decision to prevent the bad hire from accruing another positive line-item on their resume, so smart people can bounce from team to team for years without ever making a serious contribution and end out ahead.

enraged_camel• 7 years ago

>As I understand it, the premise of the argument for interviews and against work-sample testing is that candidates won't perform work-sample challenges.

My understanding, as I've heard from others, is that it's not that candidates won't perform work-sample challenges, but that they can't, due to various non-work obligations that result in lack of free time.

This leads to a situation whereby interview processes that feature work-sample tests discriminate against people with families and kids/dependents.

tptacek9• 7 years ago

Allocate N hours to your whole technical qualification process. A process that spends more of those hours on at-home work-sample problems will be more inclusive of people with time commitments than one that spends less.

kgraves• 7 years ago

Well that is kind of too bad in my books. Less candidates for me to weed out.

santoriv• 7 years ago

Also, take home assignments are often time intensive for the candidates but not for the company giving out the assignment. I’ve done remote coding assignments but then had my candidacy dismissed for such silly reasons as “you live in the wrong time zone” AFTER I had completed the assignment. I would have felt a lot better about that sort of nonsense (it’s unavoidable if you have done more than a few interviews) if I knew that the company had invested as many hours in my interview process as I had.

trustfundbaby2• 7 years ago

It's also that lots of times, the hiring company hasn't actually put prerequisite thought into how to limit the scope of the exercises they send out.

They simply tell you "this should take you 4-6 hours" and then you're left trying to figure out the tradeoffs to make. Usually that 4-6 hour exercise can EASILY take double the time, between you trying to get the best possible solution, handle all the edge cases, going back and forth with the interviewer on questions about the exercise, overthinking what the person on the reviewing end of the exercise is going to see as some irredeemable element of taste that they do not share with you etc

Then after putting in all that work, you possibly get to go through an actual interview process, get a form rejection email or even better never hear back from the company at all ... after going through that 2-3 times you learn that the upside for you is low, and you could probably make more progress with other firms in that same time.

What I personally have arrived at is that if a company isn't paying me for my time and the exercise will take over 2 hours, I simply won't participate. We'll see how it works out.

TLDR; Companies need to properly limit the scope of their take home assignments.

pmiller26• 7 years ago

Well, I “can” perform these so called “work sample tests” that a lot of companies want to give out (sometimes even before getting to speak with anyone at the company). Here’s my issue with them:

* Either they’re strictly timed, introducing an artificial sort of pressure that’s rarely present on the job, or one ends up competing against people who spend 3 or 4 times the suggested amount of time. I don’t know about you, but I can do a way better job on most of these sorts of tasks if I spend a large multiple of the expected time on it, too.

* The tasks are often poorly specified, sometimes with the justification that it tests the ability to clarify requirements. I don’t think that argument holds water, because all “clarifying requirements” does when one doesn’t have ready access to folks who can answer those kinds of questions is introduce insane amounts of latency and frustration into the process.

* Even when the tasks are reasonably well specified, the grading rubric is often kept from the candidate. At work, if I don’t know if I or my team is building the right thing or doing it in the right way, I can go ask people (PM, tech lead, EM, another team member, etc.)

* Finally, most of these “take home assignments” I’ve seen that are supposed to be in the 2-4 hour range to finish don’t get me as a candidate out of 2-4 hours of giving impromptu algorithms lectures at the whiteboard. If the prize for putting in 2-6x as much time as I would to get to an onsite interview with another company is just to come onsite with your company and spend all day doing the exact same thing, then, no thank you.

I realize that few or none of these things apply to the process at Matasano, but there is another one that does: I can already get a job that pays reasonably well, doing things I already mostly know how to do, without spending hours studying a field I know little about.

That is arguably a feature and not a bug, so, I don’t mention it as a criticism, but rather as a reason why I probably wouldn’t do Matasano’s work sample test. For the type of people they’re looking for, that’s probably fine. It’s just that I’m not one of those people.

bsder• 7 years ago

Make the work-sample challenges as determinative as possible. Candidates who do well on our work-sample challenges are presumed technically qualified; 1:1 interviews that follow it are largely pro-forma. At Matasano / NCC Group (while I was there) the bar for a technical interviewer to overrule the output of the work sample challenges was quite high.

I would argue that this worked for your hiring field because you had a secondary bar of "technical competence" but a primary bar of "motivated persistence". And your tests definitely set a bar for persistence.

The real problem most companies are currently facing is simply that no one wants to take the time to train someone. Most companies would be better served by training someone internally and then promoting them into the higher slot and then backfilling the more junior position where the stakes are lower.

The HR problem is that you have to fight to adjust someone's salary properly after that promotion. I think I have won an internal salary adjustment fight only once in all the years I have been responsible for hiring people. Hiring an outsider at $100K is easier than promoting someone and getting them a raise from $80K to $100K--and it's completely stupid.

tptacek9• 7 years ago

I think there's a lot of truth to this, and to the argument that work-sample testing really only works as well as I like to say it does when you have an engineering culture that can metabolize new hires without much previous experience in your particular problem domain. But we started doing work-samples for the opposite reason: because we kept getting bamboozled by good interviewers --- a problem I know to be rife across the whole industry. I still think you're better off keeping most of the weight of your qualification process on practical challenges rather than interviews.

We have not generally had a problem recruiting people with experience in our field, but there's obviously some selection bias in that observation.

klenwell• 7 years ago

Regardless of whether you use work-sample challenges or scripted, structured interviews, a bit of low-hanging fruit that nobody seems to pluck is: relentlessly keep candidates informed about what to expect in your process.

Amen. And I'd add: let them know when they can expect to hear back from you at each step. It can be as simple as:

"Here's my email address. You should hear back from me by the middle of next week. If you don't, please follow up with me."

Then, of course, follow up, even if it's just to say, "Sorry, we enjoyed meeting you but we were lucky to interview a candidate who was a better match for us. Thank you and good luck!"

AnimalMuppet3• 7 years ago

Here's the thing for me. I try to be in the situation of trying to find the next job while I still have the old job. That means I'm not desperate. That also means I'm still working a full-time job while I'm in the job-hunting process. For an interview, I'm probably going to have to take a vacation day. For an out-of-town interview, I might have to take two days. That's bad enough, especially since I might go through several different companies in the process of finding a job.

Now you want me to do a 2-4 hour work-sample test? And maybe 5 other companies do, too? No. Just no. I have a job, I have a life outside my job, and you're not going to abuse it with making me jump through this hoop. If I lose your job, so be it. I don't need to work for people who will abuse me in the hiring process, no matter how valid they think their reasons are.

Your intent isn't to abuse me. Your intent is to weed out candidates who were making it past your interviews, but still couldn't code. But the effect on me is still abusive, no matter what your intent is.

tptacek9• 7 years ago

The whole idea is not to make you have to take the vacation day to interview --- or at the very least, not to have to take the day unless you're assured you're technically qualified, and you're not going to get teched out on site.

If the choice is between taking a vacation day to come on site and code on a company laptop with someone staring over your shoulder, or doing almost nothing other than a take-home problem that you can do wherever and whenever you want, tell me how we'd be the company making it impossible for you to fit us into your schedule?

mike_hearn5• 7 years ago

Thanks. I'm glad to hear your description of our process, it's nice to get some positive feedback.

... candidates won't perform work-sample challenges. I've interviewed almost as many candidates as this person has (and screened out far more based on challenges) and that simply hasn't been my experience

I'm curious what you mean exactly by work-sample challenge here - I guess you mean homework assignments, as a coding test is in effect a work-sample challenge of a sort.

The belief that senior candidates won't do them is based on observation of behaviour when asking them to take 'robot' automated interviews. The people with the most experience and best known companies on their CV (the sort of places that do hire carefully) were the most likely to drop out of the process at that point. We didn't do a big study because, well, we didn't want to lose more senior candidates, but there were enough samples to see a pattern. We haven't seen that problem with video-call interviews.

I guess you're asking more about either showing open source code, or homework assignments.

High quality open source code can be a good factor in favour of a candidate and it is checked if someone puts some on their CV. Unfortunately in the past I got burned by hiring a candidate based purely on code on their github repository and a Q&A style interview, without doing a live coding test, and the person turned out to be largely incapable of writing code to the necessary standard. We never figured out why his code on github seemed so much better. There's just so much more information you get from watching someone work, like whether it takes 10x longer than it should to produce that code, or how much help they need.

The issue with homework assignments can be summed up as - I wouldn't be willing to do them if I were looking for a job right now, unless I was completely unemployed and desperate. There are very limited hours in a day and most programmers looking for a new job already have one. Also, you can't prove it was actually them that wrote the code, and you can't tell how long it took them to do so.

I make no claim our process is the best possible, or even very good. It was just descriptive and seems to work well enough - other than the one bad hire where we didn't follow the standard process, we have a pretty uniformly competent team.

tptacek9• 7 years ago

A work-sample challenge is a practical problem that mimics the actual work done on a job (the term comes from W. Edwards Deming back in the 1950s). Doing a Github-PR-style code review is a perfect example of a work-sample challenge; you can come up with any number of work-sample challenges that involve actually coding; a friend used them to recruit for his sales team by having them write outbound mails or handle simulated inbound requests.

Our work sample challenges at Latacora involve as assessment of a Django web application, an assessment of an AWS deployment environment hosting that application, and the construction of a simple scanning tool to spot the vulnerabilities discovered. At my previous firm, the challenges were similar – there was no programming challenge, but there was an assessment of a custom binary trading protocol.

I'd balk at robot automated interviews too. I'd be concerned that my responses wouldn't fit whatever weird template they were using; also, I'd be almost certain that any robo-tronic assessment thingy they were doing almost certainly wouldn't be determinative of the outcome of the qualification process, but rather just an arbitrary hurdle I'd have to jump over before the "real" interview happened. Obviously, if my options include equally attractive companies that don't present that same extra hurdle, I'm not going to bother applying!

I also think there is certainly a class of senior developer that won't do practical at-home challenges at all, no matter how carefully designed, even if they're entirely determinative of outcome and even if they're calibrated carefully to make the interview process less logistically arduous (for instance, by reducing the time commitment for actual interviews from 4-6 hours down to a single hour, and assuring before the interview happens that you're going in already tech-qualified). Certainly, work-sample qualification isn't the norm in our industry, and senior developers also tend to move from job to job through networking and very warm introductions.

What I'll contend, though, is that there's a pretty significant adverse selection problem in that pool of senior developers. I don't doubt that the bulk of them have earned their experience and reputation, but there're also a lot of developers swimming in that pool that have failed upward into it, and move from job to job largely through interviewing skill. What I've learned (or think I have) is that there's virtually no correlation between someone's ability to talk about programming and their ability to effectively deliver code on a commercial project.

What makes the work-sample process so attractive to me is that the opposite phenomenon also seems to exist: in the pool of "not labeled senior" developers, there seems to be a usefully large number of exceptionally talented developers who have, owing to circumstance, simply not acquired the resume labels to navigate themselves into good jobs. Hiring from that pool is very risky if you rely on interviews! But what we learned over the last 10 years is that you can build practical challenges and get to a level of confidence where you can rely on them over interviews and resumes, and when you do, you can hire a lot of talented people that other firms can't, because their interview processes won't let them.

momokoko• 7 years ago

Technical interviews are disguised IQ tests. IQ tests are actually not allowed in the US based on a Supreme Court decision[1] that said they were discriminatory.

We know that cognitive ability has one of the strongest correlations with job performance.

So "technical interviews" that are heavy on algorithm puzzles are just a loophole to get around the IQ test issue. They aren't actually testing domain knowledge.

[1] https://en.m.wikipedia.org/wiki/Griggs_v._Duke_Power_Co.

tjpnz• 7 years ago

That's how I view them as well. It's why a lot of people can get through them by doing lots of Leetcode in the same way that one might do lots of logic problems in order to score higher on an IQ test.

lwansbrough3• 7 years ago

At least there’s considerable variance in the problems on an IQ test so your intelligence is diversified across many problem types. The type of shit I’ve come up against in past interviews has often been essentially an all-or-nothing single problem. Which is why I now refuse technical interviews.

danpalmer2• 7 years ago

Leetcode practice gets you through a particular kind of coding test. I would say that only covers ~half what a good coding test interview should cover. It makes you good at the "how", and solving algorithmic or textbook architecture questions.

The questions I prefer to ask are about writing a basic but readable and maintainable piece of code, maybe 50-100 lines long, that requires the candidate to talk about why they are making certain trade-offs, questions that look at when they would involve other team members in decisions (when do you ask a Product Manager or Designer for help?), questions that try to understand whether they can solve problems rather than the one problem in one language given to them in the interview. I typically do those interviews in Python but some of the best candidates have never written any Python – but their communication, intuition, and problem solving abilities were great.

e10a4• 7 years ago

We know that cognitive ability has one of the strongest correlations with job performance

There is a good argument that the technical interview success is much more strongly correlated to the number of hours spent practicing Leetcode-type questions than to actual cognitive ability.

dvdbloc2• 7 years ago

And shear luck that you have seen a problem or similar before

closeparen2• 7 years ago

The process can definitely reject good candidates who are out of practice, but it’d be pretty hard to do the practice and give good performances without some underlying skill.

typon2• 7 years ago

I've noticed that exactly. IQ presumably is something inherent.. But personal experience shows that technical interviews have nothing to do with IQ. They can be easily solved by creating a dynamic programming problem pattern matcher in your brain, which is what I created after doing 300 leetcode problems. I did not become smarter in general or at coding at all, I just got good at mapping problems to binary trees or dynamic programming or string search.

dahfizz3• 7 years ago

The technical interview is not the only stage in the interview process. It might be better thought of as a way to filter out applicants who have poor innate ability and/or poor learning skills.

I would argue that being able to study leetcode and then apply that new knowledge you've learned is an important skill in and of itself in this industry. Isn't that what all of highschool and most of college is? Signalling that you a quick learner and a hard worker is important.

spunker5402• 7 years ago

Is there anything wrong with preferring people who have spent a lot of time practicing leetcode?

Most jobs prefer people with certain kinds of experience vs raw cognitive ability. Raw cognitive ability doesn’t mean much outside of entry level jobs.

I don’t believe the interview process would be any more efficient or effective for software engineering roles if they started preferring pure “cognitive ability” to leetcode effort.

lmm6• 7 years ago

There is a good argument that the technical interview success is much more strongly correlated to the number of hours spent practicing Leetcode-type questions than to actual cognitive ability.

Where is that "good argument" substantiated?

People say exactly the same thing about IQ tests, probably because it's something they'd like to be true. But it's not backed up by the evidence.

bubblewrap• 7 years ago

That would be a spectacular finding, implying everybody could become a good coder with a little bit of practice.

I'm not saying it isn't so, but I think the word is still out in that.

In any case, the solution seems simple: just practice some leetcode before the interview.

adrusi• 7 years ago

This is a common misconception about the Griggs ruling. See https://slatestarcodex.com/2017/07/26/dont-blame-griggs/

The Griggs decision explicitly places the same restrictions on college degrees that it does on IQ tests. Obviously nobody cares about this, and it turns out the Griggs restrictions aren’t that strict and most people manage to ask about college degree just fine. This changes the argument to be about whether there’s complicated case law around Griggs which makes employers figure that college degrees will probably be accepted in practice, but IQ tests probably won’t be.

There's not actually a prohibition on IQ tests in hiring, and in fact, during my recent job search, one company I interviewed with had me take an IQ test (it wasn't explicitly labeled as one, but it had exactly the sorts of questions you would find on one). The same company also had me go through a more typical algorithm-heavy technical interview later in the process.

threatofrain• 7 years ago

There's historical basis and case law to suggest that IQ exams have been used as illegal barriers on race, and it's not too far of an argument that your test could also be discriminating against a protected class. And it's potentially horrible PR.

What is required by Congress is the removal of artificial, arbitrary, and unnecessary barriers to employment when the barriers operate invidiously to discriminate on the basis of racial or other impermissible classification...
On the record before us, neither the high school completion requirement nor the general intelligence test is shown to bear a demonstrable relationship to successful performance of the jobs for which it was used. Both were adopted, as the Court of Appeals noted, without meaningful study of their relationship to job performance ability. Rather, a vice-president of the Company testified, the requirements were instituted on the Company's judgment that they generally would improve the overall quality of the workforce.
The evidence, however, shows that employees who have not completed high school or taken the tests have continued to perform satisfactorily, and make progress in departments for which the high school and test criteria are now used.[8] The promotion record of present employees who would not be able to meet the new criteria thus suggests the possibility that the requirements may not be needed even for the limited purpose of preserving the avowed policy of advancement within the Company.
In the context of this case, it is unnecessary to reach the question whether testing requirements that take into account capability for the next succeeding position or related future promotion might be utilized upon a showing that such long-range requirements fulfill a genuine business need. In the present case, the Company has made no such showing.

sien• 7 years ago

Technical interviews are done outside the US where they are legal.

Some places outside the US will do both an IQ test and a programming test.

tptacek9• 7 years ago

Maybe they are at Google, but hiring managers are not generally savvy enough to have the ulterior motive you're attributing to them; most of them are just doing what they were taught to do at previous jobs.

okaram• 7 years ago

IQ Tests ARE allowed in the US, as clearly stated in the page you linked ... "Broad aptitude tests used in hiring practices that disparately impact ethnic minorities must be reasonably related to the job."

Technical interviews are NOT a loophole about IQ tests ... they are tests that are related to the job.

blatantlies7• 7 years ago

they are tests that are related to the job

This is questionable. There is data suggesting job performance is not correlated to interview performance. Also, none of the questions in my last interviews have anything related my actual job, not even the language nor the problems I solved (since I could pick any language to write the code). The link between them are very arbitrary imo.

hn_throwaway_99• 7 years ago

I think your comment is a pretty gross mischaracterization of the relevant Supreme Court ruling. It's definitely not the case that "IQ tests are actually not allowed in the US". From the wiki page you link:

The Supreme Court ruled that under Title VII of the Civil Rights Act of 1964, if such tests disparately impact ethnic minority groups, businesses must demonstrate that such tests are "reasonably related" to the job for which the test is required.

I think it's pretty trivial for any company to argue that technical interviews are relevant to a developer job.

sgift4• 7 years ago

I think it's pretty trivial for any company to argue that technical interviews are relevant to a developer job.

Is it? Can you show, right now, that your companies technical interviews and job performance correlate positively? Cause from the things I've seen that's not checked commonly in companies and even if it is you don't hear "the technical interview is bullshit, let's do something else" as a result.

dragonwriter• 7 years ago

Note that the material you quote from Wikipedia is in error; the simple-to-meet “reasonably related” standard only applies for disparate impact cases where the discriminatory effect is based on age over 40, which is not under Title VII; the Title VII test is a much higher bar and once the disparate effect is shown requires showing “business necessity”, and even then disparate impact discrimination can be found if the employer declined to adopt available alternatives with less discriminatory effect.

gnusty_gnurc5• 7 years ago

Despite all that I must finish by observing that regardless of our best efforts in the industry, hiring is still largely random. Well designed interview processes make it slightly better than random, which is why we do them.

This article is a defense of an asymmetric process that stresses out software developers and forces them to jump through hoops. Slightly better than random selection is a garbage pay-off for the nonsense I've experienced in technical interviews.

rrhyne3• 7 years ago

Dealing with stress well is a valuable skill. Many of those that do not manage emotions well become toxic. I personally would let a few great engineers who cannot find balance through tough deadlines, un-paved roadmaps, and tricky social situations find jobs elsewhere than hire people who cannot maintain their emotions.

learc8311• 7 years ago

The kind of stress you deal with in technical interviews is nothing at all like the kind of stress you face in a functional workplace.

If your normal work environment is anything like a whiteboard interview, I want nothing to do with your company.

gnusty_gnurc5• 7 years ago

I agree that handling stress and managing emotions is a super valuable skill. But I'll hard-pass any interviews that try to stress-test my ability to maintain my emotions.

seer• 7 years ago

In which case framing the test as “technical interview” would be a bit dishonest. You should name it “stress environment coping test”.

But anyway you are right those moments do happen, but those times it’s usually “all hands on deck” kind of event where people are actively trying to help each other. Those are actually “fun” in a sense since you actually feel needed and encouraged rather than judged and scrutinized.

And each time something like this happens, it usually means a failure of the company process as a whole. Can expect it from startups but definitely not from big companies. So if you honestly want to test that out for a candidate, it puts a huge red flag for him that said company is mismanaged.

In effect you’re filtering for people that don’t know this - e.g. junior devs, or people who don’t care / tolerate it, so they won’t attempt to fix the mismanagement that allowed it to happen, the last thing you would want probably.

Thinking about it you could even stage such a contrived stress test, and actually hire people that _do_ flip out as those would probably be the ones that care enough to fix longstanding issues.

esoterica• 7 years ago

You’re misattributing the cause of the stress. Any competitive job opening is intrinsically going to be stressful because there are there are X people interviewing for 1 position and you have to outperform the other (X-1) candidates. The interview format is irrelevant. The only “non stressful” way to do an interview is to hand out $500k/year job offers to every warm bodies that hands you their resume, no questions asked. Sorry, but that’s not going to happen.

Apocryphon8• 7 years ago

I think you're getting at the core of the situation. Ultimately the interview is going to be stressful because it can only end in two ways. But I think there are ways to mitigate it, as suggested by some of the proposals in other comments.

Another cause for stress is simply because the whole process is just so damned opaque. There's not just a power asymmetry between hirer and applicant, there's also an information asymmetry. At the end of the day, you don't know what the true criteria were used to judge you. You don't know if you hit a mark and was just passed over in favor of a better candidate, or if you bombed without realizing it, or one of the interviewers simply didn't like you. And at that point it's not simply losing out on the offer, it's losing out the time spent to get a rejection that blandly says that you weren't the right fit. And sometimes they don't even remember to reject you.

Like the intrinsic stress of being at the mercy of an employer, it's part of the process that one just has to get used to. But one wonders if there's some better way. Perhaps pre-interview rubrics and post-interview feedback? Make applicants sign a waiver so they can't use it in a lawsuit. At least it'd be more transparent than the status quo.

gnusty_gnurc5• 7 years ago

I don't agree that interviews for "competitive" jobs are inherently stressful. Insofar as any interview is nerve-wracking - I agree that's unavoidable. The point of hiring is finding anyone that's capable of doing the job, not overseeing a gladiator deathmatch. I get the sense that no one is vulnerable enough to admit that hiring is a toss-up and almost intractable problem. Even more annoying is that experienced and competent workers in industry are usually going to be the people that believe they'd be good at judging candidates when there's little evidence that tech competence translates to hiring competence.

linguae• 7 years ago

The problem I have with coding interviews isn't necessarily the algorithms-based questions. I don't have a problem with reviewing undergraduate data structures/algorithms as well as those related to my subfield (for example, someone who is familiar with DBMS implementation should know the basics of a B-tree even if knowing its exact implementation requires review).

Here are the problems that I have with coding interviews:

1. Often I feel that interviewers are looking for exact, optimal solutions rather than caring about how the interviewer actually approaches problem solving. Forget about being asked FizzBuzz-style questions or about being asked to delete a node properly in a binary search tree; in the interviews I've had in the past three years, I've encountered difficult Leetcode- or ACM International Collegiate Programming Contest-style problems where it's expected that I come up with an optimal solution within 30-45 minutes. It's even worse with companies that give you a hard-level Leetcode-style problem that is automatically graded instead of being examined by a human.

2. The lack of receiving feedback about interviews that didn't go well makes the process difficult. At least on a standardized test you receive a score, and at least at the end of a final exam in class, you get your final class grade. Case in point: I've had two successful Google software engineering internships with great reviews from my mentors, and so it's not like I'm incapable of programming FizzBuzz or writing a for loop, but after three tries I haven't been able to get an offer for a full-time position there, despite making it to the on-site round each and every time. It's similar with other companies: I'm usually able to make it past the phone screen and about 50% of the time past the initial programming question, but I have a hard time making it past the on-site round for software engineering positions.

3. The sheer breadth of possible questions to be asked. Not only are Leetcode-style puzzles "fair game," but also domain-specific questions. For example, suppose I'm interviewing for a position where I'm working on a DBMS. Despite having taken graduate-level courses on the implementation of databases and distributed systems, there is still a very large amount of questions that I could be answered, including specific details of specific databases that I might not have been exposed to.

The frustration I have with the software engineering interview process is enough for me to want to change fields at times. Thankfully I've found the interview process for more research-style groups to be less about coding acrobatics and more about conveying previous research experiences and successes as well as proving competency and curiosity. I have a pleasant role as a research engineer where I'm doing very interesting research work while also maintaining my coding skills. Unfortunately such jobs are hard to find in industry, which means that one day I'm going to have to take up the gauntlet of software engineering coding interviews all over again.

xwolfi3• 7 years ago

But Google doesn't want you, you want Google. They want the process as annoying, unfair and difficult as possible cause they have entire countries that want to work there. Your internship was free so, well, it was easier to sell you to management.

Go to a small mom and pops startup where you do absolutely everything from rebooting servers to convincing clients to pay you. That'll be a thousands time more valuable than 10 years at google sleeping between desks pretending to change the color of a gmail icon :)

pcunite• 7 years ago

I failed a C++ test that was a requirement to get a job (30 minutes timed, multiple choice). I had written a C++ application that I had sold to thousands of customers (Fortune 100s in there too). I had mistakenly thought I could be a programmer at a real job.

Later, I sold $20,000 worth of source code that took me 60 hours to produce. The client thought it was easy to read.

I still don't think I'm a real programmer. The test sure seemed to prove that. But, people still buy my software today (I have other apps).

From my perspective, it seems the current technical interviews are looking for something in particular, from which you don't later actually do that for the real job. Kinda like seeing if you can survive a skydive, but you'll actually only be packing chutes after hire.

I wish there was a way to say, up front, that there are different types of programmers, vs the catch all "programmer" job title. Because it sure seems like I'm typing out code. The computer has always accepted it.

weiming• 7 years ago

I wouldn't take it personally. Language-specific tests sometimes focus on memorizable trivia instead of problem solving. Like, "How many generations does the garbage collector have?"

jussij• 7 years ago

The test sure seemed to prove that.

I would not get too hung up on those multiple choice tests.

I've been programming professionally now for over 25+ years and ran into my first multiple choice test about 4 years ago.

In a very short space of time I managed to failed three (two C# and one C++) in quick succession.

Now days, if I come across a role that requires such a test I just politely refuse and move on.

FWIW why I think these test are a waste of time and why I did so badly is because they don't actually test day to day programming skills.

The three test I did where all very similar with a lot of gotcha type of questions.

They asked questions like spot the obscure error in a very poorly written piece of spaghetti code, or tell me exactly what exception this code will throw exception etc.

As a programmer you never have to directly deal with these types of issues only because your compiler finds the errors and the documentation gives you exception details.

And in a way the tests knew this as well, because all three warned you could not switch out of the test window while doing the test.

In other words the test would have been a trivial exercise if you where actually allowed to use your day to day work tools.

throwaway033057• 7 years ago

Just another ad for a company disguised as a blog post.

dang• 7 years ago

I'm not sure why you say that here, but it's actually not a problem on HN as long as the article is interesting. I wrote about this recently: https://news.ycombinator.com/item?id=20186280.

quickthrower2• 7 years ago

aka, "hub content"

sbov• 7 years ago

The way some technical interviews test technical skills would be like testing communication skills by asking candidates to give a public speech.

Then talk about how you "dodged a bad one" because they couldn't say a single sentence without an "uhm".

ghaff• 7 years ago

Maybe not a developer job, but for jobs that actually require public presentations asking candidates to give a talk on some topic seems very reasonable. And, yeah, if they fumbled through it that would be mark down if that's something they'll do as a job.

Writing may be more important, but presentation skills are absolutely important for some jobs involving communications.

dahfizz3• 7 years ago

I don't understand this mentality at all. If I'm hiring you to do X, I want some sort of confirmation that you can actually perform X. What's the alternative? Hiring the first person who applies?

inertiatic• 7 years ago

Hiring the way hospitals hire doctors.

I don't know, do they have them dissect other species while explaining their though process?

hyperpape2• 7 years ago

To see this more clearly, consider interviewing a pilot. After establishing basic bona fides, it would be reasonable to ask the candidate about what to do in various emergency situations. Emergency situations aren’t representative of the daily work of flying, but safety is important so nobody would accuse such an interviewer of asking irrelevant questions.

Minor nit about what is really a pretty good post. This paragraph sounds quite reasonable to this non-pilot, but do you notice what it doesn't say? It doesn't tell you whether pilots are actually interviewed this way.

Here's an actual example I know of, though it's not making the point the author wants to make. At an old job, every so often, a guy used to wander over from the other side of the yard and ask us to cut a few random pieces of pipe. We then carried them across the yard for a candidate pipe welder to weld them. If the welds looked good, they'd probably hire the candidate. It was pretty close to actual work you'd do, and the results seemed ok.

I'm sure there are examples that would suit his point and be real, not just speculation.

jillesvangurp• 7 years ago

Oral exams are part of certification for pilots. Additionally, pilots have to keep log books so you can simply examine their history and progress. So, repeating that during a job interview is kind of redundant because either they are certified and current, or not. For software engineers, certifications are kind of meaningless. I actually treat them as a red flag because in my experience it's not so great consultants that tend to overemphasize certifications in their CVs to cover up that they haven't done a lot of great projects. Usually, this is obvious from their CVs.

I've been on both sides of the table for technical interviews. If we're talking, that means CVs and linkedin & github profiles were scrutinized, etc. and we're now moving to the phase where we are going to mutually find out whether there's a basis for working together. Part of that is quickly verifying those things were accurate; but most of it is simply eliminating any red flags: any obvious reasons you should not hire someone.

I usually focus on just getting people to talk about what they've recently been doing and getting them to talk about things that they are interested in technically.

When I'm interviewed myself, I mentally reverse the roles. I already know I'm good enough; the interview is about determining whether the company is good enough for me. I've declined offers because of how the interview went or because I realized the interviewer would not have gotten past my own red flags: I'd never hire them.

This attitude works well on both sides of the table. Would I want to work for me based on how I'm behaving? Is my behavior making them more or less enthusiastic about working for me? Part of an interview is that its a sales job. Imagine you get an excellent candidate and that they have multiple job offers: how do you make them pick you?

supermanfan• 7 years ago

Actually the pilot anecdote is nonsense. Emergency procedures are part of the procedures you're supposed to be able to execute, typically in a simulator.

A stronger anecdote would be asking an airline pilot to fly an aerobatic maneuver in an airliner like an outside loop or aileron roll.

(There are only a couple of aerobatic maneuvers done in an airliner. For example, when power is stuck at full throttle or the elevator forces the plane to climb, a quarter aileron role can be done for recovery.)

Source: Commercially-rated pilot.

choppaface3• 7 years ago

I’ve interviewed many developers who pass a code test, get hired, and then can’t design a basic interface for an object. Or take forever getting through a large feature. Or can’t commmunicate clearly. Or go down the rabbit hole and implement something really complex and shiny but irrelevant. (Irrelevant isn’t always bad, but does often expose a weakness in effectively interpreting user needs).

Maybe 1 in 10 engineers who pass a code test end up being actually good software engineers. The yield definitely varies by the role.

Wait, if code tests aren’t perfectly predictive, then how do interviewers get feedback about failures (or even successes) of the screening process? They don’t, because recruiters and hiring managers tend to keep that to themselves.

The story is way way more complicated than this post alludes.

ALittleLight• 7 years ago

Are people who failed code tests better?

choppaface3• 7 years ago

I know of at least a couple instances where the person failed the code test and then (according to LinkedIn) got a solid job somewhere else.

Triplebyte actually has some hard data on this trend. Many candidates definitely bomb a few initial screens, do well on later screens, and then burn out and even withdraw from later on-sites. I imagine there’s high variance. Any experienced recruiter I’m sure has observed the same pattern... bombing one code test is typically not definitive.

james_s_tayler2• 7 years ago

Yup. Just means the normal distribution applies to software engineers too, which makes perfect sense. A handful will be so god awful you'll wonder how it is they even have a 10 year career in the first place. A handful will be in the first couple of years in their career and you'd think they have been doing it for 10 years. Most are average.

But here's the kicker... in software engineering average means bad. Because average code is not good code, because only amazing code is good code. So all these questions and tests trying to figure out who's a good engineer are kind of rubbish. Mostly they will be average engineers. But companies seem to think "we only hire the best" all the meanwhile the trash-fire slowly burns.

neilmock• 7 years ago

It's the same power dynamic that will be in place forever, incumbents unable to recognize talent outside of their narrow perspective. The penalty is paid by the candidate, and the company will usually go along just fine, but it is a sad broken system for a lot of qualified people.

mlevental• 7 years ago

The penalty is paid by the candidate, and the company will usually go along just fine, but it is a sad broken system for a lot of qualified people.

i don't understand how people don't get it: if this were a malfunctioning system companies' margins/bottom lines would be affected and they would correct course. the fact that not only does this trend persist but grows signals that it's actually effective.

crdoconnor• 7 years ago

There's a lot of luck and a lot of pareto distributions for software development. There is no straightforward relationship between effectiveness and profit like there is for other careers (e.g. sales).

reallydontask3• 7 years ago

This is pretty naive.

A company I used to work at made ~$10Bn in profit per year and employed 300k people this does not mean that all its practices were profit making, some were demonstrably not profit making, even admitted to be so by SVPs. I don't mean some investment that would pay off in the future, I mean cost cutting exercises that went so far as to go past the fat and into the bone (SVPs words when we got a new CEO)

Another company I used to work at, this one employing 20 people, had a shockingly bad product. It looked dated, it was slow and it had all manner of process issues, in a word: Dysfunctional. The company was pretty profitable as it had excellent sales people. Working here really made me question a lot of things. How could a company with such a shitty, outdated, slow and unmaintainable product be so successful (for its size)?

The point is that the relationship between qualified people and financial success is far from linear.

clarry3• 7 years ago

Assuming companies are omniscient..

I suspect it's really difficult to tell from within whether your bottom line is being hurt or helped by the interview process.

The effects are not immediate, and you rarely have the possibility to see how the numbers would've looked if you had done everything differently.. of course, that goes both ways.

sgift4• 7 years ago

If this were correct startups wouldn't have a chance to dethrone an incumbent, because companies would adjust course long before.

The thing is: Some people may even see this. But then they have to convince their boss. And maybe that boss has to convince their own boss. Or HR. Or both. And then you get into "but does this mean a part of our workforce is incompetent" territory and so on.

Something doesn't get corrected just because it affects the bottom line. Someone has to fight for it and with a topic like that you probably won't win many friends by doing it.

lmm6• 7 years ago

Someone who has talent in a way that the incumbent is unfamiliar with is far more likely to be able to demonstrate that talent in a coding test (where the interviewer can't deny that their code works, even if they don't understand it) than in a conversational interview (where the interviewer can easily dismiss anything they don't understand as bullshit).

hjk052• 7 years ago

but it is a sad broken system for a lot of qualified people.

The positions get filled. The candidate who gets the job is also qualified.

There is way to much bias from people who go through these interviews and don’t get hired not because they weren’t competent and not because the system is broken. It’s just simply the case that someone else was better.

on_and_off3• 7 years ago

How could someone with 10 years experience on their CV be unable to start a new project in their own editor

This is something I need to study each time I want to change jobs. I suspect that I will have to create a new project somewhere in the interview process but it is just something I very rarely do in my day to day job, if ever. I suspect that it is the same for many engineers.

FWIW, at my current company, the interview projects come preconfigured, so you only have to import them and start coding. We have also already imported some popular libs.

Watching somebody rightfully struggle to do that is :

- a pain for the interviewee

- awkward and uninteresting for the interviewer .. I don't expect many people to have to create new projects often so it is totally ok to have forgotten it

- a waste of time for the both of us.

We also actually don't do algorithms.

We have a debugging interview, with a shitty codebase where you have to find the issues and correct them. When you are done with the bugs, you can collect your thoughts on what's wrong with that codebase so we can discuss how to do it better.

Another interview is to have to design a product similar to ours (a simplified version of course) and you have to come up with API and app technical designs.

JamesBarney2• 7 years ago

We have a debugging interview, with a shitty codebase where you have to find the issues and correct them. When you are done with the bugs, you can collect your thoughts on what's wrong with that codebase so we can discuss how to do it better.

I've always wanted to do this, but I couldn't convince any previously employers to pay for it.

PopeDotNinja3• 7 years ago

Watching somebody rightfully struggle to do that is :

Being watched usually makes me a horrible coder. I get so distracted from having a backseat driver that it can even hard to type at times. The only thing I can whip up on the spot in a readable manner is something I already know pretty well, and if the interviewer can't keep up, it becomes a big conversation about "what are you doing again?"

fourbits3• 7 years ago

Yes! I learned this in my first internship when I froze up on the terminal because I had a manager hovering. It's something that comes up a lot in my circles. I did feel a little better when I eventually taught that same manager something about SSH. That excited "Oh you can do that??" felt so good.

on_and_off3• 7 years ago

That's something I try to alleviate.

It can be pretty hard to make a stranger feel at ease in a stressful situation so I can't pretend I have a 100% success rate, especially for the one screening interview we do which is a one hour pair programming exercise. However we do it, I am going to be there and look at what you are doing.

For the onsite though, most of the time, I explain the exercise and then I leave people work while I do something else in the same room.

I stay there so they can ask questions if they are stuck but I am not hovering over their shoulders so they can concentrate.

widforss• 7 years ago

I even find it extraordinarily hard to type my password when there is any one else but me in the room.

fourbits3• 7 years ago

I had never heard of this set up a project interview test. Sounds stressful and awkward but

For me, it would possibly be preferred to algorithms because one of my strengths is reasoning about technologies. In group projects and when working with friends, I naturally take the pragmatic setup role.

Algorithms, on the other hand, I struggle with because even tonight at dinner my sister was trying to remember what word describes toxins moving up the food chain, something we learned 10 years ago.

Before she could finish her thought I was blurting out bio-magnification!

It made me think why do I remember some things so well and others really do go in one ear and out the other.

I know to an interviewer it may seem like they're testing if we have knowledge that will be applicable to the job at hand but to me, it feels arbitrary.

I am adaptable. I am not a textbook.

When I am on the other end of that table, I will have already asked for a portfolio of sorts: school work, personal projects, anything that you want to represent you.

I will spend some arbitrary amount of time finding code snippets that are problematic or interesting and I will ask you to speak to them.

I feel that will be enough to know whether or not I want you on my team.

vageli2• 7 years ago

When I am on the other end of that table, I will have already asked for a portfolio of sorts: school work, personal projects, anything that you want to represent you.
I will spend some arbitrary amount of time finding code snippets that are problematic or interesting and I will ask you to speak to them.
I feel that will be enough to know whether or not I want you on my team.

I like your approach but wonder if I would be successful if looked at through that lens—I have looked back at code written a couple years ago and was surprised to find that I was the author and would definitely be unable to speak to my choices then (though I could probably speak to how I'd do it differently now, which might be a useful signal in the interview).

blatantlies7• 7 years ago

This is something I need to study each time I want to change jobs.

+1. It is not that I don't know how to do it, but I have to practice beforehand to have it in my auto-pilot and not spend time during the interview. Otherwise I will panic seeing the clock ticking and my IDE freezing indexing files, etc.

aswanson• 7 years ago

I think a good way to gauge a persons tech ability is just have them sit with you for a debug session.

davidjnelson2• 7 years ago

Codesandbox is amazing at pre configured projects for interviews.

pytester2• 7 years ago

Fix a real bug or implement a real feature. Beyond the obvious copyright issues if you don’t get an assignment from the candidate,

Irrelevant if you've already fixed or implemented it and do not intend to use the candidate's code.

there’s no good way to make this process repeatable,

Select fixed bug/feature, isolate code, create question, use in interview, iterate on task.

no way to ensure the question always fits the available time,

Other than selecting the question such that it does and/or tweaking it such that it takes more/less time.

and no way to ensure you see a good range of programming skills (e.g. the feature may simply involve copy/pasting some existing code with minor tweaks).

Other than selecting the question such that it isn't that.

It also assumes you’re hiring Java developers for a Java codebase or Ruby developers for a Ruby codebase

I'm certainly not going to hire ruby developers for a java code base or java developers for a ruby code base, and I find it weird to assume that I would.

ajmurmann3• 7 years ago

I'm certainly not going to hire ruby developers for a java code base or java developers for a ruby code base, and I find it weird to assume that I would.

Learning a programming language is usually not the hard part unless it follows a completely new paradigm that you are entirely unfamiliar with. E.g. programming in an object oriented language if you only ever have done imperative. I've interviewed people in languages they didn't know with great success, you just gotta ask the right questions and sometimes their questions (or lack of) are more important than their answers.

hjk052• 7 years ago

You simply cannot be arguing that companies who need a developer to maintain a java code base should take special care to not consider candidates ability in java just in case they don’t know the language at all.

Yes learning a new programming language can be easy, so go do it before you apply for the job that’s looking for someone experienced in the language.

pytester2• 7 years ago

Learning a programming language is usually not the hard part

The syntax and semantics can indeed be picked up in a day or more. It's working within the ecosystem and circumventing all of the non-obvious pitfalls, knowing the shortcuts (and knowing what can't be shortcut), knowing where all the important information is located that all requires years of experience.

I've worked with developers who have switched languages and they often port across bad habits and quirks and try to make the new language more like their "home" language. Sometimes years later they're still trying to do things in a non-optimal way.

I also think that most people are more productive in their favored language (or languages) and you're unlikely to get as good quality work if you pull them away from it.

There's also a cultural element to programming languages that I think is underrated. I don't expect java programmers to have the same kind of outlook or approach to python programmers.

joezydeco7• 7 years ago

"Everyone who builds a team of developers, and I do mean everyone, rapidly gets used to people turning up to interview who cannot actually program computers, even under the most generous definitions of the term."

I've started doing a small whiteboard coding exercise during interviews, and over the past couple of iterations it's devolved down to something simple...and I mean really simple. And I'm watching senior developers fail the test.

For example: I'm trying to fill a bootloader developer position so I get a couple of senior embedded developers to the table. Passed the screen, HR likes them, yadda yadda. Some even have U-boot experience. Should be easy, I think.

So I'll ask them to perform a simple bit-flipping exercise on the whiteboard. Nothing insanely tricky and it's usually something that any register-level developer should be able to blurt out without thinking twice about it. And they are choking on the question.

It's kind of freaking me out. Is the test that hard? Is the developer that inexperienced or distant from doing this kind of work even though they're applying for it?

What am I missing here?

Quiark• 7 years ago

The high-stress environment of job interviews blocks thinking.

myself248• 7 years ago

Should the company beer fridge be in the interview room?

AnimalMuppet3• 7 years ago

Perhaps, some of the time. I doubt it's the whole answer, though. I suspect that some of the answer is stress, and some is fakers, and some is people who have been in embedded and never had to do that. (I haven't for 10 years... but I'm sure I can still do it.)

cmonnow7• 7 years ago

why is it 'high-stress' ? desire to look good ? fear of being judged as a fool?

The ability to temporarily NOT GIVE A SHIT about what others think of you is a pretty valuable psychological/leadership indicator IMO.

I would want such folks, who may not be able to code up a binary-tree search, but would work through it in the same manner they would at home in pyjamas, to be team-leads or managers over time.

maksimum• 7 years ago

Did you vet your question and its phrasing with your co-workers? What did they say?

joezydeco7• 7 years ago

They all had a direct answer or something approaching it without even pausing to think about it.

It gives me reinforcing information that the problem isn’t out of line, but you make a good point that one shouldn’t lose a problem that nobody else at the table can’t solve.

pmiller26• 7 years ago

U-boot

Off topic, but my mind immediately went to WW2 German submarines when you wrote this. I was wondering how you were screening so many 90+ year old German men for about half a second. :P

joezydeco7• 7 years ago

The full name of the project is "Das U-Boot" so you're not far off. =)

https://www.denx.de/wiki/U-Boot

kradroy3• 7 years ago

The whiteboarding in the problem setting might be the issue. How many engineers actually whiteboard regularly? It's as old fashioned as waterfall design. Everything has a REPL shell (even Java!) now. Hell, even CloudFormation stacks let you fiddle with entire architectures through a command line. Engineers have always poked and prodded at things to learn and refine them. Whiteboards are for throwing around ideas; not for solving problems.

I spent most of my grad school years researching learning and memory. The medium and environment of the fact recall/problem solving can greatly affect performance. Now compound that with the stress of your standard technical interview and you'll get some flubs.

You might want to take a step back and consider how you're asking them to show competence in addition to what you're asking them to show competence of.

joezydeco7• 7 years ago

Would it really be any better if I handed them a laptop with Notepad or some IDE and asked them to type in their ideas?

What I'm asking is the bitbanging equivalent of "store 2+2 in a variable", so setting up an entire coding ritual for 3-4 lines of code isn't really what I had in mind.

jsymolon• 7 years ago

whiteboard ... it's as old fashioned as waterfall design.

Call me old-fashioned then. I like to map out the data flow and process flow a bit. A little planning now, saves quite a bit later. Call it, pre-re-factoring.

You don't go scuba diving, hiking, camping, etc, without a plan.

clarry3• 7 years ago

It's kind of freaking me out. Is the test that hard?

You didn't show us the test so let me flip a coin...

One can get pretty far in embedded just knowing how to OR flags and mask & shift values. I guess that's something most embedded devs working at that level should be able to just blurt out. Is that what your test was about?

Then again, I'm sure you can find "senior embedded developers" who worked on the browser based interface to an appliance that runs Linux... and, well, the last few times I used U-Boot on our product I didn't really need to play flipper with bits. (Well, I did, even if I didn't need to.)

joezydeco7• 7 years ago

Here's the test. It's one I stole from an online test practice site but IMO it actually works well on a number of levels:

Give me a routine takes in an unsigned integer, pick any size you want, and returns TRUE if the binary representation has two consecutive 1s in it.

    Test data: 9 = 1001b = FALSE, 13 = 1101 = TRUE.

I'll even give them an example function prototype:

    bool adjacent_bits(uint32 x);

As I see it there are multiple ways you can write this and I really don't care how you do it. If you start with a simple case (say, a for loop), get it working, then realize you can optimize it...that's even better. It even optimizes down further to a single inline math operation.

The key is that I can see you understand bit shifting and a bit operator in C. If you've done this kind of work as your resume claims, you should be somewhere in zone on this without too much struggle.

I've seen candidates write triple-nested for loops. Bizarre interlocking bools trying to make a state machine. Hard-coded numbers that kill portability. Calls to printf where I asked for a boolean return. Attack ships on fire off the shoulder of Orion...

Perhaps, like you said, this helps weed out the people that just played with the GUI to configure something when I need people that got a driver working down at the register level. So I'm sticking with the test for now.

lmm6• 7 years ago

People who can't get a job interview far more often than people who have a job. Rejecting a large proportion of candidates at the first step is not necessarily a red flag.

ernst_klim• 7 years ago

Stress kills creativity.

https://www.writingandwellness.com/2016/11/28/how-to-keep-st...

joezydeco7• 7 years ago

But the question wasn't gauging creativity. It was an academic question.

kortilla2• 7 years ago

How could someone with 10 years experience on their CV be unable to start a new project in their own editor

Said nobody with 10+ years of experience. Starting a new project in an editor is as obscure as changing the color scheme in your editor.

nottorp• 7 years ago

Yep. I really don't remember right now. I haven't started a new project this year.

Well, I'm sure I can start one, but i'll be slow at it.

danjac• 7 years ago

Generally when I start a new project I'll do it differently from last time. Might be a different language or framework, maybe it uses docker or not, maybe the current best practice or boilerplate has changed in the last six months. It might be different for people in agencies, but then they might have some code generator or starter pack for optimization.

BigJono• 7 years ago

Technical interviews have been quite useful to me in the past.

Once I applied for a job, and received an online technical test where every single question used the wrong terminology for a bunch of stuff and half the "answers" in the multiple choice section were wrong, while some of the "wrong answers" happened to be right.

That technical interview served it's purpose perfectly, assuming it's purpose was to save me from accidentally taking a job that would have been a fucking train wreck.

james_s_tayler2• 7 years ago

Amazing.

lanrh1836• 7 years ago

The fact that interviews are so standardized in this industry is such a rarity and the pros outweigh the cons for me as someone interviewing. It means you should basically know exactly what to expect, and you have every resource, most free, to study for them. I find that a much more meritocratic process than the way hiring is done in many other industries/professions.

The alternative is widely varying interviews from one company to the next with no idea what to expect, which leads to a mismatch in expectations and mostly wasted time. Take home coding assignments is one alternative I see as well but as someone interviewing that is a much bigger time commitment and something I would not want to do especially if I’m interviewing at a few places.

bydl0coder3• 7 years ago

The problem with take home assignments is that while investing time the candidate doesn't know how it will be judged. Sometimes people want just a working prototype but sometimes they want essentially a production quality high-performance code.

scapegoat444• 7 years ago

I failed a take home b/c the company didn't host the HTML project on a server, instead they opened it via the file explorer...

wvenable7• 7 years ago

This is absolutely completely true:

"Companies do code testing because they have encountered so many candidates who look good on paper, and may even be able to talk about computers convincingly, but can’t actually write a program when asked. Any program. At all."

I interview programmers and I've encountered this a few times, and yet, I still don't do code tests. I talk to the candidate and you can easily ask the right questions to tell who can code and who can't. You can even get a judge of experience level (junior, intermediate, senior) which you won't get from a code test.

We get a lot of English-as-second-language and shy developers (who can code) and I can still figure out by asking pretty general questions about development. The idea that you need a code test to figure out who can code and who can't is false. I'm not arguing against it, I'm simply arguing that it's not strictly necessary.

gregmac• 7 years ago

What has worked well for me (at least for non-junior positions) is to just ask about their prior work. "You were on a team that developed x - great. Assume I'm an expert in that and am happy to geek out about the tiny details. What was a part that you specifically were responsible for? What was most fun/interesting about it? What was one of the challenges and how did you overcome it? What was something new you learned? If you started over today, what was a part you'd do differently, and why and how?" Really just pick something they seem excited about and dive really deep into it. "Why didn't you use y for that? How did you test it?" The key is really simple: find something they're passionate about and get them to talk about it.

The annoying thing about refining interview techniques is you only get a small view of effectiveness: you can't tell how many times when you didn't hire it was the correct choice vs passing on someone great, so I can't judge from that perspective. That said, I haven't regretted hiring anyone that did well on this style of interviewing, without another coding interview.

JamesBarney2• 7 years ago

What are the questions about development?

llamataboot• 7 years ago

I agree 100% and I would argue that unless you are pairing together just to see how you work together and for some back and forth problem solving, no other code test provides a better signal than a good conversation.

Not set questions, even, just a good conversation. What issue have they run into with Framework X, how do they refactor, what's the weirdest bug they've ever had to track down, etc etc etc.

SV seems to enjoy grueling 7-10 round interviews these days. I had one a couple months ago that was over 40 hours (of my time) and over 20 hours (of their time, that I was aware of). I just can't believe that much time provides 20-40x the value of a simple relaxed conversation.

danpalmer2• 7 years ago

I talk to the candidate and you can easily ask the right questions to tell who can code and who can't. You can even get a judge of experience level (junior, intermediate, senior) which you won't get from a code test.

I also think I can do this, but I believe most people (myself included) introduce a lot of bias when doing this. I've spoken to developers who in 15 minutes I might have written off as not very qualified, but after hours/days have realised are great at what they do, and very experienced in types of software engineering that I have had less exposure to.

I think we have to be careful when judging candidates with what amounts to "gut feeling" like this, as it's typically biased, and not aligned within the hiring team.

sgift4• 7 years ago

The thing is: The method described in the post has the same problem. It just doesn't look like it. Most of these types of interview templates are designed by one or two persons and then blindly applied. Sometimes you have a view modifications, but for the most part it's fixed.

Since every interviewee gets the same set people then start to think this is more scientific than "gut feeling" - it is not, but instead of your own biases you have the biases of the template designer.

There are probably a few companies which extensively study if their interview process for the people hired (the only subset they have access to, which already is a problem) actually matches their job performance and then try to readjust the process, but it doesn't seem to happen very often.

This also gets us back to the old problem that we have no idea what a good programmer is or how to evaluate job performance.

codeduck• 7 years ago

I find that asking "What is the worst bug you've ever encountered" is a great sounding board.

jldugger• 7 years ago

I interview programmers and I've encountered this a few times, and yet, I still don't do code tests. I talk to the candidate and you can easily ask the right questions to tell who can code and who can't.

How do you know? Is this repeatable -- e.g. can you write down the questions and the criteria such that someone who is not you can follow them and come to the conclusions you would?

Do you ask all candidates the same questions, or do you go with your gut and hope there's no systemic bias?

anbop6• 7 years ago

It’s certainly possible but it’s obviously much harder to do. Let’s say I own a professional basketball team and need to make a hiring decision on a player who walks in the door. Am I going to be able to, with just an interview, determine whether they are a currently good player, or a formerly great player who has only been coaching in recent years and has forgotten how to play? Rather than trying to determine this just by asking questions, why not take the much easier approach of putting them on the court for 2 minutes to see how they play?

irq114• 7 years ago

“Let’s say I own a professional basketball team and need to make a hiring decision on a player who walks in the door. Am I going to be able to, with just an interview, determine whether they are a currently good player, or a formerly great player who has only been coaching in recent years and has forgotten how to play?”

No, that would be phenomenally stupid. You would do what teams actually do, and look at their recent playing history.

Note that this is also entirely possible for engineers.

dvdbloc2• 7 years ago

This is unrealistic because coding in an interview is not similar to coding in reality, at least for me. When I develop software I silently get to think about reason about problems, try things and read things on Google. In an interview I am with a person I don't know staring at my hands and expected to talk and impress them, all while a timer is ticking.

AstralStorm2• 7 years ago

Unlike basketball, coding is not reproducible and has no clear rules nor set of tools even. Thus your analogy is flawed from the outset.

Additionally, a person who can adapt but has lower specific testing skill might actually be better at everything. As in your coach example, a slightly rusty to player from years past might be better than slightly better new guy with no track record. You're taking a gamble either way.

The lowest rung coding test is only good to filter out people who cannot code at all, and that if simple enough. They tend to be stupidly worded though instead.

Trying for anything tough tests for specific tricks, tooling and skills. Giving a longer time task is comprehensive but rules against people who are actually busy, presumably working, thus already kind of tested.

wvenable7• 7 years ago

why not take the much easier approach of putting them on the court for 2 minutes to see how they play?

If every basketball team did that, every player would practice for that 2 minutes of play and you'd have no idea if they would handle an entire game.

The effect is the same for developers: interviewing developers practice for coding tests; memorize common interviewing patterns, quick solutions, and junior level algorithm stuff from their university days.

If you were interviewing a player, you'd probably look to their past experience and teams they played on, their positions on the court, and you'd check with their teammates.

raxxorrax• 7 years ago

I could do that for my specific niche for programmers (c, embedded systems [for the systems I know], a bit of web stuff), but not in general for a data analyst or ML for example.

I imagine I would ask for knowledge only practical experience could provide (interest in a topic is not the greatest teacher, but a decent one)

Better than fizz-buzz I think...

But you can never be sure either way. I would think that actually giving people a chance while defining quantifiable expectations would help here. Costly and time intensive perhaps, but I don't believe there to be a reliable shortcut. Especially not if the labor market is in need of more engineers.

lmm6• 7 years ago

I interview programmers and I've encountered this a few times, and yet, I still don't do code tests. I talk to the candidate and you can easily ask the right questions to tell who can code and who can't. You can even get a judge of experience level (junior, intermediate, senior) which you won't get from a code test.

How do you know? Do you check your interview assessments against actual performance among those hired (measured how)? What about the ones you turned down, what makes you so confident that they actually couldn't program as opposed to just answering differently from how you expected?

wvenable7• 7 years ago

What about the ones you turned down, what makes you so confident that they actually couldn't program as opposed to just answering differently from how you expected?

I'm pretty forgiving -- heck it usually takes at least half the interview before the candidate's nerves settle down.

Being a developer involves being able to explain yourself clearly. You can't just sit in the dark and code, talking to no one. If I came away from an interview with the impression that you can't code then I don't want you. The candidate has to do some work too -- they are there to convince me to hire them, not just pass a test and get a grade.

pjc503• 7 years ago

How do you determine the false-positive rate of this: people who actually can code quite well but fail the interview?

speby• 7 years ago

It's interesting. I've encountered a very different animal, even without having the candidate do a "homework test" or a "coding test" during the interview.

I have found that this different animal is competent, and certainly can code, but is of the very unproductive, slow type of developer.

And so my biggest pain point is rarely around a developer's competence during the interview process (i.e. Does this person know the material and are they able to do the job?) but rather instead around their overall productivity, their "capacity" to build software using their competence.

There are plenty of people in the world who know the material but are extremely "slow" in actually getting things done and problems solved in a day job. Unfortunately, the in-interview coding tests don't do a very good job of measuring the person's productive capacity to get problems solved, only whether or not they can solve some coding problem in a 30-minute timespan.

torgian• 7 years ago

I haven’t read all the comments, but...

My question is, do companies give interviewees the option to work on a problem the company is currently facing?

If not.... why not?

Shaanie• 7 years ago

Too much effort required to maintain, interview would takr too long and good candidates may turn it down because they don't want to work for free.

ajmurmann3• 7 years ago

I work at Pivotal where we pair program with candidates on work that's usually actually in the backlog. In some cases you might have to replay a story you did already if you don't have a new one that's well suited. One of the biggest benefits is that the candidate also sees what they are getting themselves into. I don't think it would work without pair programming though and might make less sense in am environment where you won't pair either once you got the job.

ajmurmann3• 7 years ago

To me the biggest problem with algorithmic interviews is when the candidates are truly asked to reproduce a well established algorithm. Frequently at this point candidates will just have memorized the answer, but aren't actually good at algorithm thinking. I've interviewed many candidates who thought I wanted them to reproduce how a certain data structure worked although I was not. As soon as we'd deviate about the well treated path they be completely lost. I think puzzle style programming interviews that's require algorithmic thinking and good problem solving skills can be valuable. Especially talking through possible solutions and their trade-offs can be great. But only if we aren't testing that the candidate is good at memorization and studied a lot.

PopeDotNinja3• 7 years ago

I think puzzle style programming interviews that's require algorithmic thinking and good problem solving skills can be valuable

Think of it this way. Do the majority of applicants you have the luxury of interviewing think those questions can be valuable? I'm inclined to not think they are, so giving me one makes me less interested in your company. I might still do it, but it's a bit off a turn off, like if you show up to an interview 5 minutes late.

harryf• 7 years ago

Over the span of the last 10 years I've probably interviewed between 80 and 100 engineers, at some stage in the process and even ran the whole hiring process in my last company.

There are points in this blog post I strongly agree with like how disrespectful "homework" and the importance of letting a candidate code on their own machine, where they have all their shortcuts etc. setup and are able to be productive. But I've found the algorithmic coding challenge to be in equal parts harmful as useful.

The biggest problem, as I see it, is as software engineers we are strongly biased towards a rational understanding of the world. For example I've seen hiring teams argue _for_ a candidate they clearly don't like because that person aced the coding challenge. In such situations, as hiring manager, I then challenge the hiring team with "OK it's Monday morning. You had a bad weekend. And now you're in the office sat next to the candidate. How do you feel about that?" and usually the truth _then_ comes out and we avoid hiring someone that would destroy the team. The point here is we have a natural bias to see things in absolutes; zeros or ones. This _tends_ to lead to an over-focus on the coding challenge because it gives us the illusion of a black or white answer on whether we should hire.

When I've interview people, I always ask myself "what's my gut feeling about this person?". What I've found there is "my gut" is sometimes wrong in the positive direction - I liked someone but they turned into a bad hire but always right in the negative direction; if I have a bad gut feeling about someone that I can't fully explain, these days I always go with it. The "evidence" I have there was two hires that I had a bad feeling about, one I later had to fire myself and the other that stuck around for a couple of years until someone else fired them.

Otherwise, when it comes to assessing coding ability, these days most programmers have code on github, even if it's just "hobby" code. I'd rather review that, look at their commit messages etc. and get a feel for how they program when relaxed and not trying to impress.

And if we must make candidates do homework, how about contributing to the greater good and asking them to make a contribution to some Open Source project e.g. fix a bug in numpy or otherwise?

willtim• 7 years ago

I disagree with the author's explanation as to why all questions are algorithmic. I suspect the real reason is due to the US dominance in tech and the popularity of algorithmics in US academia. Indeed the parts of computer science to do with logic, formal methods, semantics, mathematics of program construction etc are, I am told, often described in the US as "Eurotheory". This is a great shame. A new graduate arrives at Google fully trained in algorithms as mandated, but is unlikely to need to implement any of them during their career. It is far more likely that they will need to glue existing systems together, consume APIs, author APIs etc. "Eurotheory" would much better prepare them for this.

As an example, consider that we could be asking questions on API design: get candidates to work through their semantics/axioms (introductory forms to create objects, methods to transform and eliminate them). Get them to write out high-level testable properties such as the way objects compose, associate, commute etc. Using types to enforce invariants etc

cjblomqvist• 7 years ago

Yes, and the algorithmic tests are highly biased towards those who've focused on that (either through academics or by self training afterwards) - and a miss on these questions still doesn't tell which is the correct story. Then again, this might not matter. Firstly, everybody knows these types of questions are to be expected so everybody cab prepare. Secondly, the problem (from an employers perspective) might not be the false negatives, but the false positives.

This in turn points to that we might try to generalize this a little bit too much, with assuming there're generally applicable ways for software developers but rather we need to adjust the process depending on the type of software development context. If you're developing a web service of minor scale, then algorithmic questions might be stupid (because the knowledge needed is rather one of network, async, web and structure rather than solving math problems), but they might be relevant for Google (where they are might be relevant because they search for developers with that mindset and who can use that mindset to come up with and solve big problems in a certain way - I don't know because I lack insight into the developer strategy at Google).

As engineers and developers my experience is we tend to love to generalize, even when it's not wise to do so.

a_imho• 7 years ago

All day interviews with no compensation are unprofessional bordering on insulting.

kstenerud2• 7 years ago

“What’s the number after F in hexadecimal?”

My first reaction to this was: "Damn... where's my ASCII table?"

Because this question can be taken two ways: "What comes after f in hexadecimal notation?" and "What is the UTF-8 code for 'G'?"

A very important thing to keep in mind for both sides in an interview is that misunderstandings will happen, for the most mundane and obvious things. How you respond to these will determine your success in interviewing. Give the benefit of the doubt until the other party has conclusively proven that they don't know what they're talking about.

xwolfi3• 7 years ago

That would be very weird to misunderstand this. I get it's stressful during interview, but "number" and "hexadecimal", as well as "F" all points exactly to 10, not 'G'.

It cannot be taken two ways, but ofc you can slip meaning, associate ideas etc, and if you fall back on your feet and joke about it quickly, nobody will mind.

kstenerud2• 7 years ago

I've had this happen quite a bit when investigating things, where I get the wrong idea from one step, and don't realize it until I've run with it for some time. This is a normal occurrence and we do it all the time. In an interview setting, with added stress of being watched and judged, the problem is magnified.

In interviews, I'll always throw in comments to help clarify if it looks like they're going off the rails, because letting them run on an incorrect assumption is only going to diminish your ability to judge their abilities. We all make mistaken assumptions all the time, about everything. The person who made no bad assumptions in an interview just got lucky.

MarkMc2• 7 years ago

I agree that the candidate should be able to demonstrate ability to write code, but I'm surprised that the author is unable to find interesting, short, diagnostically useful problems that crop up in the course of his day-to-day programming.

Often when writing code I think, "hmm, this problem would make a good interview question" so I keep a note of it. Here's a recent example from an accounting program where I was reporting debit and credit transactions: Given a list of integers, remove all pairs where the magnitude is the same but opposite sign. Eg. given [-1, 4, 6, 1, -2, -4, 4], the result should be [6, -2, 4].

For more in-depth assessment, the author dismisses the idea of asking the candidate to do an all-day task or homework because it isn't respectful of the candidate's time. So why not just pay the candidate for that time?

cbzoiav• 7 years ago

So why not just pay the candidate for that time?

If i'm working full time I probably don't want to give you my entire weekend unless you pay me silly money.

Meanwhile - at least in the UK - if I'm earning under £100,000 I might well have to fill in a self assessment solely because of your payment (I can't see you onboarding me onto your PAYE system for 8 hours of interview work). Its just not worth the hassle.

edanm• 7 years ago

The example question you give is exactly the type of "algorithmic" question people complain about, and that the article says is the only kind of question you can realistically ask. And for the same reasons.

For more in-depth assessment, the author dismisses the idea of asking the candidate to do an all-day task or homework because it isn't respectful of the candidate's time. So why not just pay the candidate for that time?

First of all, it costs money. Significant money even, if you tend to interview a lot of people.

Secondly, it's often an accounting hassle to pay money - you can't just hand someone cash, it needs to be in the form of a salary, and is not always so easy to do (and some candidates would probably be put off by it, especially if they're already working, since having a second income can get complicated).

anbop6• 7 years ago

This isn’t going to change in the near term, because the companies in the middle emulate the companies at the top, and the companies at the top are doing just fine with their current hiring processes. Sure, they’re missing out on some good people, but they have enough excess demand for employment that they can keep their cubicles well-stocked with enough C++/Java/Python coders to accomplish their business objectives and make tens of billions of dollars per year.

Telling them they should change is like telling Tom Brady he should change the way he holds the football — even if you’re right, he’s not going to listen to you while he’s still winning the Super Bowl every other year.

_v7gu• 7 years ago

Honestly, talk is cheap. If they do hire people with programming tests and keeping the performance metrics of the people they hire (as most companies do), they could have just published how test performance predicts actual job skills. Until then, the null hypothesis reigns supreme.

MarkMc2• 7 years ago

Candidates don't often appreciate the unequal disincentives for the interviewer in making the wrong call. For the company, hiring one poor programmer is usually worse than rejecting 10 good programmers. So a coding test which reduces the rate of false positives is useful, even if it increases the rate of false negatives.

As an aside, this lopsided penalty reminds me of how Typhoon forecasters in different cities systematically bias the predicted stormtrack toward their own city: https://twitter.com/m_clem/status/1081953447358984192?s=19

Apocryphon8• 7 years ago

That's certainly a valid stat that I've heard before, but one wonders then if the situation shouldn't also be improving mechanisms when managing existing employees. California, where much of the tech industry is located, is an at-will state. If a bad programmer is really so bad, then shouldn't it be easier to terminate such an individual? Because the alternative, of rejecting 10 good programmers to get to a good one, seems just as much of a waste of programmer time and resources, as hiring a bad one. Perhaps there should be a probationary period with teeth, programmers should be hired as contractors for the first month.

boblebricoleur• 7 years ago

Companies do code testing because they have encountered so many candidates who look good on paper, and may even be able to talk about computers convincingly, but can’t actually write a program when asked. Any program. At all.

Maybe I'm reading too much into this and it's just semantics quibbling but maybe talking about computers is not talking about coding ? I've never met someone able to pass for a programmer when they are not. I've always felt just asking candidates to talk about their last or favorite project help catching a glimpse of how they think about software problems.

Little anectode : my brother is the first software engineer of the family

Father : "Son, what kind of computer should I get ?"

Son : "I don't know much about choosing computers"

Father : "Don't you write software for a living ?"

Son : "Yes, exactly"

mike_hearn5• 7 years ago

Sorry, that was just a turn of phrase. For computers you can read "programming computers".

croh• 7 years ago

When I conduct interview, I ask very simple programs like fizz buzz for logic. If s/he succeeds, I focus to understand if interviewee is passionate about his craft, which is the hardest problem s/he encounter ? which is code s/he proud of ?

EnderMB• 7 years ago

The one point where I agree is, in my view, the sole reason why big companies use algorithm-style interviews.

FAANG-level companies can put out a job role and have hundreds, if not thousands of people apply. Out of all of these people, there might be dozens of people with phenomenal CV's - relevant experience to the role, mastery of a programming language or two, public publications of their work, and provable experiences through living projects.

You reach a point where it's not practical to compare these people through programming ability alone, and that's where the algorithm interview comes in. It allows companies to choose a suitable person while alienating good developers, because through sheer interest alone they can afford to do so.

I just wish that more companies were honest about why this approach is taken, because it would stop startups asking you to find a join in two linked lists when the role itself is to maintain a shitty CRUD app. These companies cannot afford to alienate good developers because they're not getting the sheer amount of interest a Google or Facebook will get.

Again, just my opinion, but people need to embrace the fact that interviewing isn't perfect. A code test should purely exist to ensure that you can weed out people that cannot do ANY programming. Outside of that, use the interview itself to determine their fit for the role.

_pmf_• 7 years ago

It’s vastly preferable to asking someone to write code with a fat marker pen whilst standing up in a tiny conference room.

Or, you know, just use a whiteboard in the interview for the same purpose you use it in meetings:

- not at all

- for high level algorithmic descriptions, data flow or architecture sketching

(Of course, the UML being demonized is a bit of a misfortune because now boxes and arrows can mean anything, but hey, better than having a visual language that has been developed by a bunch of old CIS White males, according to tech diversity Twitter.)

pjc503• 7 years ago

How could the candidate have started coding in one language, then decided actually they don’t know that language and should start over from scratch in a different one? How could someone with 10 years experience on their CV be unable to start a new project in their own editor? How could the candidate spend thirty minutes trying to generate a random number and still fail? This is crazy!

They panicked.

I would like to see a large company trying a replication study on their interview practice with their current employees. You'd need to make sure that the test was significantly different from the one used to hire them, and that they were put under sufficient economic and social psychological pressure - perhaps tell them that if they fail they will be fired and immediately escorted from the building (+), then make them do it with the CEO sitting behind them. I would expect over a 10% failure rate, including some people who are normally regarded as business-critical important team players.

You then have to hope that people lean the lesson "interviews are a poor predictor of performance" and not "some of our people are really imposters"..

(+) obviously don't actually do this, but like fire drills you need to convince people that it might be a real fire sometimes. I am also aware that there are ethics problems with performing stress experiments on employees without their consent, although nobody seems to care about this for interviewees

blatantlies7• 7 years ago

I would like to see a large company trying a replication study on their interview practice with their current employees.

IIRC Google did that and the results showed that more than half of employees wouldn't pass the interview. I can't really find the links for that bc I heard that from close friends.

abtinf• 7 years ago

I would expect a far higher failure rate, maybe closer to 80-90%, especially at a successful company with happy employees.

The engineers will be focused on building actual systems and be totally unprepared for technical interviewing.

Engineers great in one domain will be interviewed by those in different areas with different views on what constitutes “basic” knowledge.

None of their work history within the company can be used as part of the evaluation—you must simulate the problem where nearly all your work is locked up behind an NDA.

angarg12• 7 years ago

All day interviews. Many firms expect an interview for a developer to take an entire day, typically with between 5 and 8 separate interviews. This makes it hard for developers who already have jobs to attend.

This bothers me immensely. If you are job hunting, you want to apply to several jobs to get a number of offers and get an overall better deal. How do people manage this when they got a full time job? Do they literally take an entire week vacation to have 5 all day onsites?

michaelt• 7 years ago

If you're applying for your first job, you might treat job hunting like a fussy suitor problem and want to get five offers to get a good picture of your market value. You get a job where you're pretty happy with the salary, the benefits, and the work.

When you apply for subsequent jobs, you no longer need to take a large sample to calibrate your judgement; you only need to compare the job to your current one. And questions like salary and the nature of the work can be answered during the phone screen.

If changing jobs takes three days of annual leave and boosts my salary by $10,000 that seems like a pretty good trade to me.

durnygbur• 7 years ago

Do they literally take an entire week vacation to have 5 all day onsites?

That wouldn't be actually that bad as onsite is an advanced stage in the recruitment process. In the current state of the job market it's more entire week vacation to complete take home assignments and online tests.

maxheadroom• 7 years ago

I have to disagree: When you're hiring a person, you're hiring someone to join your team. Hopefully, we can agree that the majority of people in our field have the capacity to learn.

In fact, in many FAANGs, your first performance review year is "freebie" because they know that you're not going to come in and know the role and the tools used intnerally.

So, then, what is the technical aptitude test (is it really anything other than that?) for? Well, it could be used to measure a baseline for basic programming skills; however, this isn't how most technical interviews are designed.

Interviews aren't designed around the premise that knowledge on the internet exists or that your colleagues could, oft, be your first line of support (or code reviews, if the company has a good culture). Hell, there are interviews where the candidate has to code in notepad (or other text editor equivalent) because they don't want the candidate availing of things like Intellisense.

If we look at it from a different perspective, though, we live in a time when we can now put our source code online for the world to see. We can publish packages for the world to consume. We can make changes to production software (FOSS) and documentation and that's easily traceable (if you're not obfuscating your identity through sixty different handles).

It seems we don't consider these viable avenues of purview to view the candidate's ability before even asking them questions.

With that being said, are you really hiring the candidate because they would be good for the team or are you just hiring the candidate who can check off some interview-type question boxes?

I have interviewed a number of people in my day and the most excelled candidate that we interviewed and hired, because he checked all of the technical boxes, ended-up being a raging asshole and bringing the team morale down, considerably.

So, at the end of the day, you're not just hiring someone to fill a role. A single role is all but mostly dead, anymore. You're hiring someone to join a team and the human aspect should outweigh the technical aspect.

After all, just because someone checks all of the boxes, it doesn't imply that they would "be a good fit".

ChrisMarshallNY3• 7 years ago

Well, I have tens of thousands of lines of super-high-quality code, for multiple shipping apps and entire systems, used by thousands of people daily, out in the open-source domain, in dozens of repos. These have over a decade of detailed checkin commentary, with Doxygen and Jazzy docsets. I have hundreds of pages of documentation and blog posts, often intricately explaining my design process. I am constantly trying to learn the latest tech for my specialty. But I spend exactly zero time on LeetCode or HackerRank, so it's likely I'll fail Secret Santa. I doubt that I'm what you're looking for.

mike_hearn5• 7 years ago

To be clear: if you fail a coding interview question and your self-judgement is accurate then either the interview question is a bad question, or the interviewer is a bad interviewer, or the process has a problem.

I don't think many (any?) of the people I've hired this way have spent time practicing coding interviews, let alone on HackerRank. The questions asked are simply not challenging enough to require drill. If you can code competently in your daily job you can pass them.

ChrisMarshallNY3• 7 years ago

Oh, heck. I didn't even see the reply.

Probably right. Depends on the question, I guess.

You know what I mean, use questions straight out of textbooks that have only been in circulation in the last five or ten years.

I've found that it's easier all around if I simply don't hide that I'm over 40. It means a lot fewer responses to inquiries, but also a lot less wasted time. Avoids that crestfallen look when you walk in the door.

Apparently, the industry is crawling with folks that are under 35, but have the judgement and expertise you get from 30 years' of programming.

I'm also puzzled why folks would think they can learn more about me from a twenty-minute test, than by looking at ten years' worth of code in dozens of repos.

I was a hiring manager for many years. I would have drooled to see that kind of background on any one of my candidates. Still, I generally found that I was able to accurately judge their competence after about a half-hour chat (at least, I never seemed to get it wrong), and it was all about cultural fit, after that.

I suppose being an up-to-date engineer, myself, helped.

ChrisMarshallNY3• 7 years ago

Also, I know that no one will ever read this, but I figgered I'd put it there for posterity.

It's amusing, when I say "I have a published open-source portfolio that consists of XXX LoC, ten years' worth of checkin history, in XX numbers of repos.", the response is "IF what you say is true..."

Look for yourself. I can back up every single claim I make.

I don't pretend that all my code is screamingly efficient or the best-written code on Earth, but what I do have, is tens of thousands of lines of super-high-quality, well-documented, code, graphical assets (including the original vector masters), clever designs, design documents (including things like OmniGraffle originals), Medium and blog postings -often going into very deep detail about my designs, architectures -and the decisions while developing them-, GitHub issues and responses, Apple RADAR reports, and multiple published, high-quality, well-branded apps on the App Store.

Sheesh. Don't take my word for it. See for yourself.

Now, THAT being said, why doesn't everyone have a portfolio?

You ever see a designer or graphic artist going to an interview? They bring along these big black cases (although nowadays, it's probably more like a laptop case). These cases contain drawings, printouts, raw materials and sketches, etc. They are called "portfolios." Even students, fresh out of school, have them. It's considered a requirement.

No design firm on Earth would ever consider ignoring one of these, and instead, pull out a matchbook with "Draw Spunky" on the cover, and insist on that being the hiring criteria.

Which is EXACTLY what most software shops are doing, these days.

notyourday• 7 years ago

I think the vast majority of the issues come from the fundamental disconnect between what people want to believe they are hiring for ahd what they are actually hiring for.

I have been in this industry for over 25 year from being a grunt, all the way to three letter jobs. The most uncomfortable question in decision to open a job is "What are we trying to accomplish by opening this job?"

In my experience if one is cut through the posturing the answer is "We need two developers with the following credentials because we believe that by having those developers we can increase the output -- more lines of code that we need to generate to make feature we promised on a road map ship. They need to know Angular/Javascript/Css/Node/Go/React/Kafka/K8s/RMQ and be ninjas". Should one drill down what that feature is one would find "It is a new dialog box and if a user is to click this button some things will happen and X email would be sent. Our users want this. We currently use backbone but we will be using Angular in future. API is written in node but we have a few projects to explore Go and we are using RMQ for queues though there's a team that is tinkering with Kafka" That's what the manager is actually hiring for.

So the real job requirement is:

"Need to get two butts in seats to build a dialog box to trigger an API action that would publish a message to a workflow queue. Must know a bit about backbone/css/javascript, nodejs and rabbitmq" [0]

Recognizing that one wants to put butts in seats rather than someone who can spearhead the API move from nodejs to Go would change the entire dynamics. Hardcore interviews for the 1st make no sense -- if a person can think, you can teach them how to write some javascript, mangle some backbone code, create a new route in an existing Express application and publish a message to a queue.

[0] Look at the hiring ads here. It looks like everyone is hiring Chief Architects that code.

musicale• 7 years ago

Algorithm puzzles are a good test for the ability to solve algorithm puzzles (which usually means having seen a similar puzzle before or guessing the right trick.)

Which is probably correlated with solving problems in algorithm puzzle (aka "programming") contests.

If solving algorithm puzzles or competing in "programming" contests is your company's core business, then probably it's a good interview test.

In Defence of the Technical Interview

Related Stories