We analyzed 100K technical interviews to see where the best performers work

https://blog.interviewing.io/we-analyzed-100k-technical-interviews-to-see-where-the-best-performers-work-here-are-the-results/

by leeny • 4 years ago

206 226 4 years ago

blog.interviewing.io

rch• 4 years ago

Looks like Dropbox has trouble retaining tech talent?

leeny7• 4 years ago

Author here. We didn't publish a histogram showing how many users we have from each company, so I guess you'll have to trust me on this one. Dropbox has waaaay fewer engineers on interviewing.io per capita than some of the other companies... many of whom didn't make the top ten list.

In our experience, they're quite good at retaining talent.

horsawlarway• 4 years ago

Doesn't this exact comment undercut your results fairly heavily?

You're admitting that it's much more likely dropbox is a statistical anomaly because of fewer data points, rather than a robust data set.

cgearhart• 4 years ago

Or simply that they have significant bias in their interview process for selecting candidates with strong interview skills. At that point it’s tautological—if you only select people who ace interviews then you’ll have a bunch of people who ace interviews.

smrtinsert2• 4 years ago

I remember seeing a job description that was something like

  must be self starter, enjoy high energy postive environment!  enjoy coding in RoR etc

To me this reads like legacy Rails codebase people are too scared to touch that's always causing fires. No thanks.

overrun1114• 4 years ago

There has never been any rails code at Dropbox

jstx1• 4 years ago

There’s also the issue of selection bias. Maybe we’re just getting people who really feel like they need practice and aren’t an indicative slice of engineers at that company.

Or that your interview preparation platform prepares candidates better for Dropbox's interview process than it does for Microsoft's. Or that the people who were confident in their interview skills for Facebook decided not to use your platform. Or that these companies have different interview processes and selection criteria (they obviously do) so ranking "best" based on performance on different tests doesn't tell you that much.

There's hundreds of different ways to slice this data to come up with different hypotheses about what's actually occurring.

leeny7• 4 years ago

Author here. The data is mostly drawn from how people who work at these companies do in mock interviews rather than how our users do in real interviews with these companies.

tedivm• 4 years ago

The implication in your blog post doesn't make that clear-

At interviewing.io, we’ve hosted over 100K technical interviews, split between mock interviews and real ones.

wavesounds• 4 years ago

There's still the selection bias of who volunteers to do these mock interviews. Probably its the people who want practice interviewing and at dropbox those are the top performers who want to "move up" to a Google, while at Google it's the people who aren't cutting it and know they are going to have to find another job soon.

itronitron• 4 years ago

You should probably change the title of the blog as most gainfully employed people interpret 'best performers' as people that are very good at performing their job and/or trained for a specific Circus Act.

Something like "We analyzed 100K technical interviews to see which companies employ the people that we feel best performed in our mock interviews" would be more authentic

xtracto2• 4 years ago

The data and charts in the article look pretty nice!

One of the things I learned from my years in research/academia is that Design Of Experiments in itself is a pretty complicated task. Most experiments/studies are invalidated due to a huge amount of confounding factors and correlations that are not factored in for the experiments.

A cursory visit to r/science comments would show a lot of people who do science for a living providing valid criticism to published peer reviewed scientific studies due to wrong Design of Experiments procedures.

Having lived all this first hand makes me EXTREMELY resistant to take seriously the data, analysis and conclusions of the linked article.

Other than that, the effort is appreciated and I like the ideas behind interviewing.io.

hackernewds• 4 years ago

Or that less qualified people do not apply for Dropbox. It's a % acceptance rate almost - Harvard is good for having a low acceptance rate, not high.

unpopular42• 4 years ago

Alternatively, s/where the best performers work/where the best performers run from/

cscurmudgeon• 4 years ago

This study assumes that the companies have similar rates of people wanting to leave.

WYepQ4dNnG6• 4 years ago

I think people put too much effort and time in this whole interview business. My suggestion, based on my experience, is to spend a reasonable time like a month, brushing up on most common data structures and algo.

Then just take your chances. Rinse and repeat.

jonshariat• 4 years ago

Depends on your goal, some people want as many high offers as they can to drive up their comp even higher.

When you're young it might be worth it but to me the cost/benefit isn't worth spending months over preparing.

pizza234• 4 years ago

I have exactly the opposite opinion, for couple of reasons:

1. interviews are black boxes - there's no feedback; getting feedback from a professional interviewer is the best way to understand one's weaknesses

2. dealing with pressure in simulated environment can help (a lot, for some) to handle pressure in a real one.

I had no idea, for example, why I didn't fare well in a certain type of interview, until I did a mock one.

bckr• 4 years ago

I think interview practice with real interviewers and expert feedback is an excellent investment.

WYepQ4dNnG6• 4 years ago

My strategy is simple: I do interview practice by simply interviewing with companies that I am not really interested in joining. That allows me to practice not only the technical interview but also all other steps involved.

Last time I interviewed, I did 7 of these interviews, targeting: startups, pre-ipo, mid public companies, 2 of the FAANG I would never work for ...

I did that while prepping for interviewing at the companies I was really interested in working for. And eventually it all worked out and got a few offers, of which two from FAANG of which one from one of the two FAANG I really wanted to join.

chillacy• 4 years ago

based on my experience

I will throw out counter-experience that I did not succeed at technical interviews until I treated it like a fulltime job. In the past I just brushed up and failed Google's interview a handful of times before I decided enough was enough.

My takeaway ultimately is that how much time and effort one chooses to put in is entirely personal based on how much they want one of these jobs + what their gaps are in algorithms and data structures. Some people are going to need more effort than others.

b2000017• 4 years ago

the process is incredibly inefficient and disrespectful. people have already paid for university degrees and getting a degree is no joke, certainly not at a european university. graduating without programming skills is impossible.

what companies should do is ask you to walk them through a piece of code of your choice and discuss that and/or hire you for a month as a contractor and pay you a fair market rate.

prepend3• 4 years ago

graduating without programming skills is impossible

I’ve encountered many who graduated from bachelor and even PhD programs without programming skills (even from European universities), so your statement is false.

It also seems pretty weird to assume that a degree or certificate is foolproof for demonstrating programming skills. That seems like a lack of critical thinking skills.

ex_amazon_sde• 4 years ago

graduating without programming skills is impossible

I interviewed a lot of people and quite a few could not implement fizzbuzz.

overrun1114• 4 years ago

hire you for a month as a contractor and pay you a fair market rate.

Huge risk for anyone who currently has a job. I'd much rather the technical interview.

paxys4• 4 years ago

people put too much effort
spend a reasonable time like a month

Spending a month on interview training is exactly what I would consider unreasonable effort.

Here's a scenario – I have worked at big tech company A for a decade writing backend code. Big tech company B has many openings for senior backend coders, and is desperate to fill them. B's recruiters are hounding me every day to consider a switch. The job looks interesting, and the salary and benefits are great.

Should be a perfect fit, you say? Except if I interview with B today I'll get rejected at the very first coding stage. The barrier to entry they have created for themselves is me taking time away from my current job to solve programming puzzles, solving them perfectly in an interview setting, then throwing away all of that knowledge. They are never going to make me go through this effort unless I am truly unhappy at my current role.

This is the exact reason companies are finding it so hard to hire engineers, and why they have to pay them so much to switch.

vkou• 4 years ago

Have you ever ran interviews? I have, and there's an incredibly wide range of candidates, all of whom with fantastic pedigrees, but some of whom are completely incapable of demonstrating their problem-solving ability. Likewise, some candidates have poor pedigrees, but make it clear to me that they can think, communicate, and code.

I'm not saying that some of my rejections can't do that. But I can't measure what I didn't observe.

I also get mountains of emails from recruiters about how deeply impressed they are with my skills, and how I should go work for their firms. I know that they are completely full of shit, because from looking at my LinkedIn, they have no idea whether or not I have any of those skills.

gcheong9• 4 years ago

This is why I don't believe it when companies say they are desperate to hire.

hiq3• 4 years ago

If B is desperate, they'll just increase salary and benefits until enough people able to pass the interview start applying. Adjectives like "desperate" don't reflect what actually happens: they priced the cost of possibly not hiring soon, of using a different hiring method and of changing the benefits, and they computed (badly?) the optimum to be where they are.

If they become more desperate, things could change, but they don't and things stay the same.

kieselguhr_kid10• 4 years ago

This is a lot easier to say when you're established in the industry and you have a "feel" for interviewing. When you're breaking in, you have to learn all the computer science shibboleths and industry jargon before people will take you seriously as a candidate.

agumonkey2• 4 years ago

It's insane to me that we have to rehash everything every time. You graduate, you leetcode, you rehearse for a round of interview, and then you have to that all over again. So boring.

commandlinefan4• 4 years ago

They all mostly ask the same sets of questions, too. If you get turned down by one interviewer, just keep track of what questions they asked, go look up the answers to those questions and you'll most likely be asked the same question by the next interviewer.

lmarcos• 4 years ago

A month? That's what I would consider "too much effort". What I do is:

- read the whole website of the company I want to work for

- apply

That's it. If after N years of experience I'm not able to pass the tech interviews then I'm not a suitable candidate for them, and I simply accept that. To me, spending a month memorizing and refreshing concepts I learned when I was at uni, feels like cheating (because I'll forget what quickly studied in a few days).

I do "study" on my own constantly, though. I'm always reading tech books and trying new things. I do it at my own pace. Perhaps that's the reason why I don't prepare myself for interviews.

selcuka2• 4 years ago

If after N years of experience I'm not able to pass the tech interviews then I'm not a suitable candidate for them, and I simply accept that.

I wonder if you are thinking of the same tech interviews that anybody else here is talking about. Those tests are almost inversely correlated with N years of experience. Without specifically studying for those tests you would definitely score lower than an inferior (but well prepared) candidate.

adossi• 4 years ago

No Facebook/Meta? That’s a little surprising to me.

bagels• 4 years ago

In all the graphs save one.

sirmike_• 4 years ago

Honestly the only thing I see out of this is that Dropbox is a huge paying customer of this service.

robbywashere_• 4 years ago

The fallacy to these findings and conclusions is that you're measuring processes of your own brand and flavor; the biases are so deep you likely don't recognize them. 'technical', 'problem solving', 'communication' these are all of your own construction, setting, presentation and scoring. Look at the failings of the IQ test for example.

I'm curious if people just really need to learn the talk and the walk of a silicon valley employee to land a job at a FAANG.

pas• 4 years ago

Look at the failings of the IQ test for example.

can you elaborate on that please?

stale20022• 4 years ago

The fallacy to these findings and conclusions is that you're measuring processes of your own brand and flavor
Look at the failings of the IQ test for example.

You are mis-interpreting the point of all of this. This website does practice interviews. It doesn't measure how good of an engineer someone is, or how smart they are, and they don't claim to do so.

Instead, it measures how good they are at tech interviews. And for that purpose, the study works well enough.

seoaeu• 4 years ago

It would be interesting to show error bars on the graphs. Seeing 95% confidence intervals would give a much better idea of whether we're looking at meaningful signal or just noise

nevermindiguess• 4 years ago

I would never want the best developers. Imagine the giant pain in the back of these primadonna always thinking they can get paid more in their next job. And how you should give them this or that, or how you should change your development stack, language, processes, chair color, etc. I only need very few geniuses and a lot of normal ones. 80/20, remember?!

vdnkh• 4 years ago

Why do you think the best engineers are hard to work with?

1270018080• 4 years ago

Your definition of best engineer doesn't sound like they're the best engineer.

mcbuilder• 4 years ago

So, the best developers are always primadonna's now? Maybe the ones who grind leetcode all the time because they have something to prove are the primadonas, and you are somehow selecting for them.

For skills, balanced team makes sense. You put a 10x type person in a room full of 0.5x people then yeah they'll start complaining about the chair color because the job sucks and they would kill for the chance to get on a better team.

b2000017• 4 years ago

this

abeppu• 4 years ago

The good news for you is if the target is to hire mediocre developers, you're likely to hit it.

criddell2• 4 years ago

Damn, you must have had some rough hires.

The very best people I've ever worked with don't fit any of those characteristics you've mentioned.

Although I will say if a developer of yours wants a different chair, get them a better chair. They aren't that expensive and they are durable. Sometimes the overhead of getting $1000 purchases approved adds up to almost the same as the purchase itself. Give your developers some leeway to order equipment, software, books, etc... without approval up to some reasonable limit. It won't cost the company that much and your people will be happier and feel trusted.

Oras• 4 years ago

always thinking they can get paid more in their next job

Isn't that the definition of career progress? I don't think it applies to developers only.

vmception• 4 years ago

If you’ve hired engineers from some of the companies in this post, have they performed better than others?

It should be: have you hired engineers from interview.io

This author is sadly blinded by the echo chamber of their own creation

b2000017• 4 years ago

you should have analyzed 100k interviews to figure out how to reverse the leetcode trend

seabriez• 4 years ago

"best performers work" With a title like that you really just cannot take this study seriously lol. Not to say it's not interesting but that is one crazy claim, at best title should be "most effective interviewees." Also, I work in FANG but signed up for this website and can't even participate, so how you chose all these candidates is also questionable.

tmp_anon_22• 4 years ago

The logic being "They work at Google/Dropbox/etc... therefore they are the best performers" really irks me.

yrgulation• 4 years ago

The author confuses successful products with high performing tech and high performing engineers. I’ve met many brilliant indie engineers that build highly performant code that almost no one knows about yet it provides these people with a steady stream of income and they would never work at faang or unicorns.

criddell2• 4 years ago

I was thinking the best performers may very well be at one of those companies, but there's a pretty good chance that they never did any kind of coding interview. Maybe they were hired through a merger or acquisition or perhaps their reputation or portfolio is more than enough.

overrun1114• 4 years ago

I don't see this claim being made anywhere. The article claims the opposite: the best performers on their technical interview happen to be from Dropbox/Google etc.

spupe• 4 years ago

Many others have commented on possible biases here. That's true, but I also find this a really interesting observation based on a huge dataset. Once you can correlate it to actual job performance - performance reviews, whether you were fired or not, maybe even salary -, it might be one of the best ways to see if technical interviews actually measure anything significant.

sublimefire2• 4 years ago

Every big corp has that data. Remember Google ditching brainteaser questions due to the lack of correlation between the interview and job performance.

kderbyma• 4 years ago

this is laughable to me. Dropbox is so bad on all accounts. it's my least favourite cloud storage product

hparadiz• 4 years ago

They used to be the best but then ran their client software into the ground. Weirdly the Linux client is still the old school one and since I only use Linux it's fine for me.

magneticnorth• 4 years ago

This is super interesting! Worth noting that another possible title is "... where the best performers were trying to leave during the study timeframe".

Really interesting to see dropbox so high - would be curious to see some other data to corroborate that they (at least used to) employ the best engineers.

From my time interviewing, I've seen clusters of very good candidates often be more reflective of which top companies were having a hard time, internally or externally. There was a while where my company hired a lot of people from Uber; right now we're getting Amazon and Facebook/Meta.

WYepQ4dNnG6• 4 years ago

hired a lot of people from Uber; right now we're getting Amazon and Facebook/Meta.

good luck with that

lhnz2• 4 years ago

It's interesting that Google was not higher for communication. Aren't they known as the big tech company with the "writing culture"? I guess 'communication' means something different here.

jldugger2• 4 years ago

Aren't they known as the big tech company with the "writing culture"?

Are they? I thought that was Amazon.

lhnz2• 4 years ago

Oh interesting. I'd heard about Google's code review culture and thought there was lots of other written content, too.

renewiltord• 4 years ago

I wonder if people give folks high communication scores as a consolation prize. Like, “he didn’t do well on anything else but he can talk well”.

Anyway, interesting results. Let me do some Dropbox outreach and see how they are.

duxup• 4 years ago

I wonder about when you don’t need “the best”. Everyone needs solid role players too…

Nobody ever talks about getting good, but not “the best”.

chewbacha• 4 years ago

Oof, non-zeroed charts are very deceptive in enlarging differences that might be within the error margins.

jenoer• 4 years ago

I was curious what one of these would look like as a "regular" chart.

Here's what it looks like: https://imgur.com/a/BkzEoGV

mooreds2• 4 years ago

Thanks for that. Still looks significant, but much less striking.

drc500free• 4 years ago

In this case, is 0% passing really a meaningful bar? You could flip the chart upside-down and do it on fail rates, and the differences would look like this.

kieselguhr_kid10• 4 years ago

I'm glad to see this getting roasted in the comments, as it's a really good example of how companies put out self-serving pseudo-statistical nonsense in an effort to promote themselves.

There's no effort to quantify what "technical" or "communication" skills are - these are left to the interpretation of a the interviewer. It makes no effort to show where these engineers are going, what the interview process they're completing looks like, what impact demographics had on this, etc.

I find this stuff repugnant. It perpetuates the myth that there's something really special about Silicon Valley engineers, while making only lazy and perfunctory efforts to examine any alternative explanations than "this is where the rockstar ninja coders work." Shameful.

ttyprintk• 4 years ago

At least they left in the "Unemployed" category.

rootusrootus2• 4 years ago

It perpetuates the myth that there's something really special about Silicon Valley engineers

Does this myth still have much traction? If anything, my general regard for engineers in the bay area has steadily declined in the last few years. There are so many really worthless folks who have only figured out how look like they have a clue, but go any deeper and they flail. I know I'm painting with a broad brush, and that's not fair, but most of the great engineers I work with are at various other places around the country, not California.

kieselguhr_kid10• 4 years ago

Personally, as a (now relocated) Bay Area native, I agree. Over, I think there's still a lot of prestige for large Silicon Valley firms, even if some of the gloss has justifiably started to fade.

This article certainly assumes that myth is still in place.

kjeetgill2• 4 years ago

Hey, it's not our fault! Where else are posers gonna flock to to pose? I swear, even with all the remote work, more than 95%* of the people that worked here 5 years ago still do. We live here.

* made up statistic.

emerged2• 4 years ago

I know a ton of engineers. Of them all, those working FAANG are profoundly less skilled than the others. It’s impossible to miss its so obvious. Maybe my social network is an outlier, but I really really doubt it.

ummonk5• 4 years ago

"There's no effort to quantify what "technical" or "communication" skills are - these are left to the interpretation of a the interviewer."

And yet, these scores are measuring something and averaged across tens of thousands of technical interviews, you have enough statistical power to average out the particularities of each interviewer.

I'm sorry you find the results repugnant, but the results are what they are. And the article did have a large section on limitations of their analysis.

kieselguhr_kid10• 4 years ago

It's not the results I find repugnant, it's the assumption that the results have any real world validity. The _something_ they're measuring is as likely to be demographic biases as it is technical or communication skills.

leeny7• 4 years ago

Author here. Yes, the skills are left to the interpretation of the interviewer, but most of our interviewers are senior engineers at FAANG. We've done quite a bit of work internally to make sure your interviewers are well calibrated, and we have a living calibration score for each one (calibration is based on how the interviewees they interview end up doing in real interviews).

The interviews in question are a mix of algorithmic interviews and systems design interviews.

Also, if I ever use "rockstar" or "ninja" in my posts, I hope someone finds me on the street and punches me in the face. I'd deserve it.

TurkTurkleton• 4 years ago

Also, if I ever use "rockstar" or "ninja" in my posts, I hope someone finds me on the street and punches me in the face. I'd deserve it.

Are you sure you'll never find yourself blogging about bands' flamboyant frontmen (or frontwomen), or covert agents in feudal Japan?

kieselguhr_kid10• 4 years ago

Is there any statistical reason I should assume an interviewer at FAANG has some kind of special insight into what good technical and communication skills are? Were any attempts made to adjust for demographic biases? Is it possible, for instance, that Dropbox engineers are disproportionately taller white dudes with nice hair? I know that sounds a bit unserious, but all of those factors would increase the likelihood of a higher score.

I appreciate that you never used "rockstar" or "ninja". That dig was a bit unfair of me.

bryanrasmussen2• 4 years ago

I suppose Douglas Hofstadter is en route to punch you right now

CobrastanJorji4• 4 years ago

FAANG interviewer here. I've conducted many hundreds of interviews for multiple FAANG companies. The totality of my training in how to interview is about 4 hours (when I combine the training of each company). I have zero confidence in the usefulness of calibrating my answers and expect that neither I nor anyone else who does this would reliably score the same person with the same score most of the time, outside of a small percentage of outlier candidates.

junon3• 4 years ago

senior engineers at FAANG

This means absolutely nothing. It's a well known fact that SV titles hold next to no weight.

potatolicious• 4 years ago

"but most of our interviewers are senior engineers at FAANG"

Wait, so the result is that "interviewers from FAANG companies rate highly interviewees from FAANG (and FAANG-adjacent) companies"?

Or maybe more causally: "those who have already passed FAANG-style interviews are more likely to pass interviews conducted by FAANG people"?

I appreciate the mission here - but if the idea here is to give people a fair shot even if they don't have the pedigree of FAANG, building FAANG interview styles into the system seems counter to your stated goals. If anything these results are concerning - you can interpret the findings in (at least) two ways:

- these companies hire or produce superior engineers, the results you got are indicative of a broader higher caliber of engineer in those companies.

- the interviewing exercise is optimizing for "people who can pass/have already passed FAANG-style interviews", which rolls in all of the myriad biases of FAANG hiring and perpetuates them.

psyc• 4 years ago

While I agree that this particular exercise is riddled with problems, I simply cannot image Hacker News rolling over and accepting evidence-based answers to questions of this nature, regardless of where the data came from or what the methodology was.

tomp• 4 years ago

um are we looking at the same graphs?

the second best performing company on "technical" and "problem solving" was Bloomberg, literally the opposite of Silicon Valley

rocgf• 4 years ago

While there may not be any reliable measure of communication skills across the industry, the fact that the data was based on scores given my a large amount of people, that by definition means that it's accurate.

Think about it carefully - if people rate people as being good at communication, then there is no reason to quantify it any other way. There are some obvious flaws here, like the quantity of data and it's normalization, but it's basically a tautology.

ClumsyPilot• 4 years ago

"Think about it carefully - if people rate people as being good at communication, then there is no reason to quantify it any other way. There are some obvious flaws here, like the quantity of data and it's normalization, but it's basically a tautology."

So if a lot of people rate you as a great leader, you are a great leader? Even if you lie to their face and delover terrible results? Objective reality doesn't matter?

So the greatest leader in the world is in North Korea?

Communication is a clearly measurable skill. Just because a lot of people have been sold a lie, that doesn't make it true

overrun1114• 4 years ago

It perpetuates the myth that there's something really special about Silicon Valley engineers

Why are you sure it's a myth? My prior would be to believe that engineers at the most exclusive companies with the highest hiring bars that pay 3-5 times more than average would be better programmers. The article is just one data point confirming what intuitively should be true.

If Silicon Valley engineers are no better than anywhere else then someone should notify the execs at FAANG, I'm sure they'd be interested to know they are dramatically overpaying for talent.

I don't understand what is so uncontroversial about this. SV companies recruit the best talent from around the world and it's where the best talent wants to work. Similarly, the best financial talents are in NYC and London, the best actors are in Hollywood etc.

kevin_thibedeau• 4 years ago

There are capable people who don't want to live where 1000sqft home costs $1M. Never mind all of the other problems with the region. Selection bias doesn't prove anything about the people you aren't selecting for.

kieselguhr_kid10• 4 years ago

I've lived in Silicon Valley and I've lived outside of it. I'm sure it's a myth. The higher rate of pay is not an indication of higher skill.

xtracto2• 4 years ago

It perpetuates the myth that there's something really special about Silicon Valley engineers

In my [disillusioned] experience, this holds true: Silicon Valley engineers are very good for building throwaway MVPs that they won't have to maintain more than 3 years.

I've been very disillusioned by the quality of the software written by Silicon Valley companies, but in hindsight it makes sense: "Run fast break things" development culture resonates with the "raise VC money every 18 months" business culture, and then look for an exit in 5 years tops. There is no incentive in Dev or Business to really develop good software.

decebalus1• 4 years ago

I cannot take any of this seriously.

- First of all, it assumes that `interviewing.io` is some sort of certification standard (which I'm willing to bet is the actual point of publicizing these 'studies' in the first place. It's 'fact' manufacturing)

- Then there's selection bias about engineers actually using one platform vs the other

- Touting the data set size in order to give the 'study' some credibility is a red flag for me. You can analyze millions of the same technical interview and deduce all sorts of conclusions.

- The use of 'best performers' is deceitful. It means 'best performers in the interview context'. But using it in the context of where do they work, it implies something like 'the best performing engineers are at company X'. Which is bullshit. More like 'best trained engineers to pass these interviews work at company X'.

Garbage. I'm flagging this as it's nothing more than self-serving marketing.

mooreds2• 4 years ago

Or maybe this is a signal to short dropbox, since so many of their high performers are interviewing?

laluser• 4 years ago

I work there. Our attrition is not any different than other companies. We have magnitudes less engineers than the other companies mentioned, so most likely it's the few that interview on average are better from Dropbox than the average of other companies. That's all.

listless• 4 years ago

Does...does Dropbox own interviewing.io?

dxbydt• 4 years ago

I honestly don't care which company retains the best performers. Is none of my business. Maybe if you are an investor in these companies, might be a useful signal to know such trivia. But from a candidate standpoint, this page is super useful - https://interviewing.io/recordings

Simon_O_Rourke• 4 years ago

If Dropbox engineers are so good across the board, why are they interviewing at all? Surely the desire to move away from this kind of technical nirvana would be almost zero?

jldugger2• 4 years ago

Dropbox was famously leading the SWE comp charts[1]. In particular, they focused on equity grants both pre- and post-IPO.

Well, live by the equity sword, die by the equity sword: SP500 is up 16 percent in the past year, DBX is down 10 percent. Even over the long term, DBX is down 15 percent since their IPO 5 years ago, while SP500 is up 77 percent. It's the kind of performance that results in layoffs[2], which will probably hit your most expensive / experienced engineers first. Glassdoor ratings suggest remote work did not go well when combined with the layoff.

[1] https://medium.com/@paysa/thinking-inside-the-box-spotlight-... [2]: https://www.reuters.com/article/us-dropbox-layoffs/dropbox-t...

paxys4• 4 years ago

What this really tells me is that the best engineerineers at Dropbox are looking to quit.

sublimefire2• 4 years ago

^^^ this LOL

agumonkey2• 4 years ago

Interesting interpretation :)

But where would the best go ? where the worst engineers are ? to appear even smarter ? or just a bit below ? to avoid toxic workplace.

i_love_music• 4 years ago

Yes, best interpretation yet. This data is whack. I would take it all with a massive planet size grain of rice.

pgroves• 4 years ago

And the implication is the 'quality' of engineers at the companies is actually reversed - the top performers at Dropbox are struggling and leaving while the under performers at FANG are struggling and leaving.

gcheong9• 4 years ago

"If things go well, they skip right to the technical interview at real companies (which is also fully anonymous)."

So the most I'll get out of this is to skip the initial screening interview and jump straight to the "real" interview? Or do I even get that?

toddm• 4 years ago

So our best and brightest are working for a company whose product is a GUI for an FTP site. Good to know.

commandlinefan4• 4 years ago

our best and brightest

interviewing.io's best and brightest.

gcheong9• 4 years ago

"Of course, the really interesting question in all of this is the holy grail of technical recruiting: Does performance in interviews reliably predict on-the-job performance? "

And until you can really say for sure this is the case, any speculation about the value of technical interviews other than just being a barrier to entry is really moot.

w0mbat• 4 years ago

Why does Dropbox software ship an amateurish Electron app then?

brimble• 4 years ago

Sometimes I look at the credits lists for very impressive, native 90s desktop programs, that were made in a reasonable amount of time by a small group of people who, at least to some degree, had worse tools than we do now, for a smaller market and lower pay. Usually to be shipped on CD, so they even had to mostly work and not have too many bugs at launch, on top of that. And I wonder WTF happened to our industry and what the armies of programmers at all these companies are doing.

subsubzero• 4 years ago

Dropbox was known as a hard place to interview since 2014, the tech companies that had the hardest technical interviews are Quora, Palantir and Dropbox(honorable mention fog creek)(this was from a few years back so things may have changed). Just because the company makes it extremely difficult to get in does not mean the company is generally all around awesome or pays great or employs the greatest engineers. It optimizes for people who generally come straight out of an elite CS program with those learned concepts fresh in their mind, and for people who grind out on leetcode or who are great at interviewing. Of the three companies I mentioned above I would not work for any of them now.

paxys4• 4 years ago

Yeah "we are more selective and exclusive than Google and Facebook" was Palantir's whole schtick when they originally took off. And a lot of very smart engineers bought it.

Turns out it takes a lot more than a high leetcode bar for your interviews to run a successful company.

FranksTV• 4 years ago

When developers change roles so frequently, being an employer with a reputation for being full of smart people is a good recruiting tactic.

Not because you get good work done, but because people will apply because you look good on a resume.

babyshake• 4 years ago

Are you suggesting that these jobs don't mainly consist of quickly banging out dynamic programming and DFS/BFS traversal algorithms as quickly as possible while a stranger stares at you? From interview experience, I assume this is what engineers mostly do at these companies all day.

mrits• 4 years ago

We make you prove you know how to implement things correctly in the off chance we ever give you enough time to do so.

vonseel• 4 years ago

My brother is an MD and I showed him some Leetcode prep material since I am studying for interviews. Specifically, a problem from Grokking the Coding Interview.

His response - “It would be helpful if they then showed how that is used in real world code“. I had to tell him I don’t think these kinds of scenarios come up often in real-world use cases, haha. They are essentially coding puzzles to filter for people who are good at solving coding puzzles, which may or may not directly translate to being good at writing application code.

at-fates-hands• 4 years ago

There's a local agency here in the city I currently reside in that touts they're hiring percentage is lower than Harvard's acceptance rate. They're telling you up front its really hard to get a job there and its a very "bougie" place to have on your resume - as if they're on the same level as the FAANG companies.

The interview process is fairly long, accompanied by an office tour which touts the numerous "Freebies" they offer their employees. Then you find out after the entire process that you'll be getting paid 4050K LESS THAN the industry average. At the time I interviewed with them, I had five years experience in UI/UX and mobile development. What they offered me was essentially what I was making as an entry level dev.

It was easy to turn them down. No amount of free Red Bull is going to pay my mortgage.

alskdjflaskjdhf• 4 years ago

Since a good bit before 2014, even. They used to recruit very heavily from MIT (while Palantir was overrepresented at Stanford and Quora at Harvard--all of these reflecting the almae matres of the founders). Note that this was before the popularity of Leetcode and the whole cottage industry around trying to game algorithmic-type interviews. I'm not sure if similar companies founded today would push these algorithm-heavy interviews as hard, since they've probably lost some signal now & prevailing attitudes have changed a bit.

At any rate, it doesn't surprise me at all that Dropbox engineers do better than FANG engineers on these technical metrics. The average Dropbox engineer is almost certainly a bit smarter and a bit better at algorithms than the average FANG engineer. Of course those attributes don't automatically translate into being a better engineer, though, nor do they automatically translate into company success or anything.

jameshart2• 4 years ago

All this post says is that among the candidates who scored highest on this model, a higher proportion worked at these companies.

But there's no inverse analysis: of people who worked at these companies, how predictive was that overall of a higher score on this particular assessment?

'Our five highest scores ever were all people who wanted to leave FooCo' tells you little about the overall quality of FooCo employees. Maybe the rest of them are terrible and these five needed to get away?

ummonk5• 4 years ago

No you're misinterpreting it. Go back and reread the graphs.

jameshart2• 4 years ago

Ah, you're right. The graphs don't actually show the data they describe their methodology as collecting. That fills me with confidence.

valleyjo• 4 years ago

My conclusion so that Dropbox has a lot of smart people who want to leave. Higher numbers of interviewees means more folks want to leave and are practicing to do so.

hussainbilal• 4 years ago

Rather than showing the top ten for statistical significance, wouldn't it make more sense to use PCA on the ratings and show each component's top 10 instead?

gcheong9• 4 years ago

Does this happen in any other field? If I was a doctor and wanted to work at some other company, would I need to study the MCAT every year in order to pass a screening interview based on one possible question? What's the equivalent in other fields? Closest I can think of is an acting audition but even then they give you the script beforehand. I'm beginning to think that the industry somehow settled on this approach not so much as a skills verification process but, by making the process so onerous on the candidate, talent retention is a lot easier.

visarga• 4 years ago

I'm beginning to think that the industry somehow settled on this approach not so much as a skills verification process but, by making the process so onerous on the candidate, talent retention is a lot easier.

There are plenty of applicants who just can't do the job, and the cost of restarting the search all over again is high. It's important for a team to vet their new hires.

commandlinefan4• 4 years ago

applicants who just can't do the job

While that definitely matches my observations, the current interview grind:

a) filters out people who can do the job too

and

b) somehow also manages not to filter out people who can't, either.

driverdan• 4 years ago

Welders typically have to demonstrate their welding skills. That's the only one I can think of though.

dgunay• 4 years ago

At least a welder actually welds stuff during their day-to-day. Still looking for a job that needs people who can implement efficient graph traversals though.

overrun1114• 4 years ago

Software engineering is the only field I know of where someone with zero pedigree or education can make as much as a doctor. If you can pass a grueling technical interview then you can get a high paying job (200-500k+) no matter who you are or what your background is.

The tradeoff is that you have to go through the same grueling process every time you change jobs. I think the tradeoff is well worth it and would not like to see software become more like medicine where you have to put in 10+ years of schooling to prove your worth.

gcheong9• 4 years ago

I don’t need a 200k+ job at a FAANG, but it doesn’t seem to matter as FAANGS are perceived as having the best hiring practices (because they wouldn’t be so successful otherwise, right?) so the entire industry just blindly follows suit.

ThrowawayR22• 4 years ago

"If I was a doctor and wanted to work at some other company, would I need to study the MCAT every year in order to pass a screening interview based on one possible question?"

If you were a doctor, you would have gone through 4-5 years of supervised residency after medical school (https://en.wikipedia.org/wiki/Residency_(medicine)) to guarantee some minimum level of competency before becoming a practitioner. Pay is notoriously poor and the hours are extremely long.

Programmers have it exceedingly good and the only reason for that is that software is still in its growth phase. I doubt we'll all be still riding the gravy train in another 25-50 years.

gcheong9• 4 years ago

Yes but once you get through your residency your expected earnings go up quite a bit and you don’t go back to square one. If someone had told me that I’d still need to prep for leetcode interviews at age 55 I’d have chosen the MD path given a choice, but I’m also old enough to remember the days when you could get a decent coding job based mainly on your experience so from my perspective things have in general gotten demonstrably worse in terms of process for anything other than a lottery-ticket FAANG position.

steven-xu• 4 years ago

Kudos to interviewing.io to share this analysis. I agree with the many issues in methodology and analysis that others have raised here, and I agree there's a risk that a face-value reading of the blog post is highly misleading. But this is true for all data, and poo-pooing the analysis without crediting the sharing just leads to less sharing. To be clear, I'm supportive of the criticism, but let's also give credit where it's due.

Technical interview performance is a high stakes field for which almost all data is cloistered in individual hiring companies or secretive gatekeepers. In my mind, all efforts, even imperfect ones, to share data is a great step here. We should encourage them to continue to share, including pursuing options to anonymize the raw data for open analysis. The field deserves more transparency and open discussion to push us all to be better.

lowbloodsugar• 4 years ago

How is it that Amazon was better than both Facebook and Google overall, but worse when comparing Technical, Problem Solving and Communication. What are the other undisclosed ratings that makes Amazon better overall?

lamontcg• 4 years ago

Going to be real interesting to see how I do at technical interviews whenever I decide to jump back into the job market.

At the job I'm leaving tomorrow, I just did a 2 hour long video session / training where I started by teaching how to read call graphs, and that led to over a hour of me trying to sort out a horrible performance issue in real time.

The problem, solution and the iterated debugging to deal with all the edge conditions that the extensive unit tests called out (and I wrote all the unit tests that blew up, so I get to take credit for all that -- although I also wrote the bug I fixed) should show that I'm very high functioning engineer. And I had identified the problem previously at a higher level and had a fix that papered over the problem, but during the video I correctly figured out that the real source of the problem was deeper in the code, and had existed before the change which surfaced the problem, and managed to do a data-driven analysis to track down the perf bug and go from 15% of CPU time in one subsystem to 1% of CPU time in the same subsystem for a 15x speedup on my problem (and probably closer to a 90x speedup for the customers who were reporting it--including a large customer everyone is familiar with here due to headlines they're involved in).

Meanwhile I forgot that it was obj.send(:method, *args) in ruby and tried to obj.call(:method, *args) and had to look that up because my brain was derping a bit on that, and the night before I forgot it was JSON.generate in ruby and not encode/decode and just in general my brain is a mash of too many different programming language syntaxes. At one point I caught myself trying to use `%` for comments because I had been doing Matlab writing an Iterated Chebychev Picard Method IVP ODE solver the prior weekend. If I can't work with the command line or an IDE and with google I'm just going to be a mess of trivial mistakes due to crossed wires.

I've also never reversed a linked list in my life and the correct answer to that question is probably to never use a linked list due to TLB cache thrashing at the very least.

pnathan• 4 years ago

I always enjoy your work, @leeny. I remember quite a few years ago you were working as a recruiter and I liked working with you then, although we didn't wind up with a placement for me.

Best wishes!

k__• 4 years ago

Now throw in a bunch of engineers from one of the countless small businesses so we get a feeling for what those numbers actually mean.

calchris42• 4 years ago

Despite all the caveats / problems on statistical significance and methodology, 1 key takeaway:

“Unemployed” engineers communicate better than those at Uber, Twitter, Amazon, and Google.

oneepic• 4 years ago

Not to mention every other company outside of the top 10.

numbers• 4 years ago

This reminds me, a friend of mine recently indicated that Dropbox has become an Amazon graveyard due to the mass exodus of talent from Dropbox to other companies that feel like Dropbox did about 5 years ago. Without their perks and fancy food, Dropbox is a boring product company with not much upside at this point.

nosefrog2• 4 years ago

As an ex-Dropboxer, Dropbox asks legitimately tough questions. I only got in because I got asked the exact set of questions that I could figure out the answer for, once I joined and went through interview training I realized I would have failed about half of the questions that Dropboxers ask.

I'm also not sure how it is at other companies (at Google but haven't gone through interview training yet), but Dropbox's rubrics are also pretty strict. Doing "well" on a question requires getting through multiple parts normally with close to zero bugs that you don't catch yourself.

paxys4• 4 years ago

You just described the entirety of the tech hiring experience. Sure some people are bonafide geniuses that can crack the hardest problems in their sleep. The majority of tech workers, however, are ones who got lucky with the specific combination of questions asked in the interview. Maybe they had seen the question before. Maybe the solution just "clicked". This is why the best strategy for acing such interviews is – just apply to a lot of companies.

ctvo2• 4 years ago

Of course, but this is a gradient. Struggling startup may ask a question that 80% of engineers are capable of answering (through luck, exposure, experience -- whatever dimensions).

Dropbox asks difficult questions, and it's hard to discern why. I don't believe the problems at Dropbox are particularly difficult relative to its peers. I don't think they innovate at a clip that's outsized, etc.. But they do this, and their engineering culture focuses on this.

3minus1• 4 years ago

This just sounds like Dropbox, Google have a similar interview to interviewing.io. So if you're able to clear the Dropbox interview bar, you'll do well on their test. It doesn't really say anything about general work performance, only performance on a specific interview.

MathMonkeyMan• 4 years ago

There's something that I like about the writing style. I can't put my finger on it. "Sober," maybe.

bradlys• 4 years ago

Interesting how everyone here reads the title differently than me.

I read “where the best performers work” as not where do the best coders/employees/whatever work… I read it as people who perform the best work. Where do people who do the best performance work? And in the context of interviewing - I don’t find this particularly weird to look at. Interviews are a performance.

That said - I’m apparently a weirdo!

Also, limiting this to companies with 50+ employees who have used the platform is going to definitely only let very large companies in the analysis. Even where I’m at now (1500 eng) I would be surprised if 50+ had used the platform because 75%+ were hired in the last 1-2 years.

Also - the way everyone here critiques the rating system - what a joke. You think the rubric at these big companies is really any better? You just find a way to shove your core feeling into the rubric and that’s it.

mbrodersen• 4 years ago

I have worked with a few developers who went on to work for FAANG companies. They were OK but not the best I have worked with. They did however have an inflated opinion of their own skills. Perhaps that helps when interviewing?

We analyzed 100K technical interviews to see where the best performers work

Related Stories