Amazon's Current Employees Raise the Bar for New Hires

http://online.wsj.com/news/article_email/SB10001424052702304753504579285133045398344-lMyQjAxMTA0MDAwNzEwNDcyWj

by glaugh • 13 years ago

45 71 13 years ago

online.wsj.com

armon2• 13 years ago

I've worked at Amazon, both in the warehouse management division as well as AWS, and the whole "bar raiser" phenomenon is mostly just marketing fluff. There are a handful of engineers who don't mind interviewing (or at least hate it less than most), and they get picked often to be part of the interview circuit. After some number of interviews, you are considered one of the mystical "bar raisers". However, in my experience, they are not any different from other interviewers. More experienced sure, but the difficulty of questions is highly variable, and honestly there isn't much culture at Amazon to worry about culture fit.

RankingMember2• 13 years ago

Can you compare it to any of the other places you've worked to give some perspective to your comparison? Like "isn't much culture" compared to working at IBM or "isn't much culture" compared to Google/Zappos/RedGate?

jw_2• 13 years ago

I found there was a culture of constant overtime and firefighting. I also found they made a point of letting people know they were replaceable when they came onboard.

It's a big company though - my experience at other large corporations was that this sort of thing varies wildly from team to team.

The only other "culture" at Amazon was the constant cult-like references to a set of principles - people would actually talk about "bias for action", "frugality" and whatever in their normal daily conversations. It was kind of strange.

Happy to be working elsewhere now.

hirammcdaniels• 13 years ago

Amazon has a certain body of engineers that get into the official culture, get involved in broomball and wing-eating competitions and the like, but there are far more engineers who pretty much ignore the whole thing. As far as I could tell, the biggest piece of their culture is now a bunch of bitter engineers on internal mailing lists slowly working up the courage to leave for more enjoyable companies.

That said, different teams at Amazon were SUPER different. Some teams went out drinking together twice a week, some teams were super extraverted and into video games, but many other teams were a bunch of quiet, serious introverts. TEAM fit was super important. This of course absolutely sucks for you if you get roped into interviewing for one team and you're a much better culture fit for a different team. Too bad. Rejected.

IBM has a culture, it's just a boring culture. Pretty much a bad IBM culture fit is a person who's passionate about something in life. I'm sure there are parts of IBM that are different (it's huge), but I was just glad that I didn't have to wear a suit.

Google-like companies are pretty much the opposite of IBM. If you don't have some weird interest you're passionate about, you kind of stick out.

armon2• 13 years ago

I've also worked in university research group settings, as well as a few startups in SV. I'd say there is a much stronger sense of the "whole" in these smaller groups, where everybody feels personally responsible for the success of projects. The smaller groups also take on the dynamics of the members, and the more eccentric personalities shape the culture and make the group a more fun and interesting place. You think of coworkers as friends first, coworkers second.

At Amazon, you don't get any of that. It is a very dry culture. It's clear that everybody is a coworker first. Eccentricity is largely looked down on (after all you need to worry about reviews and promotions, etc). Instead you get a sort of blah corporate culture, everything is grey, you talk about frugality as if somehow that's great. Rarely do you find super motivated engineers. Most people are fairly jaded, and aren't that committed to the success of projects. It's just a 9 to 5.

ignoramous5• 13 years ago

Bar Raisers do go through a round of training every year or so. Regular interviewers themselves are encouraged to attend a program called 'Making Great Hiring Decisions' every once a year or so.

AFA engg roles are concerned... because the interview process is distributed, and because Amazon is growing too big too fast, there is a high chance that a lot of 'B' candidates get hired, but the general consensus is that those 'B' candidates can't survive the break-neck pace of innovation and that they'd generally not be able to match up, let alone keep up with other performers and do the work that's expected of them exceptionally well. Since promotions are merit-based, and due to the stack-ranking employed by Amazon to rank the enggs, things only get diff for 'B' candidates.

Daniel_Newby• 13 years ago

... the general consensus is that those 'B' candidates can't survive the break-neck pace of innovation ...

Judging by the stream of Kindle regressions, Amazon could use a bunch of slow, reliable engineers grinding out the innovations of the 80s.

DannyBee• 13 years ago

"but the general consensus is that those 'B' candidates can't survive the break-neck pace of innovation and that they'd generally not be able to match up, let alone keep up with other performers and do the work that's expected of them exceptionally well. Since promotions are merit-based, and due to the stack-ranking employed by Amazon to rank the enggs, things only get diff for 'B' candidates."

This is probably completely wrong , unless all of your B candidates are being hired at very low levels only, and none of them ever get to participate in the promotion process.

Additionally, most tech companies are really bad at dealing with low performers in general (they tend to stay around too long).

In any case, my sources are completely empirical, having managed large numbers of people at a number of tech companies (and thus watching things like what happens to B candidates during cross-organization score calibrations, etc), as well as mentoring and helping managers with some of these issues.

I'd love to be wrong, but I have yet to see a company where the hiring of B candidates did not have significantly more impact that was expected.

JonFish85• 13 years ago

"With a word, they can veto any candidate, even if their expertise is in an area that has nothing to do with the prospective employee's."

Ah good, giving some extreme power to someone otherwise completely un-involved in the hiring process. Why should Rachel in accounting get to veto Sara as an engineer?

"Sailesh Rachabathuni, who developed software for Kindle devices before leaving Amazon in 2012, says he once vetoed a candidate for a programming job because the candidate didn't know much about a specific programming language, a detail others missed."

Great! A game of software trivia! Could be relevant, could just be this person being a douche, hard to say.

trentmb• 13 years ago

Pull the ladder up after you climbed it.

yapcguy• 13 years ago

Exactly.

Why hire someone into the company who might end up competing against you?

The veto is a gift for Machiavellians lurking in the officefor

twistedpair2• 13 years ago

Rung Raisers?

benjaminwootton• 13 years ago

I came here to make that point.

I once worked where a startup where almost everyone in the 15 person tech side of the company would interview every candidate.

On the rare occasions that all of us thought a candidate was a hire (1 in 30 candidates maybe) then the owner of the company could and often would still veto them if he wasn't happy.

Once pretty much all of us agreed that a candidate was useless, but the owner overruled us and bought him in anyway.

I have to say that guy actually ended up being quite successful at the company, so maybe the owner was better at spotting talent and we should have just skipped the whole charade.

Alas, it was a very frustrating and arbitrary system to work under, both for us employees and the candidates.

ignoramous5• 13 years ago

I'm a SDE at Amazon, but I don't speak on the behalf of the company...

AFAIK, there are different Bar Raisers for different roles. For instance, a QAE Bar Raiser has no say whatsoever who gets hired for a SDE role, let alone someone from the Accounting Dept. vetoing a Engg. candidate (vice versa is also true).

walshemj4• 13 years ago

Nah 99% chance Mr Rachabathuni is a douche

theboss• 13 years ago

That's what it is like already. Except instead of trivia, it is the same recycled data structures and algorithms questions, with some specific questions about topics on one's resume.

IMHO the hiring process will change a lot in the next few years.

blazespin• 13 years ago

Bar raisers have a reputation bringing on super stars at the company. They track everything. I worked for Amazon for awhile.

Isofarro2• 13 years ago

It depends on which part of the Amazon organisation.

The ones in Amazon Instant Video in London are useless. Particularly in identifying suitable candidates for the AIV Retail Website stack. They've burned out their hiring credibility in London to the point that the three hires to London in 2013 have been from East Europe. Plus churning and burning through university interns. It's been a desperate mess for a good two years now.

I hear positive things about the AWS hiring process.

mingfli• 13 years ago

I'm sure there's context missing in that quote. Bar Raisers would not veto someone just because they didn't know specific things about a programming language ... unless they claim to be masters of a certain language.

jeffbarr2• 13 years ago

I was about to say the same thing. The only reason that a bar raiser would ask the question would be if the applicant claimed to have the knowledge. If your resume claims a certain number of years of experience with a language, and you are asked how well you know it, you'd better be very confident before you say that you are an expert.

mjt0229• 13 years ago

I worked for 5 years at Amazon, and bar raisers were carefully selected for their ability to ask questions that probed much more deeply than this. I'm shocked that such a trivial detail would be used to disqualify someone by a bar raiser, and guess that it's more the result of faulty reporting than faulty interviewing. Not that mistakes aren't made (what interview process is perfect?), but this is hardly typical.

kevinmchugh• 13 years ago

It's possibly more than trivia. I expect a candidate to be able to have a conversation about whatever language they know best and have used the most. They should be able to speak about that language's strengths and limitations and mitigation strategies. If a candidate couldn't talk about languages they claimed to use, that would be a huge red flag for me.

If the interviewer is just throwing out questions about keywords and language implementation that are irrelevant to the candidate and the position, that counts as trivia. It's hard to say from the quote whether or not it was a trivial question.

vivekl• 13 years ago

I am an SDE with Amazon having worked across AWS and the retail side. I have done over a hundred interviews in the past 2 years for SDE loops. Each of these involved a debrief led by bar raisers. I have yet to experience a "veto" by a bar raiser as outlined in this article. As far as I have experienced, the bar raisers' questions are not typically any more complicated or harsher than other interviewers. The main role of the bar raiser is in fact during the debrief, i.e. after the interviewee has completed the interviews.

At Amazon candidates are evaluated on an array of competencies. In addition to technical competencies, particular attention is paid to Amazon's leadership principles (the fact that these are displayed right at the careers page: http://www.amazon.com/b?node=239365011 should highlight how important these are in the culture). I love that culture-fit and leadership qualities play as big a part as technical abilities in Amazon's decision making process unlike what I have seen in some of the other companies people mention in the same breath as Amazon. In my experience a good bar raiser acts as an unbiased advocate for the greater good of the company. As a result, they are NOT from the same team with the open position (this doesn't mean that an accountant is judging an engineering candidate or vice-versa mind you, they just happen to be from a different org) which allows them to look-past the immediate need the team might have which could blind them to some of the obvious flaws in the candidate. In some ways they are also acting on behalf of the candidate by ensuring the match works both ways. I have seen several situations where the bar raiser determined that the "team" or the "role" were poorer fit for the candidate than the other way around and went out of their way to facilitate a correction.

That said, there is no question that good bar raisers are just as, if not harder to come by than good employees! It is a thankless, demanding assignment to take. I have met some absolutely amazing bar raisers and some others that are just darn right atrocious. Considering how quickly Amazon is growing, they definitely need more of them but that risks damage through poorer ones entering the system.

jasoncrawford3• 13 years ago

Don't trust reporting to be precise and clear in every detail.

I worked at Amazon. They don't put accountants as bar raisers on engineering loops. There is a bit more specificity than that.

Regarding “didn't know much about a specific programming language”, I would guess that the candidate represented himself as fluent in that language, or had worked in it for many years. A good bar raiser doesn't play trivia.

seivan• 13 years ago

Some HR goon wants to tout their new revolutionary system. The only way they actually look like they get work done.

walshemj4• 13 years ago

Any experienced HR person woudl be having nightmares god only knows what discriminatory hiring practices are going on here.

ignoramous5• 13 years ago

It is not that Bar Raisers get away with just the Veto. Usually the process involves discussion between every one involved in the interview loop, including the manager, BR, other engineers, and the HR. A final "hire" or "no hire" call is made based on "data-points" collected over the course of the interview / interactions had with the candidates by the people present in the interview loop.

As evident, BRs clearly have a final say in the hiring process, but they don't always go about vetoing every candidate just because they can. They usually justify it with valid "data-points" and if one or two others have "data-points" to make a compelling argument against what the BR says, then they usually do so. And BR do listen to others who have contributed to the interview loop... they aren't in the mold of dictators exactly.

Even BRs get evaluated for performance... since the HRs are also involved in the discussion, one can be sure that no HR would tolerate discrimination, let alone 4-8 other engg or managers on the interview loop.

Again, just my take on things, I do not speak on the behalf of my employer.

codex3• 13 years ago

I imagine using a bar raiser increases the randomness of the interview process at Amazon, as it effectively reduces the number of interviewers down to one in the veto case. Thus, more noise. However, reducing the false negative rate isn't a high priority in most organizations, and Amazon is so desperate for employees that you often see the same person interview multiple times until they get in, so perhaps it doesn't matter.

ignoramous5• 13 years ago

The last part may be true. I have first-hand experience of this... Amazon does tend to "recycle" candidates, and likes to give candidates that came close to a "hire" or candidates that were unlucky and perhaps had a "anti interview loop" (as Steve Yeggie would put it) of sorts a second, or a third, or a fourth chance even.

rhizome2• 13 years ago

Amazon is so desperate for employees that you often see the same person interview multiple times until they get in

This doesn't make sense to me. Wouldn't they hire them first time around if they were truly desperate?

codex3• 13 years ago

Because the interview process is structured to "raise the bar" but is also highly random. As with any tech company, you have to run the gauntlet to get a hire. It's politically easier to have candidates reinterview when they fail rather than get permission from executives to "lower the bar."

jayhuang2• 13 years ago

"he once vetoed a candidate for a programming job because the candidate didn't know much about a specific programming language, a detail others missed"

He appears to be pretty proud of this. Most top companies actually care less about knowing specific languages as opposed to having the critical thinking skills they're looking for.

Not sure him vetoing a candidate because of a lack of knowledge in one language was a plus for Amazon.

ignoramous5• 13 years ago

Amazon doesn't place emphasis on language-specific skill-set. But there are certain job roles that require specific skill set, for instance, someone interviewing for a Software Development Engg role might get asked a lot of Algorithms, Data Structures, Databases, Operating Systems and so on. Someone interviewing for Web Development Engg role might get asked questions about JavaScript and HTML. Same thing goes for SDEs being hired for Machine Learning or Big Data teams... there are times when these teams are on look-out for experts in certain fields.

But yes, generally, AFA technical roles are concerned, Amazon doesn't place emphasis on language-specific skill-set.

jayhuang2• 13 years ago

Yep, exactly.

I interviewed for a Front-end Development Engineer position with Amazon, and aside from the poor handling by the recruiter, the on-site interviews covered a very well-rounded set of topics that were extremely relevant to the position. One of the better interview experiences I had in terms of topics tested IMO.

Jun8• 13 years ago

"To become a bar raiser, a worker generally must have conducted dozens or hundreds of interviews, and gained a reputation for asking tough questions and identifying candidates who go on to be stars."

Typical interviewing scenario where P(FA)>10^10 and P(Miss)=0. Such schemes create groupthink problems later on.

jessaustin• 13 years ago

...dozens or hundreds of interviews...

When does a "bar raiser", like, work? I've never been part of an org large enough to afford this sort of distraction.

parennoob• 13 years ago

John Sullivan, a San Francisco State University management professor, said Amazon's protracted hiring process is an important signal for applicants that Amazon is a tough place to work, with a lot of pressure.

A high-pressure interview, to work at a high-pressure, slightly-better-than-averagely paid job, that is then touted in the Wall Street Journal as being "bar-raising"?

No thanks. I guess Amazon is yet another company that failed my interview process.

amzbr3• 13 years ago

There are a lot of factual mistakes with this article.

(see notes at the end before flaming)

Amazon interviewers typically go through a long "shadowing" process where they calibrate their questions and evaluation criteria. You can't just go on to interviewing candidates on your first week. Also, interviwers interview for their own "job families", so e.g. SDEs and SDMs will typically interview technical roles and likewise for non-technical roles. All interviewers are also required to enter detailed "feedbacks" (if you've been put-off by the interviewer furiously typing away, please pardon them - they're not emailing, just taking detailed notes).

Amazon bar-raisers go through additional rigorous shadowing process that typically lasts many dozen of interviews and calibrations. An Amazon BR is more like a "moderator" in the interview loop than an all-powerful veto-machine. Veto is a very powerful tool, to be wielded extremely sparingly. BR on a loop is an "external" third-party who makes sure no self-interests are in play while hiring a candidate - i.e. no team is compromising on quality just to fill open positions.

Also, it's patently false that BRs ask extremely hard questions. Infact, most questions of BR focus on soft-skills and Amazon's leadership principles (Google them, we take them very seriously!). Also, BR's do loop within their job families and anything otherwise may happen only under extraordinary circumstances.

Notes: Amazon being as large it is, is comprised of thousands of autonomous groups and hence some candidate's experience may be sub-par and might have been contrary to what I've written. If you had a bad experience while interviewing at Amazon, please reach out to your recruiter who'll then forward your feedback to BRs. We take instances of bad candidate experience very seriously and will try to fix deficiencies if any. While recruiting, we consider the candidate as our customer and strive to make their interviewing experience delightful.

NB: Yes, I'm an Amazon BR. Posting using a throw-away account.

amzbr3• 13 years ago

The above comment is my own and I do not speak for my employer.

codex3• 13 years ago

Rather than use a BR to make sure that a group doesn't compromise on quality, why not just have interviewers interview candidates outside of their group, or interview them for a generalist pool which are assigned to groups later, or randomly? That prevents the BR from inserting subjective bias into the process. Everyone has bias, but the best way to cancel it out is to distribute influence, not concentrate it, nor attempt to "fight" the influences of other interviewers.

amzbr3• 13 years ago

That's one of the ideas often toyed with in Amazon. It hasn't worked out so well in Amazon because of rapid expansion of the company. As an SDE/SDM you have only so much time to devote to hiring, and with lot of pressure to hire, people like to devote time to hiring for their own team. Hence, the BR becomes the "balancer" to the process.

A BR can not be a "bar raiser" on his own team's interview loops.

WalterSear2• 13 years ago

Amazon is a sweatshop, from bottom to top.

Source: relative who worked for Amazon and had frequent interactions with Jeff Bezos.

cantcatch222• 13 years ago

It's a large company with diverse departments and diverse teams within those departments. You cannot generalize the sweatshop mentality to the whole company.

Source: Worked on an Amazon team that did not strike me as a sweatshop. Also these views are my own and I do not speak for Amazon

WalterSear2• 13 years ago

A fish rots from the head down.

jw_2• 13 years ago

One fun benefit of having such a "tough" (low probability of hire, not necessarily difficult-question tough) hiring process is that smart people constantly get weeded out, which makes filling spots extremely tough, which means that your team is constantly understaffed. This of course means your existing staff are overworked and turnover is high, meaning you now get to try to fill even more spots.

ck2• 13 years ago

ebay has 30k workers? no way.

they don't have inventory, they don't police listings, they don't do support by email anymore

what exactly are all those 30k people doing?

bane• 13 years ago

This happened to me. I got pretty far into a TPM interview and had a couple interviews with people that were so far out of my area they may as well have been working in a different company.

I still had fun and don't regret not working there, but thought it was odd that somebody who interviewed me for logistics backup plans had anything to say in my hiring process.

lukethomas• 13 years ago

In context, Amazon is also one of the worst performing companies in the Fortune 500 for employee retention. An average employee lasts 1 year: http://www.slate.com/blogs/business_insider/2013/07/28/turno...

mvgoogler• 13 years ago

I would take that report with a huge grain of salt.

First - it's based on dubious sources "_Payscale's report recorded data collected from 250,000 people working at Fortune 500 companies in the past year (July 1, 2012 to July 1, 2013)_" It's not from official company data. There is little to no information about how they collected the data.

More importantly, median tenure says little about retention or employee loyalty in company that is doing a lot of hiring - e.g. if a company hired 50% of it's total work-force in the past year, the media tenure will be around one year.

Another potential source of skew would be if the data included contractors, which are often limited to one year of tenure.

jasoncrawford3• 13 years ago

Does that count warehouse / fulfillment-center employees? If so you can't compare it to other tech companies, because they will skew the average.

johnrob• 13 years ago

It sounds like the amazon hiring process takes a long time, which probably also contributes to their goal: employees who stay long term. Filtering on quality gives you little guarantee as to how long someone will stay (in fact, I wouldn't be surprised if those attributes were reversely correlated). But a slow hiring process will definitely reduce the field to those looking for a long term commitment.

midko• 13 years ago

Somewhat offtopic but I think there's a chance someone here might be able to help.

I had applied through Amazon's website for a graduate position nearly 2 months ago and according to the dashboard, my application is still under review but I have never heard from anyone. I think my application has been stuck in some limbo and is now invisible to HR. I tried looking for a relevant email so that I can follow-up but this has been impossible. Granted, I found a graduate email I had written down 1 year ago but so far haven't had any reply and the email seems deprecated because its not advertised on the website anymore. Even on on-campus events, Amazon recruiters would give only the website address, no emails.

Could anyone perhaps point me to someone I could ask to have a look?

edit: To clarify, I am certain the application has not been rejected. I also do believe I am at least worth looking into hehe ;)

mratzloff3• 13 years ago

In my younger days I interviewed with them twice, but every aspect of the hiring process was so protracted that I ended up accepting jobs with other companies instead. Inevitably, several days after I had started at the new company I would get a phone call from Amazon to schedule the next round of interviews.

This was before they were as desperate as they are today, but even then the questions were hardly brutal.

Here in Seattle the reputation most often associated with them is "sweatshop". This isn't universally true, of course--some teams are better than others--but just about everyone I know who works there always tells me, "Thankfully I work on a good team. I wouldn't want to be one of the guys working on XYZ, though. Those guys are chained to their desks." The people who had negative feedback about their own team didn't work for them anymore, of course.

Anyway, churn and burn. Bring on the H-1Bs!

mratzloff3• 13 years ago

By the way, Jeff Bezos owns both Amazon and the Wall Street Journal, which published this particular fluff piece.

vonmoltke• 13 years ago

WSJ is owned by Dow Jones & Co. Bezos has nothing to do with them.

Isofarro2• 13 years ago

Bezos owns the Washington Post, not the Wall Street Journal.

mratzloff3• 13 years ago

Shoot, you are correct! Thanks for the correction.

cantcatch222• 13 years ago

You are thinking of the Washington Post

mathattack2• 13 years ago

As a general policy, this seems fantastic. It just doesn't sound like they're scaling the culture carriers fast enough.

* Bar raisers may be asked to assess as many as 10 candidates a week, for between two and three hours each, including paperwork and meetings—all while doing their regular full-time job, be it in finance, marketing or product development.*

20-30 hours per week for a non-job activity? This can't possibly be correct. Or does Amazon have a workaholic culture that supports this - 20-30 hours for those hungry after their 50-60 hour workweeks?

I heard a story about Google that the employees who were vetoed as candidates but hired anyways actually outperformed, because they had something about them that was so valuable that it was worth trumping the process.

As someone who has hired in the past, I have gotten in trouble by letting too many people have vetos. You wind up having to accommodate so many pet peeves ("They don't have 5 years of specific experience", "I don't like their school"...) that it's impossible to get good candidates.

jasoncrawford3• 13 years ago

When I was at Amazon, BRs did maybe 5 interviews a week, and that was considered a lot. If BRs are really doing 10 a week these days, it's probably due to a temporary shortage of BRs.

mathattack2• 13 years ago

Still a lot if you take the paperwork seriously. 10 hours falls into the 20% above and beyond category. It's a big investment in the importance of culture.

Amazon's Current Employees Raise the Bar for New Hires

Related Stories