A Review of Airbnb's SRE Hiring Process, from someone who just interviewed there

http://redbluemagenta.com/2012/08/15/review-of-airbnbs-interview-process/

by philiph • 14 years ago

35 59 14 years ago

relation• 14 years ago

There was too much emphasis on tool choice/religion, Linux trivia, and getting the ‘right answer’ versus solving engineering problems, where multiple tools might have to be glued together to come up with one of many solutions.

airbnb seems to be missing the mark. tools come and go. the 'right answer' doesn't matter as much as the reasoning.

the ability for one to assimilate him/herself quickly and productively into the environment is what they should be looking out for.

georgebarnett2• 14 years ago

I'm not in a position to comment on Airbnb's process here simply because I haven't experienced it, so this comment isn't aimed at anybody in particular.

I find it remarkable how many skilled engineers reach for the toolchain first and then find problems to solve with whatever tool they have chosen. Even worse, they often reshape the problem to fit a particular tool and then solve it badly.

One of the downsides of the Github/BitBucket open repository movement is that there are now zillions of tools for these engineers use to go looking for problems.

Kind of reminds me of the database as a hammer story:

When I first wanted to spend some time learning about relational databases, my then boss told me a story about database people, which I’ve found to be remarkably true. The story is, RDB people have found the most beautiful, wonderful, perfect hammer in the whole world. It’s perfectly balanced – not too heavy, not too light, and swings just right to pound in a nail just right every time. The grip is custom-made, fitted to the shape of the owners hand, so that they can use it all day without getting any blisters. It’s also beautifully decorated – encrusted with gemstones and gold filigree – but only in places that won’t detract from how well it works as a hammer. It really is the greatest hammer ever. Relational database guys love their hammer. It’s just such a wonderful tool! And when they make something with it, it really comes out great. In fact, they like it so much that they think it’s the only tool they need. If you give them a screw, they’ll just pound it in like it’s a nail. And when you point out to them that dammit, it’s a screw, not a nail, they’ll say “I know that. But you can’t expect me to use a crappy little screwdriver when I have a magnificent hammer like this!”

  http://scienceblogs.com/goodmath/2008/01/22/databases-are-hammers-mapreduc/

dfc3• 14 years ago

"Github/BitBucket open repository movement"

Did you mean to say "open source"? I am not familiar with the github movement.

georgebarnett2• 14 years ago

What I mean is any piece of code that's open for re-use on GH/BB.

I've seen many repo's that aren't open source in the GPL/BSD sense but are open source in the "here's the code now its your problem" sense. Sorry if this has resulted in confusion.

MichaelGG• 14 years ago

I was asked how I could coordinate actions between many machines, and I told them that I could probably use Chef to do a lot of that, which was met with a huge red “no” response. I then had to play bingo and landed on the “correct answer” of “Zookeeper.”

I'm curious about this. I'm only vaguely aware of either system, but I thought they solved rather different problems. Chef handles system configuration (like things in /etc, or "configure memcached to listen on this IP and port"). Zookeeper is more of an API to deal with distributed locking/sync issues for an application itself.

Was this interview question less of a "bingo" and more of a "one-side really misunderstood something"? Or how do Zookeeper and Chef compete?

cparedes5• 14 years ago

Author here. They do, though for a certain set of problems, you can get away with using things like Chef for coordinating lots of information between a lot of machines.

My point, however, was that tools should never be the focal point of solving a problem - if you recognize the problem first as a distributed locking problem, or as a configuration problem, then you can start deriving a solution (and maybe that solution might have Zookeeper in it, maybe you'll write a new piece of software, maybe you'll use Chef, etc.)

ryanpers7• 14 years ago

One of the biggest problems as I noted in a different comment is that chef doesnt solve _software_ deployment. Configuration management and "best effort file pushes" is not really appropriate for running a website. Sure you could do something with chef, but waiting 20 minutes for a rollback would be pretty lame-o.

The issue here is software push and service orchestration. How do you do a rolling restart of a cluster in an automated fashion?

As for 'configuration management', I feel like puppet, chef, etc arent catch all solutions for 'i want my config to be the same on all machines' for the above mentioned 20 minute cycle.

cglee• 14 years ago

This breaks the golden rule of relationships: don't kiss and tell. Interviewing is the first step in forming a relationship, and it looks like Airbnb dodged a huge bullet by not engaging further.

zavulon• 14 years ago

I agree with you, and would never do something like that myself - however, AirBnB is not without fault here either. They've conducted an interview in such a way to piss this guy off enough so he goes ahead and blogs about it. Usually we only see posts like that about interviewing at Google, Microsoft, Amazon, etc - so if I was AirBnB, I would definitely look into their hiring process.

Swizec• 14 years ago

"Don't kiss and tell" is male code for "yes, yes I did in fact have sex with her".

I think it's important to share interviewing stories. Both for other startups to learn from, and for future employees to know what they're heading into.

Or is it not every manager's wet dream to have good guys right out of university who won't hide their true skills by being incompetent/nervous at interviewing?

Conversely, is it not every young guy's wet dream to be able to display their true skill in an interview, rather than mess it up by being bad at interviewing?

Where are they to learn if nobody shares the stories?

troels4• 14 years ago

Yes, it did seem a bit out of line. He could have sent the entire post as an email to one of the founders instead.

philwelch5• 14 years ago

Forgive me for being a little naive here, but I have a genuine question. Is there any actual harm that could come of this, aside from people looking down on the author for writing and publicly posting it?

learc836• 14 years ago

I see no problem with this, how are interview processes going to change unless we talk about them? There's a real problem with companies joining the Google interview process cargo cult.

Also the job == relationship metaphor breaks down fairly quickly. What applies to one doesn't apply to the other just because you say it does.

drewcrawford4• 14 years ago

I don't know; it's certainly given an unknown candidate a lot of press. Perhaps AirBnB has dodged a bullet, but if you take his statements at face value (e.g. he doesn't want to work at such companies) then he's lost nothing and potentially gained several offers from better fits.

Harkins• 14 years ago

That's much nicer for the ones with all the money and power than for the one person looking for work. Or a landlord. Or a client. Or a partnership. Or an acquisition. Etc.

philwelch5• 14 years ago

I'm going to go ahead and say a few things, slightly in defense of Airbnb from the perspective of a fan, an observer, and an occasional customer, and also to get these points out of the way.

It's really hard to know the right way to hire SRE's/sysadmins/systems engineers, from a startup's perspective. If your founders are highly technical, they're more likely to be developers, which is a similar profession but frankly different enough to note. In some cases they will be designers, or non-technical people entirely. As your company grows, you need to hire developers first, so at best, you find a (decent) solution to that problem. As we already know, though, nearly every company that interviews and hires developers still does a shitty job at it--not because it's their fault but because it's a genuinely hard problem. It's only after you've grown awhile and your developers get tired of doing all the ops shit that you think to hire SRE's. But at this point, your culture and hiring practices are already pretty set, and set by people who aren't actually SRE's. It can be fantastically difficult to rollback this kind of momentum even if you do manage to get a top-notch SRE or three onboard.

There's another side of this, though. It's both a root cause for the problem and good news for Airbnb: Airbnb's competitive advantage is not going to be having the best SRE's. They're not AWS, Heroku, or Google. So it's not really a problem if they're bad at hiring SRE's, because most companies get away with being bad at things that aren't central to their business. What Airbnb gets right, they get fantastically right. This isn't to say they shouldn't do a better job hiring SRE's. They should. But they're still a great business despite this.

noomerikal3• 14 years ago

I agree and would only add that of the many purposes of raising some $120 million is to put appropriately experienced individuals in a leadership capacity. Appropriately experienced meaning management that understands how to maintain a happy medium without bias towards development or operations.

johnx123-up• 14 years ago

At the tail end of the interview loop, I was asked how I would improve Airbnb’s site.

May refer these guys http://labs.agriya.com/burrow

drewcrawford4• 14 years ago

I'm going to say something potentially very stupid hoping someone will correct me: I don't understand why technical interviews exist in the general case.

I mean, I understand why companies like AirBnB would like them to exist. What I don't understand is why engineers participate in them.

I totally get selling yourself to the potential employer. Spending an hour on the phone talking through a technical problem. Having IRL conversations where you see if you "gel" with the team in a bunch of hard-to-quantify ways.

But why would any engineer in today's market take two days to respond to asinine pet/trick questions that random engineers lob at them in order for a shot to spend additional time for people higher up the food chain to lob hypothetically less-asinine questions in a way that makes them directly comparable with the other 19 people jockeying for the spot?

Do companies like AirBnB believe that there are top-tier people with nothing better to do? Or do they believe they have some perk that is different than the perks of every other startup currently hiring that would motivate top-tier people to create such time?

Maybe my perception of market demand leads me to overestimate it. I get so many "please work for us" emails (from actual founders, even, let alone recruiters) that my inbox is out of control. I've gotten a job offer while writing this comment. I can't even reply at that scale. Perhaps one offer a month is lucky enough to receive an actual e-mailed decline; I never even consider doing a phone interview. Am I a precious snowflake, is my experience non-representative?

borski5• 14 years ago

I don't write negative comments, but this was impossible to not respond to. Honestly, you sound less like you're precious and more like you're pretentious.

While I'm very happy for you receiving job offers all the time (congratulations!), there are definitely reasons for an engineer to take an interview; in particular, falling in love with what a company stands for. The idea that all engineers are mercenaries ready to work for anyone and sell themselves to the highest bidder is an aged one, and one I had hoped was no longer true. If I fell in love with a company (as has happened in the past), I'd certainly be willing to jump through some hoops to "prove" myself to them - if that means going through an interview process so they can fall in love with me as much as I have with them, so be it.

When I see a woman I like, I don't go up and tell her about all the other women that want to sleep with me. Instead, I flirt, I jump through some hoops, make her jump through some of mine, and ultimately establish a strong connection. Or not; that happens too. Same goes for finding work.

noomerikal3• 14 years ago

How could you fall in love with a company without working for them? Isn't that synonymous with the high school scenario of falling in love with the girl that you never said hello to?

gaius• 14 years ago

Two days vacation burned is a lot of commitment up front from a candidate. Maybe it's intended to act as a filter.

atirip• 14 years ago

You didn't read the story and the comment you responded, did you?

Because your example with "woman I like" is not valid. There was no woman _you_ liked. There was woman who liked _you_ (recruiter e-mailed). And that _woman_ expressed desire to sleep with _you_. And after that, _you_ strongly advocated to start flirting and "jumping through some hoops" with that woman. May I ask why?

drewcrawford4• 14 years ago

> in particular, falling in love with what a company stands for

People can purport to stand for just about anything, but unless there is a track record there of standing up for [principle] in the face of [adversity], it's just a politically expedient claim. There are wonderful exceptions, but most ordinary people and most ordinary companies don't deduce their day-to-day behavior from first-order principles, and at any rate, companies the age of AirBnB don't have the track record to enable an observer to distinguish principles from marketing claims.

> The idea that all engineers are mercenaries ready to work for anyone and sell themselves to the highest bidder is an aged one, and one I had hoped was no longer true.

This is not my view; nobody is saying take the highest offer, and I have said absolutely nothing about money at all.

My question is: why do engineers participate in technical interviews? Insofar as I understand it, your response explains why an engineer might (I would say improperly) create a post-hoc rationalization and "fall in love" with the company after-the-fact, but under your own logic it doesn't explain why the engineer takes two days off to go in the first place. Assuming that "falling in love" is even something you can do in two days, the candidate hasn't done it at the point this decision is made.

philwelch5• 14 years ago

Maybe you just are that awesome. Developers are in higher demand than SRE's, which is one thing. But I think it's highly unusual to get a job offer without an interview process. Hiring is a big commitment and choosing the right people to hire is a tough problem; so is for choosing the right place to work at. I certainly think it's worth at least a couple days to make the right decision.

learc836• 14 years ago

>But I think it's highly unusual to get a job offer without an interview process.

I don't really think it's the presence of an interview process that the OP is complaining about. Rather it's that the interview process can take 2+ full days.

Do a phone interview, a 2-3 hour relevant work sample test and a 1-2 hour in person interview--then if you're still want more, pay me for a short project (that I can complete nights and weekends).

davidw• 14 years ago

Maybe his comment comes off as "I'm too cool", but there's something to what he's saying.

After the Nth email from someone at Google, I got curious and said, sure, we can start the process.

The first phone screening was like "Unix Trivia Time". I knew lots of things, but some others just weren't things that matter to me day to day, so while I know they're out there, I don't recall the specifics. Generally, I'd know what man page to go look up, or I might also considering a certain search engine to find information. The interviewer said that I wasn't supposed to do that. This went on for what I recall as an hour or two, and I did pass, and got invited to do some other interview.

But the more I thought about it, the more I realized I did not want to spend any more time doing stuff like that: they don't have offices anywhere near where I live, nor any offices for technical people in parts of Europe that I consider pleasant places to live, and I don't want to live in California either. And supposedly it's pretty difficult to land a job there anyway, so the odds would be to spend N evenings/hours doing more silly interviews just to get turned down, or even accepted and be faced with the prospect of moving somewhere I don't want to be.

So I politely let them know that it had been decided that Google was not a good fit for my current employment needs, and that I wished them luck in their candidate search.

noomerikal3• 14 years ago

With a LinkedIn profile that has at least three positions with experience in "startup" technologies, I believe your experience is very representative. Economically speaking, we all know there is a surplus of development-related positions (in key startup cities, i.e., sf, la, nyc, etc). What generates this ridiculous level of interview practice is the notion of "hacker-rockstar-ninja-beard" guy or gal and the inane belief in the existence of only two camps of engineering talent, 1.) hacker/rockstar/whatever and 2.) the rest being shit.

Clearly, this isn't the case.

dfc3• 14 years ago

Can anyone with hiring experience comment on the relative merits of publising a post-interview write up like this? (Not just from airbnb's perspective, also future employers) My initial reaction is that nothing good can come from this but I do not have a lot of experience in hiring or recruiting.

joering2• 14 years ago

Remains me of a recent almost interview I almost went through. I have 15 years of experience in IT and basically over past 7 years I built two large healthcare-related CMS systems. The company found me through one of clients that uses my product for 5 years and is very happy with the value 2 cost ratio.

So I go in and first question I hear. "can you describe how is TCP/IP built". This is for web development position, you know code igniter, php, jquery, that kind of stuff. I looked at them for about 45 seconds, got up and said "thank you for your time" and left before they had a chance to say anything.

Dude, wtf, are you serious? Its not like i couldnt remaind something about tcpip from the times of my b.d. 10 years ago, but seriously unless we are going to invent new transportation protocol for the whole human civilization to use, why on earth would you ask me that, when you hiring me to be a web developer. What a joke!

ryanpers7• 14 years ago

leaky abstractions. It's important, but there is also a cargo cultism of asking questions in this certain way that is imitated in form only from how people envision google demands their engineers.

ryanpers7• 14 years ago

bad interview process... but they are correct in the sense that chef and puppet solve problems, but that problem is NOT software and clustered software deployment. You could hack it on top of it sure, but it'd be more straight forward to basically start from scratch. Hell bash for ssh loops would work better.

Perhaps they really were looking for someone who already thinks they way they do, and that is valid. they probably dont need you in their office for a few weeks saying "i could do this in chef if..." until they fired you for "poor fit".

Also dont forget that tech/aspy folks just are... blunt in a way that most people interpret as "insensitive". Maybe they're just being efficient? You write a blog, seem pretty eloquent, so I'm guessing you are NT, so perhaps not a good fit after all.

cparedes5• 14 years ago

Author here.

NT? (edit: you mean non-technical? I'm definitely not non-technical - I work as a systems engineer by trade.)

And, yes, you don't need to use Chef or Puppet to solve the software deployment problem. You use it to ensure system state, so you know you can throw software on top of it without any problems. (this can go on into a debate about golden images vs. configuration management, but I think this is a different discussion.)

That's probably true as well, though... I've been in shops without Chef or Puppet, and it's really not a big deal for me, either. The thing that stood out was the justification for not choosing Chef or Puppet, not really the fact that they weren't using it.

philwelch5• 14 years ago

NT == "neurotypical", i.e. not autistic.

ryanpers7• 14 years ago

NT = neurotypical, standard lingo in "aspy" forums. You're probably pretty normal. Expect eye contact and all.

On a technical note, it's not clear to me how you KNOW that chef or puppet has solved your system state problem. Now if its limited to doing super basic stuff, ok great, but then you really arent adding enough value to justify the huge cost of these systems.

I'd also like to hear what you have to say about "throw software on top of it"... for example, I want all 40 webservers to have the same build revision of our codebase. But I need positive feedback - and a gradual rollout possibly. Eventually run puppet clients really dont tell us this. Oh wait, I mean I want to rollback to the previous deployed version, because it's broken horribly.

These practical software deployment issues I find just are not addressed by the system configuration management tools like chef, puppet.

And that is why they are so passionate about it, because a million people have told them 'just use chef' and yet chef doesnt solve their problem at all, nor does it solve any fraction of their problem either.

learc836• 14 years ago

I really feel like most of these companies are just trying to emulate Google's hiring process rather than finding something that actually works.

How many people have actually been hired after going to one of these? Every job I've gotten has been through networking, back channels, and a token interview.

2+ days of anyone's time is too much to ask without compensation.

A Review of Airbnb's SRE Hiring Process, from someone who just interviewed there

Related Stories