Why We Do All-Hands Support at Olark

http://www.olark.com/customers/why-we-do-all-hands-support

by bcx • 12 years ago

92 49 12 years ago

bcx4• 12 years ago

It's pretty interesting to think about how to scale rotations of All Hands Support. It's one thing when you are like 4 people, but as you build a company there are a lot of resource constraints to think about. What's the biggest company you guys have seen that still does rotations?

jorts2• 12 years ago

I believe Zappos makes all new employees do a one time two-week rotation on the support team, regardless of their position.

bcx4• 12 years ago

I've seen it done at on-boarding with companies like Freshbooks and Zappos, but I am not sure how often that experience is renewed. (i.e. the rotation aspect being more than one time)

Engineers at Zappos do support for just a few hours a year doing peak season after that onboarding. With a program called "Holiday Helpers" - http://blogs.zappos.com/blogs/zappos-family/2011/12/20/holid...

sadfaceunread• 12 years ago

Certainly a big issue is scale. For example if your company grows sufficiently large, there are people who do not know how at least one of your products works at your company.

You could fix this by partitioning who supports what, but even that won't work completely. Is your accountant going to do support for your enterprise or consumer products?

jorts2• 12 years ago

That's a great idea. I'd be really interested to see how this works out as you scale. How do you continue to consolidate and relay feedback from customers to the rest of the team that isn't on the support rotation?

bcx4• 12 years ago

Consolidating feedback is a problem for any team at scale regardless of who is doing the support. Dedicated customer service teams have a more complete view of what's going on with customers, than the average rotated employee, but at the same time both need a way to bubble up that feedback to the product team.

More importantly, it's essential to understand the difference between the problem you head about today, verses the problem that customers have been facing for a long time. (I do think that support rotations cause a bit of a recency affect in prioritization for better or worse)

In the long run my argument is the closer you are to your customers the better insight you'll have in general. Because we all make a ton of decisions that aren't clearly specified, and we want everyone on the team to have the best intuition for the customer when they make those decisions.

sunir4• 12 years ago

Things like JIRA tickets to collate feedback and build a list of interested customers who will be potential beta customers.

kyrra• 12 years ago

National Instruments does something interesting with their customer support. NI is one of those companies that hires mostly college grads then tries to keep people around forever. If you are a college grad hired into NI, will will spend 6-12 months in their customer support division. This applies to almost everyone regardless of degree.

They do this for a few reasons from what I understand. It helps indoctrinate newhires to the NI culture. It also helps the people figure out what department they will enjoy and be a good fit into, as they will get exposure to a large part of the companies product portfolio.

bri3d• 12 years ago

I think Datalogix does this as well, at least for many entering graduates.

Serow2252• 12 years ago

The MathWorks (makers of MATLAB & Simulink) do this too. For their new hires coming from an MSc/PhD, you spend the first 12-24 months in the Tech Support group as a Tier I with a rotating schedule of 1 week of phone/email support and the next week spent working on a project of your choice with one of the many Dev/QA/BaT/doc/etc groups. The idea is that once you've completed a handful of different projects, you've had a chance to figure out what you like to work on, what group you'd like to work with, and the groups get a chance to see what it's like to work with you for several months on a real project. Once you've been there for 12-24 months you typically apply for a job somewhere else in the org, although some people find that they enjoy doing support and move up into Tier II/III. It's not like doing support for Comcast, you get to talk to a lot of smart people with interesting/challenging questions and issues :)

_pmf_• 12 years ago

As Mathworks customer, I've come to hate this, since this involves talking to clueless people with superficial information. Every question that involves something slightly more complex is dispatched to deeper support levels and regurgitated in a more or less screwed up way to me by the tier one guy/gal.

elliotz• 12 years ago

It's a great policy. Stripe was doing this 2 years ago: http://blog.alexmaccaw.com/stripes-culture Anyone know if they still do it, even after growing?

brianpgordon• 12 years ago

This is in pretty stark contrast to yesterday's article about devops:

Somewhere along the way, however, we tricked ourselves into thinking that because, at any one time, a start-up developer had to take on different roles he or she should actually be all those things at once.
What began as an experiment aimed at increasing software quality has become a farce, where the most talented employees are overworked (while doing less, less useful work) and lower-level positions simply don't exist.
Large companies love this, as it means they can hire far fewer people to do the same amount of work. In the process, though, actual development becomes a vanishingly small part of a developer's job.

http://jeffknupp.com/blog/2014/04/15/how-devops-is-killing-t...

I certainly wouldn't be happy with a developer position that made me do tier 1 customer support. That stinks more of "absurdly cheap" than "connecting with customers."

sunir4• 12 years ago

My experience says that if you're responsible for building a product, it's way cheaper to be directly in touch with the end customer so you can fix what you're building. It's better than having a legion of product managers, sales people, business owners in between you and the user.

If you're a developer who is oriented around building products that sell and are loved by customers, it's pretty addictive to talk to customers to find out what they think of what you built.

chris_mahan• 12 years ago

My experience is that it's a magnitude cheaper to have the developer talk frankly (no managers, marketers, lawyers in the room with them) BEFORE the product is designed.

quanticle• 12 years ago

I disagree quite strongly. In theory it works like you suggest - developers see problems, and they're motivated to fix them in a permanent fashion so they won't ever have to deal with them again. In practice, there are multiple equilibria. The good equilibrium is what you describe and what I've summarized above. The bad equilibrium is that the team is short staffed and developers are too busy putting out fires to actually implement the architectural changes required to prevent the fires from starting in the first place. If you're building an organization where developers are expected to do operations work, you have to think about the latter case. If you don't, you can very easily fall into a trap where your core infrastructure stagnates because developers are too busy coming up with the next hack to fix the ongoing outage to step back and think about broader changes that will make the entire system less painful in the future.

In fact, this is one of the things that Steve Yegge cites in his famous essay [0] about how Google, organizationally, is better than Amazon:

    And their operations are a mess; they don't really have SREs and they make 
    engineers pretty much do everything, which leaves almost no time for coding - 
    though again this varies by group, so it's luck of the draw.

At a small company like Olark, they may be able to get away with this because their product is small, their team is small, and so by necessity they have to do everything. But once their product grows, or their customer volume grows, it's very easy to get sucked into a very painful trap by having your engineers be your ops and your customer support.

[0] https://plus.google.com/+RipRowan/posts/eVeouesvaVX

bcx4• 12 years ago

Brian great comments, but I think there are a few things to think about from Jeff's comments.

Having an engineer respond to level-1 support is actually incredibly expensive. Especially, given that you could hire someone else who had a job specifically to respond to those issues.

In the cases where companies do choose to put engineers on frontline (once they've scaled to the point they can afford to hire dedicated support), they need to make a choice about the kind of organization they want to build.

The organizations Jeff is talking about and the one you are alluding too sounds like a pretty horrible place to work, where people are overworked, and resources are not allocated where they need to be to move the product forward.

Building a great organization is a challenge, and we choose to build an organization where everyone on the team has a strong connection to the customer. We do this with a strong understanding of how our team wants to grow and specialize. When done right, all hands customer service can work really well to align the team around the customer.

But you point is well taken, at many startups this idea that engineers should be doing "one more thing", when they are already responsible for "all the things" could be overkill. My advice would be to first figure out how to divide up responsibilities so people could focus, and then have teams interact with their customers every once in a while to remember what it's like to use their product.

I think you'll be surprised at how many engineers actually enjoy talking to the people they are building their product for.

varkson2• 12 years ago

I feel the opposite. I'm a dev who is also support. I don't generally get the 'please reset my password' kind of requests, as we don't get them very often, but help users with our system and work on fixing bugs they report.

I find you get a different perspective on what customers find easy and hard and this helps me develop my projects in a way that our customers will understand.

weixiyen• 12 years ago

Highly doubt this scales past 30 employees unless your customer base is really small. Customer service should really be summarizing concerns, and you should verify the concerns by looking at data to see if it's really a concern worth alleviating.

wdewind• 12 years ago

Etsy does this (or did while I was still there) at 500+ employees. While they have dedicated CS reps, they also rotate everyone in the company. My experience with it was (literally) mixed.

-As an eng I was frustrated by it because I felt it got in the way of "real work" (I don't know if this was right or wrong - it just felt frustrating at the time).

-As a manager, I was frustrated by it because my engineers were constantly disappearing for customer support rotations.

But then I watched one of my friends literally eliminate 400+ hours a month of customer support time (as well as the less measurable corresponding customer frustration) by writing a small tool in an hour only because he had that first person experience with the issue.

punkrockpolly• 12 years ago

I agree that there are benefits to cross-training employees, which is especially useful when they discover new better ways of doing things, or automate a process. The counter example can be developers who are bad "people persons" and might quit or dislike their job as a result this strategy. I think the important part is that the company is transparent about goals and takes employee feedback seriously.

ziziyO• 12 years ago

I had an inverted experience like this at my job. My job title is Technical Support. I do internal and external support for our company. We use Salesforce. By sheer luck I convinced my boss to allot me full system admin access to our Salesforce instance. I've gone to town improving our customer support workflow for our 100+ phone and e-mail reps by creating small tools and tweaks using Apex/VF Pages among other things.

I had no prior experience with Salesforce when I first requested access. I'm no Salesforce Architect, but I have a great grasp on the internals of it now all because my boss had a little faith in me.

kevin• 12 years ago

37signals does this and they support a massive user base. http://signalvnoise.com/posts/3676-everyone-on-support

iamleppert• 12 years ago

This is just a note but you guys really need to work on making your chat thingy less annoying. I'm reading this on an iOS device and there was no way to dismiss the bouncy-puppy chatbox that was begging me to talk to someone. It was so distracting that I rage quit your blog post.

JoeAltmaier• 12 years ago

It means you hire folks who can do customer support. That's definitely not an introverted engineer. Sounds like the benefits might even be worth the extra hiring filter.

imsofuture2• 12 years ago

As an introverted engineer Olarker, I strongly disagree that introverted engineers can't do customer support! :)

I do agree that it is a great hiring filter though: we definitely want to work with people that can maybe step out of their comfort zone a little and get a new perspective on the product they build, and the people that use it. You don't have to be a social butterfly, but you shouldn't need to wall yourself off from your customers, or the rest of the company for that matter.

rcfox• 12 years ago

As an introverted engineer, I beg to differ. If you can't spend 10 minutes in a very directed conversation with a stranger who wants to use your product, that's an attitude problem, not a introversion/extroversion problem.

Introverted doesn't mean anti-social or shy.

tekalon• 12 years ago

Not always. I'm an introvert and half of my position is a customer support role. It's pushed me, helped me grow and yes, it exhausts me on some days. It works best when you have it only on occasion and if you are able to learn from it to make a better product (as the article says).

fishtoaster2• 12 years ago

I worked at a SaaS startup that did this, and found pretty similar benefit: quicker fixes, and a better understanding of real customer needs.

Eventually we made it optional though, because we found that a minority of our developers really disliked it. Which I can understand- not everyone enjoys interacting with customers and answering dumb questions like "can you reset my password?"

I personally found it really valuable, though. After the hundredth password reset request, I was pretty motivated to make our password reset function easier to find. :)

Some pitfalls to be aware of, though:

1. This only works when the people doing support feel empowered to improve things, either because they can do it themselves or because they feel they are listened to by whoever can. For example, I can say to our founder over lunch: "People have been complaining about X a bunch this morning. How about we do Y to address that?" Then we'd discuss it, and I could build it that afternoon. For smaller fixes, it was often just posting on our group chat: "Hey, I'm fixing A by changing B" and merging it if no one objected.

Without this ability, though, it could easily get frustrating. If you're being forced to deal with repeated customer issues you have no power to address, it can feel more and more like an unproductive burden.

2. It can have trouble scaling with the complexity of your product. We were targeting small businesses when I started, and I was able to answer most questions about our product by myself. As we got bigger, though, and started introducing more complicated features aimed at the needs of larger companies, there were more and more features that I didn't have a solid grasp on. I probably could have kept the whole product in my head a lot longer if I were a full-time support worker, but as a dev I only did 5-10 emails per day,a nd didn't have the time to devote to learning some of the more esoteric aspects of our product.

3. Not everyone is cut out for customer service. We tried to hire good developers with personalities we wanted to work with. That often, but not always, overlapped with the type of communication skills necessary to be an effective support person. We had a few developers who lacked the patience to deal with slower customers, or weren't really good at explaining things clearly to them.

djloche• 12 years ago

" If you're being forced to deal with repeated customer issues you have no power to address, it can feel more and more like an unproductive burden."

You have summed the standard customer support experience so distinctly.

fishtoaster2• 12 years ago

Yep. I worked tech support part time in college- I remember. :)

When implementing an "everyone does tech support" system, though, remember that at least some of your employees won't have much trouble leaving if they find themselves hating the experience. When they have to deal with customer complaints, but can't do anything to fix them, they may well move to a company where that's not the case.

wpietri• 12 years ago

Great points.

I think #1 is a feature, though. One of the ways a growing company can go bad is to make it hard for most people to solve customer problems.

The megacorp customer support model is basically just to put all the disempowered support people in Nowheresville, so that nobody "important" knows how much the customer experience sucks. Which is part of what allows startups to come in and savage their business.

But there's another path: you can also avoid the customer and staff frustration by empowering them to fix problems, and working hard to keep it that way as the company grows.

rhizome• 12 years ago

We had a few developers who lacked the patience to deal with slower customers, or weren't really good at explaining things clearly to them.

Here's where we see "culture fit" is arbitrary.

mfish• 12 years ago

my company did this (out of necessity) back when there were ~10 of us. As a young developer, I didn't know any better, but I dreaded it. I was SO glad when we moved to a dedicated support team. Also, in my experience, if you are hiring good people, a dedicated customer support person is going to give your customer a MUCH better experience.

vvvVVVvvv• 12 years ago

Obviously, but being exposed to real issues first hand will certainly give you a better insight on why it should be fixed.

Some kind of reality check.

batmansbelt• 12 years ago

Not having to deal with customers is a huge perk for a developer.

fernandotakai• 12 years ago

well, depends. i like having to deal with customers since that makes me able to build a better product -- dealing with customers makes you realize what they want.

(disclaimer: i work at olark)

imsofuture2• 12 years ago

It's also awesome to interact with people who are legitimately happy about the stuff that you build.

varkson2• 12 years ago

Personally I enjoy it and consider it the best thing to happen to my career. Couldn't do it all day every day but doing support on the side is rather enjoyable IMO.

sunir4• 12 years ago

I love this talk by Kevin Hale about their all hands support at Wufoo. https://www.youtube.com/watch?v=fAnl8ZGHVXU (alternate. https://www.youtube.com/watch?v=wYIH2qTizUU)

The most interesting thing is that they gave every engineer time to fix the $#@! they heard on support. 30% of their effort was on internal tools.

Disclaimers. I also work at Olark. I also think Kevin Hale is (almost) perfect.

emilsundberg• 12 years ago

I love the talk. It's too sad to see Wufoo these days. All their great ideas about support disappeared after selling it to SurveyMonkey.

jefftchan• 12 years ago

I think doing all-hands support in a way that's efficient (not taking too much out of engineer's time) and scalable requires a well-designed system to support it. Quizlet, where I've worked before, does a good job of this. See: http://quizlet.com/inside-quizlet/quizlets-incredible-feedba...

tphan• 12 years ago

We considered using Olark for a client's website one time but, at the time, their embed code didn't work properly with Google Tag Manager. Whilst we ended up using another service, I did receive an email from Olark a few months later saying that they had addressed the issue, which I appreciate.

digitalabyss• 12 years ago

I’m a IT department of one so I get a lot of those "reset password" requests and have to do a lot of hand holding I feel I should not have to do. Its 2014 and a certain level of computer literacy should be expected of you if you want to function in corporate America. (or whatever country you are from). I have a Masters of Science in IT Security, a B.S. in electronic engineering and my RHCE with 10 years experience and feel that being pulled away from "real work" to reset a password is extremely annoying. However its a lot less annoying when the client services team is not available and the receptionist passes the call directly to me. In most cases the client tends to be very friendly and thankful and makes you want to help them.

The problem with this is geeks are generally not business people and do not know if the client is up-to-date on their billing or if they are notorious for trying to get free work and need to be kept in check. When I do take personal interest in solving a clients issue I risk promising things I should not be promising them or putting way more time and effort into a client who is already behind on billing than I should be.

I also have a very hard time quoting a client an appropriate amount. I work for a interactive and design agency and when you work with us you have a team of individuals working for you including but not limited to an account manager, producer, designers, art directors, user experience, strategy, front end developers, back end developers, system administration in some cases. Having a team of 5-10 specialists working for you is expensive. Just having a 1hr meeting with everybody is probably going to get billed at around $1000. I feel really weird giving clients numbers like this and responding to the sticker shock.

MisterBastahrd• 12 years ago

That's neat, but when you've already optimized your password reset as well as you possibly can, now you're just throwing time and money out the window because of a dumb company policy and probably pissing off your most talented developers too.

Some customers are just incredibly stupid. That's who Tier 1 support is precisely for: the kind of people who would suffocate in a wet paper bag. I walked into my manager's office one day and said enough. I developed 10 products used by over 250k users and I was the only developer at the company. Never again was I going to reset another password. I followed best practices and made it as easy as possible. Any further difficulty on the part of a user was not my problem.

lsiebert• 12 years ago

Some customers need hand holding.

One thing to consider is an inverse of customer support. Have your dedicated customer service rep handle only repeatable issues with a script.

Then you just say, "I'm going to escalate your issue to our Solutions Department, who are responsible for that type of issue. They will be able to walk you through the process."

But really, whatever you do, if you do email/chat support I'd be feeding every chat session/email exchange into an expert system, and trying to generate responses when possible.

paulhauggis• 12 years ago

This sounds like it works well for Olark.

However, I've seen companies use this to stretch the responsibilities of the development team to include customer support..and not hire anyone for customer support.

It was a disaster and I hated it...and eventually quit because of it.

Why We Do All-Hands Support at Olark

Related Stories