Billing Incident Update

http://www.twilio.com/blog/2013/07/billing-incident-update.html

by agwa • 13 years ago

109 44 13 years ago

jmandzik• 13 years ago

This is about as good as you can handle an incident like this, PR wise.

ajtaylor• 13 years ago

Simple, honest, open and straight to the point. Nice to see that they recognise it's not just about the money, but the trust people had in Twilio. I'm looking forward to reading the promised post-mortem, because I can't recall reading one from them before.

RobSpectre9• 13 years ago

It has been some time since we have had an incident this disruptive for our customers. Looking forward to delivering that investigation summary to you as soon as we feel we got it right.

aroman• 13 years ago

Very well handled. Facts, gestures of good will, and straight-forwardness.

Call me crazy, but I tend to think issues like this ultimately serve, if handled well, to improve a company's public standing.

Though I suppose that depends on how one defines "well"...

jkestner• 13 years ago

You're not crazy. I've found that a small customer service interaction early on (regardless of fault) helps solidify a good first impression. Not that I intentionally create problems...

michaelrbock• 13 years ago

This isn't crazy at all, in fact it's been proven to be true.

The Marriott hotel chain found that guests who had something bad happen during their stay but which the Marriott fixed were actually more likely to return than those who had nothing bad happen at all.

Source: www.cbsnews.com/8301-505125_162-48540174/how-customer-complaints-can-improve-business/

smackfu• 13 years ago

Reminds me a bit of people who say "Apple is the best! My phone died but they replaced it on the spot."

mikeash2• 13 years ago

Thinking about it, it makes sense to me.

Everybody makes mistakes, and everybody experiences bad luck.

If you only experience good times with a company, then you have no idea how they respond to bad times. They might be great, or they might be awful. You just have no idea.

If you experience a problem with a company, then you know how they respond to problems. If they do poorly, you'll avoid them. But if they do well, you now know that they handle problems well.

The time then comes to choose a company for something. Which do you choose: an unknown, or a company that you know handles problems well?

zaidf• 13 years ago

Call me crazy, but I tend to think issues like this ultimately serve, if handled well, to improve a company's public standing.

I royally screwed up the embed script testing for our very first customer resulting in their entire site not functioning and a panicking customer on the other end. I ended up writing a 3 page post-martem and apology. We debated a bit internally to be as open as I'd been, essentially admitting basic sloppiness on my part and promising to learn from the mistake if given a chance.

Fast forward 7 months, the same customer(originally acquired from a cold call) recently introduced us--without asking--to one of the top seed funds locally.

agilebyte• 13 years ago

Happened to me. I made an SQL update without specifying the conditions on a live database (project running, surprise surprise, late). Our team has sent some flowers and chocolates too. I am going to do the same thing next time I screw up, and I will.

These days it is too much about the electronic and not enough of the "physical" in my mind.

stinky613• 13 years ago

In 1982 some nutjob went around to supermarkets and pharmacies and put potassium cyanide in Tylenol bottles. Johnson & Johnson responded by issuing a recall for ALL Tylenol products nationwide. This incident is anecdotally credited with having directly led to those paper or foil seals on over-the-counter pill bottles.

https://en.wikipedia.org/wiki/Chicago_Tylenol_murders

BHSPitMonkey• 13 years ago

Wow. That is utterly tragic and horrifying.

erichurkman• 13 years ago

https://en.wikipedia.org/wiki/Chicago_Tylenol_murders is the link in case anyone has not heard of the case.

colkassad• 13 years ago

Correct me if I'm wrong, but I believe that also lead to the seals we see today on a wide variety of products.

RobSpectre9• 13 years ago

Thank you very much for that - our work is still not done. Through the night we have identified what we believe to be the root cause and have reconstructed the timeline that produced the incident.

We still owe you a full technical post mortem of the incident and we are responding to all the servicing fee reimbursement requests at help@twilio.com.

Deeply appreciate your support - more to come.

benjamincburns3• 13 years ago

Replace the word "company" in "a company's public standing" with "well-established startup that's already known for good service" and I 100% agree.

If Twilio's overall service wasn't excellent, people wouldn't be nearly as willing to view this from the positive angle. If Twilio weren't a startup - if it were, say, SalesForce.com - this also would hurt more than it would help.

Fortunately, Twilio seems to know this. They're absolutely in no way resting on their existing positive reputation. Even though they've earned the trust and respect of so many customers, they're still treating each and every customer as if they must still continue to earn their trust and respect.

Word of advice to Twilio execs: when this is all done and over with give your PR people a bonus and/or a raise. They did a fantastic job here in a nearly impossible situation, and I'm sure the recruiters are circling like sharks.

RobSpectre9• 13 years ago

Truthfully? Our PR guy is on his honeymoon. Not kidding.

Glad to hear you got the information you needed when you needed it - still a lot of work to do to resolve this to our customers' complete satisfaction.

We'll get it done.

throwaway1979• 13 years ago

That's how it is done. Classy.

mikegirouard• 13 years ago

It was surprising to read they will be reimbursing any overdraft fees caused by the problem. Good move.

jpiasetz• 13 years ago

At the end of the day aren't they likely liable?

benjamincburns3• 13 years ago

I don't think they are, but it's the right thing to do. One would hope the banks would be understanding enough about the situation (haha, I know, right?) to reverse the fees. But even in that case Twilio is still causing you to, at a minimum, call your bank, wait on hold, spend 15 minutes getting transferred around to someone who has the ability to reverse the fees, etc. If they want minimal impact for their customers, they'll just reimburse the fees.

I played this game with Dreamhost and SunTrust back with their whole billing debacle in Jan 2008. Dreamhost didn't respond to my support requests to reimburse "non-sufficient funds fees", The bank manager at SunTrust said that regardless of the reason funds weren't available in my account when the other transactions posted, so the NSF fees that I was incurring were legitimate.

In the end I think I paid a little over $300 for their mistake.

cosmie• 13 years ago

I would hope they have a business liability insurance policy, in which cases it should cover the costs associated with this unfortunate mess, including any reimbursed fees.

grimlck• 13 years ago

Why are you surprised they are reimbursing overdraft fees? That is the bare minimum that I would expect from any company which made this mistake

maaaats2• 13 years ago

Of course it is to be expected, but unfortunately one cannot expect it happen. If you get it?

RobSpectre9• 13 years ago

Think it is the least we could do.

ckdarby• 13 years ago

So many updates on their status page! One of the first companies I felt like I knew wtf was going on during the issue.

dustingetz• 13 years ago

And you can see status from two years ago! Google won't even recognize that gmail is down unless 5% of requests are failing (!!!! Try getting your work done all day if 1/20 actions fails.)

RobSpectre9• 13 years ago

Very glad you found the stashboard useful during the incident. This is actually one of my favorite open source projects of ours.

If you would like a similar service for your company, check it out here: http://www.stashboard.org

brianstorms• 13 years ago

Twilio should really consider a lighter gray color for their text on the white background. Right now the text is simply too legible still! Why, I can almost read it, if I crane my neck and look at the screen at just the right angle. What could they have been thinking? Hopefully they will take this feedback to heart so that future urgent messages like these will be completely obscured.

maaaats2• 13 years ago

Kinda ironic that this comment shows up as light grey on a grey background. :)

RobSpectre9• 13 years ago

Appreciate the feedback. Will get it fixed.

dredmorbius• 13 years ago

I've been a fan of ContrastRebellion, though I've lately simply taken to modifying websites' CSS that I visit often enough that it annoys me. E.g., I've swapped the border and content background colors on Hacker News.

What the heck, here it is: http://stylebot.me/styles/2945

I use Stylebot, but Stylish or other CSS management plugins should work just fine, copy/paste the CSS source.

contingencies• 13 years ago

For anyone looking for multiple (redundant) SMS provisioning, I have had not perfect but pretty good experience with http://fortytwotele.com/ over a good few years. I am not connected to them at all, just throwing the option out there. As a bonus, they're European, which means you are better positioned for compliance with EU privacy laws for non-US clients.

bdcravens• 13 years ago

Interesting. I just came back from an event here at Lone Star Ruby Conference they sponsored (movie - they used Twilio powered service to pick the movie we watched at a local brew theater). Would have made for some interesting questions had more known about this problem :-)

josh2600• 13 years ago

Congrats to Twilio on a textbook example on how to handle a billing outage. While I hope to never to through this I do admire a team that can execute like this.

Cheers, from one telco nerd to your team of hackers.

RobSpectre9• 13 years ago

Always appreciate your support Josh. much work to do before this incident is resolved fully.

happywolf• 13 years ago

Nice response, bad text color. The gray on bright white background really strains my eyes. Ouch.

neurostimulant• 13 years ago

I use iReader extension whenever the website's color scheme strain my eyes too much. It's essentially a clone of "Reader" functionality on safari.

https://chrome.google.com/webstore/detail/ireader/ppelffpjgk...

RobSpectre9• 13 years ago

Good feedback. Thank you.

brador• 13 years ago

Why not batch process billing every hour or 30 minutes? That way you can run simple checks to look for anomalies.

RobSpectre9• 13 years ago

It's a good question. As our service is delivered through a utility pricing model, most of our API requests are billing events - each call, text message or recording ends up having a debit to the account balance associated with it. As you can imagine then, our auto-recharge feature could be called any moment to make sure the balance doesn't run out.

That said, we do have the monitoring you describe in place which is how we were alerted to the incident and limited the damage to 1.4% of the account base.

Billing Incident Update

Related Stories