How upgrading to Google Apps for Business killed my company's email for 6 hours

http://floriancornu.blogspot.com/2013/05/how-upgrading-to-google-apps-for.html

by ValentineC • 13 years ago

143 110 13 years ago

floriancornu.blogspot.com

bravura• 13 years ago

tldr; Author experienced an 6-hour outage in Google Apps service, after upgrading to a paid Google Apps account. This behavior is not documented.

sneak2• 13 years ago

Is "outage" the right word? That suggests that it's an error, or down for everyone.

They really screwed the pooch, here.

PommeDeTerre• 13 years ago

It's still an "outage" if even just one user is affected. In fact, it's still an "outage" if the service is unavailable (which includes being temporarily unusable) for some period of time, and no users are actively affected by it.

300bps2• 13 years ago

Is "outage" the right word? That suggests that it's an error, or down for everyone.

As someone who has been in IT for 20 years, I can tell you're in IT. This type of semantics drives business users crazy.

Business User: Hey my system is down!

IT Person: No, it's not down.

Business User: I can't log in to it.

IT Person: We are doing scheduled maintenance so the system is unavailable, but it's planned maintenance so it's not down.

darrellsilver2• 13 years ago

The same thing happened to us as http://www.thinkful.com/, only it took 18 hours to come back! Customer support was hilariously useless as well: "Yes, happens to everyone." "No, there's nothing we can do about it."

For us, accounts that were forwarding mail someplace else continued to work, but anyone that had to log into Thinkful's gmail account didn't.

corresation3• 13 years ago

Customer support was hilariously useless as well: "Yes, happens to everyone." "No, there's nothing we can do about it."

The time delay is ridiculous and unacceptable, but it sounds like customer support did the most they can possibly do in that case: Engineering had implemented a (questionable) migration policy, and there are no toggles or switches that customer service can fiddle with to make it happen more quickly.

taga• 13 years ago

Disclaimer: I work for Google

Google's support is a complete joke. The only way to get things fixed in a timely manner (if at all) is to ask someone who works for Google to escalate internally. Then you stand a chance. Otherwise you're SOL.

On top of that the Gmail team routinely ignores internal feedback before rolling out new features or UI changes claiming the Googlers do not represent average users.

I would recommend the OP to contact a Googler friend if they have one.

Google's arrogance and disrespect for its users will be its undoing.

rasterizer• 13 years ago

So the upgrade took a few hours to complete and it didn't happen instantly in defiance to the laws of physics. Next time play it safe and upgrade during the weekend.

The post is dated May 27, is Google planning to announce a new feature for Apps this week and this is some sort of a preemptive PR attack?

robryan2• 13 years ago

6 hours... We run a 4 person company and 6 hours during the weekend would still be a problem for us.

Given a company larger than 10 might actually upgrade (small amount of shared email accounts) I could see it causing big problems for some.

justincormack6• 13 years ago

Really? Do you run a highly reliable email service yourself then? I certainly would not use gmail if that were the case. You should seriously look at why you are so reliant on email, as it is not really a reliable service, the mail servers at the other side could easily delay mail for 6 hours, it is only a best efforts eventually consistent protocol with very long delays allowed (you don't normally get bounces for days if the mail is queued for retry).

VLM2• 13 years ago

How do you get 24x7x365 coverage with only 4 people? The "standard" is it takes about 5 to fill a position 24x7x365 over very long term on a larger average with absolutely no failure (unstaffed) tolerated. As in, 50 people can fill 10 positions absolutely positively all the time, but it doesn't downscale well at all to just 5 people and one desk. You can "fill" ten positions with a lot less than 50, but even at scale there's going to be a heck of a lot of time when there's only 8 or 9 people there hence the quotes around "fill". For instance you can "fill" 10 positions with 30 people but there's going to be a heck of a lot of time where the supposed 10 positions only have 7 (or even fewer) bodies actually on deck.

Depending on the business sector of course. If you're a stereotypical weekday business then a 6 hour outage at 9am on a weekday would be a disaster, but a 6 hour outage from midnight to 6am on Sunday morning wouldn't even be blinked at because no one cares.

frankacter• 13 years ago

To play devil's advocate:

1. There is no indication in the upgrade process that the upgrade would "take a few hours to complete". Quite the opposite, it indicated there would be no downtime.

2. The customer service reps seems to have little insight into what was happening with the account which is a bit scary. I'm always nervous of black hole "processing windows" where all you can do is wait and hope for the best. It's bad when it is customer facing, it's worse when it is customer service facing.

3. Upgrading on the weekend could potentially be more troubling given that customer service may not be staffed with the quantity/quality to fix issues if they occur. Not saying this is or is not the case for Google, but running into problems on a Saturday morning and getting a "call us back during normal business hours" is just as troublesome.

What should have happened in my opinion?

1. Google should document that there can be a temporary delay of x - x hours while the upgrade happens.

2. Florian should have scheduled the upgrade ahead of time with the team, letting them know that worse case there may be an outage of 'x' minutes/hours/days.

3. Google customer service should have better tools as to upgrade process for both the admin (customer) and customer service rep so that it is not a black hole of 'wait and hope'. Even a step 1 of x, or you are number # in queue, or estimated time, etc.

mehwoot• 13 years ago

So the upgrade took a few hours to complete and it didn't happen instantly in defiance to the laws of physics.

What law of physics necessitates that such an upgrade takes 6 hours?

chii• 13 years ago

What law of physics necessitates that such an upgrade takes 6 hours?

the one where the customer isn't paying enough money to have somebody working 24/7?

If your business depends on a particular service, you don't accept something vague - you get things signed in contract with legally enforcible SLA's. If you can't afford that, then you just have to live with shitty service.

Negitivefrags• 13 years ago

It wasn't that it took a long time. It's that the interface showed a state as if all the mail was deleted. That could have been devastating.

We recently upgraded to paid Google Apps for Business and we didn't get any downtime. This is how it is supposed to work.

Unfortunately we also had irritating problems after the upgrade. We upgraded because our customer support email address had run out of mail quota. Paid google apps has a 10x higher quota, but upgrading to premium didn't increase it.

After ringing customer service they refused to increase it immediately and said that it would be increased "in a couple of months time".

Meanwhile hundreds of our customers were going unanswered.

I suppose I can imagine a scenario in which they would want to wait until after credit card chargeback window before lifting a quota limit, but their support department should understand the paralysis of a business offer to solve the problem.

We had to create a temporary support2@mycompany.com email and manually port email across which wasn't fun and played hell with our ticketing system.

nitid_name• 13 years ago

Why not pull down/delete old mail off your support@example.com email address?

I can't imagine you have 5 gigs (or however many they give you these days) of tickets that can't deal with a few days of cold storage.

corresation3• 13 years ago

Most people would assume that this is nothing more than an administrative change on the account, with no long drawn out six hour process. And if there is an upgrade time required, it absolutely should have in-your-face warnings given the importance of email 24/7 for many organizations.

Even if we buy that this is more than an administrative change and it somehow moves to better hardware, this is a problem that I would think that Google would have built to a mostly transparent process -- at most long term archives are unavailable after a very brief initial migration, etc.

bigiain4• 13 years ago

That exactly the sort of assumption you can't afford to make if you've honestly got company-destroying problems if your email goes down for 6 or even 24hrs.

It's easy in hindsight, but I've seen that happen before, and I have no doubt that if you'd considered the risk and (perhaps ironically) googled for information, you too would have known about this.

On the other side of the coin (and perhaps the reason Google haven't cared enough to fix the problem), SMTP is nicely designed so as to not result in this sort of thing losing any mail - "well behaved" email systems will just queue and retry mail for 5 days if needed.

_b8r0• 13 years ago

So the first thing about Google Apps is that you still need to back it up. Because it's (mostly) pretty easy to use it's easy to fall into the trap that it does everything right first time and that your data is safe when you make a change, unlike say running your own infrastructure. It isn't.

surferbayarea• 13 years ago

We had another issue, where the catch-all address started bouncing. Apparently they have changed the way they handle email routing. So if you upgrade to the 'new google apps', you better be very careful about your email routing rules. Went through a very similar experience with google customer care. Again turned out to be a bug in their system. But what sucks is for 3 days our emails were bouncing! And we're a paid google apps customer...Would be happy to pay 5X, just want some sort of customer service + acceptable SLAs on issues. And yes, please have a test team go over important features like enterprise email! I know it's Google policy to not have features go through a testing team, but maybe they should consider it for Google Apps.

/end{rant}

nivla• 13 years ago

If you are relying on a third-party cloud email service, please keep a regular backup of your data. From random hacking attempts leading to a account lockout to incorrectly being marked for spamming violations, anything could go wrong. The easiest way is to install Thunderbird and let it automatically sync and backup your emails to your computer.

VLM2• 13 years ago

From past experience at an email service provider, admittedly a small one, and quite a few years ago, the most likely failure mode related to internet outages knocking out the customer's internal email unlike a locally hosted email server which customers could never seem to understand, and DNS issues (failure to renew, got scammed / domain stolen because of noobs, etc). Probably the single biggest support problem was people (on either side) getting on spammer lists so genuine business communication can no longer flow.

Generally speaking the web interface being down, or the imap server being bonkers, was very rarely if ever a problem.

dangrossman• 13 years ago

Rackspace Mail. $2/mo/mailbox. 100% uptime SLA. 24/7/365 phone, email and chat support -- you can call and talk to someone with power to solve your problem at 3AM on Christmas Eve. Big mailboxes, configurable backups, good spam filters. Instant e-mail notifications in your choice of native desktop/mobile clients with IMAP push, or use their hosted webmail.

If you aren't fully addicted to the Gmail interface, there are better options for business mailbox hosting.

lubos• 13 years ago

Or $19/month get a Mailgun.net account (also run by Rackspace) and use it as a redirection service into individual free gmail (or google apps) accounts.

dools• 13 years ago

I do the same thing for free with my domain registration company and then pop all my messages off gmail using fetchmail for good measure. I then send through an external smtp server. All the fun of gmail, no reliance on Google. I actually also tried to pay them before setting that all up and they couldn't sort it out.

thibaut_barrere• 13 years ago

Similarly, if you use DNSimple you can enable email-forwarding on your domains [1]. That's the solution I use for my SaaS product so far.

[1] http://support.dnsimple.com/articles/email-forwarding

andybak• 13 years ago

No reason not to use a 3rd party IMAP/POP and still use the Gmail front-end.

Best of both worlds.

mike-cardwell• 13 years ago

You're just adding more points of failure.

cpswan• 13 years ago

The polling interval for POP (there is no IMAP) can be up to an hour, which I find sufficient reason not to be happy with that approach.

JohnTHaller6• 13 years ago

There were multiple reasons right in his post. Including the actual support options like being able to call someone at 3am. And the service level agreement that states it has to stay up or you don't pay for it. And the regularly scheduled backups that you know are occurring. Gmail has none of that.

msy• 13 years ago

I've been using Rackspace mail for a few years now, flawless service and shockingly good customer support - helpful, competent people available instantly, cannot recommend enough. Given our email accounts our the skeleton keys of our digital lives relying on a free service, or even a paid service where support is something the company sees as a numbers game seems madness.

My only gripe is the lack of support for two-factor authentication, apparently this is under development currently.

kngspook2• 13 years ago

My only gripe with Rackspace is how long it takes for them to push out a new version of Exchange when they're released... Was something like a year and a half after Exchange 2010 was released before it appeared on Rackspace? Curious to see how long before 2013 appears...

Otherwise I love them.

Osmium2• 13 years ago

I've recently converted to Fastmail.fm, which seems great and they have a nice web interface too. They're now owned by Opera, so I assume their data retention policies must be consistent with EU law, though I think they have servers based in the US so I'm not sure what that means in practice.

What I'd like is a service that kept my emails encrypted on-disk with my user password. Not because I think emails are private -- they're not in the sense that I fully expect someone could intercept them in transit, even if that would be illegal -- but just because it would give reassurances that my emails can't be data-mined for advertising purposes à la gmail, or if the mailserver had a major security vulnerability, a hacker couldn't copy all my emails wholesale.

But I don't know of any service that provides this, nor of a way to do it myself easily (any mailserver I set up myself would probably have far more vulnerabilities than one I rent from someone else!).

JohnTHaller6• 13 years ago

Unfortunately, most folks would not pay the extra fees that would be necessary for implementing a service like this. And the NSA wouldn't like it as they couldn't just ping Google's server and read all your mail from Gmail.

voidlogic• 13 years ago

Or a $10/mo Digital Ocean VM running Zimbra 8 Community Edition.. https://s3.amazonaws.com/uploads.blog.zimbra.com/wp-content/...

ishbits• 13 years ago

Hows the Rackspace web mail interface?

ozataman2• 13 years ago

That's great - I love worthy and proper competition combatting nonchalant/tyrant attitude of incumbent giants. It is so annoying to deal with customer service reps that act like they are your boss and you're inconveniencing them through your questions. The stronger the company, the more of those reps you'll get.

Thanks for posting this!

eli2• 13 years ago

SLA is sort of a red herring here.

If my email goes down, getting a partial or full refund on my $2/month mailbox fee is the least of my concerns.

e1ven• 13 years ago

The point of the SLA here isn't so much to save you the $2, but to make it cost them $2 * N accounts. This gives them an economic incentive to avoid problems.

rgbrenner2• 13 years ago

I used to love rackspace... but had a really bad experience recently. I was using Rackspace Files as a CDN for my ecommerce site. One of my customers emailed me saying that the graphics were missing from the site. Turned out 2 of the 3 zones were down, according to status.rackspace.com. They fixed it about 12 hours after it started IIRC. They never sent an email notifying me of the down time.

Then -- and this is where they destroyed my trust in their service -- I took a look at the status panel the following day, and they cleared the history of the downtime (it shows 30 days of issues). Instead of showing the issue, they marked it as green and deleted the messages describing the issue.

That day, I stopped using Rackspace for everything. (This was about a month ago)

rgbrenner2• 13 years ago

Interesting.. I just looked at their status panel, and the downtime is there, including all of the messages. They were definitely missing the day after it happened.

Still disappointed they don't send out emails when there's downtime..

Would probably use them again the future. (This was my one negative experience with them, btw)

Bjoern• 13 years ago

100% uptime is technically impossible. For eMail its not even necessary as failure to deliver will result in automatic retry for some time.

gte910h• 13 years ago

100% uptime SLA doesn't mean it will be up 100%, it means you get paid for every X is isn't up in that period.

fareesh• 13 years ago

I've known of a few cases where the Gmail account associated with Google Apps becomes unreachable. Rather than upgrading to the paid plan, most of the people involved just opened a support ticket which was resolved in a few hours.

codev2• 13 years ago

I had a similar situation where I switched a 50 person charity to Google Apps. They had a 30 day trial period and during that time the CEO's credit card was stolen and replaced. I changed the information but the Apps interface said 'No valid payment information stored' so I called their support and they assured me the new card was stored and everything would be fine.

Surprise, surprise on the day the account totally shut down. I called in a panic and after a lot of fiddling about (and reassuring me that they had recorded that I thought there might be a problem) they got it back up with everyone's mail after almost a whole day of downtime.

To add insult to injury Google don't run a charity program in the UK despite the non-profit laws being a lot stricter than the US and it being easy to validate a charity - so they're paying full rate for Google Apps for Business.

shrikant2• 13 years ago

That last part is not entirely accurate. I did some IT volunteer work for a charity very recently, and they definitely had a free Google Apps for Business setup. This is in addition to a reasonably generous grant in the form of AdWords credit.

ValentineC2• 13 years ago

Would you have any idea who we're able to contact about this? I'm involved in a registered UK charity and we were looking to use Google Apps before, but can't justify the monthly cost of $5 per account.

Thanks!

codev2• 13 years ago

Google Apps for Nonprofits page states it's only available in the US:

https://support.google.com/nonprofits/answer/1614602?hl=en&t...

If you could ask someone involved in that charity how they got free Google Apps I'd appreciate it as I work for a number of charities and the costs of Google Apps are a significant admin overhead. I've tried calling the sales people and they say it's coming soon but they've been saying that for over 5 years now.

vidyesh• 13 years ago

tl;dr : Whenever you click on 'Upgrade Now' button just click it and go to sleep. Once you wake you everything should be fine.

Author was awake and saw some horrible things.

biot• 13 years ago

The solution to this is for Google to explicitly tell the end user that the upgrade involves a migration to bigger and better infrastructure, during which their accounts will be inaccessible for a number of hours. Then let the user schedule the migration to minimize the inconvenience. During the migration, show the account status as "Migrating..." instead of empty. Problem solved.

dpcx• 13 years ago

But that would involve Google adding documentation to their system. Which, from my perspective, seems very difficult for them to manage.

Silhouette3• 13 years ago

Sorry, but that isn't even close to a solution. The problem here is that a business-critical facility was off-line for pretty much an entire business day, at who-knows-what cost to the business.

Many small businesses that use facilities like this are trading internationally or otherwise need 24-hour functionality, so the idea that you could just schedule something to "minimize the inconvenience" doesn't really help them at all.

cstejerean• 13 years ago

Just because it doesn't work for everyone does not mean that a scheduled outage would not be a significant improvement for large percentages of customers. Most businesses can find a 6 hour window.

bigiain4• 13 years ago

I have some clients who are paying MessageLabs something like two orders of magnitude more than Google charges per seat. They would, undertstandably, scream blue murder if their entire corporate email went down for 24hrs.

I have other clients who wouldn't dream of paying $5 per user per month, because they get free email forwarding with their domain name or cheapo web hosting.

I have _many many_ clients, who if I gave then sufficient notice and said "we're upgrading some infrastructure, we need to take the mail server down for up to 24 hrs, does after close of business the Friday after next suit you, or would you prefer we did it Sat night/Sun morning?", would be _perfectly_ happy. (with the possible exception along the lines of "oh, we're at a trade show that week, can you move it forward or wait till a week later?")

I wonder how big the crossover is between companies who really absolutely _have_ to have 24/7/365 email uptime at better tha 3 nines, but who haven't already got something more robust in place than a handful of personal gmail accounts?

I can't help but think they're "doin it wrong"…

JohnTHaller6• 13 years ago

It wasn't business-critical before, since they were using the free version. Businesses don't use free services with no SLA for business-critical things. Unless they don't know how to do business.

rexreed• 13 years ago

I've been POPing my email since the 90s... am I doing it wrong? Yes, accessing email archives on multiple machines is a kinda-headache (I replicate my mailboxes), but never lost email.

brokentone2• 13 years ago

Killed it for 6 hours... not convenient, but probably shouldn't make big IT changes during business hours and without warning staff. Our company's move from GApps to Office365 + Outlook has been much more systemically affecting...

JohnTHaller6• 13 years ago

I think the issue is that this wasn't a big IT change. It was a simply addition of functionality to a cloud provider, which is SUPPOSED to be easy. Of course, they were using the free version of Gmail, which has no SLA and can be shut off at any time for any reason. Which means they didn't consider email that mission-critical.

kngspook2• 13 years ago

How are you liking Office365?

brokentone2• 13 years ago

Hate it.

qwerta• 13 years ago

And where is your backup? Oh wait you dont have a backup...

mrt0mat02• 13 years ago

I used to be IT manager of a company that wanted to save money and they had me switch them to google business. it was a smooth transition, but the company wanted it's employees (50ish non tech salespeople) to still use Outlook. It was ok. google had an app that allowed this. All was good. I left the company but my friend took my job from me when i left. He now has come across the problem of google not supporting the newest outlook. he gets new computers in and has to downgrade the outlook just to get email to work. headache

Karunamon• 13 years ago

They recently got the Office 2013 connector out there, for what it's worth.

Agree that the transition took ENTIRELY too long.

mrt0mat02• 13 years ago

thanks. i hope he noticed. i'll mention it to him.

joemaller1• 13 years ago

Different issue with Google Apps: We work with some big pharma companies. During the free evaluation, Google flagged two of our partners and one of our project managers as spammers and silently suspended their accounts. We hosted our email somewhere else.

thracky• 13 years ago

I migrated a company with 60 franchises from a broken, old, 25 MB per mailbox mail system, to Google Apps last year.

The biggest suggestion I have for anyone wanting to use the paid Google Apps is to go through an authorized reseller. You will get better pricing, and they generally have faster access to higher level support than going directly through Google. I had an absolute nightmare of a time with client support (construction workers with little computer literacy) but any legitimate problems during the transition were quickly dealt with by Google via the reseller.

Also, good luck getting a response from the Google Apps sales team in the first place.

wc-• 13 years ago

For whatever it is worth, here is my experience from a few hours ago: my google apps trial ended this morning. Google emailed me, I clicked the billing link in the email and entered my info. After submitting, my account was immediately reactivated and I received a few emails that had arrived while the account was suspended. No downtime or anything... Maybe it has to do with how much data your accounts contain when you switch over to paying for access?

fpgeek• 13 years ago

For a trial, I'd expect you started on the paid infrastructure from day 1, even though you hadn't paid yet. That would mean no migration and no downtime.

How upgrading to Google Apps for Business killed my company's email for 6 hours

Related Stories