Gmail back soon for everyone

http://gmailblog.blogspot.com/2011/02/gmail-back-soon-for-everyone.html

by mjfern • 15 years ago

79 50 15 years ago

gmailblog.blogspot.com

ck2• 15 years ago

How on earth do they backup all that to tape?

Must be a massive amount of tape.

amock• 15 years ago

If you look at http://www.oracle.com/us/products/servers-storage/storage/ta... it talks about the reduced cost of ownership for a 20PB tape library, so there are tape products available for large volumes of data. The SL8500 supports 500PB of raw storage.

jedsmith2• 15 years ago

If everybody filled their mailbox, 500 PB would back up 66.1 million Gmail accounts. Since they don't, we're probably looking at everybody on Earth: 73.8 MB per mailbox, which sounds like a pretty good shot at an average.

Looking at the SL8500, I think I'd intentionally lose data just so I could play with it to restore it.

bdonlan• 15 years ago

The hard part is backing up that kind of volume of data frequently enough that you always have a recent backup :)

snewman• 15 years ago

To quote from The Ringworld Engineers: "The Kzinti Patriarchy is not normally terrified by sheer magnitude." I can safely say that the same statement applies to Google.

sacrilicious• 15 years ago

Total conjecture, but 3 ways come to mind: De-duplication, compression, and hierarchical storage management. Keep less-frequently accessed parts of the data 'Nearline', and snapshot the lower-demand, older data slightly less often. Continuous data protection appliances were probably designed and tuned specifically for this problem set, and one could assume it was built into the original architecture so that an appropriate amount of the infrastructure already had backup space in mind and accounted for.

tdfx• 15 years ago

"If you were affected by this issue, it’s important to note that email sent to you between 6:00 PM PST on February 27 and 2:00 PM PST on February 28 was likely not delivered to your mailbox, and the senders would have received a notification that their messages weren’t delivered. "

That's pretty scary to me. I would've hated to have gotten any e-mail alerts or payment due date reminders during that time.

tomjen3• 15 years ago

Since they got a warning that the email address was temporarily unavailable (at least that's what they should have received) any well configured mailserver should have resend the email.

kgo3• 15 years ago

Yeah, scary would be no bounce, no delivery. Bouncing is relatively graceful.

wtallis2• 15 years ago

From a bounce I got:

    ...

    From: Mail Delivery Subsystem <mailer-daemon@googlemail.com>
    To: [me]@gmail.com
    X-Failed-Recipients: [someone else]@gmail.com
    Subject: Delivery Status Notification (Failure)
    Message-ID: <20cf30433ea07a2745049d4fe20e@google.com>
    Date: Mon, 28 Feb 2011 04:12:14 +0000
    Content-Type: text/plain; charset=ISO-8859-1

    Delivery to the following recipient failed permanently:

         [someone else]@gmail.com

    Technical details of permanent failure: 
    Account disabled

    ----- Original message -----

    ...

Not exactly the kind of thing that would make you think resending would get the message through.

there2• 15 years ago

i love how they keep mentioning "0.02%" so you think it's some insignificant number, but it's actually something like 150,000 accounts.

mhansen• 15 years ago

0.02%

mitjak2• 15 years ago

And how many of those are likely to be active and be expecting serious email instead of newsletters and spam?

kenjackson• 15 years ago

I presume the same percentage as all gmail accounts. I saw nothing that indicated that these users were less active than anyone else.

Unlike Hotmail, I think a very large percentage of Gmail accounts are pretty active.

foobarbazetc• 15 years ago

In what world does that matter?

I think the word "tape" tells you all you need to know about how serious this issue is.

kgo3• 15 years ago

150,000 accounts out of SEVEN HUNDRED AND FIFTY FUCKING MILLION had trouble accessing email for LESS THAN A SINGLE DAY on A WEEKEND. No accounts or emails were deleted.

I'd argue that is an insignificant number.

there2• 15 years ago

you'd probably have a different attitude if it were your account.

bkhl• 15 years ago

I really hope that I'm not using anything you created on the web. I would hate to be a customer of a service whose owner think 150,000 users are insignificant.

edanm• 15 years ago

Not that I'm disagreeing with you, but the 750 million figure is way off. According to Wikipedia:

"As of November 2010, [gmail] had 193.3 million users monthly"

ZoFreX• 15 years ago

While I partially agree with you...

> No accounts or emails were deleted

This is not true. During the affected time period, those accounts could not receive email, and sent out permanent failure bounce messages. If you were one of the affected and you were sent an important email during that time... You didn't get it, and you might never get it.

ronnix• 15 years ago

It's been more than a day for me. It's Tuesday and I still have not regained access to my GMail account...

panarky• 15 years ago

And that's bad compared to what exactly?

What alternative is online 24x7, with no downtime for backups, upgrades or power failures?

Does on-premise Exchange do better? Lotus Notes? sendmail?

mda• 15 years ago

Even if you assume there are 200 million active accounts, %0.02 of it makes 40.000, not 150.000

akent• 15 years ago

"We released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email. When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version."

Some bug!!

Cariapa2• 15 years ago

I can't remember having this sort of problem with yahoo mail. Then again they don't build their own filesystem.

jasonkester• 15 years ago

Yahoo is far worse. When they lose your mail, it's gone.

Yahoo Mail lost two of my accounts over the years. The first time was early in the service, and they seem to have lost all their accounts at once. Suddenly you couldn't log in, and there was an apology message saying that you'd need to create a new account. And that you couldn't get the same email address back, which is why my @yahoo.com address had a 2 at the end of it for several years...

Until they locked me out of it one day. As in, I checked my mail at night, then in the morning my password didn't work anymore. Since they have no human beings working for them, an entire month's worth of attempts to get access back resulted in nothing but a big pile of form letters and links to the Forgot Password page.

So no, you're mistaken. Yahoo does have this problem. If you have an account with them I'd strongly advise you to back everything up. Then switch to Gmail.

iwwr• 15 years ago

Yahoo also charges a monthly fee for access to pop3. Their web interface is slower and buggier than gmail's. There is free unlimited storage, but a smaller limit on file sizes. The virus scan on outgoing messages is usually slow; if gmail is doing virus scans, it's on the backend, it does not delay the user.

Cariapa2• 15 years ago

Oh, I'm on Gmail already, trust me. But thats because I like the faster interface, search capabilities, being able to check multiple mail accounts, gtalk etc. Yahoo is a great repository for my non-friends and family email (semi-spam).

I guess I've been very lucky that never happened to my Yahoo account.

btilly• 15 years ago

It is worth noting that this is the first significant data loss issue that I am aware of involving gmail. As much as the incident sucks for those involved, that is a pretty good record overall.

bane• 15 years ago

This is great, last night I went to bed thinking "How could Google just walk away from this problem without fixing it, don't they have backups, what about corporate users?"

And this morning "we have backups, we're restoring, oh and we're trying to fix what caused it in the first place"

l0c0b0x• 15 years ago

Would it be safe to assume then, that Google tests it's software upgrades with 0.02% of it's gmail account? (However many thousands clients are affected, it's probably better than affecting ALL their clients).

I wonder how they choose their 'test' client base.

Gmail back soon for everyone

Related Stories