Gmail is having an outage

https://outage.report/gmail#2019-03-13

by gdeglin • 7 years ago

316 123 7 years ago

byoung2• 7 years ago

I was trying to send an email and kept getting "Oops there was a problem" errors, but it appears to be working now.

It isn't just gmail, Google Music, Google Play, Google Drive, and others are also impacted. Seems to be a large scale Google outage of some kind.

jjeaff• 7 years ago

Maybe unrelated but a large portion of the Google Cloud web Console was down yesterday for hours.

noitsnot• 7 years ago

Add YouTube to the list.

sitkack• 7 years ago

Youtube was giving me 500s in ways that have never occurred before.

jsaxton86• 7 years ago

YouTube seems to be in this amazing degraded state for me where videos play just fine, but it isn't showing me ads.

cazum• 7 years ago

Google photos is absolutely struggling to load my jpegs right now, though I'm not sure how separate that service is from Drive, if at all.

awill• 7 years ago

I would bet they're both using the same storage layer (Google Cloud Storage?)

r_deckard• 7 years ago

I also have problems with firebase virtual device testing service and google play console api sice yesterday. I keep getting 500 as response, dont know if related though.

sureaboutthis• 7 years ago

No issues here in the Midwest.

cableshaft• 7 years ago

Same, at least not yet. Chicago area here. It does seem slower than usual, but might just be my imagination.

quickthrower2• 7 years ago

All Ok in Sydney

laurencei• 7 years ago

same - I'm in Sydney - no issues (so far).

Perhaps it's user based, rather than location based?

codemati• 7 years ago

Seems to be. I'm on the same network as others not having any issues.

PudgePacket• 7 years ago

Having issues in Sydney :)

rorykoehler• 7 years ago

Half of my colleagues had issues in Sydney. OK in Singapore apart from images attaching to docs in drive

AceyMan• 7 years ago

Can confirm wrt Drive. I was just at work trying to upload files (and one-deep folders) and barely anything completed. I finally gave up.

Our edge/gateway is in metro SEA, fwiw.

ardenpm• 7 years ago

Definitely having problems in Melbourne, Australia

prawn2• 7 years ago

In Adelaide also. Things I'm experiencing:

  can't reliably download attachments in Gmail
  messages not getting through in Hangouts
  constant "Oops" auto-save modals when typing Gmail messages

Gustomaximus• 7 years ago

Brisbane too - Google Photos would sync about the time you wrote this comment.

samplatt• 7 years ago

GMail's working fine for me (8min later) in Melbourne, Australia.

fourier_mode• 7 years ago

Works fine in Illinois.

mnyary• 7 years ago

Having problems in Budapest, Hungary.

mav3rick• 7 years ago

What's the tech scene in Budapest like ?

samstave• 7 years ago

He told you already, Hungry.

gabeh• 7 years ago

Google seems to acknowledge the reports now. https://www.google.com/appsstatus#hl=en&v=status

gowld• 7 years ago

https://status.cloud.google.com/

https://status.cloud.google.com/incident/appengine/19007

" Elevated error rate with Google App Engine Blobstore API and App Engine Version Deployment

Incident began at 2019-03-12 19:49 (all times are US/Pacific).

Mitigation work is currently underway by our Engineering Team. We will provide another status update by Tuesday, 2019-03-12 20:45 US/Pacific with current details. "

etaioinshrdlu• 7 years ago

Status page is down for me.

cheerlessbog• 7 years ago

That page is not very color blind friendly, incidentally. The information is distinguished only by color.

tomcam• 7 years ago

Aren’t the icons all different?

crazygringo2• 7 years ago

The orange dots are much darker and the green ones are much lighter.

I'd assume anyone who is colorblind can still distinguish between the two based on brightness and not color?

Is there any reason to think that's not the case?

R4V3• 7 years ago

Hit that feedback button in the bottom corner.

0x8BADF00D• 7 years ago

Noticed this a few minutes ago. It’s pretty rare for Google to have an outage, don’t they have massive replication and failover?

baroffoos• 7 years ago

Youtube was down for a few hours late last year.

rubicon33• 7 years ago

Fine here in CA.

stevenwoo2• 7 years ago

I'm in Mountain View, receiving/sending email works but downloading full size attachments does not for me personally.

taeric2• 7 years ago

I just emailed some folks images. Got a few warnings of something going wrong, so I then checked in my outbox. It showed that the images went, but there was a message that the "virus scanners were down, download at your own risk." Or some such. Seems to be back to working now, so I just assumed it was a time delay.

Edit: Just saw it again. Full message is: Gmail virus scanners are temporarily unavailable – The attached files haven't been scanned for viruses. Download these files at your own risk.

First time I've seen something like this, actually kind of glad they had this failure case coded for. :)

phantom784• 7 years ago

Also Mountain View, but had issues sending/receiving as wells. I'm on Comcast, you?

jszymborski• 7 years ago

And in Montreal, Canada too

AndrewWarner• 7 years ago

YouTube seems down too:

https://outage.report/youtube

the4ner• 7 years ago

drive, maps, photos all having issues as well

simonhorlick• 7 years ago

I'm getting a ton of 503 Service Unavailable errors from the Google Drive API. I can't wait to see what caused such a large outage.

carmate383• 7 years ago

Can't attach files to drafts nor send emails from Philippines

bluenose• 7 years ago

Elizabeth Warren is winning :(

nickchuck• 7 years ago

SHE'S HACKED THE SYSTEM

jhgg• 7 years ago

The root of this is definitely GCS. We noticed some keys in a GCS bucket go inaccessible since like 19:15-19:20ish PDT, and then we noticed increased timeouts/503's ramping up at 19:30 and plateauing at 19:45. But error rate for us seems to have gone back to nominal levels since 20:10... but the keys that originally went unavailable are still returning 503's.

Fun times.

jedmeyers2• 7 years ago

GCS might be relying on the same system as Gmail/YouTube does, so it’s not necessarily GCS itself. Spanner?

parent5446• 7 years ago

Blobstore

mehrdadn4• 7 years ago

It will be very interesting to see what timing issues might be involved if this is indeed due to Spanner.

ohaideredevs• 7 years ago

If Google's GCP marketing material is to be believed, you are truly on their infrastructure on GCP, so if GCP is having issues so are Google's main services.

jchallis• 7 years ago

Down in Singapore

dickeytk• 7 years ago

I had trouble replying to an email but it worked once I removed the images inside the quoted signature. Makes me think this is file-related.

minikomi• 7 years ago

Street view is iffy as well

graposaymaname• 7 years ago

Fine here in Kerala, India.

EpicEng• 7 years ago

Working fine in Austin TX

giarc• 7 years ago

Down in Calgary still (10:24pmMST). Was going to do work but not confident my email drafts will save.

awal2• 7 years ago

My bad guys. I'm running out storage space on my google account and I think I might have pushed it over.

mehrdadn4• 7 years ago

On Gmail or on Drive? :-)

jedmeyers2• 7 years ago

Isn't it shared between services?

awal2• 7 years ago

Would you believe I have over 10GB of old emails sitting around?

sn_master• 7 years ago

Google Photos too. Unable to download images :(

nukemandan3• 7 years ago

First time this has happened to me. Historic moment. Might be time to get less completely reliant on google?

nukemandan3• 7 years ago

AND we are back

nukemandan3• 7 years ago

May I ask - why the downvote? I know it was a blip, but still, a reminder for a moment that most of life relies on them. And some of my data is not replicated else ware... that is changing.

diminoten• 7 years ago

I personally downvoted because of how a single incident is being extrapolated into a new trend by you, and that feels unhelpful/fear mongering.

zerop• 7 years ago

Is Gmail onto kubernetes?

ttul• 7 years ago

I think it may be a touch beyond Kubernetes.

hknd• 7 years ago

Google is not using kubernetes internally but it's predecessor "borg". Kubernetes was build from scratch with the same idea, but not used internally.

https://www.quora.com/How-are-Borg-and-Kubernetes-different

londons_explore• 7 years ago

The general sentiment seems to be that Kubernetes is better than Borg in almost all ways.

The only thing keeping Google on Borg is the massive amount of work to migrate.

BucketSort• 7 years ago

Living for the postmortem on this one.

bsilvereagle• 7 years ago

As far as I know, there wasn't a public postmortem for the Oct 16, 2018 YouTube outage. Based on that, I'm not expecting one for this outage.

jpatokal• 7 years ago

"Red" incidents on the GCP status dashboard, like this one, will typically have public incident reports.

Disclaimer: I used to work in GCP Support, but no longer do.

afniljl• 7 years ago

YouTube ain't a Gsuite service with SLA. So this is much more likely to have a public report. Whether it's detailed or not is another thing. Also it's probably due to the underlying GAE or GCS issue is my guess.

jimmy_ruska• 7 years ago

Sorry, Bill tripped on the server cable

sachin18590• 7 years ago

lol. email outage is the new dog ate my homework

vinayan3• 7 years ago

In CA.

Colaboratory Research is experiencing issues loading and saving notebooks in Google Drive.

Sad :(.

wybiral• 7 years ago

Happy 30th birthday, Internet. Enjoy your consolidated cloud infrastructure.

kerng2• 7 years ago

Fitting that something like this happens when we celebrate 30 years of the decentralized world wide web!

wereHamster• 7 years ago

Only (some) Google services are affected. My email still works, as it's hosted elsewhere.

ian0• 7 years ago

Email is working for me, but attachments are failing. Opening up the console and it has a message about a database connection loss.

skilled• 7 years ago

Attachments are definitely down for me. Can't view received images.

ssambros• 7 years ago

If anyone is interested: Memorystore for Redis is experiencing intermittent failures during instance creation. Existing instances are not affected.

ghthor• 7 years ago

Mercury, god of communication, retrograde is kicking it up a notch this year.

908087• 7 years ago

Google forgot to charge their energy crystals?

mholt3• 7 years ago

Of course this would happen while I'm trying to submit my thesis at the last minute. https://twitter.com/mholt6/status/1105703745143205889

nurettin• 7 years ago

Time to register for a competitor email service that doesn't use google infrastructure. Live.com, maybe?

diafygi2• 7 years ago

Good thing https://send.firefox.com/ was launched today :)

mholt3• 7 years ago

Except I'd still have to send the link in our email thread!

cedricium• 7 years ago

Google Maps (specifically Street View) is showing up as a black screen for me as well.

xbmcuser• 7 years ago

Seems to me a location specific breakdown as all services I tried like gmail, photos and youtube all are working for me

config_yml• 7 years ago

Probably caused by a typo in the configuration for the new region they just launched ;)

aj7• 7 years ago

Is having to load two tabs of gmail to get my mail to actually appear also a type of outage?

bluedino• 7 years ago

So this is why we couldn't send emails last night (9pm est) Watched Youtube instead :)

Tsubasachan• 7 years ago

Ah gmail. Where spam goes to die.

QuamStiver• 7 years ago

Happy that I was sleeping at that time.

thrusong• 7 years ago

I thought outages weren't supposed to be posted here? Every time I've posted one about Facebook, it's been flagged and removed.

Gmail is having an outage

Related Stories