Google Cloud Is Down

https://news.ycombinator.com/item?id=20077421

by markoa • 7 years ago

1395 593 7 years ago

news.ycombinator.com

https://status.cloud.google.com

https://status.cloud.google.com/incident/compute/19003

Status page reports all green, however the outage is affecting YouTube, Snapchat, and thousands of other users.

walshemj2• 7 years ago

just had rolld20 in the USA blow out a game I wonder if it is effected

zoidq• 7 years ago

Looks to be working in the UK

walshemj2• 7 years ago

Network peering probs in Canada - looks like an unrelated problem

RappingBoomer• 7 years ago

youtube streaming is also down

horyzen• 7 years ago

Yeah I was having trouble accessing my Gsuite apps, had a couple of 502s, which led me to check HN. While it doesn't give me 502 now, it's abnormally slow.

nkassis• 7 years ago

The GCE console also affected, couldn't send a support ticket just getting errors.

darkof• 7 years ago

That feeling when you open https://console.cloud.google.com and see that you don't have your Kubernetes clusters and CloudSQL databases, but CTA to create first.

vpontis2• 7 years ago

Gosh, this was so scary... I thought someone had hacked in and deleted everything...

I hope they come back. This is still pretty scary

stnmtn• 7 years ago

Same, my Manager called my and said "everything is down".

So I wander over to my Firebase console, and there's no database loading. Thank god for twitter, and people also saying that they have the same issue or I would have for sure though we've been hacked.

I hope this is a good wake up call for everyone. I know that I'm going to think more about how we do backups and fail-safes

hanniabu• 7 years ago

And here I thought I was having a bad day with Google Play not loading

jjeaff5• 7 years ago

Same. I was thinking, oh, my db cluster must be having trouble recovering. Couldn't get any response through kubectl. Logged in to the cloud console and it looks all brand new, like I have no clusters setup at all.

Of course, this is 2 weeks after switching everything over from AWS.

dpau• 7 years ago

my vm instances are all still there, can even log in via SSH in the compute engine tab. looks like they got a reboot 15 min ago. just restarted some processes but lost my progress on about 12hrs of computing time, i'm guessing it's going to be hard to get a refund..

flamingcow2• 7 years ago

[I am the Cloud SQL tech lead]

This is a networking issue, and your data is safe. Cloud SQL stores instance metadata regionally, so it shares a failure domain with the data it describes. When the region is down or inaccessible, instances are missing from the list results, but that doesn't say anything about the instance availability from within region.

captn3m09• 7 years ago

Mumbai region here, and GKE seems to be fine. (Accessing it from Bangalore)

electron0zero• 7 years ago

Have services running in europe-west1, everything loads and working fine for me. looks like europe-west1 is not affected.

digaozao5• 7 years ago

I cant see any gke cluster in Brazil, or any VM.

miller_joe2• 7 years ago

I'm seeing that with northamerica-northeast1. I can't access anything over the network in that region and most of the GKE clusters and VMs in that region aren't listed in the console

stuck_in_the_ma• 7 years ago

Google Play is also experiencing massive issues.

snisarenko2• 7 years ago

The status page took a while to show issues. My app was down, and Twitter knew google cloud was down before the official status page.

vinayan3• 7 years ago

Looks like only GCE is down according to the status page now. I'm able to access my console for instances and GKE clusters.

snisarenko2• 7 years ago

My site runs on Google App Engine and its down as well.

sladey• 7 years ago

I happened to be initializing a GKE pool upgrade just as this occurred. The upgrade is now stuck according to the console.

The interesting thing is that a couple of minutes before everything went wrong, kubectl returned a "error: You must be logged in to the server (Unauthorized)" error

miller_joe2• 7 years ago

They initially opened an incident against GCE (https://status.cloud.google.com/incident/compute/19003) then opened one against Networking (https://status.cloud.google.com/incident/cloud-networking/19...).

The networking incident looks like the one to follow for updates now.

murat1243• 7 years ago

GCP has been down since 11:50am and they acked it 35 mins later. They're great at leaving their customers in the dark.

copperx• 7 years ago

Not much different from AWS, from what I've heard.

numbsafari• 7 years ago

Definitely the case. Neither are super great at this. One issue is that issues that may 100% impact individual clients may only impact a vanishingly small amount of their overall service load. That mismatch between customer and provider experience is one of the ugly aspects of public cloud providers.

msbarnett• 7 years ago

Yeah, Amazon is the master of having their status page read all Green while half of US-East is in the toilet

xerxes9012• 7 years ago

Couldn't load the support console to "me too" this one either!

macintux2• 7 years ago

And thus was ruined hundreds or thousands of pleasant Sunday afternoons.

I don’t miss being on pager duty one bit. I see it looming in my headlights, sadly.

xerxes9012• 7 years ago

Spare a thought for the pleasant Australian early Monday mornings too! Always a rude awakening...

jagtesh• 7 years ago

Multi-cloud for those times when you really need that level of availability and can afford it.

jjeaff5• 7 years ago

It's not even about being able to afford it. Some things just don't lend themselves to hot failover. If your data throughput is high, it may not be feasible or possible to stream a redundant copy to a data center outside the network.

newsbinator• 7 years ago

It's the Queen's birthday, a Monday off here in New Zealand...

... but not for everybody now.

Scoundreller5• 7 years ago

So what happens when the crown changes? They change the holiday? Immediately? For the next year? Sounds like a bit of a nightmare.

bowmessage• 7 years ago

Do you work at G?

macintux2• 7 years ago

Nope. I was more thinking of everyone else.

jacques_chester2• 7 years ago

The only response is to wait for Google to fix it.

Nothing you or I or the pager can do will speed that up.

I am aware some bosses won't believe that and I am not trying to make light of it. But there really isn't much else to do except wait.

warp_factor• 7 years ago

Or you wait for Google or you are frantically trying to move everything you got to AWS.

mandatory• 7 years ago

Yep, I can no longer see my Cloud SQL database - it's as if I've never created one at all. Really hoping this is just an issue displaying it and that Google hasn't punted my infrastructure and backups.

vpontis2• 7 years ago

Praying isn't working. Now, I'll try sobbing :(

Havoc• 7 years ago

Systematic problem solving. I like it

dodobirdlord6• 7 years ago

Reports claim that it's a network congestion issue primarily in the northeast United States. It seems doubtful that any data has been lost, but you probably can't see resources due to requests not getting through. I hope that hearing this helps you to feel better.

flamingcow2• 7 years ago

[I am the Cloud SQL tech lead]

ksajadi• 7 years ago

GCP status page is worthless as it's always happy and green when production systems are down and then they might acknowledge something an hour later

JimboOmega• 7 years ago

Just like AWS, then. "Some users are experiencing increased error rates" = "Everything has been down for hours"

bsimpson2• 7 years ago

"Everything is fine, unless you're Carl. There's a massage outage, but only at Carl's house. Sorry, Carl."

djsumdog3• 7 years ago

I remember when S3 was down and the status was green because the updates for the status page with pushed via S3.

Analemma_• 7 years ago

Azure too. During the most recent outage a couple weeks ago their Twitter account acknowledged the incident an hour before the status page did.

So no matter where you go for your cloud services, you're guaranteed a useless status page. Yippee.

lionradio• 7 years ago

I think this might be a static page they are hosting on Akamai?

Godel_unicode2• 7 years ago

Doesn't look like it:

https://www.whoishostingthis.com/#search=status.cloud.google...

actuator4• 7 years ago

AWS is no better. Something from 2015 I remember: https://twitter.com/SIGKILL/status/630684777813684224?s=19

duxup4• 7 years ago

I swear most status pages are run by folks who aren't "there".

hinkley7• 7 years ago

I haven’t written a status page in a while, but the rest of my infrastructure starts freaking out if it hasn’t heard from a service in a while. Why doesn’t their status page have at least a warning about things not looking good?

prepend3• 7 years ago

It’s an easy problem to fix as basically services should emit performance data, openly, and the status page should just summarize that. So if a service doesn’t report out, it’s assumed down or erroring out.

Having an excel file where people enter statuses is not very useful to me as a customer. That’s more like a blog.

domenici2000• 7 years ago

They update the page manually.

emilfihlman• 7 years ago

Digitalocean has the same issue: status pages are actually manually updated and no live data is fed into them.

intellix• 7 years ago

Was noticing massive issues earlier and thought that maybe my account was blocked due to breaching from TOS as I was heavily playing with Cloud Run. Then I noticed gitlab was also acting up but my Chinese internet was still surprisingly responsive. Tried the status page which said everything was fine and searched Twitter for "google cloud" and also found nobody talking about it. Typically Twitter is the single source of truth for service outages as people start talking about it

arach• 7 years ago

Confirming issues on our end. I'm able to load up my console but when I go to Kubernetes Engine, I don't see my clusters. I'm monitoring closely on twitter

tr33house• 7 years ago

Took me a while to track latency issues to GCP. Wasn't expecting it. This also seems to affect some GAE instances and some of their products like google photos. At least according to my observations

different_sort2• 7 years ago

I see this as well.

bilal4hmed• 7 years ago

gitlab is slow too

amenod2• 7 years ago

Slow is understatement... some pages on gitlab.com take minutes to load, and jobs take tens of minutes to start.

EDIT: It's been like that since at least 12h ago though. Not sure if it's connected to Google Cloud?

dsumenkovic2• 7 years ago

Yes, it was definitely related to GCP services.

GitLab is no longer seeing errors and Google Cloud has resolved the issue as of 23:00 UTC yesterday. Any further information can be found on the issue at https://gitlab.com/gitlab-com/gl-infra/production/issues/862

klon• 7 years ago

Anyone using both AWS and GCP that can form an opinion on availability of both? As a GCP customer I am not very happy with theirs.

pm904• 7 years ago

GCP is incredibly bad at communicating when there are problems with their systems. Just terrible. Its only when our apps start to break that we notice something is down, then look at the green dashboard which is even more infuriating.

timdorr2• 7 years ago

AWS is often the same way. No one seems to be good at communicating outage details.

jjeaff5• 7 years ago

Their dashboard does show red on GCE and networking right now, for what it's worth. https://status.cloud.google.com/

hinkley7• 7 years ago

What aren’t these on separate systems? I never had the impression that google cheaps out on things but this sounds exactly like the sort of shit that happens when people cheap out. Not even a canary system?

tubaguy50035• 7 years ago

Obviously we don't know what the extent of the issue is yet, but afaik there has never been an AWS incident that has affected multiple regions where an application had been designed to use them (like using region specific S3 endpoints). GCP and Azure have had issues in multiple regions that would have affected applications designed for multi-region.

votepaunchy• 7 years ago

like using region specific S3 endpoints

AWS had the S3 incident affecting all of us-east-1: “Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.”

https://aws.amazon.com/message/41926/

theevilsharpie• 7 years ago

I find GCP quicker to post status updates about issues than AWS, but GCP also seems to run into more problems that span across multiple regions.

I'm overall happy with it, but if I needed to run a service with a 99.95% uptime SLA or higher, I wouldn't rely solely on GCP.

kenhwang2• 7 years ago

AWS has what feel like monthly AZ brownouts (typically degradated performance or other control plane issues) with the yearly-ish regional brown/blackout.

GCP has quarterly-ish global blackouts, and generally on the data plane at that which makes them significantly more severe.

jjeaff5• 7 years ago

Are there any services that track uptime for various regions and zones from various providers? It's rare that everything goes down and thus the cloud providers pretend they have almost no downtime.

dimes• 7 years ago

I use both services heavily at work. The networking in GCP is terrible. We experience minor service degradation multiple times a month due to networking issues in GCP (elevated latency, errors talking to the DB, etc). We've even had cases where there was packet corruption at the bare metal layer, so we ended up storing a bunch of garbage data in our caches / databases. Also, the networking is less understandable on GCP compared to AWS. For instance, the external HTTP load balancer uses BGP and magic, so you aren't control of which zones your LB is deployed to. Some zones don't have any LBs deployed, so there is a constant cross-zone latency hit when using some zones. It took us months to discover this after consistent denials from Google Cloud support that something was wrong with a specific zone our service was running in.

AWS, on the other hand, has given us very few problems. When we do have an issue with an AWS service, we're able to quickly get an engineer on the phone who, thus far, has been able to explain exactly what our issue is and how to fix it.

captn3m09• 7 years ago

Just curious, is this on a specific region(s)?

jpmattia2• 7 years ago

We've even had cases where there was packet corruption at the bare metal layer,

I'd love to know how this happens in the modern world. I've seen it myself only once (not GCP, but our own network with cisco equipment.)

Is something in the chain not checking the packet's CRC?

rexarex2• 7 years ago

AWS has better customer service and I don’t remember the last time there was a huge outage like this besides the S3 outage

dodobirdlord6• 7 years ago

There was a terribly day 2-3 months back in us-west-2 where CloudWatch went down for a couple of hours and took out AutoScaling with it, causing a bunch of services like DynamoDB and EC2 to improperly scale in tables and clusters, and then 12 hours later Lambda went down for a couple of hours, degrading or disabling a bunch of other AWS services.

solidasparagus3• 7 years ago

I've heard from people who have worked with both AWS and GCP that AWS has far better availability.

thruhiker3• 7 years ago

I've also heard similar from a teammate who previously worked with GCP. That said I know several folks who work for GCP and they are expending significant resources to improve the product and add features.

landon32• 7 years ago

u.s. west: all our cloud compute is inaccessible rn.... our API is down, can't ssh into the servers, and also can't see them on the dashboard.

colinbartlett3• 7 years ago

Google Cloud is the number 4 most monitored status page on StatusGator and Google Apps is number 12. In addition, at least 20 other services we monitor seemingly depend on Google Cloud because they all posted issues as soon as Google went down.

It's always interesting to see these outages at large cloud providers spider out across the rest of the internet, a lot of the world depends on Google to stay up.

hazeii3• 7 years ago

...and only the paranoid survive?

MrMorden• 7 years ago

Just because you're paranoid, it doesn't mean they're not out to get you.

FPGAhacker2• 7 years ago

I guess we know what steam uses (the store at least).

xNevo• 7 years ago

No issues for me. Maybe they have a failover mechanism?

colinbartlett3• 7 years ago

I don't know about Steam, but I know Apple must use Google Cloud: https://www.apple.com/support/systemstatus/

hhs• 7 years ago

"a lot of the world depends on Google to stay up."

Yup, I'm trying to check the Associated Press News right now and it's having trouble connecting to "storage.googleapis.com".

nabla9• 7 years ago

This feels like 80's.

When the mainframe is down terminals are useless.

pmlnr3• 7 years ago

Yep. The cloud is just a lot of cheap hardware acting together as a shitty mainframe.

tus87• 7 years ago

So that's why I can't login to YouTube this morning...

different_sort2• 7 years ago

I was playing around this afternoon with appengine, and thought I broke one my projects when I started getting 502 back.

There appears to be some irregularities on consumer services as well that are of course certainly related, youtube was behaving a bit oddly for me.

The impact seems to be cascading down from just GCE to other services as well - that status page certainly does not reflect the reality of the situation. You can't even sign into GCP right now, and things that run on GCE, like appengine seem impacted.

derekhh2• 7 years ago

We will provide more information by Sunday, 2019-06-02 12:45 US/Pacific.

I'm not seeing anything at 12:47.

derekhh2• 7 years ago

https://status.cloud.google.com/incident/cloud-networking/19...

404 - Impressive

ryacko3• 7 years ago

Cloud status dashboards seem to be hosted on the same cloud, which doesn't say much about redundancy.

timdorr2• 7 years ago

They are updating the root cause issue, which is networking, here: https://status.cloud.google.com/incident/cloud-networking/19...

Next update is in about 25 minutes.

rwbhn• 7 years ago

gmail also down/super slow atm for me (East Coast, USA)

codys• 7 years ago

Other google services are also affected:

https://www.google.com/appsstatus#hl=en&v=status

https://i.imgur.com/pcqwwA4.png

mixedbit• 7 years ago

Good that Google+ is up again

dang• 7 years ago

https://news.ycombinator.com/item?id=20077275

https://news.ycombinator.com/item?id=20077293

PerfectElement2• 7 years ago

Just 2 weeks after I migrated a DB cluster from Azure to Google Cloud thinking things would get better.

duality• 7 years ago

FWIW they might still be.

AaronFriel2• 7 years ago

Just remember that "better" can be entirely based on when you pick the starting and ending points of your graph ;)

XCSme3• 7 years ago

It's just like the stock market.

user59944613• 7 years ago

It's the yearly outage. You will be fine for next year.

Better than the monthly outage from Azure.

leowoo91• 7 years ago

0 issues at compute, reporting for europe-west3-b,

pgoodjohn• 7 years ago

Everything looking normal on our GKE / CloudSQL stuff (eu-west1)

wichert• 7 years ago

gcloud tells me:

WARNING: The following zones did not respond: us-west2, us-west2-a, southamerica-east1-c, us-west2-b, southamerica-east1, us-east4-b, us-east4, us-east4-a, northamerica-northeast1-c, northamerica-northeast1-b, us-west2-c, southamerica-east1-b, northamerica-northeast1, southamerica-east1-a, northamerica-northeast1-a, us-east4-c. List results may be incomplete.

Luckily for us eu-west1 seems to be working normally.

echelon• 7 years ago

Nest is down for me right now.

It's amazing how far-reaching outages can be these days.

unicornmama• 7 years ago

There are dozens of us.

andrewprock• 7 years ago

Code reuse is a wonderful thing, until it's not.

hn23• 7 years ago

Code reuse? Couple if years ago some started to call servers "cloud" but the cloud is just that: computers managed by someone else. If the whole service layer relies on Three big companies, there is a problem..

squarefoot2• 7 years ago

And Gmail too doesn't feel very well today.

  [21:55:19] POP< +OK send PASS
  [21:55:19] POP> PASS ********
  [21:55:21] POP< +OK Welcome.
  [21:55:21] POP> STAT
  [21:55:21] POP< -ERR [SYS/TEMP] Temporary system problem.
  Please try again later.

ToniCipriani• 7 years ago

Pretty much Gsuite is out: https://www.google.com/appsstatus#hl=en&v=status

megous• 7 years ago

At least G+ is working. /s

hazeii3• 7 years ago

IMAP as well - for some considerable time now.

agentile• 7 years ago

Yes experiencing IMAP auth/connectivity issues here as well, along with outgoing SMTP

benbristow• 7 years ago

Wondered why Snapchat was being weird today. Thought it was my pi-hole setup blocking something from working, but nope, it's Google!

brown9-23• 7 years ago

When talking about GCE being down please also mention what regions you are talking about

chupasaurus6• 7 years ago

In this case it's a luck if any are working correctly, a problem is global with some exceptions.

brown9-23• 7 years ago

seems to be some comments here of some regions functioning ok, although perhaps it’s not 100% in all regions

sgammon2• 7 years ago

us-central1 us-west1 us-west2

is what I’ve heard so far. east seems to be OK, and Europe too

digaozao5• 7 years ago

South America is down for me. But youtube works OK, I can watch Google IO and they talking about all Google greatness while their outage impact second class clients on their cloud.

javabean_3• 7 years ago

vimeo.com is down.

boulos8• 7 years ago

Disclosure: I work on Google Cloud (but disclaimer, I'm on vacation and so not much use to you!).

We're having what appears to be a serious networking outage. It's disrupting everything, including unfortunately the tooling we usually use to communicate across the company about outages.

There are backup plans, of course, but I wanted to at least come here to say: you're not crazy, nothing is lost (to those concerns downthread), but there is serious packet loss at the least. You'll have to wait for someone actually involved in the incident to say more.

odiroot• 7 years ago

including unfortunately the tooling we usually use to communicate across the company about outages.

There's some irony in that.

boulos8• 7 years ago

Edit: and I agree!

I’m not in SRE so I don’t bother with all the backup modes (direct IRC channel, phone lines, “pagers” with backup numbers). I don’t think the networking SRE folks are as impacted in their direct communication, but they are (obviously) not able to get the word out as easily.

Still, it seems reasonable to me to use tooling for most outages that relies on “the network is fine overall”, to optimize for the common case.

Note: the status dashboard now correctly highlights (Edit: with a banner at the top) that multiple things are impacted because Networking. The Networking outage is the root cause.

Twirrim• 7 years ago

AWS experienced a major outage a few years ago that couldn't be communicated to customers because it took out all the components central to update the status board. One of those obvious-in-hindsight situations.

Not long after that incident, they migrated it to something that couldn't be affected by any outage. I imagine Google will probably do the same thing after this :)

k_bx• 7 years ago

Even more irony: Google+ shown as working fine: https://i.imgur.com/52ACuiY.png

SmokeGS2• 7 years ago

nothing is lost

except time

sdan5• 7 years ago

and a nice Sunday afternoon

stanfordkid• 7 years ago

Can't use my Nest lock to let guests into my house. I'm pretty sure their infrastructure is hosted in Google Cloud. So yeah... definitely some stuff lost.

countbackula• 7 years ago

And reputation. With this outage the global media socket is going to be in gCloud nine.

jussij• 7 years ago

and reputation.

123jay7• 7 years ago

This happened to Amazon S3 as well once. The "X" image they use to indicate a service outage was served by... yup, S3, which was down obviously.

hinkley7• 7 years ago

One of the projects I worked on was using data URIs for critical images, and I wouldn’t trust that particular team to babysit my goldfish.

Sounds like Google and Amazon are hiring way too many optimists. I kinda blame the war on QA for part of this, but damn that’s some Pollyanna bullshit.

boulos8• 7 years ago

To clarify something: this outage doesn’t appear to be global, but it is hitting us particularly hard in parts of the US. So for the folks with working VMs in Mumbai, you’re not crazy. But for everyone with sadness in us-central1, the team is on it.

digaozao5• 7 years ago

It seems global to me. This is really strange compared to AWS. I don't remember an outage there (other than s3) impacting instances or networking globally.

captn3m09• 7 years ago

Some services are still impacted globally. Gmail over IMAP is unreachable for me. (Edit: gmail web is fine)

the-rc• 7 years ago

Fiber cut? SDN bug that causes traffic to be misdirected? One or more core routers swallowing or corrupting packets?

ls612• 7 years ago

I’m from the US and in Australia right now. Both me and my friends in the US are experiencing outages across google properties and Snapchat, so it’s pretty global.

falcon2_0• 7 years ago

It seemed to be congestion in the North East US.

bufferoverflow• 7 years ago

Well, lots of revenue is lost, that's for sure.

ChuckMcM• 7 years ago

I'm guessing this will be part of the next DiRT exercise :-) (DiRT being the disaster recovery exercises that Google runs internally to prepare for this sort of thing)

foobarbazetc2• 7 years ago

Seems to be the private network. The public network looks fine to us from all over the world?

nodesocket3• 7 years ago

Not on my end. Public access in us-west2 (Los Angeles) is down for me.

ohazi• 7 years ago

including unfortunately the tooling we usually use to communicate across the company about outages.

So memegen is down?

dosy• 7 years ago

You're brave to jump on here when on holiday!

Shouldn't that outage system be aware when service heartbeats stop?

Could this be a solar flare?

gingabriska• 7 years ago

We are also hosted on GCP bit nothing is down for us. We are using 3 regions in US and 2 in EU.

ikiris• 7 years ago

go/stopleaks :)

foota• 7 years ago

Hm, isn't releasing go links publicly also verboten? :)

breadandcrumbel• 7 years ago

What can be the reason for the outage? Can it be a cyber attack to your servers?

Double_a_92• 7 years ago

but there is serious packet loss at the least.

Can confirm with Gmail in Europe. Everything works but it's sluggish (i.e. no immediate reaction on button clicks).

wenbin2• 7 years ago

btw, Google Analytics realtime is down as well.

bbayer2• 7 years ago

I wasn't aware that outage and had small heartattack when I saw huge drop of visitors. I think other metrics are also affected.

mileycyrusXOXO• 7 years ago

Thanks, I noticed I didn't have any users after pushing an update and thought I broke something.

sdan5• 7 years ago

Github contribution graphs are also gone

sadris• 7 years ago

So is Youtube.

mtarnovan• 7 years ago

Is shopify on google cloud? i noticed they are having issues too

brown9-23• 7 years ago

yes

captn3m09• 7 years ago

I'd reported a networking issue to Shopify yesterday, which they said was resolved upstream (GCP?). Could it be?

drchiu• 7 years ago

Yes it’s affecting their system as well.

crispyporkbites• 7 years ago

Sunday evening is a pretty big time for ecommerce. Losing 1000s of dollars of sales here :-(

40acres• 7 years ago

Can't wait for the postmortem!

murat1243• 7 years ago

My money is on config push.

bpye• 7 years ago

I would question why they are doing a rollout on a weekend?

seibelj• 7 years ago

Not sure if related, but I was going to a BBQ yesterday and myself and 3 other people got lost because Google Maps app glitched out, directing us to the wrong places. If you search twitter for #googlemaps tons of people have the same issue. Surprised no one has posted about it.

w_s_l2• 7 years ago

You know this reminds me of a bad taste that Google Sales team left when I asked for some of my billing that I was unaware of running after following a quickstart guide.

AWS refunded me in the first reply on the same day!

GCP sales rep just copy pasted a link to a self support survey that essentially told me, after a series of YES or NO questions that they can't refund me.

So why not just tell your customers like it is? Google Cloud is super strict when it comes to billing. I have called my bank to do a chargeback and put a hold on all future billing with GCP.

I'm now back to AWS and still on a Free Tier. Apparently the $300 Trial with Google Cloud did not include some critical products, AWS Free tier makes it super clear and even still I sometimes leave something running on and discover it in my invoice....

I've yet to receive a reply from Google and its been a week now.

I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature.

WC3w6pXxgGd• 7 years ago

Anything created in-house at Google (GCP) is typically created by technically-proficient devs, those devs then leave the project to start something new and maintenance is left to interns and new hires. Google customer service basically doesn't care and also has no tools at their disposal to fix any issues anyway.

The infinite money spout that is Google Ads has created a situation in which devs are at Google just to have fun - there really is no incentive to maintain anything because the money will flow regardless of quality.

Source: I interned at Google.

marcinzm2• 7 years ago

Isn't it also that promotions at Google are based on creating new products/projects rather than maintaining existing ones? So engineers have a negative incentive to maintain things since it costs them promotions.

kerng3• 7 years ago

I'm not sure why you are downvoted - seems like a reasonable insight and explanation for the drop in quality and weird decisions Google is making recently.

panopticon• 7 years ago

From what I’ve been told, the issue is that the people with political capital (managers, PMs, etc) are quick to move after successful launches and milestones. No matter how many competent engineers hang around, the product/team becomes resource and attention starved.

espeed• 7 years ago

What was the quickstart guide?

mcintyre1994• 7 years ago

The thing that worries me most about Google Cloud and these billing stories is that I’m assuming if you chargeback or block them at your bank then they’ll ban all Google accounts of yours - and they’re obviously going to be able to make the link between an account made just for Google Cloud and my real account.

ganeshkrishnan4• 7 years ago

They WILL absolutely block and suspend all accounts indefinitely. They have terminated accounts for credit card failing transactions.

I really wanted to try out their new autoML but I was paranoid of entering my credit card and getting banned from Google

kerng3• 7 years ago

Google is well known for not caring about small shops, only if you are a multi million dollar customer with dedicated account manager you can expect reasonable support. That's been the case forever with them.

fredthomsen2• 7 years ago

Does Amazon treat smaller customers any better? I am genuinely asking, as I have no clue.

bscphil3• 7 years ago

I asked for some of my billing that I was unaware of running
I have called my bank to do a chargeback

You're issuing a chargeback because you made a mistake and spent someone else's resources? And you're admitting to this on HN? I'm not a lawyer, but that sounds like fraud and / or theft to me.

prepend3• 7 years ago

It’s not, read the terms of your credit card. It’s basically “I didn’t intend to buy this. I tried in good faith to contact the merchant for return and support. I was ignored. I’m contacting you.”

It’s pretty convenient for companies like Comcast and Google that have poor customer service.

Can_Not2• 7 years ago

GCE charged me for "Chinese egress" but doesn't provide me a way to block China via firewall or other methods. They have the ability to check and bill me for it but if I want to use the same logic for a firewall rule I'm on my own. That sounds like theft and or fraud to me.

OP sounds like they're just defending their selves from ambiguous draconian billing robots.

lucb1e• 7 years ago

Are you seriously complaining about having to pay for using their resources? I understand that you're surprised some things aren't covered in the free trial or free credit or whatever, but getting $300 free already sounded a little too good to be true (I heard about it from a friend and was dubious; at least in Europe, consumers are told not to enter deals that are too good to be true), you could at least have checked what you're actually getting.

I think it's weird to say you get credit in dollars and then not be able to spend it on everything. That's not how money works. But that's the way hosting providers work and afaik it's quite well known. Especially with a large sum of "free money", even if it's not well known, it was on you to check any small print.

mcherm• 7 years ago

Are you seriously complaining about having to pay for using their resources?

I didn't read it that way. I thought they were complaining about poor customer service that made it difficult to understand the bill or respond to it appropriately.

Exuma• 7 years ago

https://downdetector.com/

Pretty much every service is down

lachs• 7 years ago

It's frightening how many services rely on gcloud and which impact this downtime has

markoa• 7 years ago

We're hosting an open global Zoom call for all engineers affected by the outage, join us at https://zoom.us/j/793450725

avocado4• 7 years ago

One click on this link and it instantly starts streaming your webcam footage to everyone in the chat room.

Sebguer• 7 years ago

This is a local zoom setting, you can change it.

chrisseaton2• 7 years ago

Use 'Turn off my video when joining meeting' in Zoom.

briandear• 7 years ago

Put some clothes on.

revvx3• 7 years ago

This is weird, I had Zoom installed a long time ago but uninstalled according to their instructions [1]. I'm a macOS user.

As soon as I clicked that link, the client downloaded a PKG file, installed itself and launched itself without asking me if I wanted to share my camera or audio.

I uninstalled according to their instructions again, searched for all "zoom" files in my disk and rebooted.

This leads me to believe that following their uninstall instructions is insufficient, and there are hidden files left on my computer.

Sorry in advance for the off topic message

[1] https://support.zoom.us/hc/en-us/articles/201362983-How-to-u...

Causality1• 7 years ago

Why? Do you expect to be able to do something about it or did you just want somewhere classier than Twitter to complain?

sdan5• 7 years ago

Weird for Twitter to still be up and fully functioning. I thought they migrated everything to GCP this/last year?

sorenn111• 7 years ago

Not the main functionality of the service, just lots of data analysis tooling. nothing that end users would notice

sdan5• 7 years ago

Interesting. Thought I had read some posts of them migrating their data, but you could definitely be right.

f_martinez• 7 years ago

We are on region us-east1 and our systems are still up. Specifically, we are on us-east1-b.

jshprentz• 7 years ago

The two Google Cloud networking incidents are:

Incident #19008 began at 2019-06-02 12:48. https://status.cloud.google.com/incident/cloud-networking/19...

Incident #19009 began at 2019-06-02 12:53. https://status.cloud.google.com/incident/cloud-networking/19...

Times are US/Pacific

pm904• 7 years ago

They both seem to say the same thing....

boulos8• 7 years ago

dang et al.: I’d make 19009 the URL for this (GCE Just reported first, but this is an outage / networking issue in the Eastern US).

AnssiH3• 7 years ago

#19008 mentioned by parent is not the GCE one, there are two networking ones: #19008 and #19009.

The #19008 (networking) says there will be an update by "13:30 US/Pacific", but as of 17:05 there is no update.

Similarly, #19003 (GCE) says there will be an update by "16:00 US/Pacific", but no update as of 17:05.

All the latest updates seem to only be in the third incident #19009 (networking).

cheeze3• 7 years ago

Looks like they are having trouble updating their statuses. 19008 was supposed to be updated over an hour ago. Meanwhile, 19009 has the same comment posted three times. I'm guessing internal tools are barely working at best.

londons_explore7• 7 years ago

That is normal for Google status pages.

They don't want to admit fault or place blame because there can be legal and commercial ramifications, so they can only say canned responses.

jamisonbryant• 7 years ago

I've noticed problems on GDrive (GSuite) and YouTube as well. Connected?

mindcrime• 7 years ago

FWIW, Youtube is fine for me, but I'm seeing intermittent errors saving updates to a document in GSuite. I had thought the latter error was a problem with the wifi where I am, until I saw this. Now I'm not so sure. HN is loading fine on the same wifi...

ikeboy• 7 years ago

So that's why YouTube was being weird. I thought it was an extension problem or something.

sarim• 7 years ago

Funny how as soon as I realized that Gmail and Google Sheet aren’t working properly I rushed to HN to figure out what’s going on. I love this community!

sgammon2• 7 years ago

So far the Ko list:

GCE, GKE, BQ, Pub/Sub, GAE

asia-south1 us-west1 us-central1 us-west2

captn3m09• 7 years ago

Our GKE stuff in asia-south1 (Mumbai) is up, and the GKE console works fine here. (Bangalore).

maz1b• 7 years ago

Ironically, I moved all of my objects off GCS today.

javabean_3• 7 years ago

could this be the result of another BGP hack ? cyberwarfare ? I am just speculating here big time.

pishpash• 7 years ago

Let's see if perfect leetcode skills will save the day. /s

CSDude• 7 years ago

With Google Cloud incidents, most of the time whole regions fail, and with AWS generally only a region fails. Of course there would be exceptions, but Google Cloud does not make me feel safe as an outsider (and a user of multi-region AWS)

gist• 7 years ago

Right now this is at the 'good at least it's not mine to fix or worry about' kind of like 'and the reason I choose IBM' [1]. I can just sit back and wait for gmail to work correctly. Now at the point it starts to last what I would consider a long time well then I will have things to worry about.

One thing with gmail though. When it's down it's similar to a snow storm if you only do business in a city. Everyone is impacted and everyone understands a missed deadline is unavoidable.

[1] For those not old enough to know what I mean read this: https://www.ibm.com/ibm/history/ibm100/us/en/icons/personalc...

cobaimelan• 7 years ago

seems like only us down

d2xdy2• 7 years ago

Was certainly an interesting alert when my Cloud Functions started reporting downtime. Among the many things that dip in and out on what seems like a monthly basis, I’ve not seen them just drop out in quite a while. Hopefully they get things sorted out. I can’t really imagine what it looks like internally when this level of outage is going on, but I want to think everyone is fairly collected

javabean_3• 7 years ago

Nice overview : https://downdetector.com/

XCSme3• 7 years ago

I see a lot of Nest complaints, isn't there any security issue if Nest goes down?

pmlnr3• 7 years ago

I see people complain about Nest and the only thing I can think of is "what on earth where you thinking to have a door or thermostat that doesn't function without internet?!"

mrb2• 7 years ago

My whole house is covered by a Nest Secure alarm system. At the moment my Nest Guard is telling me "it's offline". But if someone breaks in the house, the alarm will still ring in the house, however I believe I won't be notified on my phone (can't even log into the Nest app), and the alarm monitoring center (operated by Brinks under a partnership with Google) probably won't be notified as well, which sucks. I'd love to test it right now, but my daughter is sleeping and I don't want the alarm to wake her up...

I wonder how often outages occur with other alarm monitoring companies. They certainly do occur, but customers don't have a lot of visibility into them.

sandov• 7 years ago

Wow. Everyone depends on Google.

actuator4• 7 years ago

Instagram is on GCloud? Facebook has a heavy investment in their own DCs, I thought they would have transitioned to them by now.

sneak2• 7 years ago

Just because you run your own ram and compute doesn’t mean you run your object store, too. Running drives is a commodity now, and crypto is cheap since aes-ni.

No idea if this is what instagram does or not, just in general. Drives are hot and need lots of power and it’s expensive to out-S3 S3.

luu2• 7 years ago

No, the graphs are highly misleading because they autoscale the y axis to the highest point and the they do very loose string matching to detect errors.

If you click through to something not on Google cloud, you see moderately elevated error rates (e.g., Instagram is up by 4x) but if you click through to something actually on Google, you see very highly elevated error rates (e.g., roughly 50000x for Snapchat).

If you read the "error reports", they actually report that Instagram isn't down (same for Twitter if you check Twitter). The error report detection seems to be just string matching. Here's an actual "error report" from downdetector that's the caused of allegedly elevated error rates:

my twitter timeline: why isn’t snapchat working? anyone’s snapchat not working? snapchat’s being dumb. rip snapchat.

The Twitter "error report" is literally a report that Twitter isn't down.

ccarpenterg• 7 years ago

Google Colaboratory is back. At least I can access my github's notebooks and public notebooks from google drive.

cjohansson• 7 years ago

On days like these I’m glad I don’t use any of the affected services

biohax20152• 7 years ago

Does anyone know if this is a regional or global outage?

I can see my GKE clusters in one region but not in another, so I am guessing it's the former.

Looks like we'll need a cluster in each region going forward...

boulos8• 7 years ago

To help others: which two regions?

biohax20152• 7 years ago

us-central1 is up

us-east4 is down

XCSme3• 7 years ago

I can not access vimeo from EU, not sure if related to Google though.

peterwwillis• 7 years ago

The last big outage, iirc, was Google didn't test their rollback procedures for router upgrades. I'll be very interested to hear if it's yet another change control problem that caused this outage.

amasad• 7 years ago

Anyone experiencing issues with GCS? Seems highly intermittent and dependent on the location the request is coming from (maybe that's because it's a networking bug).

The status page says GCS is fine but that's highly unlikely.

ejanus• 7 years ago

I just can't reach google apps on my HTC m9 since yesterday. I am in West Africa. My Whatsapp crashed too and I lost all my previous threads. Is my issue related to Cloud being down?

captn3m09• 7 years ago

What I've realized from this: Google doesn't have an official status page for GCP. There are a few unofficial ones, but nothing official that I could find.

cheeze3• 7 years ago

Like this?

https://status.cloud.google.com/

captn3m09• 7 years ago

I meant Twitter Status Page (in my defense it was 3 am). The only ones are: @gcpstatus and @gcp_incidents, both unofficial

codemati• 7 years ago

https://tour.golang.org is also down

outside1234• 7 years ago

It seems crazy to me that Google Cloud can have this level of instability but I, on the other hand, can never remember google.com going down.

Why are they operating one with a different networking infrastructure from the other?

chupasaurus6• 7 years ago

AFAIK google.com is running independently on a different infrastructure, but some of other services are relying on GCP hence the problem affects them too.

sakisv2• 7 years ago

They did have a major outage for a few minutes on 2013: https://www.theregister.co.uk/2013/08/17/google_outage/

escapecharacter• 7 years ago

So this is now the longest outage ever, right?

savrajsingh• 7 years ago

Sounds like a good design — but in the early days of google cloud — 2008 — you’d get the google homepage on a bad request

vl• 7 years ago

It's completely different mode of failure - when part of search is down somewhere, it's just routed to the same service in other location. If part of the GCP is down anywhere, some customers are affected.

Since original Google infrastructure was developed specifically for first kind of services, cloud org still has problems adopting it to its needs.

chpmrc• 7 years ago

This might be all in my head but I've been experiencing really bad latency for like an hour, while browsing, and then I read this.

m0zg2• 7 years ago

Prediction: the final postmortem will say "someone pushed a bad config", just like most of the previous postmortems (and most of the internal postmortems as well, for Borg-based services). This is the cause of most other outages in other cloud providers as well. A really hard to solve problem.

krackers• 7 years ago

Multiple regions seem to be affected though. Wouldn't it make more sense to start out config pushes with a single region before expanding it to avoid these types of wide outages?

m0zg2• 7 years ago

It's a networking outage. Google is a well known user of SDN. Networks, by virtue of connecting things, necessarily affect more than one region.

bluefox• 7 years ago

You look up. A single ray of light has made its way to the earth. Some day, you hope, the sky will be clear again.

typenil• 7 years ago

Was trying to set up SSL on a GKE cluster today. Guess I'll have to wait for tomorrow if I want to be able to tell my mistakes apart from Google's.

wenbin2• 7 years ago

forbes.com is down also?

AdmiralAsshat• 7 years ago

And Thunderbird suddenly kept throwing prompts for my Gmail passwords, even after I sign back in. I hope it's related.

captn3m09• 7 years ago

It says "could not reach imap.gmail.com" for me.

titzer• 7 years ago

This is your yearly reminder to resist centralization of the internet.

quickthrower23• 7 years ago

Yes, I haven't read the Bitcoin is down or BitTorrent is down status yet.

jasonvorhe• 7 years ago

Because Internet Exchanges and carriers never had network issues.

eitland• 7 years ago

Because then at least you and your colleagues has a chance to get some work done.

el_benhameen• 7 years ago

That explains why my google home thing thinks it’s sub-zero even though I’m warm in shorts and a t-shirt.

https://pasteboard.co/IhBsyrsO.jpg

lewis1028282• 7 years ago

So just got a new phone and trying to resync my 2FA for AWS but I can't because Gmail is down. Ffs.

netmonk• 7 years ago

There is a loop in the spanning tree.....

finphil• 7 years ago

Working here but slow (I'm based in Central Canada).

unilynx• 7 years ago

G suite was broken for me 20 minutes ago (in Europe) but is working now. Perhaps things are starting to come back?

based2• 7 years ago

https://www.reddit.com/r/sysadmin/comments/bw1gye/most_googl...

https://www.zdnet.com/article/some-internet-outages-predicte... 768k Day

_Marak_• 7 years ago

This should be voted higher up.

According to https://twitter.com/bgp4_table, we have just exceeded 768k Border Gateway Protocol routing entries, which may be causing some routers to malfunction.

dreamer_soul• 7 years ago

Isn't it weird that it's happening now even though that number was surpassed nearly a month and half ago?

juanuys• 7 years ago

Will this affect more than just Google? I haven't seen any outages from other cloud providers.

namibj• 7 years ago

packet.net was hit. Specifically, also their San Jose DC. Internet only. It took less than an hour to recover. More than 20 minutes. I didn't ping it continuously, and I can say that the traceroute got stuck in Frankfuhrt (Where my ISP and their ISP first (as seen from me) meet).

I was actually surprised, as they tend to have excellent networking. Now I'm not nearly as distrusting as I was initially, knowing it was likely their ISP getting screwed by routing table overflow.

tntn• 7 years ago

There goes 3 nines for June and for Q2. I guess everyone gets a 10% discount for the month? https://cloud.google.com/compute/sla

gundmc2• 7 years ago

A couple more hours and everyone will get 25% off for June.

twistedpair• 7 years ago

Does that apply to the rest of June?

Might be a good month to rebuild all your models ;)

CamelCaseName• 7 years ago

Ironically, the SLA page returns a 502 error.

londons_explore7• 7 years ago

The discount seems way too small.

I would pay a premium for a cloud provider happy to give 100 percent discount for the month for 10 minutes downtime, and 100 percent discount for the year for an hour's downtime.

gundmc2• 7 years ago

Any cloud provider offering those terms would go out of business VERY quickly. Outages happen, all providers are incentivized to minimize the frequency and severity of disruptions - not just from the financial hit of breaching SLA (which for something like this will be significant), but for the reputational damage which can be even more impactful.

Johnny555• 7 years ago

Just take the premium that you'd be willing to pay and put it in the bank -- the premium would be priced such that the expected payout of the premium would be less than or equal to what you'd be paying.

Besides, a provider credit is the least of most company's concerns after an extended outage, it's a small fraction of their remediation costs and loss of customer goodwill.

ddalex• 7 years ago

The discount seems way too small.
I would pay a premium for a cloud provider happy to give 100 percent discount for the month for 10 minutes downtime, and 100 percent discount for the year for an hour's downtime.

It takes a lot of effort (exponential) to reliably (I. E. Designed to fail-working) build something that is guaranteed to have this level of uptime at these penalties.

So I'm sure that I can build something that works like this, but would you pay me $100 per GB of storage per month? $100 per wall-time hour of CPU usage? $100 per GB of Ram used per hour? Because these are the premium prices for your specs.

OkGoDoIt• 7 years ago

Remember to request the credit!

From that linked page:

"Customer Must Request Financial Credit

In order to receive any of the Financial Credits described above, Customer must notify Google technical support within thirty days from the time Customer becomes eligible to receive a Financial Credit. Customer must also provide Google with server log files showing loss of external connectivity errors and the date and time those errors occurred. If Customer does not comply with these requirements, Customer will forfeit its right to receive a Financial Credit. If a dispute arises with respect to this SLA, Google will make a determination in good faith based on its system logs, monitoring reports, configuration records, and other available information, which Google will make available for auditing by Customer at Customer’s request."

baalimago• 7 years ago

Playing "is google on this website or not?" just became so much easier, simply see if website works or not.

Scary stuff. What happens when Murphy's law decides to crash things even more?

ramon2• 7 years ago

No wonder I was trying Webcodesk right now and it's not working it's all firebase, yay!

ramon2• 7 years ago

It seems that it`s focused on the database stuff like firebase and firestore.

dodobirdlord6• 7 years ago

Apparently (from further up the thread) it's a network congestion issue causing extremely high rates of packet loss. I imagine pretty much anything that's homed in the effected regions will be degraded or unaccessible.

sneak2• 7 years ago

A demo of “too big to fail [via antitrust]”?

noncoml• 7 years ago

There go the nine nines for this year? Is it more like four nines or so now?

skizm• 7 years ago

Ah, wonder if this is affecting Google's SSO. It was super slow when I was trying it just now.

squarefoot2• 7 years ago

I appears that logging into the webmail solves at least POP mail problems. I tried my mail client and failed, then attempted a login to the webmail which worked. Gmail then asked me to confirm my recovery address and cellphone, which I did, and finally loaded the inbox page. I immediately attempted a connection through the POP client and this time it worked.

It might be something security related if it triggers a mandatory identity confirmation.

edit: I tried to send me a mail from another account and it worked but out of 4 or 5 mail checks at least two failed giving the same error.

[23:44:27] POP< -ERR [SYS/TEMP] Temporary system problem. Please try again later.

The problem seems much more complex.

filvdg• 7 years ago

Level(3) one of the biggest backbones has issues too, might be related

https://downdetector.com/status/level3

Yrlec• 7 years ago

Now is a good time to point out that the SLA of Google Cloud Storage only covers HTTP 500 errors: https://cloud.google.com/storage/sla. So if the servers are not responding at all then it's not covered by the SLA. I've brought this to their attention and they basically responded that their network is never down.

crazygringo• 7 years ago

Ironically I can't read that page because, since it's Google-hosted, I'm getting an HTTP 500 error... but which means at least that service is SLA-covered...

Cloud services live and die by their reputation, so I'd be shocked if Google ever tried to get out of following an SLA contract based on a technicality like that. It would be business suicide, so it doesn't seem like something to be too worried about?

moon2• 7 years ago

Its funny how searching for "Google Down" on Google and filtering for results within 1 hour yields nothing.

stvkoch• 7 years ago

My gmail it's down!

StaticRedux2• 7 years ago

This must be why GitLab is giving me shit. They recently made the switch. Wonder if there is any second guessing going on over there right now.

Uninen2• 7 years ago

They have a incident report page up at: https://docs.google.com/document/d/1RM3QnuJ4FPH10J3UrJS0T26d...

StaticRedux2• 7 years ago

Using Google Docs to report problems with Google, including Google Docs. Irony at it's finest.

kerng3• 7 years ago

I was just thinking the same. One of their reasons (besides being paid a lot by Google) was stability.

dsumenkovic2• 7 years ago

You can find some more data about GitLab availability after the move to GCP here https://about.gitlab.com/2018/10/11/gitlab-com-stability-pos.... As we're trying to stay transparent as always, we'll definitely let everyone know if are going/thinking about the change.

eisvogel• 7 years ago

gmail is down in Australia...

atopuzov• 7 years ago

Hug your on-call engineer.

cedricium• 7 years ago

Majority of G Suite services are suffering service outages: https://www.google.com/appsstatus#hl=en&v=status

tlogan• 7 years ago

I also noticed that Google search stop indexing news articles.

So I searched for "gmail down" on bing and I got some results [1]. But searching on Google for "gmail down" does not return any results [2].

[1] https://www.bing.com/news/search?q=gmail+down&qs=n&form=QBNT

[2] https://www.google.com/search?q=gmail+down&source=lnms&tbm=n...

griffinkelly• 7 years ago

Looks like Google Analytics isnt reporting stats either?

_xerxes_• 7 years ago

Nest is down too, not surprising given they are part of Google. What I don't understand is why I can't still control my devices over my local network. Why does the system even require access to Google servers?

fredthomsen2• 7 years ago

That made me rage the other day when I had an internet outage. I am honestly fed up with Nest esp after this new Google Nest branding, and just disconnected my thermostats from the internet. I rarely changed it from my scheduled settings via the app anyway.

jansan• 7 years ago

Makes me wonder What your thermostats are doing on the internet in the first place.

kibwen• 7 years ago

Let's make the following the tech mantra of the next decade: "the Internet Of Things can never possibly work until the LAN Of Things does".

nektro• 7 years ago

the IoT trend really needs to stop

Raidion• 7 years ago

It doesn't, but it sure was easier implimenting one connection through Google APIs instead of two: one through Google and one through local network.

ryacko3• 7 years ago

I have not implemented additional APIs by running a local DNS server.

fdgwhite• 7 years ago

I tried multiple times to setup a Google Wifi router today. Wifi would work but the app said it was offline. Perhaps I am not insane or incompetent after all

ecabuk• 7 years ago

My builds are failing because it cannot download the chromium.

Error: Download failed: server returned code 502. URL: https://storage.googleapis.com/chromium-browser-snapshots/Li...

the-dude• 7 years ago

Huawei just flexing its muscles. Nothing to see here, move along.

pitaj• 7 years ago

Looks like my Gmail is back, but I don't have any emails from while it was down. Yikes.

Edit: just got one email from the downtime, so perhaps my initial conclusion was incorrect

Stryder• 7 years ago

Whoa, if confirmed this would be really bad. Downtime happens, but data loss is much worse.

BillinghamJ2• 7 years ago

This is normal handling for SMTP - the sending server just tries again later.

chx• 7 years ago

These things happen. That's OK. Here's what's not OK:

We are investigating an issue with Google Compute Engine. We will provide more information by Sunday, 2019-06-02 12:45 US/Pacific.

The next update is at 12:59. Just ... no.

CallumRobertson2• 7 years ago

Snapchat is fixed but snap maps is still disabled

CallumRobertson2• 7 years ago

Snapchat is back up but snap maps is down

nmstoker• 7 years ago

Seems to have coincided almost exactly with my Chromecast stopping displaying my photos (in ambient mode).

fourty2• 7 years ago

google services also f´ed up here in middle europe; cannot reach anything google related from hamburg, germany.

londons_explore7• 7 years ago

It seems the AdWords anti-spam system is down, which means anyone can put a billion dollar bid on every keyword and get their ads showing on every Google search for every query.

Systems that fail 'open'...

remote_phone• 7 years ago

Phase 2 of mitigation was completed a few minutes ago and GCE is expecting improvement shortly.

nopriorarrests• 7 years ago

https://status.cloud.google.com/incident/compute/19003

We will provide an update at 16:00 US/Pacific.

it's 16:22 and no updates were posted. a bit unprofessional..

bbayer2• 7 years ago

I think nobody mentioned but Adsense also is not updating for couple of hours.

joaomacp• 7 years ago

I was finishing a university assignment with the deadline 90mins away.

I wanted to upload a video of the project to YouTube and add a link to it in the report. YouTube takes a long time to process the video, and then says it's unavailable.

I go to Vimeo: it's down.

I upload the video to Dropbox, and copy its link to the report.

But my report was a Google doc. And when I tried to export it as PDF (which I had not done yet) it couldn't do it. I never hated google more.

Eventually the video went through to YouTube, and I could export the PDF on the third try, but this really made me conscious of my dependance on Google.

misrab• 7 years ago

Anything to do with China?

bahmboo• 7 years ago

No, not really.

rum33• 7 years ago

Keep on centralizing the internet with your stupid clouds and this is what happens.

AngeloAnolin• 7 years ago

Does Google also have some sort of listing on which consumer apps are particularly affected (i.e. Gmail, Hangouts, Docs, Sheets, etc).

The cloud components may be directly affected but for consumers, there's nothing which will provide info on what consumer facing services are getting some issues.

tcarn• 7 years ago

Once again HN proves to be the best status monitor.

ecthiender• 7 years ago

In India, I could access Youtube, Gmail (web) and Google cloud console and GKE and Compute Engine instances in south-east asia region.

bprasanna• 7 years ago

https://status.cloud.google.com/incident/compute/19003 Does status update mean only the status. Won't there be words like "sorry", "apologize" & "inconvenience"! Only PR is responsible for those words?

jwandborg• 7 years ago

Apologies are unnecessary on status pages. You can be sure that they are sorry.

Google Cloud Is Down

Related Stories