GCP Outage

https://status.cloud.google.com/

by thanhhaimai • 1 year ago

1468 478 1 year ago

status.cloud.google.com

thanhhaimai• 1 year ago

The status page is green, but there are outages reported: https://downdetector.com/status/google-cloud/

ransom15382• 1 year ago

Why can't companies be honest with being down. It helps us all out so we don't spend an hour internalizing.

We are truly in gods hands.

$ prod

Fetching cluster endpoint and auth data. ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=503, message=Visibility check was unavailable. Please retry the request and contact support if the problem persists

jeanlucas• 1 year ago

Because there are contracts related to uptime :)

9rx6• 1 year ago

The program that updates the status page is hosted on Google Cloud.

oxymoron• 1 year ago

Because a lot of the time, not everyone is impacted, as the systems are designed to contain the "blast radius" of failures using techniques such as cellular architecture and [shuffle sharding](https://aws.amazon.com/builders-library/workload-isolation-u...). So sometimes a service is completely down for some customers and fully unaffected for other customers.

rozap• 1 year ago

Please, won't somebody think of the KPIs.

kingstnap• 1 year ago

Because they have unrealistic targets so they make up fake uptime numbers. 99.999% would mean not even having an hour of downtime in 10 years.

I remember reddit being down for like a whole day or so and they claimed 99.5% in that month.

voytec4• 1 year ago

Why can't companies be honest with being down

SLA agreements.

rapus952• 1 year ago

if half the internet is down, which it apparently is, it's usually not the service in question, but some backbone service like cloudflare. And as internal health monitoring doesn't route to the outside through the backbone to get back in, it won't pick it up. Which is good in some sense, as it means that we can see if it's on the path TO the service or the service itself.

supportengineer4• 1 year ago

Nobody gets a promotion, that's why.

DrBenCarson2• 1 year ago

Whichever product person is in charge of the status page should be ashamed

How could you possibly trust them with your critical workloads? They don't even tell you whether or not their services work (despite obviously knowing)

FireBeyond• 1 year ago

Yeah, my company of hundreds of people working remotely are having 90%+ failures connecting to Google Meetings - joining a meeting just results in a 504.

nerdsniper• 1 year ago

Why even have a status page? Someone reported that their org of >100,000 users can't use Google Meet. If corps aren't going to update their status page, might as well just not have one.

https://www.google.com/appsstatus/dashboard/

https://status.cloud.google.com/index.html

Edit: The GCP status page got updated <1 minute after I posted this, showing affected services are Cloud Data Fusion, Cloud Memorystore, Cloud Shell, Cloud Workstations, Google Cloud Bigtable, Google Cloud Console, Google Cloud Dataproc, Google Cloud Storage, Identity and Access Management, Identity Platform, Memorystore for Memcached, Memorystore for Redis, Memorystore for Redis Cluster, Vertex AI Search

supportengineer4• 1 year ago

Who gets a promotion from a working status board?

SOLAR_FIELDS2• 1 year ago

There's no situation where the corporation controls the status page where you can trust the status page to have accurate information. None. The incentives will never be aligned in this regard. It's just too tempting and easy for the corp to control the narrative when they maintain their own status page.

The only accurate status pages are provided by third party service checkers.

paulddraper5• 1 year ago

might as well just not have one

This is my position.

nikcub• 1 year ago

I have zero faith in status pages. It's easier and more reliable to just check twitter.

Heroku was down for _hours_ the other day before there was any mention of an incident - meanwhile there were hundreds of comments across twitter, hn, reddit etc.

milesward4• 1 year ago

It's updated now, shows the impact to console, dataproc, GCS, IAM and Identity Platform: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

jorts• 1 year ago

Here's the incident: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

deathanatos2• 1 year ago

It was nearly an hour into our company's internal incident channel on this for GCP to finally declare that yes, in fact, things on fire.

… I get that PR-types probably want to massage the message, but going radio dark is not good PR.

matdehaast• 1 year ago

Not just GCP, most of Googles services are out of action

milesward4• 1 year ago

I'm on a meet, in cal, editing a dozen docs, in GCP, pushing commits and launching containers; it's not clear yet what exactly is going on but it's certainly intermittent and sparse, at least so far

parpfish• 1 year ago

stop it. you're overloading their system by doing three things at once. let the rest of us have a turn.

sourthyme• 1 year ago

Maybe cloudflare?

nickporter• 1 year ago

having issues with cloudflare as well

aetherson4• 1 year ago

Cloudflare status page reports an issue: https://www.cloudflarestatus.com/

aetherson4• 1 year ago

We're experiencing intermittent slowness and timeouts on our GCP everything.

dorkitude• 1 year ago

Same here. Even the page to submit support requests is down.

Cloud console does nothing.

They should host their support services on AWS and vice-versa.

milesward4• 1 year ago

I just logged into several of my GCP accts, everything popped up, multiple home regions.. I wonder what % of folks are feeling this right now.

aetherson4• 1 year ago

Does anyone know if it's region-specific? We're experiencing it and are in us-west-1.

data-ottawa• 1 year ago

Us-central-1 as well

vbb• 1 year ago

europe (netherlands) region as well

izolate• 1 year ago

us-east-1 too

johannes5117• 1 year ago

Frankfurt seems to be down as well

mccoyc• 1 year ago

Can confirm us-east1 (and possibly us-south1) are having VPC host reachability problems.

NeT8235• 1 year ago

south korea as well

tridao• 1 year ago

it's due to IAM and global

ea016• 1 year ago

Google Cloud Storage seems to be down or very slow

digest• 1 year ago

love how their status page is green with no issues detected!

pbmango• 1 year ago

https://www.canva.com/design/DAGqKquGD-c/xtRObgH1r_4RoulPAys...

hambro2• 1 year ago

@dang could you merge this and https://news.ycombinator.com/item?id=44260669?

toomuchtodo• 1 year ago

No notifications for mentions, have to email the mods at the hn@ email address.

hambro2• 1 year ago

I think I was a bit optimistic in the response time from mods. This thread won the popularity contest quite well...

Thanks for letting me know about emailing the mods, refreshingly explicit to send email.

cwillu• 1 year ago

Do we know if email is still working? kidding-but-not-really-because-gmail…

niij• 1 year ago

Experiencing 504s in Google Meet.

Google Cloud Console won't load.

ransom15382• 1 year ago

Yeah their status page is all green nothing to see here (but all production systems are down).

alexcroox4• 1 year ago

Cloudflare KV is also having an outage. I wonder who is reliant on who here.

hackermondev• 1 year ago

seriously doubt Google Cloud is relying on Cloudflare KV lol

dlewis1788• 1 year ago

Looks like more than KV is having an issue. Just tried to load dash.cloudflare.com and no bueno.

eterm• 1 year ago

Cloudflare speedtest is down too, I assume because of this?

voxadam• 1 year ago

Works for me in Portland on Quantum Fiber.

clairegraham• 1 year ago

Our site depends on Workers and KV and it's very broken right now. Can't login to the Cloudflare Dashboard either.

gfs• 1 year ago

Appears to be a separate incident: https://news.ycombinator.com/item?id=44261064

reassess_blind• 1 year ago

Two big cloud provider outages at the same time? Has to be related surely.

siliconc0w• 1 year ago

Gemini API isn't working for me :/

conroy• 1 year ago

We're in us-west-1 and seeing issues across Cloud Run, Cloud SQL, Cloud Storage and Compute Engine.

oalessandr• 1 year ago

Having issues with services in cloud run as well

zacharynewton• 1 year ago

Ahhh, explains why some of my apps are going crazy... Couldn't read a message from my kids pre-school

Thankfully we use AWS at work for everything critical

atonse2• 1 year ago

Getting a lot of errors for Claude Sonnet 4 (Cursor) and Gemini Pro.

Nooooo I'm going to have to use my brain again and write 100% of my code like a caveman from December 2024.

bicx• 1 year ago

I was in the middle of testing Cloud Storage file uploads, so I guess this is a good time to go for a walk.

matsemann• 1 year ago

A good excuse for adding error handling, which otherwise is often overlooked, heh.

orangebread• 1 year ago

lmao i refuse to write code by hand anymore too. WHAT IS THIS

sujayakar• 1 year ago

switch to auto mode and it should still work!

ashu14617• 1 year ago

GPT is working in agent mode, which kind of confirms that claude is hosted on google and GPT probably on MSFT servers / self hosted.

burntalmonds• 1 year ago

Same here. Getting this in AI Studio: Failed to generate content: user has exceeded quota. Please try again later.

crocowhile• 1 year ago

openrouter.ai is down for me

sunir• 1 year ago

I chose sepuku.

Xavez• 1 year ago

Apple’s local models looking better each day :’)

nolist_policy• 1 year ago

Google's local models as well (Gemini Nano/Gemma 3n)

cryptonector• 1 year ago

Devs before June 12, 2025: "Ai? Pfft, hallucination central. They'll never replace me!"

Devs during June 12, 2025 GCP outage: "What, no AI?! Do you think I'm a slave?!"

atonse2• 1 year ago

100% agree... I even thought "ok maybe I'll clean up the backlog while I wait" but I'm so used to even using AI to clean up my JIRA backlog (using the Atlassian MCP), so even that feels weird to click into each ticket, just the way I used to do it TWO MONTHS AGO.

This is a good wake-up call on how easily (and quickly) we can all become pretty dependent on these tools.

thefourthchime• 1 year ago

So true

sva_• 1 year ago

It appears like "Devs" is not a homogeneous mass.

christianqchung• 1 year ago

Goomba fallacy

robin-a• 1 year ago

Cursor throwing some errors for me in Auto Agent mode too.

gigatexal• 1 year ago

YouTube is also very flakey.

Brystephor• 1 year ago

some core GCP cloud services are down. might be a good time for GCP dependent people to go for a walk, do some stretches, and check back in a couple hours.

mlb_hn• 1 year ago

Our GCP is down

milesward4• 1 year ago

What region?

ashu14617• 1 year ago

I think multiple regions are down. asia-south, us-east atleast are impacted.

sergiotapia• 1 year ago

xAI having problems, Supabase down, Discord can't upload images to share in chat. Seems like a major backbone outage.

saltcod• 1 year ago

We're investigating right now. Looks like a potential issue with Cloudflare.

atsaloli• 1 year ago

https://www.cloudflarestatus.com/incidents/25r9t0vz99rp

faizanrupani2• 1 year ago

You're right, https://www.cloudflarestatus.com/ is showing outage, which cause google gcp outage, and claude outage.

theflyinghorse• 1 year ago

identitytoolkit.googleapis.com is 503-ing on us, my whole customer success team is locked out from our platform

madjam002• 1 year ago

Google Maps not loading, thought it was my 4g, go to see if my connection works by loading Hacker News, GCP Outage XD

evtothedev• 1 year ago

Yarn package registry also appears to be down.

tom1337• 1 year ago

npm is, registry.yarnpkg.com is only a CNAME to npm

kfarr• 1 year ago

Yes Firebase auth is down and affecting many apps, on Discord and Slack groups tons of others are corroborating. A bit disappointing that there is no post on the status page for nearly 30 mins: https://status.firebase.google.com/

kentlyons• 1 year ago

It just updated. Maybe affected by their own outage!

ashu14617• 1 year ago

Just proves how shady the status page and sla stuff is

_kush• 1 year ago

Supabase is also down

vpuna2• 1 year ago

Yes my project on Supabase is down as well.

andrewmcwatters• 1 year ago

Ah darn it. My Spotify DJ just stopped working.

capital_guy• 1 year ago

BigQuery is completely dead

ddtaylor2• 1 year ago

Smells like BGP since there are services people claim have nothing to do with GCP being affected. OpenRouter is down, Lovable is down, etc.

DrBenCarson2• 1 year ago

npm as well

koito17• 1 year ago

Initially attributed the unresponsiveness of `npm install` to npm (the CLI tool) in general. Tried using bun to install dependencies, saw the same result -- but with actual logs instead of a vague spinner -- and decided to check Hacker News.

Getting 504 errors on anything from registry.npmjs.org that isn't cached on my machine.

brown9-2• 1 year ago

perhaps Lovable uses GCP somewhere in their stack?

thallium205• 1 year ago

AWS seems fine though. My bet is Cloudflare.

CSMastermind2• 1 year ago

AWS and Azure both had outages.

TN1ck• 1 year ago

Cloudbuild completely down for us. Getting "Visibility check was unavailable" errors.

quectophoton3• 1 year ago

Twitch was broken too: https://status.twitch.com/incidents/b79nyp1yhxql

EDIT: Updated link to point to the specific incident.

rplnt• 1 year ago

Is Amazon running Twitch on Google Cloud (at least partially)?

quectophoton3• 1 year ago

I don't know, at this point I don't know who uses what. This is maybe unrelated but even BunnyCDN has an incident from a few hours ago (https://status.bunny.net/incidents/6g27lbtp67m4).

Seeing how everything seems to be broken everywhere, I'm very much looking forward to the post-mortem.

rcfox• 1 year ago

I'm able to login to the GCP dashboard, but it isn't able to find any of my projects.

keizo• 1 year ago

Yup, intermittent db connection issues and cloud storage problems.

admissionsguy2• 1 year ago

Wish there existed a decentralized network connecting computers around the world

redman25• 1 year ago

Crazy, they could call it the "internet" or something like that... kind of rolls off the tongue.

tsouth• 1 year ago

Everyone is down. Cloudflare has problems too. All auth providers broken.

traeregan• 1 year ago

For us Cloud SQL instances are toast but App Engine Standard instances are still serving requests. Google Cloud console is borked too, mostly just erroring out.

ekojs2• 1 year ago

Super duper frustrating having the status page being green. Why can't Google do this properly?

supportengineer4• 1 year ago

Those responsible have been sacked.

181728282861772• 1 year ago

Those responsible for sacking the people who have just been sacked, have been sacked.

imzadi2• 1 year ago

Can't reach my nest thermometer, but their status page says it's fine lol

charliemeyer• 1 year ago

the real concerns in life

andrelaszlo4• 1 year ago

This is pretty crazy :D How did it affect you?

imzadi2• 1 year ago

I almost died

tmiku• 1 year ago

Looks like I'm about to start learning which of my time-killing websites are hosted on GCP - The Ringer is down, and since Spotify owns them and is a major GCP customer, it looks like they've been hit by this. CRAZY that the GCP status page is still green.

ashu14617• 1 year ago

When you deploy code generated by Gemini :D

whitedurna• 1 year ago

i think it'll be disaster.

whalesalad4• 1 year ago

Meet is also down for me right now. Cannot attend any video calls.

braunshedd• 1 year ago

Our GCP workloads are unavailable across several US regions. The GCP console is intermittently unavailable for most pages.

Crossing my fingers for a quick resolution.

morgandoane• 1 year ago

Storage, CloudRun, Firebase...... All down....

dana321• 1 year ago

Auth, GCP, Windsurf,Augment Code,Udio, the list is endless.

Facebook, Reddit and Hacker News is still up, but thats about it

vpuna2• 1 year ago

is supabase on GCP ? My Supabase projects are down.

duckarmada• 1 year ago

Supabase is on AWS, but this is looking like an upstream Cloudflare issue. https://status.supabase.com/

devmor2• 1 year ago

Not just GCP. AWS and Cloudflare too.

Did someone screw up BGP again?

remram3• 1 year ago

Source? We didn't see anything wrong with AWS here.

akash8400• 1 year ago

GKE workloads are also affected.

meltyness• 1 year ago

Can't upload discord attachments from mobile.

Axsuul3• 1 year ago

Does anyone know if instance-to-instance networking has been affected? My Redis instance has been throwing a lot of connection errors.

markbnj• 1 year ago

We're not seeing any connectivity issues between pods and vms in our vpc, but your mileage may vary.

Axsuul3• 1 year ago

Thanks

181728282861772• 1 year ago

YouTube was down for me for some time

dbacar• 1 year ago

kaggle not responding correctly, is it related?

faizanrupani2• 1 year ago

https://www.cloudflarestatus.com/ is showing outage, which cause google gcp outage, claude outage, firbase outage https://status.firebase.google.com/

andrelaszlo4• 1 year ago

How would Cloudflare's outage cause a GCP outage?

I'm sure it's not entirely impossible, but sounds backwards to me. Sure - a lot of the internet relies on Cloudflare, but I'd be very surprised if GCP had a direct dependency on Cloudflare, for a lot of reasons. Maybe I misunderstood your comment?

paulddraper5• 1 year ago

"No major incidents" as of 11:37 PDT.

https://status.cloud.google.com/

File that in the status pages worth ~0 category.

jschroeder• 1 year ago

Status page is showing green because GCP admins can't login to change it ;)

tiagod• 1 year ago

Getting Gateway timeouts on docker hub. Maybe related? I can pull images.

Example: https://hub.docker.com/layers/library/eclipse-mosquitto/late...

ZiyadFarhan• 1 year ago

this aint looking good yall

sleepybrett• 1 year ago

npm registry happen to be hosted on gcp, because that seems to be down as well.

supportengineer4• 1 year ago

Interesting how I landed here. I was having trouble with Nest. Then I went to Down Detector. I noticed many sites having a simultaneous uptick. Then I came to HN, and found this link at the top of the front page.

geekamongus• 1 year ago

I usually just go here first.

ryanscio• 1 year ago

Same here with npm

https://status.npmjs.org/incidents/dn5mcp85737y

waythenewsgoes• 1 year ago

Status pages at cloud providers aren't usually based in reality -- usually requires VP level political games to actually get them changed especially for serious outages.

rvnx3• 1 year ago

It looks like that it is a central service @ Google called Chemist that is down.

"Chemist checks the project status, activation status, abuse status, billing status, service status, location restrictions, VPC Service Controls, SuperQuota, and other policies."

-> This would totally explain the error messages "visibility check (of the API) failed" and "cannot load policy" and the wide amount of services affected.

cf. https://cloud.google.com/service-infrastructure/docs/service...

EDIT: Google says "(Google Cloud) is down due to Identity and Access Management Service Issue"

VWWHFSfQ• 1 year ago

There are multiple internet services down, not just GCP. It's just possible that this "Chemist" service is especially externally affected which is why the failures are propagating to the their internal GCP network services.

rvnx3• 1 year ago

Absolutely possible. Though there is something curious:

https://www.cloudflarestatus.com/

At Cloudflare it started with: "Investigating - Cloudflare engineering is investigating an issue causing Access authentication to fail.".

So this would somehow validate the theory of auth/quotas started failing right after Google, but what happened after ?! Pure snowballing ? That sounds a bit crazy.

gcpman• 1 year ago

perhaps the person who maintains Chemist took the buyout

https://www.businessinsider.com/google-return-office-buyouts...

mrGomesDev• 1 year ago

I use Expo intermediation for notifications, but with this Google context, I imagine that FCM is also suffering, is that possible?

rvnx3• 1 year ago

Very likely. Firebase Auth is down for sure (though unreported yet), so most likely FCM too

chief_jeef• 1 year ago

Firebase status page has acknowledged it as a "global issue". https://status.firebase.google.com/

A contact in google mentioned to me that some bad update to Google Cloud Storage service has caused some cascading issues affecting multiple GCP services.

0xffany2• 1 year ago

Everything appears to be down as of 18:43 UTC... https://downdetector.com/

patapong• 1 year ago

Perhaps their detection logic is running on Google cloud /s

throitallaway• 1 year ago

I believe Downdetector displays user reports.

sillypuddy• 1 year ago

Well that's interesting. I wouldn't expect AWS or Microsoft 365 to be affected by a Google outage.

paxys5• 1 year ago

Who said it's a Google outage?

AlienRobot• 1 year ago

Wait, it's all Google?

plateng000• 1 year ago

"always has been"

deepsun4• 1 year ago

Google was the first to report probably.

bananapub3• 1 year ago

all cloud

voytec4• 1 year ago

Yeah. This service was presenting charts likely probed from inside GCP. I was on a call with a Google rep, someone pointed out that "AWS is also down" and I foolishly said something about "possible BGP attack" out of spite, before checking AWS availability myself. Shame on me.

toast02• 1 year ago

Didn't have the feeling of a BGP issue, most services I was working with were reasonably quickly returning failures, as opposed to lingering death.

yard20103• 1 year ago

I love this kind of fake news. It's like that scene from Scary Movie (can't remember which one) in which someone says "I heard the japs took out one in Kikoman" :')

peanut-walrus2• 1 year ago

Downdetector in incidents like this is 100% misinformation.

johanyc• 1 year ago

Why

deadbabe• 1 year ago

If LLMs are down work grinds to a halt until they return. Just the new era now.

kachapopopow• 1 year ago

I just realized that the reason the status isn't updated is cause they can't access it lol.

paulddraper5• 1 year ago

Don't host status pages (or their dependencies) on your own infra lol.

Seems obvious.

deathanatos2• 1 year ago

It should be obvious because both AWS and Azure have done this in the past and shown what a bad idea it is…

Axsuul3• 1 year ago

How do you know that?

fidotron• 1 year ago

It's completely nuts that Firebase has this: https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2e...

"Firebase Data Connect unavailable due to a known Google Cloud global outage"

While the Google Cloud status page https://status.cloud.google.com/ says "No major incidents" and everything is green. So Google Cloud know there is an outage but just deem it not major enough to show it.

Edit to add: within 10 minutes of this post Google updated their status page. More curiously the Firebase page I linked to has been edited to remove mention of Google Cloud in the status and now says "Firebase Data Connect is currently experiencing a service disruption. Please check back for status. ".

kjuulh4• 1 year ago

Something must be preventing them updating the status page at this point. Of course they could still deem it not enough, but just from my limited tests, docker, buf, etc (it may not be GCP that is down, but it is quite the coincidence). are outright down. I'd wager that this is much more widespread.

sss111• 1 year ago

I'm actually on a bridge call with Google Cloud, we're a large customer -- I just learned today that their status page is not automated, instead someone actually manually updates it!

codergautam• 1 year ago

Maybe the outage is preventing them from updating that specific page? Hmm

EDIT: Looks like it has been updated now (6:49 PM UTC)

samdung2• 1 year ago

:))))))

devMem• 1 year ago

I hope this is the case, or google is super unreliable for production grade work.

alexcroox4• 1 year ago

Almost an hour to update the page...

artooro2• 1 year ago

Anytime there is an outage that affects App Engine, Google can't seem to get their status page updated for an extended period of time.

samdung2• 1 year ago

GCP just updated their status

aetherson4• 1 year ago

More likely they are unable to update their own status page, but in either case not covering themselves in glory over at GCP right now.

alexcroox4• 1 year ago

CF too: https://www.cloudflarestatus.com/

cherioo• 1 year ago

This extra funny that GCP status page even includes a “last updated” time, which is exactly built to convey possible failure to update in cases like this

No major incident as of “ Last updated time: 12 Jun 2025, 11:48 PDT”

ike2792• 1 year ago

Maybe their dashboard is hosted on GCP and they are displaying a cached version. :-)

blibble• 1 year ago

lies, from big tech?

say it's not so!

Workaccount2• 1 year ago

I asked testing to see if it was up, and it pointed out that Google shows nothing but Nest is showing an outage right now, lol

https://status.nest.com/posts/dashboard

octo888• 1 year ago

Status pages are PR. It gets the same PR treatment as anything else

shmatt• 1 year ago

IIRC status pages drive customer compensation for downtime. Updating it is basically signing the check for their biggest customers, in most similar companies you need a very senior executive to approve the update

On the other side of this, Firebase probably doesn't have money at stake making the update

aiauthoritydev• 1 year ago

It is not the status page that drives customer compensation. It is downtime.

refulgentis5• 1 year ago

Nah, its just some client side caching / JS stuff. Clicking the big refresh button fixed it for me, 15 minutes before OP noted it.

(n.b. as much as Google in aggregate is evil, they're smart evil. You can't avoid execs approving every outage because checks without some paper trail, and execs don't want to approve every outage, you'd have to rely on too many engineers and sales people, even as ex-employees, to keep it a secret. disclaimer: xoogler)

(EDIT: for posterity, we're discussing a "overall status" thing with a huge refresh button, right above a huge table chockful of orange triangles that indicate "One or more regions affected" - even when the "overall status" was green, the table was still full of orange and visible immediately underneath. My point being, you gotta suppose a wholeeee bunch of stuff to get to the point there was ever info suppressed, much less suppressed intentionally to avoid cutting checks)

cyberflame5• 1 year ago

Services are recovering in some locations it seems - Discord is healing

nailer• 1 year ago

AWS has this all the time. If you need to know if a service is down in a region, check for other engineers talking about it on X.

paxys5• 1 year ago

Kinda funny that the top post on HN titled "GCP Outage" links to the Google Cloud status page which shows...no outage.

kgwxd• 1 year ago

Sorry, after decades of being hard wired, I just installed a PCIe Wifi6 card on my desktop. Internet took a dive the second I got it connected. Must have done something wrong.

happy-camper2• 1 year ago

Text messaging on Android is broken

happy-camper2• 1 year ago

Text messaging for android is broken as well

ipsum22• 1 year ago

Cloudflare is down too. From https://www.cloudflarestatus.com:

Update - We are seeing a number of services suffer intermittent failures. We are continuing to investigate this and we will update this list as we assess the impact on a per-service level.

Impacted services: Access WARP Durable Objects (SQLite backed Durable Objects only) Workers KV Realtime Workers AI Stream Parts of the Cloudflare dashboard Jun 12, 2025 - 18:48 UTC

Edit: https://news.ycombinator.com/item?id=44261064

paulddraper5• 1 year ago

Broken link? EDIT: Weird, definitely was just empty

ipsum22• 1 year ago

Should work, but its also on the front page.

0xy• 1 year ago

Seems like a major wtf if Cloudflare is using GCP as a key dependency.

a2128• 1 year ago

Some day Cloudflare will depend on GCP and GCP will depend on Cloudflare and AWS will rely on one of the two being online and Cloudflare will also depend on AWS and the internet will go down and no one will know how to restart it

0xffany2• 1 year ago

Does anyone know of a good dashboard to check for such BGP routing anomalies as (apparently) this one? I am currently digging around https://radar.cloudflare.com/routing but it doesn't show which routes were actually leaked.

I would love if anyone has any good tool recommendations!

SparkyMcUnicorn• 1 year ago

I don't know if I've seen CF Radar before. That's pretty cool!

Here are some others, although some seem to be experiencing issues due to the current outage I can only presume.

- https://atlas.ripe.net/probes/public

- https://www.ihr.live/en/global-report

- https://www.ihr.live/en/network

- https://bgp.he.net/

- https://ioda.inetintel.cc.gatech.edu/dashboard/asn

ddtaylor2• 1 year ago

I am a newb at this too, but is it "normal" for the "Announced IP Address Space" section to have that large jump from addresses like that?

estees_ecstacy3• 1 year ago

go https://status.gcp.databricks.com/

cluckindan• 1 year ago

BGP attack?

mrngm• 1 year ago

My default go-to: https://bgp.tools/

Why would you think this outage is (internet) BGP related?

whalesalad4• 1 year ago

Cloudflare runs all their own bare metal servers. Seems odd that they would be impacted by Google cloud. Same can be said for all the other issues on downdetector. This points to a broad issue at the core internet which could certainly be related to BGP.

stepri• 1 year ago

They have updated the status page finally https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

acureau• 1 year ago

Well this explains the issues I've been having with Spotify through the last hour.

artooro2• 1 year ago

What's crazy is that RCS messaging is down as a result of this outage. It shows how poorly the technology or infrastructure was designed.

foota• 1 year ago

Isn't RCS basically just instant messaging? I don't know why it's surprising that it would be down.

roywiggins• 1 year ago

I'm not sure any single company could have an outage that would take out SMS globally, but RCS is presumably more centralized.

treesknees• 1 year ago

Thanks for mentioning this, I’m seeing this too and couldn’t figure out why.

whalesalad4• 1 year ago

should have used Erlang

wbl3• 1 year ago

That explains why I couldn't get the photo of my parents dog today.

dcchambers• 1 year ago

Oh my god is that why my RCS chats were failing earlier?!?!

compscidr• 1 year ago

well this explains so much lol

ilovebabyyoda• 1 year ago

It looks like more than GCP: outages reported across the board including aws

https://downdetector.com/

koliber• 1 year ago

About the only thing not down is down detector.

tonyhart7• 1 year ago

god send omg, imagine down detector is down lmao

anyone know what tech stack they use and where they host

ekojs2• 1 year ago

https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue

IAM issue huh. The post-mortem should be interesting at least.

yard20103• 1 year ago

Ha. With all this soviet style euphemism I rather read the onion instead.

bananapub3• 1 year ago

It’s not a euphemism - every outage, including the 99.9% that don’t end up on HN gets a postmortem document written about it, which is almost always a fascinating discussion of the technical, cultural and organisational situation that led to an unexpected bad thing happening.

Even a few years ago senior management knew to stay the fuck out except for asking for more info.

0xCAP• 1 year ago

No major incidents

… Proceeds to show worldwide degraded service level alerts.

jimt1234• 1 year ago

Yep. Self-reporting status pages are pretty near worthless. At my former large company (not FAANG), we weren't allowed to update the status page until we got VP approval, which also required approval from both PR and Legal. It would take a lot more time and effort to get those approvals than to just fix the problem and move on.

iFred2• 1 year ago

SLA contracts, clawbacks, and performance obligations make these pages a bit of a minefield for CSPs. When I was at a top-tier CSP, we had the status page that was public, one that was for a trusted tier of customers, one built for a customer-by-customer basis, and one for internal engineering.

xan_ps007• 1 year ago

Where are the AI agents?

dionys• 1 year ago

Poor agents, finally taking a break

kyleee• 1 year ago

The AI is over employed

Jayakumark• 1 year ago

Someone must have checked in AI Generated code :-)

cyrux004• 1 year ago

GPay which is a widely used payment service in India is down as well

edm0nd• 1 year ago

India is having a really bad day today

unsupp0rted• 1 year ago

The last few times this happened I wouldn't have thought "So this is the day AI takes over".

But this time...

agawish• 1 year ago

GCP status page now reflect the issues, looks like Google Cloud Dataproc, Google Cloud Storage and Identity & Access Management

SenpaiHurricane• 1 year ago

Now my api can not connect to PostreSQL...

sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3

dineshsingh1• 1 year ago

https://downdetector.in/

DonHopkins2• 1 year ago

Damn you Bart Simpson!

https://en.wikipedia.org/wiki/Bart_Gets_Famous

dinvlad3• 1 year ago

#HugOps

devmor2• 1 year ago

Incident report published: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

james_m1231• 1 year ago

My firebase hosting and firestore db are back online, but GCP console and Google SQL instances are still having serious issues as of 7:00pm UTC.

phoenix98• 1 year ago

when its going to be fixed i am seeing now more and more services are down?

chupamela• 1 year ago

internal systems at google are currently broken.

phoenix__19982• 1 year ago

when its going to be fixed i am seeing now more and more services are down?

dinvlad3• 1 year ago

Surprised no one else mentioned "it's always DNS" yet :-)

phoenix__19982• 1 year ago

when its going to be fixed , i am seeing now more and more services getting outage started with IAM ?

kjuulh4• 1 year ago

I wonder what the damage ($) for having a good portion of the internet down for an hour or two ;)

throwcarsales• 1 year ago

My friends and I are even having trouble getting Rcs text messages to send.

coryvirok• 1 year ago

Shameless plug for https://rollbar.com

Good luck out there!

CyrMeta2• 1 year ago

if everything down at the same time - No one is mentioning an attack on us cloud services ? ( China or Russia ) Maybe ?

cyberflame5• 1 year ago

They've now added this as a major incident - before it just was listed under overview

CyrMeta2• 1 year ago

if all services at down at once, no one is thinking or mentioning a potential attack on US cloud providers ? (China or Russia) Maybe ?

dpedu• 1 year ago

reCAPTCHA affected? I couldn't log into my local utilities website due to a reCAPTCHA error. Downdetector agrees, but I interpret that site as dubious.

marifjeren• 1 year ago

Yeah recaptcha is down intermittently

ashwinsundar• 1 year ago

Interesting that all Digital Ocean services are fine...

baq• 1 year ago

One of these days in which the young engineers learn the concept of 'counterparty risk'.

DonHopkins2• 1 year ago

What is this Touchable Grass stuff I keep hearing of?

bsamson05• 1 year ago

So frustrating, but here's a link to track status of this outage: https://status.anthropic.com/incidents/kn7mvrgb0c8m

CrimsonCape• 1 year ago

I'm having trouble getting any Street View imagery. Can anyone else confirm?

NameError• 1 year ago

Yep, street view is not working at all for me

estee_ecstacy2• 1 year ago

Seems recovering now

estees_ecstacy3• 1 year ago

Seems recovering now

estees_ecstacy3• 1 year ago

seems recovering

jamesrwhite• 1 year ago

Seems like a wider issue at Google than just GCP, the Sheets and Chat APIs are also returning similar "Visibility check was unavailable" errors.

yunwal2• 1 year ago

Presumably many Google products run on GCP

throwaway7783• 1 year ago

Even though BigQuery is not listed in affected services, we see errors connecting to it

tecleandor• 1 year ago

It's listed by regions :(

leoh2• 1 year ago

If Google Chat is down per https://www.google.com/appsstatus/dashboard/, the ability for Google engineers to communicate among themselves impaired, despite SREs having IRC as a backup.

iamdelirium• 1 year ago

Google Chat wasn't down for me throughout the entire incident.

miohtama• 1 year ago

Someone actually uses Google Chat...?

asadm• 1 year ago

it's the best

leoh2• 1 year ago

Almost everyone inside Google

00deadbeef• 1 year ago

Google has a chat product?

sebzim4500• 1 year ago

TIL Google chat hasn't been killed yet

bananapub3• 1 year ago

it at least used to be standard and fairly well known practice for non-sres to use the irc bridge.

the much more disastrous situation would have been the irm fallback.

donalhunt• 1 year ago

They have irc services internally (or at least did when I was there 10-ish years ago).

jim180• 1 year ago

Claude Code is down :( too lazy to do manual conversion from Cocoapods dependency to SwiftPM

rectang• 1 year ago

Haha, I don't ordinarily spend a lot of time in the Google Cloud Console but just now I was debugging a squirrely OAuth issue with reCAPTCHA failing to refresh several days running. I'm getting this weird page error, and I think, "Is this an issue with my organization? [futz futz futz] Hey wait is GCP actually down?" And it turns out to be the top discussion on HN. XD

dgellow2• 1 year ago

Well, good luck to all googlers dealing with this, that's not fun :(

enahs-sf• 1 year ago

Would be comedy if one of the progenitors of this took Sundar’s buyout offer yesterday and let the world burn today.

AIorNot• 1 year ago

sheesh so many side-affected issues accross all systems, maybe big tech companies like google shouldn't have laid off all those engineers.. https://www.google.com/appsstatus/dashboard/incidents/Eab7zG...

but no tech bros, just keep following your ketamine addled edgelord when he did this with twitter..

dang• 1 year ago

Related ongoing thread:

Ask HN: Is Firebase Down? - https://news.ycombinator.com/item?id=44260669

cyberflame5• 1 year ago

Everything except us-central1 is back up - it's recovering now though

pancomplex• 1 year ago

thank god hn is hosted on a single bare metal server, free of all this bloat.

cddotdotslash• 1 year ago

I can recall plenty of times HN has been down.

rldjbpin• 1 year ago

it has already been facing some issues in the past couple weeks. e.g. the "front" page was broken randomly for past dates (like https://news.ycombinator.com/front?day=2025-05-30 or https://news.ycombinator.com/front?p=2)

as per their api documentation [1], it might be linked to firebase, which might explain this?

[1] https://github.com/HackerNews/API

admissionsguy2• 1 year ago

Google denies the outage. https://x.com/Google/status/1933246051512644069

baggy_trough2• 1 year ago

"clearing cache and cookies"? what is this, 1997?

bosmanos• 1 year ago

lol

andrelaszlo4• 1 year ago

First, check that nobody else in your family is making a call on the phone line that your modem is connected to, then make sure to disable your Internet Explorer add-ons before trying again.

ElijahLynn• 1 year ago

for those who boycott X:

https://nitter.net/Google/status/1933246051512644069

desktopninja• 1 year ago

Borg and K8s were fighting for resources, so Gemini decided to take out DNS. Now a sysAdmin has to step in.

* just trying to add a little humour. pretty stressfull outage. grarr!!

lawrenceyan• 1 year ago

Solana is up ¯\_(ツ)_/¯

pier25• 1 year ago

"All locations except us-central1 have fully recovered. us-central1 is mostly recovered. We do not have an ETA for full recovery in us-central1."

kubectl_h• 1 year ago

An hour later and everything is a mess in central-1. They seemed to jump the gun on that one. Doesn't matter if some dinky service like "AutoML Vision" is working, if GCS isn't, then they shouldn't post an optimistic message.

_2d30• 1 year ago

This is at least why Claude is dead: https://status.anthropic.com/incidents/kn7mvrgb0c8m

Also spotify isn't working for me so I assume that's also related.

These are my most important productivity resources! Sad!

estee_ecstacy2• 1 year ago

sentry is down https://status.sentry.io/

cyberflame5• 1 year ago

Root cause has been identified and it's being resolved/monitored now

johnnyApplePRNG• 1 year ago

This appears to be continuing to cascade over an hour later... wow... more and more services mentioned as completely down on the outage page.

Kind of nice to not be glued to AI chat prompts for a while to be honest.

sigmaball• 1 year ago

Guess they used Jules to code their services :)

alexcroox4• 1 year ago

2 hour outage at this point

plerpler• 1 year ago

GCP Artifact registry still down... Not accepting image push and showing 500 status code

augbog• 1 year ago

Cloudflare Outage also just updated

Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information

togume• 1 year ago

Is GCP the third party?

itdependsnet2• 1 year ago

Any chance this is the root being that so many different services are effected? https://github.com/kubernetes/kops/issues/17433

yunwal2• 1 year ago

I doubt gcloud would be affected by an aws-specific cni. Unless maybe enough AWS users have a GCP backup environment that they flipped on all at once, but it seems unlikely

itdependsnet2• 1 year ago

good point. I took that as simply the example that they had in front of them but a generic issue.

jamie0• 1 year ago

https://cloud.google.com/kubernetes-engine/docs/release-note... google did release an update to gcp k8s today, seemingly shortly before the outage

pikdum• 1 year ago

Was just about to do a demo, but Google Meet was down. Tried to use Jitsi as a fallback, but couldn't log in because Firebase was down too. Ended up using a Slack Huddle, lol.

zeke• 1 year ago

mapbox maps seemed to be down for a few minutes about an hour ago. I wonder if it is related.

ferchysmn• 1 year ago

Hola

jwatte• 1 year ago

Let's say a typical base service (network attached RAM or whatever) has 99.99% reliability. If you have a dependency on 100 of those, you're suddenly closer to 99% reliability. So you switch to higher-level dependencies, and only have 10 dependencies, for a 99.9% reliability. But! It turns out, those dependencies each have dependencies, so they're really already more like 99.9% at best, and you're back at 99% reliability.

"good enough" is, indeed, just good enough to make it not worthwhile to rip out all the upstreams and roll your own everything from scratch, because the cost of the occasional outages is much lower than the cost of reinventing every single wheel, nut, bolt, axle, bearing, and grease formulation.

geocrasher• 1 year ago

https://soundcloud.com/ryan-flowers-916961339/the-internet-i...

quyleanh• 1 year ago

Look like affect to Cloudflare as well [1]

  Update - Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency.
  
  Jun 12, 2025 - 19:57 UTC

1: https://www.cloudflarestatus.com/

asim• 1 year ago

Just our bi-yearly reminder of our over reliance on cloud providers for literally everything. Can't say there's an answer beyond trying to build more independent tech but we know how that goes.

ocdtrekkie• 1 year ago

Hilariously, I did not know about any outages today during the workday because we discourage cloud service usage and nobody complained about anything breaking. :)

ManBeardPc2• 1 year ago

Yet migration to the cloud continues, driven by people arguing that doing it yourself is too complicated and expensive. Let’s see how long until one outage takes down the global economy for multiple days or weeks.

JohnMakin• 1 year ago

"The cloud" isn't the problem nor is migration to it. The problem is single points of failure in a vast, super-connected, global network.

To demonstrate my point, say someone like cloudflare opted to get "off cloud" and run their own datacenters. Half the web would still go down if they had some issue in their datacenters. If anything, the economy of scale and resiliency of a huge cloud network is far beyond what any single operator can ensure for their own service. If this wasn't the case, cloud services couldn't be as profitable as they are.

It isn't the panacea people seem to think it is. One critical service going down, regardless of whether it's cloud hosted or not, has ripple effects in the broader network.

LZ_Khan• 1 year ago

Is this the new Y2k?

riknos314• 1 year ago

Actual incident link posted: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

makk• 1 year ago

And THAT, Smithers, is why we wear hardhats on the job.

kodisha• 1 year ago

Waiting for downdetector.com to respond...

ManBeardPc2• 1 year ago

The cloud enables you to scale. It allows us to distribute systems across multiple regions and data centers. Seems that this is true for outages as well.

The PHP application I wrote as a student running on a single self-hosted server had a higher uptime than any of the cloud providers or redundant system I have seen so far. If you don’t need the cloud for scalability, do it yourself and save yourself the trouble and money. Most companies would be better off investing into some IT staff instead of giving away their systems in the hands of some proprietary and insanely complex cloud environment. You are becoming dependent on someone you don’t know, have no control over and can’t talk with directly. Also the single point of failure is just shifting: from your system to whatever system is managing the cloud. Guess one advantage is that you can shift the blame to someone else…

herpderperator• 1 year ago

When Google said GCP is "down", did it affect entire availability zones within a region? For people who designed redundant infrastructure, did your backup AZs/regions keep your systems online?

slt2021• 1 year ago

so much for System Design interview and bs gatekeeping...

jlhawn• 1 year ago

The outage was global. For my team specifically, a global Identity and Access Management outage meant that our internal service accounts could not refresh their short-lived access tokens and so different parts of our infrastructure began to fail over the course of an hour or so, regardless of what region or zone they were in. Services were up, but they could not access critical GCP services because of auth-related issues which resulted in internal service errors for us.

To give an example, our web servers connect to our GCP CloudSQL database via a Cloud SQL Auth Proxy (there's also a connection pooler in between but that also stayed up). The connection to the proxy was always available, but the proxy wasn't able to renew auth tokens it uses to tunnel to the database, regardless of where the webserver or database happened to be located. To mitigate this in the future we're planning to stop using the auth proxy and connect directly via mutual TLS but now it means we have to manage TLS certificates.

marcinignac• 1 year ago

Can't wait to see how charts are going to look like here on the project we have developed for Maintel https://variable.io/maintel-digital-landscape/. It shows availability across multiple services as a landscape. Expecting to see a lot of spikes tomorrow.

biglyburrito• 1 year ago

I wonder how many SLAs Google blew out today with this outage.

knuppar• 1 year ago

Spotify was not loading, thought my 5G was bad, used YouTube Music instead without issues. Hmmm...

GCP Outage

Related Stories