Google Cloud Is Having IO Issues in US-EAST1

https://status.cloud.google.com/incident/compute/19012

by Fabricio20 • 7 years ago

120 57 7 years ago

status.cloud.google.com

teovall• 7 years ago

This outage is affecting Discord.

https://status.discordapp.com/

bpye• 7 years ago

Discord seems to be preventing new connections.

VolatileVoid2• 7 years ago

Is it just me or are Discord's status updates legendarily bad to the point of being entirely useless? As a user of the discord app, don't tell me the "API is having issues." Just tell me, "Discord is currently down. We are working on it and apologise for the disruption." Everything else is unnecessary line noise.

p1necone• 7 years ago

Mentioning the cause of the issue seems better than just "shits broke yo" to me.

Fabricio20• 7 years ago

The Discordapp status page is written to by developers, and as a developer I really like the insights as to what is going on. As a user, I expect you to just look at it and see it's "Having Issues, We are working on it".

scottmf• 7 years ago

We use Discord as an interface for moderating our app and monitoring spam/reported content so this kinda sucks. Not super important though.

seriesf2• 7 years ago

Being taken down by slow IO in a single cloud zone seems to implicated their overall architecture.

jrockway2• 7 years ago

Looking at their job postings, they use Cassandra and Elasticsearch, so I am somewhat surprised that they didn't survive this outage. (I have not ever run Cassandra at scale, but their website says in the first paragraph that it's designed to handle regional failures.)

If I were on GCP and had paying customers, I would use Cloud Spanner. It's expensive, but it's a good piece of technology.

mostlystatic• 7 years ago

Cloud SQL has been down for me for 45 minutes now. The status page still doesn't say, but it says this in the Known Issues for technical support:

"We are experiencing an issue with Cloud SQL instances hosted in the us-east1 region, beginning at Saturday, 2019-12-07 10:00 US/Pacific. Symptoms: Some Cloud SQL instances hosted in this region are becoming unavailable, and are refusing connections. Self-diagnosis: Connections to the Cloud SQL instance are rejected. Workaround: None at this time Our engineering team continues to investigate the issue. We will provide an update by Saturday, 2019-12-07 12:04 US/Pacific with current details."

Illniyar• 7 years ago

Well here goes the 4 nines.

microcolonel• 7 years ago

No worries, that's what the other datacenters are for. :- )

Jasper_• 7 years ago

The beauty of distributed systems is that you spread your services across lots of different computers, so that when one of them goes down, they all go down.

erik_seaberg2• 7 years ago

“A distributed system is the one that prevents you from working because of the failure of a machine that you had never heard of.”

itronitron• 7 years ago

and typically that machine is the authentication server

BlueTemplar• 7 years ago

It's not really distributed if there's a single point of failure?

eyjafjallajokul• 7 years ago

Not sure what you mean here. Would a similarly architected non-distributed system have higher levels of availability?

WilliamEdward• 7 years ago

I think they're being sarcastic? The idea with a distributed system is if one part goes down it can still mostly function, which google have managed to fail at somehow.

xvector• 7 years ago

He was joking :) a buggy or poorly designed distributed system might behave more like a house of cards than independent silos

exabrial• 7 years ago

Interesting, do they use a SAN or is it local storage?

tyingq• 7 years ago

They offer both: https://cloud.google.com/compute/docs/disks/

Discord's description of the issue sounds like issues with either zonal or regional SAN storage.

ithkuil3• 7 years ago

Google Cloud Persistent Disks are implemented on a kind of Storage Area Network (AFAIK it's based on IP networking, IIRC it's iSCSI over Colossus /D, not on fiberchannel or other off the shelf SAN tech)

joatmon-snoo2• 7 years ago

It's not iSCSI (at least I don't think so).

The closest thing to public literature about Colossus and D that we've ever published is the Procella paper, which describes the abstractions that Colossus provides for it. In some ways it's similar to GFS (RPC interface, writes are generally append/overwrite) but many things are completely different now.

vl6• 7 years ago

It’s a custom log-based block device implemented on top of Colossus/D. As such it connects to storage nodes using regular data center network. This is why (in some cases) both IO and network-intensive workload may compete for the same bandwidth on the host.

ikiris• 7 years ago

Honestly, I'd describe it closer to BFM: black fucking magic.

Some stacks are mostly normal, some are really odd... and then there's storage.

Animats2• 7 years ago

Is this the beginning of Google's wind-down of "cloud services", as part of the de-emphasis of Alphabet and the refocus on the core ad business?

ilogik• 7 years ago

yes, they're moving out of a profitable business where they have a lot of knowledge about.

and they start this move with an outage during the busiest time of the year

why not?

leesalminen• 7 years ago

yes, they're moving out of a profitable business where they have a lot of knowledge about.

Rackspace is trying to get out of the public cloud game. We had $20k/mo account with them and the rep called me every month asking if we wanted their help (professional services) migrating to AWS. I was baffled but they admitted they don’t want to be in the Cloud game anymore and wanted to get customers off their platform.

heybrendan• 7 years ago

Respectfully Mr. Nagle, I expect more intelligent {commentary,questioning} from you, especially given your breadth of experience and background.

A mere infrastructure issue, that occurs with all providers from time to time, would hardly constitute a service "wind-down". Based on a very cursory review, GCP's official deprecation policy [1] doesn't seem all that dissimilar from everyone else.

In short, to answer your question: No.

[1] https://cloud.google.com/terms/deprecation

Animats2• 7 years ago

My point here is that, while this is a big issue for Google Cloud customers, it's not a big deal for Google. Worst case for Google is they issue a credit for more Google Cloud services, not to exceed half the monthly payment.[1] Google Cloud is, what, 3% of volume? AWS is Amazon's big profit center, but for Google, it's not as important. Nothing to bring in top management for on a weekend.

Now if ad revenue collection went down...

[1] https://cloud.google.com/compute/sla

hn_throwaway_99• 7 years ago

I am in general a big fan of GCP, especially for startups. In many cases I find I can be productive faster in GCP compared to AWS, in the sense that it's easier to do "wrong" things in AWS if you don't have a top notch devops person.

That said, it seems like GCP has a real, significant problem with overall reliability. While I don't have numbers to compare, seems like I see frequent outage notifications on HN for GCP. They really need to focus more on reliability than new features.

RKearney• 7 years ago

I am in general a big fan of GCP, especially for startups.

The major disagreement here from me on this is that GCP does not support IPv6 (outside of IPv6 termination on load balancers) and I do not agree with startups launching without IPv6 support in 2019.

GhettoMaestro• 7 years ago

As a network person thank you for bootstrapping IPv6 where you can. Most folks still don’t care :-(.

eloff• 7 years ago

I agree it's important, but it doesn't really bother me since I'll put Cloudflare and or a network load balancer in front of anything external facing anyway. Internal things being ipv4 is fine. Other benefits aside you halve your bandwidth bill by putting Cloudflare in front.

ljhsiung• 7 years ago

In case you're curious, here's some logs about their status https://status.cloud.google.com/summary

Looking at Cloud Compute, the one responsible for the outage today and I think the big one a few months ago (13 hours!!!), they still do achieve 99% uptime (having been down 81 hours total this year).

While 99% uptime "sounds" great, this is paltry in comparison to AWS, which has set the standard for 99.99% (also known as "4 9s") in reliability, translating to about an hour of downtime per year.

Just gives you some more perspective and concrete numbers. Perhaps one of the reasons you find GCP "faster" is due to more devotion of resources to "speed" over "reliability".

lostmyoldone• 7 years ago

Where is AWS logs for all service outages, partial or not?

Can't seem to find them, only "selected" outages. The recent, several hours long, service degradation in Frankfurt EC2 isn't in the historical list, seems like it is only in the rolling status history?

AWS and GCloud seem to also be reporting disruptions completely differently, AWS reports on tons of smaller pieces which makes any "aggregate" uptime become something completely different than if you report on larger aggregates as GCloud seems to do.

Unless I'm missing something, I don't see how one could compare comparing service availability reasonably without running large numbers of "canary" instances on both providers to actually measure aggregate availability?

jordanthoms• 7 years ago

I don't think you can judge uptime for either AWS or GCP from the status pages, both of them are slow to update their pages when issues start and plenty of less major issues (which could still bring down a individual customer) never make the status page at all.

blaisio• 7 years ago

I agree that they seem to have problems more frequently than AWS. I've been told it is frequently due to their network, which is incredibly advanced and efficient but also incredibly complex and therefore more easily broken.

synack• 7 years ago

The most recent update indicates that this outage is only affecting a single availability zone. https://groups.google.com/forum/m/#!msg/gce-operations/oYoFJ...

Google Cloud Is Having IO Issues in US-EAST1

Related Stories