Google Outage in Europe

Assuming the chance of an outage is actually static. If there is one (highly reliable and trusted) provider with five products one year, and three providers with ten products the next year, the chance of you seeing an outage has gone way up because of surface area.

querez• 5 years ago

Except that Google services don't run within GCP, but apart from it (mostly on Borg I'd guess).

deepstack2• 5 years ago

More reason to promote for self-host, and decentralised systems like smtp, matrix.org, ActivityPub. The idea of having all the data and server concentrated in a few player such as google, amazon, ccp is not reliable for the digital operation of the planet.

sva_• 5 years ago

Almost like bad weather. Perhaps we'll have cloud outage forecasts at some point.

yunohn2• 5 years ago

There’s always this question on every “X is down” post on HN.

Quite boring commentary tbh.

afc• 5 years ago

I think it's just you, probably recency bias. I recall the days of Gmail being globally down for hours, more than once (3 times, IIRC), back in 2009.

The-Bus• 5 years ago

I also can't think of the last time I saw Twitter's Failwhale, which almost became their default mascot at one point.

(Yes, I know they discontinued it, but I haven't seen the service "over capacity" in some time. I say this as a casual web user of it, not an active account holder or app user).

riffic4• 5 years ago

ask anybody who researches complex and distributed systems and see what they say.

1cvmask• 5 years ago

And many people blame their internet provider as google outages are rare.

vanderZwan• 5 years ago

I don't think it's that so much as the fact that that half of the internet services that rely on Google along the line somewhere (but not visibly to the users) are down, at least for me. And yet the Google search page can be reached (again, speaking from my European perspective, sample size one). To make it even more confusing: these issues are limited to my WiFi. If I use my phone connection as a hotspot the problems disappear.

So from the perspective of a regular person it is very reasonable to conclude that it is their internet provider that is down.

kaba0• 5 years ago

And I think no internet provider has such a good name where an outage on their part would be unheard of. Much more so than google.

ratww• 5 years ago

Now that I think of it, I might have blamed my provider or at least restarted my modem if Hacker News was ever down...

rablackburn• 5 years ago

It’s the perfect connection test.

9 times out of 10 if you can’t connect to hn the problem is you, not them. Probably even 99/100

veltas• 5 years ago

People and software, when Facebook was done my mobile thought the internet was unavailable.

intunderflow• 5 years ago

I felt a great disturbance in the force, as if millions of pagers suddenly cried out in terror

kroolik5• 5 years ago

There were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around december/eoy, thus more releases are made.

https://en.m.wikipedia.org/wiki/Google_services_outages

oblio3• 5 years ago

He he. It's kind of hilarious. Maybe they should stagger reviews to ensure high availability? :-D

kroolik5• 5 years ago

Nothing wrong with it. People make errors, distributed systems aren't easy. More frequent changes - more likely a bug was introduced.

My post is just a speculation, let's wait for the actual technical details doc from Google.

moffkalast• 5 years ago

Then they get fired instead, well done ha.

kroolik5• 5 years ago

Why would you fire an engineer you have just spent millions to train?

evercast• 5 years ago

Comments like this make me wonder if people really expect engineers to be fired because of an outage? I do not work at Google, but none of my workplaces would fire engineers because of a failure. Mistakes happen. As long as they are not repeated, everything is good.

If your company fires people in situations like this, run away and never look back.

hdjjhhvvhga• 5 years ago

It makes no sense at all. After the outage you have not only a review of the causes and appropriate remedies, but also more experienced people who are now more aware of possible consequences of seemingly unrelated actions and will take extra care not to make these things happen in the future.

Also, such cases are rarely the "fault" of a single person. Or, the direct/immediate cause is often not the main one.

delroth• 5 years ago

Perf has been over for almost a month now, and the evaluation period was over more than two months ago.

kroolik5• 5 years ago

Then it might, indeed, be unrelated. Thanks for clarification!

samhw4• 5 years ago

Could it not be knock-on effects of earlier changes? Or indeed post-review changes from scared/precarious engineers?

dspillett• 5 years ago

and the evaluation period was over more than two months ago

Excellent time to slack off a bit, make a few mistakes, then come next eval you can point to a marked improvement over the intervening six-to-nine months!

inglor• 5 years ago

I would be very interested in any research about code quality with relation to promotion packets. Ideally acamdemic.

I am not sure where to look for it

rajin444• 5 years ago

Is there an objective way to measure code quality?

We barely understand the systems we’re writing code for - I don’t see how you could objectively judge the code to manage those systems without fully understanding the systems first.

You could have wave some metrics like number of bugs or test coverage, but I can’t think of any (aforementioned included) that aren’t subject to tons of confounding variables.

de6u99er• 5 years ago

Some people are definitely not gatting a raise.

jankeymeulen• 5 years ago

More likely someone actually will. A blameless postmortem will be written, and the people that will fix the bug or systems issue will have something to work on with high visibility and high impact, which tends to translate in good performance ratings.

(Googler, opinions are my own)

trhway• 5 years ago

November 11 is a public holiday in US. And it seems it has in recent years became a company holiday in many companies too. May be that break in established over many years cadence of workday and holidays plays a role here, like say affecting smoothness of support transition between US-EMEA-APJ.

mikelward• 5 years ago

Veterans Day is not a day off for US Googlers.

kixiQu• 5 years ago

Far more likely IMO to coincide with trying to get Exciting New Features out before release freezes for some conference or other.

deanCommie• 5 years ago

And this is why nobody serious should build on GCP.

Google's more interested in placating their primadonna engineers than solving customer problems.

bamboozled• 5 years ago

There is also a lot of pressure on engineers around black Friday and Cyber Monday to get things done before any change embargoes come into place. This is coming from someone who just worked 16 hours straight to scale things before a change freeze.

I don't know how or if this would impact Google, but I'm sure someone there has at least thought about those dates.

alpb• 5 years ago

evaluation and promotion window ends around december

Googler here, that is not when those promotion/perf evaluation windows happen.

peteri• 5 years ago

Docs was a bit flaky earlier on.

tomudding• 5 years ago

Probably related to the Google Cloud "service disruption" that started earlier this morning: https://status.cloud.google.com/incidents/1xkAB1KmLrh5g3v9ZE...

It was very frustrating to see DNS working perfectly fine (since DNS is always the problem) and the connections just timing out.

demosito666• 5 years ago

I was also getting 500 error in GCP console

ddek• 5 years ago

Yep, we're hit by that right now. It's not a total outage, we're losing about 10-15% of our calls to bigquery from within cloudfunctions. What we have using a VPC connector is ok. Fortunately areas affected are ancillary (mainly monitoring, ironically), our service is still running.

elp• 5 years ago

The Google imap service has also been unstable in South Africa this morning.

ulzeraj• 5 years ago

Didn't even noticed. Work doesn't use any GCloud components and personally its been a while since I've degoogled myself. No Google docs, mail or search. Not even the quad 8 nameservers.

yosito2• 5 years ago

Not sure why you got downvoted. I'm also a degoogled European and didn't even notice.

rossdavidh2• 5 years ago

Downvoted from people who are jealous, perhaps?

dredmorbius• 5 years ago

Also reported / dupe here: https://news.ycombinator.com/item?id=29197479

bla3• 5 years ago

Looks like it might be resolved now.

nudpiedo2• 5 years ago

Seems that google is not anymore the company it used to be advertised. The myth prevails though.

It’s a pity as some of its services are frankly among the best, and I depend on them.

southerntofu3• 5 years ago

Seems that google is not anymore the company it used to be advertised.

Do you mean about the "Don't be evil" slogan it started with? Even back then, it was a pretty obvious move for any movie villain.

Or do you mean you actually bought into the reliability promises of cloud providers thinking downtime would not exist anymore?

nudpiedo2• 5 years ago

due their reliability and over the tp. It always overfelivered. during its first decade, If anyone read any of the books from its founders, papers, stories related to the company and them it seemed to be the new "non zero day" infrastructure, however the longer the company alive is the more it feels just like any other.

Unrelated: not sure what is going on on HN since a few months but almost all comments I get to write get automatically downvoted, even little things that no one should really care much about.

rzr• 5 years ago

Let me crosslink to invidious tracker:

https://github.com/iv-org/invidious/issues/2577

tobyhinloopen• 5 years ago

We just released a rewrite of an (web)app today so lot’s of support calls and mails. Really unfortunate timing, hah

rossdavidh2• 5 years ago

So what you're saying is, you brought down Google with your rewrite. :)

intsunny• 5 years ago

Outage reports in UTC, amazing!

(glares at AWS)

tex0• 5 years ago

Hahaha. This one made me laugh hard.

andrelaszlo• 5 years ago

No it will make you laugh in about 30 minutes. Oh wait, DST. Never mind.

NaturalPhallacy2• 5 years ago

I want to smack whomever invented time zones.

Everyone is bad at them. Trading desks, international trade offices, space related offices all have a set of clocks on the wall because even really smart people just suck at figuring out time zones. I even use https://everytimezone.com/ in lieu of a set of clocks.

What a needless complication. I wish we could all just switch to UTC and stop daylight savings time.

judge2020• 5 years ago

I can't imagine anyone taking kindly to a change as big as eradicating time zones. Get rid of DST, yes, but only if the U.S. goes up an hour (the extra sunlight at 6pm during DST is nice).

iso12108• 5 years ago

I do laugh at people that say "Using the cloud means less downtime than your own server", then things like this come along :D

HatchedLake721• 5 years ago

It’s still less downtime than your own server and you have hundreds of engineers 24/7 ready to diagnose and fix issues

tester34• 5 years ago

It’s still less downtime than your own server

what makes you think so, actually?

iso12108• 5 years ago

Really, downtime of google so far this year: More than 0 minutes. Downtime of my own server: 0 minutes.

Of course I'm not trying to run a massively scalable service coping with millions of customers, because I don't need that.

southerntofu3• 5 years ago

This is so misinformed by industry propaganda. Modern cloud services are very often unavailable from a region without any apparent reasons. Or the service appears to be available but some specific feature doesn't work. Or it's available but just really slow or dropping packets.

When you have a "simple" (already quite complex) BGP(TCP(HTTP)) tunnel, chances are things just work and it's easy enough to diagnose issues. Between anycast, auto-scaling, and WAFs (among others) you have added so many layers of complexity that the chance of a "random" error somewhere in the stack has dramatically increased and diagnosis is close to impossible.

It used to be simple that web services are either up or down. Now it's not so obvious anymore, and that's definitely not being reflected in the 5 9's SLAs falsely promised by cloud providers. I can say with confidence most selfhosted systems i've been close to have much better uptime than modern cloud services and are much cheaper to service and maintain.

i5heu2• 5 years ago

Right?

My private website on my RPI is running now 2 Years without a problem and only minimal downtime due to rebooting for the new kernels.

It is amazing how much uptime you can achieve with a 5$ Computer in comparison to a 1730000000000$ (1,73 tera $) Company. Even if you compensate for dynamic content.

exikyut4• 5 years ago

(Google is currently processing multiple terabits of traffic per second using several billion dollars' worth of distributed infrastructure.)

remram4• 5 years ago

Maybe Google's servers are also running without a problem and you can't reach it for a different reason. Self-reported uptime is not a good measure of server availability.

Was your home internet available all of the time? How many times did you reboot your modem?

hulitu• 5 years ago

Maybe they are just upset because they got fined. It's common by big companies.

southerntofu3• 5 years ago

Or maybe this is unrelated and it's just one of the many cloud outages contradicting their uptime promises. Welcome to the reality of cloud computing, where everything's foggy and the hallucinogenic fumes made us believe downtime was a thing of the past.

gabkins• 5 years ago

I do wonder how many companies and businesses got affected by the outage.

natch• 5 years ago

EU to AI:

Here's a €2.4 billion fine for you.

AI to EU:

OK, human.

blauditore• 5 years ago

ITT: "My rasperry has been up for 2 years with almost no downtime, what is up with these suckers?"

Nextgrid2• 5 years ago

It's valid criticism for projects that don't need the scale or features that cloud-based solutions offer.

A single machine won't give you the same level of scaling and management features, but also won't have hundreds of distributed moving parts that could break and take your service down as a side-effect.

Spivak• 5 years ago

I feel like you are selling the cloud short for SMBs. The majority of my career has been spent working for shops that have their own datacenters or colo. It's so freaking cheap to run a huge service it's not even funny. And it's a dream to have so much compute and storage floating around that you can "waste" it without even thinking -- it's already paid for!

However, when your small the economics flip. You want to run as little as possible as cheap as possible. And a load balancer pointing at a two asgs in different availability zones is by far the cheapest way to get something production ready on the internet. You can go from your garage to a 100M business and probably never need to graduate from this architecture.

remram4• 5 years ago

"For a Linux user, you can already build a Google yourself quite trivially with wget -R and grep."

ac50hz• 5 years ago

I guess the network admin who skipped the classes on Network Fundamentals, has moved on from their recent placement in OVH, to Facebook, then to Google.

Next stop AWS, firewall maintenance division?!

yosito2• 5 years ago

Degoogled European here. Genuinely didn't even notice. Google can fuck off.

Google Outage in Europe

Related Stories