Three people in critical condition from Google data center 'electrical incident'

https://www.sfgate.com/news/article/google-electrical-incident-injures-3-17360321.php

by Jimmc414 • 4 years ago

143 69 4 years ago

tatersolid• 4 years ago

Terrible, I hope these folks are OK long term.

I recall a similar incident, causing us to enact our DR plan a decade ago. A truck driver had a medical issue and drove into the transformers outside a Rackspace facility. Emergency crews required everything to go dark for the rescue. High-power electric delivery is the Achilles heel of the cloud.

inferiorhuman• 4 years ago

Doubtful they'll be okay long term. Even if they survive the burns, electrocution does a number on most of your organs including your kidneys. I'm in no way to evaluate the veracity of the claims (which are mostly: electricity damages the body in a lot of different ways), but here's a graphic description of the damage electrocution does to the body:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6026718/

What I'd read in a non-academic context is that the kidney damage suffered from electrocution takes more than a few years off your life expectancy. I couldn't (quickly) find an authoritative source discussing long-term survival but burns and acute rhabdomyolysis (like the paper above is talking about) on their own are no joke.

It'll be interesting to see if Google opens up about what went wrong.

ars5• 4 years ago

It's unlikely to be both an arc flash and electrocution, and to survive for any length of time.

Electricity powerful enough to cause an arc flash would instantly kill if it went through a body. It was more likely a tool shorted two conductors, and the arc flash harmed them, but no electricity went through them.

xxpor2• 4 years ago

Arc flashes are no joke: https://www.youtube.com/watch?v=G0WlqC7MkS0

belkarx2• 4 years ago

Beat me to it :)

I was going to post this one: https://inv.bp.projectsegfau.lt/watch?v=PO6see7_ODY (this is an invidious link; it's an alternative frontend to youtube)

teeray• 4 years ago

It’s like every panel has a claymore mine in it

mlindner2• 4 years ago

I prefer this video as it has slow motion footage of the flash. https://www.youtube.com/watch?v=6hpE5LYj-CY

bearmode• 4 years ago

If you watched the guy's video above, you'd see a much better slow mo of the flash at 1:30

the8472• 4 years ago

Unless you have solid-state circuit breakers. https://youtu.be/szhDolRV2S0?t=2268

xxpor2• 4 years ago

That's seriously amazing, do you know if there are solid state breakers for high voltage? 480v is of course a lot, but if these guys were working on a distribution box outside the actual DC, I wouldn't be surprised if it wasn't fully stepped down yet.

msoad• 4 years ago

Hope those folks are going to recover.

Hope Google releases details on how this took Google Search down? In theory the entire data center can vanish and everything should stay functional

quickthrower2• 4 years ago

Probably unrelated?

belkarx2• 4 years ago

Arc flashes, for the uninitiated, are short-circuits caused by electricity flowing through through air (partially ionized) between two live conductors. The arc of electricity causes an arc blast, which is a huge fiery shockwave that can (and does) blast through metal and all surroundings

HKH2• 4 years ago

Like a giant welder?

ta2234234242• 4 years ago

More powerful than that. Think molten metal going everywhere or vaporizing into the air.

atoav2• 4 years ago

You are forbidden by law (and mandated to report) such arcs in Europe. In Germany people who work in such environments have to have a few special certificates and PPE that looks like they are trying to defuse a bomb.

Arc flashes like that can burn with 10000 to 15000 degrees celsius and if you are close by not only will they send you flying, you are probably dead before you hit the ground.

And trying to extinguish such an arc can be quite a challenge.

mlindner2• 4 years ago

More than that: https://www.youtube.com/watch?v=6hpE5LYj-CY

jfoutz• 4 years ago

Yes. Like this, but not intentional - https://www.youtube.com/watch?v=GMbN9nb3qyk That alone looks very scary to me, indoors? accidentally? gah.

moring• 4 years ago

I don't get it. There is a huge arc in that video, but no shockwave.

edit: It's explained further down the thread.

noja• 4 years ago

Is this at all related to use of DC-power in datacentres?

andruby• 4 years ago

Afaik DC or AC can cause similar arc flashes. Typically you do need high voltages to bridge larger gaps.

n0mn• 4 years ago

What conditions can cause this to happen, and how does one prevent it?

albrewer• 4 years ago

Basically, the dielectric constant of air is not infinite. If the electrical potential is high enough between two points, air can conduct electricity.

If you put a small bit of conductor near two legs of a high voltage system, and that conductor has a path to ground, you can get now get an arc between a voltage source and the conductor (e.g., someone with a screwdriver loosening a screw is too close to a high voltage leg). This initial arc ionizes the air, which makes a pretty great conductor, so the amount of current flowing through the air increases really quickly as if it's flowing through an open circuit. The arc is superheated, can burn through steel (or flesh) quickly, and expands.

One of the hazards of working on high voltage is gasping in shock after an electrical arc - you inhale superheated plasma and it cooks your lungs instantly, so you suffocate.

Anyway, to prevent this, you don't cut corners as an electrician - you have proper insulation on all your tools and body parts to prevent this sort thing. You isolate legs of high voltage circuits, or de-energize the circuit you're working on with proper lock-out tag-out procedures.

mc322• 4 years ago

The question is what caused it?

One scenario is equipment failure. On-site person was not familiar, called for back-up, back ups did not don proper PPE nor follow proper protocol. Another scenario is the equipment failed in a manner not envisioned by the designers of the protocols and were really unlucky.

In any case, this is really sad and bad and hope they recover.

gaudat• 4 years ago

It may be something simple as a shor tbetween the phases. In power engineering these are called fault conditions and the currents can run up to tens of thousands of amps before the circuit breaker acts. Not sure if you can imagine the magnitude but that is why the standard PPE in working with on-line low voltage (<1 kV) switchgear is a full body silver arc flash suit.

Now imagine if the datacenter guys need to wear that suit before going down the racks...

ars5• 4 years ago

I was testing a light bulb socket one time, and accidentally shorted it - the short set fire to the metal of the bulb socket (I assume it was aluminum), which proceeded to burn until it was all consumed, despite me pulling the plug.

It produced a jet of flame and sparks many feet tall. I had no idea this was even something that could happen.

I can't even imagine how bad a powerful arc flash would be.

colechristensen• 4 years ago

Yeah metal fires are no joke. Hard to start but when they do…

kadoban2• 4 years ago

Not _exactly_ just a metal fire, but along similar lines: thermite reaction videos are fun to look at on youtube.

In person I get the sense that they're too scary and unpleasant to really be a good time to be around, even if well controlled.

pflanze4• 4 years ago

I assume it was aluminum

Maybe it was magnesium? That is known to burn once ignited. Aluminium AFAIK doesn't (unless it is powder).

zaarn2• 4 years ago

Aluminium burns too, just harder to get going. Almost all the metals can be coaxed into burning in various ways, all metal fires are an absolute disaster when they happen.

c_o_n_v_e_x3• 4 years ago

We had an arc flash incident at a Digital Realty DC years ago. The project was to build out a data hall to be used by a prominent social networking website. Someone attempted to incorrectly terminate a wire on the UPS battery bank.. huge number of lead acid batteries all wired together. The technician had a sizeable portion of skin burnt off one arm.

pvaldes• 4 years ago

My sympathy with the injured workers and their families.

The technical aspects of the accident are fascinating, of course, but the main point here is how this people will learn to adapt to their new life. And they will. They still have a loving family and friends and I'm sure that Google will do their best to support them also. They probably will need skin grafts and surgery but, as they were found breathing and conscious, I'm confident in that they will survive this horrible experience. Life deserves to be lived even if personal circumstances are hard.

captainredbeard• 4 years ago

This message of hope and resilience is wonderful

Overtonwindow• 4 years ago

I wonder if this is what caused the outage yesterday.

Three people in critical condition from Google data center 'electrical incident'

Related Stories