Senate passes bill to decrease grid digitization, move toward manual control

https://www.utilitydive.com/news/senate-passes-cybersecurity-bill-to-decrease-grid-digitization-move-toward/557959/

by howard941 • 7 years ago

246 179 7 years ago

inlined• 7 years ago

There’s a reason our nuclear system still operates on floppy disks.

96fps• 7 years ago

Floppy-disk based systems are not the only way to ensure security/reliability, but it's far better than a half-baked system that isn't properly audited. Systems we rely on must be dead simple and reliable.

minton• 7 years ago

Do you have a source on this?

lucasmullens• 7 years ago

Google it, it seems to be a generally known fact: https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-nucl...

tntn• 7 years ago

https://www.gao.gov/assets/680/677454.pdf

Page 15.

pfranz• 7 years ago

https://www.cnn.com/2016/05/26/us/pentagon-floppy-disks-nucl...

It was a popular tech story a few years ago.

floatingatoll• 7 years ago

https://hn.algolia.com/?query=nuclear%20floppy&sort=byPopula...

dzhiurgis• 7 years ago

This is when I head off to n-gate.com

drngdds• 7 years ago

I'd assume that's more about not fixing something that isn't broken. (Because if you break it, you might accidentally annihilate humanity!)

buboard2• 7 years ago

There was no alternative to floppy disks when they were built though. Was there a deliberate choice to use an older technology?

e2le• 7 years ago

Why not simply stop networking these systems together and/or connecting them to the internet?

colechristensen• 7 years ago

It doesn't stop attacks.

Stuxnet successfully attacked air gapped systems.

the-pigeon• 7 years ago

Still makes them exponentially harder since you need to get someone to connect an infected device to the internal network somehow.

simonh• 7 years ago

It does stop attacks. It just doesn’t stop all attacks. Just because a security measure, by itself, isn’t perfect doesn’t mean it isn’t worth implementing.

DEADBEEFC0FFEE• 7 years ago

Arguably that was a failure of physical or personnel controls. Someone was able to take media in. A physical device was allowed to be connected.

beenBoutIT4• 7 years ago

IIRC, stuxnet successfully targeted and exploited weak humans. The air gap didn't fail, an idiot worker with a USB drive was the vector.

antpls• 7 years ago

Maybe there are still benefits to have a network, but not using internet. Couldn't we build a network with a subset of internet features but which are all formally proved, and let humans take care of the rest ?

effie• 7 years ago

Internet is fine as a network, it is resilient to damage. What is sufficient to make it secure is to restrict the network features of the facility server software to one command, SHOW_STATUS. Control of operation over network should be impossible.

jshowa34• 7 years ago

There would be no way to do reporting on any of the devices in the field without some form of network connection. This delays response time if, for example, a recloser failed to operate, nobody would be aware unless someone physically checked the device. With networked SCADA systems, this becomes an immediate notification to the control room.

I'm a fan of manual systems, but within reason though. The point is, you're never going to be 100% protected. People can still break into substations and compromise the grid rather easily.

toomuchtodo8• 7 years ago

Read only monitoring is fine. “Write” ability must be heavily permissioned, audited, and governed. Failing that, no physical “write” access.

probablypower2• 7 years ago

This is exactly what is done in practice. Critical systems (i.e. control room software and hardware) are air gapped and operate in a DMZ.

The only real vector one has into such a system (assuming you can't physically access it) is by attacking the 'inputs' to the SCADA network.

Attack vectors like stuxnet or petya require physical access to these systems (i.e. a usb key getting plugged into an operation terminal).

bdamm• 7 years ago

But... airgapping doesn't stop damages from Aurora type of attacks. So even with top-level grid controls airgapped, if the attacker is at the switch for enough vulnerable lower-level consumers and can synchronize enough switches (car chargers, solar inverters, local substations, etc) the potential damages can still be very significant. As we saw in the Ukraine power grid attacks, you don't even need to go after critical systems. Quantity has its own quality, as the saying goes.

AmericanChopper• 7 years ago

This is exactly what is done in practice.

I think you’ve got this backwards. This is exactly what is done _in theory_. In practice many of these systems are not. When I was working as a consultant, I saw so many exposed SCADA systems.

Whatarethese3• 7 years ago

There are also NERC compliance requirements that require USB ports be disabled at and OS or BIOS level.

kevan• 7 years ago

There's definitely some advantages to using age-old practices. It's easy to explain how trust and verification works in a system when 2 people have separate physical keys and you need both to perform an action. It's a lot harder to explain things like modern crypto to most people. Until a critical mass of the decision-making chain understand the tech we may be better off not using it for mission-critical things.

Alternatively, this could also be a misguided effort that holds back smart-grid tech in America by a decade. It's really hard to tell.

Finnucane• 7 years ago

Does smart-grid tech work any better than smart-home tech? So far most 'smart' tech hasn't had a good track record on security or reliability, certainly not at the level you would trust your major infrastructure to it.

penagwin3• 7 years ago

Keep in mind most "smart home" technology is marketed at consumers who have no idea what SSL/TLS is (nore do many care sadly even though they should), and are meant to be as cheap as possible.

While I have no clue what the security or reliability looks like in industrial applications, I would hope they evaluate their solutions with reliability in mind.

sephamorr• 7 years ago

For the utilities, absolutely. The utilities have hard requirements of delivering power to all customers while maintaining grid and frequency stability, so the ability to shape demand and supply quickly is at least one very valuable knob.

jhayward• 7 years ago

Yes. The modern sensors and controls being built in to US electric transmission and distribution systems have resulted in a far better run grid: more stable, more reliable, more efficient, fairer, and enabling things like real-time market making which enables rapid improvement using new technologies such as grid storage batteries, etc.

It does come with some vulnerabilities. I'm not sure what the best way forward on those is; I suspect this bill's effects, if any, will not result in a more resilient or less vulnerable grid system. Attackers will just find different leverage points to attack.

xvilka• 7 years ago

It is just because ICS/IoT vendors do not care/know a thing about cybersecurity. If you push security development practices, it should really improve the situation. Just prevent an access to the market until they pass the red team audit. Would surely motivate them enough.

toomuchtodo8• 7 years ago

I have seen a contactor SCADA controller (high voltage physical relay) for a high voltage transmission line at a Florida substation on the public internet (no airgap, no vpn) during an audit.

Apply brakes while we work on the steering. This is good policy. “Internet Of Shit” is no joke.

nitrogen• 7 years ago

Industrial controls at least work most of the time. Can't quite say that about consumer gear.

jacquesm4• 7 years ago

That does not surprise me in the least. Some of the stuff I came across doing DD would make your hair stand on end, after 130 companies I'm more surprised by the ones that do it right than the ones that have no clue.

crankylinuxuser2• 7 years ago

In a hacking group I'm part of, we had an address that ended up being a passwordless VNC to what looked like a centrifuge controller. It was at a place whose IP tracked to CERN.

We reported it as fast as we fucking could. It took about 6 hrs for them to get it off the public internet with no password.

ethbro5• 7 years ago

My intuition from experience is that in sufficiently complex systems (basically all modern hardware + software systems), things break from humans misunderstanding or not knowing the rules far more often than from physical failure.

If there's a reason for using an electronic system, by all means.

But for God's sake air-gap it, compartmentalize functionality with clearly defined interfaces, build simple fault-tolerant monitoring, and make final controls hand-on-metal.

When multi-million dollar warships are crashed because of poor UX, I have zero faith we're safely managing complexity.

qserasera3• 7 years ago

When multi-million dollar warships are crashed because of poor UX, I have zero faith we're safely managing complexity.

My understanding of the incident is that the auto-detection functions failed and when human functions were not done per SOP due to exhaustion and senior command elsewhere the vessel came into jeopardy. Then when inexperienced and untrained crew members tried too late to evade. An apparently too complex UX was sending ship controls to more than one console. This culminated into one of the multi-million dollar warships crashing.

russdill• 7 years ago

More than one large power outage has been caused by a human operater manually grounding live transmission lines by mistake.

hinkley2• 7 years ago

Nearly this exact scenario has been bugging me for a couple months. I started thinking about it due to deployment issues.

Why don't we have a proverbial two key system for certain activities like deploying to production or deleting user accounts, or backups?

We make do with things like pull requests, but everywhere we've ever had retros, I've run into situations where someone did something very stupid, somebody else rubber stamped it, and now we have a mess. Approvals on an action do not approximate the ritual sobriety that is embodied in two independent humans deciding to turn a key.

There are some people for whom these two actions can be considered equivalent, but we tend to bag on them for their meticulousness.

I think I'm wanting a tool where I type "deploy 1.0.12345 to production" and it sits there waiting until someone else types "deploy 1.0.12354 to production", and then it stops us because one of us transposed some digits.

eximius• 7 years ago

Someone did release a Pam module? That required confirmation before someone sudo-ed. I thought it was Stripe but I can't find it right now.

stouset2• 7 years ago

It's not necessarily for deploys (though you certainly could make a workflow that uses it to do so!) but I developed a dual-control plugin for sudo while at Square. We use it extensively.

https://github.com/square/sudo_pair

In action: https://raw.githubusercontent.com/square/sudo_pair/master/de...

buro9• 7 years ago

https://blog.cloudflare.com/red-october-cloudflares-open-sou...

harshaw• 7 years ago

we have some of these systems at AWS where you need two people to approve a commit, or a type of change operation. Basically, introduce friction at various points in the name of safety.

rudolph9• 7 years ago

It probably will yield good results for the next decade but will set a bad precedent for the world in general and back us into a corner as confidence in the security for this stuff improves and the next decades passes.

swagasaurus-rex• 7 years ago

Software does not appear to be on a trajectory of increased security or reliabilty. It has always been hard to verify correctness, let alone measure how vulnerable and easy to exploit software is. Testing software has not been a suitable replacement to a large technical user base constantly tweaking and improving said software.

If software were to become reliable, I'm sure those pioneers in reliable software can pave the way to reasonable use in mission critical, high value espionage targets.

Until then, legislation like this will pave the way back to what has worked for centuries.

gremlinsinc• 7 years ago

Why not create a closed government only internet or intranet where all grids connect to each other but none have access to the main internet. No computer in the network can connect to the internet and the govnet, maybe even have a different protocol and other unique features. They could also use ai to determine if a command to say shut something important off makes sense if not notify someone in charge before following through on suspect actions.

IAmEveryone2• 7 years ago

The “smart grid” we are working towards does things like scheduling your AC and dishwasher to turn on/off depending on supply and demand fluctuations. To that end, every node, both consumers and energy producers, needs to be connected. To do this completely independent from the existing internet would amount to doubling the cost of our communication infrastructure. It would also defeat your purpose, because suddenly everyone on the grid has access to the network again, and could try to attack it.

You also need some connection, somewhere, to the “regular” internet. As just one example, consumption data is the basis for billing, which needs to connect to financial infrastructure.

And, finally, even air-gapped systems are vulnerable. See Stuxnet.

Nasrudith3• 7 years ago

Logistics and attendant expenses most likely - especially with practice to gain any actual benefits.

Say everyone has a registered nonrepudiatable cryotographic certificate - a decent idea in theory. However apply it to every accessible grid communication part including ones which may cycle in and out and could have secrets extracted. Suddenly it doesn't provide as much benefit if a hacked SmartMeter can get in anyway and there is deniability. Better than nothing but you could have likely gotten better bang for your buck with strict firewall rules or hiring a swarm of consultants.

Similarly vetting commands has two problems - one is that reaction time may be critical. Second the judgements themselves are complexity and may be a source for errors - especially when they are "triage" ones that cause a shutdown or damage to avoid greater damage

throw0101a4• 7 years ago

Why not create a closed government only internet or intranet where all grids connect to each other but none have access to the main internet.

An air gap didn't help Iran against Stuxnet, or the US DoD against agent.btz:

* https://en.wikipedia.org/wiki/2008_cyberattack_on_United_Sta...

It wouldn't hurt, and it's probably worth doing anyway as a risk mitigation, but keeping it completely sanitary would be impossible.

userbinator• 7 years ago

Good to see some more "common-sense" in contrast to the "Internet everything" trend that's taken hold in some areas of the industry. Some of the control systems in the national grid have been in use for over a century, and are likely to continue working indefinitely if kept maintained.

avip2• 7 years ago

We need smart-gridness to reduce our dependency on idle fossil-burning power plants as a safety net to balance the now much less predictable grid. Strictly ecologically speaking, this seems problematic.

probablypower2• 7 years ago

This is an important point.

Increasing renewable penetration into a power system leads to:

-> reduced inertia

-> increasing Rate of Change of Frequency (RoCoF)

-> reduced time to react

If you rely entirely on manual systems to react to frequency changes on a grid with high renewable energy penetration (e.g. the Irish grid), you'd really struggle to avoid daily blackouts without decommissioning a bunch of wind farms.

It is by using such automated control schemes that countries like Ireland are able to push the limit on wind power penetration.

If I was being truly cynical, I would think that a push towards mandating manual control (rather than mandating that it is available as a fallback) is actually a way to limit growth of the renewable energy sector.

ethbro5• 7 years ago

Or, nuclear.

wongarsu3• 7 years ago

While a smart grid is incredibly useful to solve this, is it strictly necessary?

For example you could build an array of flywheels that charges when the frequency is above 50Hz and discharges when the frequency is below 50Hz.

No network connection necessary, and if you want to you can build it entirely analog. If all you really want is inertia and reaction time it could even be as simple as dumb three phase motors with weights.

msisk6• 7 years ago

This.

As someone responsible for the IT systems reliability of a grid operator I have to wonder if this really is a backhanded attempt to limit to the growth of renewables.

We run load frequency control on the grid every 4 seconds to keep the grid in balance. Hard to see how that'd work with manual controls on today's complex grid.

scythe• 7 years ago

I agree that the grid needs to handle variable sources and storage units, but how much of the problem can be handled by storage units? Molten salt storage looks better every day and it can be deployed anywhere with simple equipment.

https://www.energy.gov/articles/department-energy-announces-...

Physical grid improvements may be as important as computational ones. More available and cheaper electricity is crucial to reducing the use of gas-powered building heat systems, which represent a large chunk of CO2 emissions:

https://www.eesi.org/files/climate.pdf

jmartrican• 7 years ago

US Gov taking steps to prevent the Butlerian Jihad before it even begins.

Havoc• 7 years ago

Good thinking security wise but could be a serious setback for efficiency

beenBoutIT4• 7 years ago

They could still collect data locally and use ML to optimize it.

jacquesm4• 7 years ago

Efficiency does not trump security for such installations.

Nasrudith3• 7 years ago

It may be an unpopular opinion but I disagree - security is a cognitively biased function owing to historical roots but it actually seldom comes up which fails to justify its costs and there is little incentive to gain from attacking it.

Even if a bad actor did bring down the grid all it would do to concrete capabilities is GDP damage that they could easily get credit for and still leave a pissed off superpower able to make an example. Even actual rival status like say China were to do it would be shooting themselves in the foot such that something stupid and self destructive that if the PRC decided to simply start bombing Beijing it would do less damage.

Besides for decades a few tens or hundreds of people could go out, buy semi automatic rifles and start casually shooting substations and walking or driving away until someone arrests them. That would cripple power infastructure far worse than any hacker could as it would take longer to replace and be a distribuited issue to fix.

Given actual frequency even with lax security and pessimistic assumptions efficiency is a likely winner. Logistics and not superweapons win wars.

Avamander• 7 years ago

Instead of fixing the laws and forcing proper auditing there should rather be regression in scalability, response and monitoring?

droithomme2• 7 years ago

Very good news.

Mandate paper ballots next please.

Whatarethese3• 7 years ago

Punch cards. Scantrons. Stone Age shit for elections.

beenBoutIT4• 7 years ago

Anything that the public depends on for survival could be a great candidate for this type of restriction. Think of the hundreds of lives that would have been saved had this type of law been in place for airlines and their aircraft. A guy aware of the risks running himself into a tree in a Tesla is one thing, an unaware set of pilots with hundreds of passengers on a public aircraft is another.

incompatible• 7 years ago

Some accidents caused by automation failures would have been avoided, but how many others would have been caused by human failures?

beenBoutIT4• 7 years ago

If the automation is actively saving lives by correcting human failures then they need to log those events silently and forward them to whoever manages the pilots.

IAmEveryone2• 7 years ago

Since 1970, air travel has increased by a factor of 7, while fatalities have decreased by >80%. Combine these figures and you get a safety improvement by a factor of about 70.

We are clearly doing something right, and automated systems replacing error-prone humans may well be a part of it. I don’t quite get HN’s infatuation with the mythical perfect pilot.

tlear• 7 years ago

This is very good news. Maybe few years late but better now. Current situation is a disaster waiting to happen

Stuxney penetrated a heavily guarded, monitored, secured with air gap facility. US power grid systems are child’s play in comparison

tomohawk2• 7 years ago

This is a huge indictment of the IT industry. We need to kick the shiny rock mentality, and actually engineer systems that are safe and secure.

Just look at how some companies interview and select people. They do so on the basis of cleverness, not carefulness and attention to detail.

booleandilemma• 7 years ago

The attitude in the industry towards developers is pretty much:

You know Java? Cool. My kid is learning that in high school.

boogiewoogie• 7 years ago

You set the bar high. Some are more like, "Talk to this guy, he's a computer geek."

jacquesm4• 7 years ago

Just look at how some companies interview and select people. They do so on the basis of cleverness, not carefulness and attention to detail.

That's a direct reflection of how we develop software. New and shiny trumps reliable and boring any day of the week. That, coupled with an industry that does not even recognize the concepts of liability and defective product is a surefire recipe for a very large disaster at some point in the future. And after that it will get a lot easier to get budget for security, testing and all those other things that companies see as unnecessary cost.

CobrastanJorji• 7 years ago

Even if software were perfect, this makes sense as a defense-oriented decision. It's harder to hack a physical switch. Air gaps can be bridged.

There are also plenty of reasons to automate things. It allows for for faster reactions to problems and interesting new ways to reroute power. Both approaches have pros and cons, and there are good reasons for backing either approach.

Russia attacked Ukraine in a new way, and we responded by trying to become more well-defended against that new attack. Much as I agree that coding interviews are problematic, I fail to see how your point follows.

throwawayjava• 7 years ago

It is possible to write safe and secure software. It's expensive, but it's possible.

Digitizing the grid has enormous upside, to the tune of billions in savings and improved resiliency/response to weather related outages. It we could do it safely and securely it'd be a no-brainer.

We're just trading a devil we know for a (preventable) devil we don't.

BTW: digitized grids aren't even necessarily more vulnerable. In a complex system, the increased latency and miscommunication opportunities introduced by human operators are also a potential attack vector...

tomohawk2• 7 years ago

The values that we hold in the industry are often the very things that lead to problems. We love generic things that can flexibly serve multiple purposes. And then we end up with a data port (USB) that is also used as a power source. What could possibly go wrong if I plug my computer to get power, but that same connection can be used to attack my computer? We love the cloud, but what could possibly go wrong if I hook my front door lock up to it? If the IT industry built cars, rolling down the window would occasionally cause the undocumented ejection seat to kill the passenger.

aswanson• 7 years ago

Even old-school industries aren't immune. I had an embedded hiring manager ask me the burning stick, 45 minute timing question along with how many interrupts there were on a processor i used five years ago. A lot of engineers are doltish in general when assessing others.

jlarocco• 7 years ago

On the other hand, maybe the problem with our industry is that we try to solve every problem with computers and software when simpler and more secure physical solutions can do a better job in some cases.

exabrial• 7 years ago

I will probably quote this comment.

It does take both kinds of kinds... creative thinking for hard problems, but attention to detail towards the path of correctness. I completely agree the "cleverness" is massively over-emphasized during the interviews for most companies

rossdavidh• 7 years ago

As a person with two electrical engineering degrees and 15 years' experience as a professional programmer, I am nearly speechless at such a rational, foresighted, and prudent action coming from our government, in regards technology. It's like some bizarro-world where we have leaders who use the power of government to encourage industry to be careful and secure. I will wake up soon, and realize this was all a dream.

ip26• 7 years ago

I suppose a captured regulatory body is right twice a day

craftinator• 7 years ago

This put a genuine smile on my face =p

threezero• 7 years ago

For some reason this reminded me about the still-unsolved San Jose substation physical attack. The shooters did a lot of damage but didn’t interrupt much power.

https://en.wikipedia.org/wiki/Metcalf_sniper_attack

tantalor• 7 years ago

The opposite of "digital" is "analog", not "manual".

The old phone networks were analog and vulnerable to cereal box whistles.

wtdata• 7 years ago

Analog can still be automated. The right word here is indeed manual since it needs human intervention.

2_listerine_pls• 7 years ago

Electronics also need to be protected against EM weapons.

buboard2• 7 years ago

Just hope this doesn't snowball

This is quite an indictment of the security community.

dreamcompiler• 7 years ago

I have mixed feelings about this. In some ways it's a good idea but the demonization of "digital" is misguided. Some of us know how to build secure digital hardware and software but there are no incentives for companies to insist upon such designs, and there are many active disincentives for doing so. The incentive structure needs to be changed to fix this, and then the technology will follow. Demonizing a technology is short-sighted and dangerous.

Whatarethese3• 7 years ago

This seems like a terrible idea. Computer controlled systems lead to optimization and smarter decisions.

rubidium• 7 years ago

Not always in my experience. It’s balanced with “Out of sight out of mind”. There’s a benefit to making certain critical decisions manual, as it forces some review and awareness.

Plus the current track records of internet security for physical control systems is terrible.

liveoneggs• 7 years ago

should have used erlang?

tootie• 7 years ago

"bill that will study ways to replace automated systems with low-tech redundancies to protect the country's electric grid from hackers."

The headline seems to be inferring more than the content is saying. The intent is not to rollback digitized systems, it's to decrease dependency on vulnerable systems. And the bill is only to commission a study on what systems could benefit from manual fallback, nothing is slated to be implemented yet.

cududa• 7 years ago

Thank you for the only substantive comment here!

CharlesMerriam2• 7 years ago

This is an important step in safe guarding our collective experience of power plants.

Consider that the way we currently interact with power plants: "Someone needs to stay behind and manually make the boilers explode! I'll do it! Tell Laura I love her." pales in drama to "OK, let's see, Power / Input / Boiler / Max, confirm, confirm, confirm. OK, let's run!"

vinay_ys• 7 years ago

It is quite alarming if we need to move away from digital connected tech for electrical grid monitoring and management. What about banks? What about hospitals? storage facilities? record keeping? These are the core underpinnings of a functioning society. We are already using digital-only systems for these. If these can be disrupted or corrupted, then society will come falling apart quickly. If we truly cannot secure a digital system for electrical grid and hence we should move away from digital, then we should be doing the same for these other critical systems as well. Is this even practical now? Shouldn't we be going down the path of inventing chips, systems, algorithms, protocols and proof methods to secure these systems?

asperous• 7 years ago

There's probably an ideal system down the route of more high tech, but in the real world they would never be implemented perfectly. As complexity increases so does room for security issues.

I don't see the problem with striking a balance with technology and putting in low-tech solutions for security reasons. So yeah I think many of those systems should implement similar measures.

nwrk• 7 years ago

Isn't this great success of 'Russia/China' so-called hacking ?

Corrado• 7 years ago

This reminds me of the 2004 Battlestar Galactica reboot. All of the ships in the fleet are modern and get hacked and taken over by an outside force. The "Galactica" is being decommissioned because it's old and it's systems are linked in with the rest of the fleet. Hence it is able to withstand the attack and save humanity.

spacemanmatt• 7 years ago

We are capable of protecting our assets. We just have an administration that prefers to sacrifice efficiency to lock in a dying energy industry.

Senate passes bill to decrease grid digitization, move toward manual control

Related Stories