Hackers breached Colonial Pipeline using compromised password

https://www.bloomberg.com/news/articles/2021-06-04/hackers-breached-colonial-pipeline-using-compromised-password

by 1cvmask • 5 years ago

240 138 5 years ago

nabla9• 5 years ago

In this and age, if companies are not giving employees FIDO2 security keys it's just incompetence.

arkadiyt• 5 years ago

The account didn't have any kind of 2fa - even SMS would have stopped this breach. Obviously hardware keys are the gold standard but I wouldn't trivialize the amount of work it takes to roll that out to an organization.

Godel_unicode• 5 years ago

I would, it's pretty trivial (I have done it for multiple organizations of different sizes/technical competencies). Especially for something like a VPN; at this point if you use a VPN solution which doesn't support MFA it almost certainly has other security issues as well.

ineedasername3• 5 years ago

I have a family member who had a very junior role at a big finance company 15 years ago and they used hardware keys. Seems like national infrastructure should require similar levels of security.

qbasic_forever• 5 years ago

For any sysadmins or future sysadmins reading this though, do NOT put your service's 2FA on SMS. Do not use SMS for anything other than sending your grandparents a happy birthday message. It is trivially broken with a sim card hijack and we have seen many high profile incidents of people being hacked precisely through this vector.

kevin_thibedeau• 5 years ago

These days cheapskate companies will insist on a validator app on employee provided phone rather than take the hit on capital expenditure for a proper key. It would be nice to have legislation that bans that practice.

contravariant• 5 years ago

For most things regular any kind of MFA is good enough. Even just a password can be acceptable if it's not too important.

For stuff that ends up in international newspapers you definitely want more than that though.

altdataseller• 5 years ago

Doesn’t Okta solve this problem?

1cvmask• 5 years ago

They failed to implement multi-factor authentication on their VPNs. This is one of the most common attack vectors. VPNs by themselves are not enough. Over two thirds of corporate attacks with VPNs have been through compromised static credentials.

There are numerous ways of integrating VPNs with MFA ranging from a RADIUS integration to a SAML one. At Saas Pass the admin can choose and implement up to 15 different methods of MFA to VPNs ranging from FIDO keys to push approval to scanning encrypted barcodes to SMS (not recommended) etc...

Disclaimer: Worked on the design of some of the mfa interfaces for the SAML based VPNs and the access control policies.

kevin_b_er• 5 years ago

And disused account, which probably means an employee who left.

gabereiser3• 5 years ago

This. Have a solid offboarding checklist that includes VPN credentials or any access to systems that employee might have. At this point I don’t have any sympathy for the hacked when it’s as simple as this.

samstave• 5 years ago

I'd like to subscribe to your newsletter.

Seriously though - in your comment, you described the landscape of knowledge whereby it is EASY for any given corp to mis-step without a 'you' on staff or contract.

Now, as the NSA has released in the past best practices around security and such (like how to lockdown a linux box for example)

Wouldn;t it be prudent for the .gov to be able to present a salient security posture recommendation in such a format as you present above?

Meaning - WHY THE FUCK DONT WE HAVE A SECURITY.GOV WEBSITE THAT STATES EXACTLY HOW TO SECURE YOUR [THING] and CLICK HERE FOR EXPERTISE ON HOW TO SO???

We are in the Digital Nuke age. We should act as such.

You get the idea...

Strs2FillMyDrms• 5 years ago

While my dentist was performing a procedure on my teeth he received a call from a former student now teacher at the university.

The issue? A group of old professors chose to stop teaching a particular procedure.

Reason? Market prices of that procedure had began to be driven down by competing professionals.

She sounded so disappointed, as she should.

Never underestimate the power of greed against the well being of everyone else.

Spooky23• 5 years ago

We do. The entire security industry is built around the NIST framework.

If you want more explicit guidance, IRS Pub 1075, DoD STIGs and other guides will literally tell you exactly what to do.

Egregious failure is always driven by lack of funding and/or motivation. Other failures are usually a result of a lack of defense in depth or compensating controls that missed something.

eecc2• 5 years ago

I guess as long as IT engineers are kept outside of the “adult room” and treated like gifted children coaxed with shiny toys.

And besides, I only recently discovered - after proudly presenting myself as having a Masters in Engineering - that the title Engineer is considered poorly, a synonym to machine operator.

hsbauauvhabzb2• 5 years ago

MFA doesn’t solve targeted phishing and that’s something people keep forgetting. Sprays are just the best ROI, but if everything had MFA it would quickly create a phishing as a service business model.

gruez2• 5 years ago

MFA doesn’t solve targeted phishing and that’s something people keep forgetting.

any VPNs support u2f?

foxpurple• 5 years ago

If you use the physical tokens it does solve that because the tokens won’t authenticate on the scammer site.

jl2718• 5 years ago

Personally I think RADIUS is a terrible idea. If that thing is compromised, or even the network or name service to connect to it, the attacker gets everything. Cert-based auth gives every client the ability to independently authorize every login without any ability to learn or compromise credentials. And there are lots of cert-based MFAs that require nothing more than a few lines in an Apache config, or even handled through the operating system to set up a vpn handshake.

One of my biggest contributions at one place I worked was to dismantle their radius auth and replace it with token cert auth. It’s hard to write about that on a resume, because people tend to be more impressed with complexity than simplicity.

er4hn2• 5 years ago

"Upgraded legacy, obsolete, AAA service with modern and standards compliant solution"

And if it is ever asked about, boy do you have some great stories about upgrading from a system whose protocol name referred to dial in modems and used MD5 to obfuscate passwords.

g_p3• 5 years ago

One issue in the critical infrastructure space is that many of these MFA solutions are less robust and resilient than no MFA. Anything based on calling out to an external service (Duo, for example) is likely to be less resilient [1] than the static, no-external-dependencies system they use for plain password Auth.

Clearly moving towards complex systems like RADIUS and SAML introduce even more moving parts that sit with interconnects between trusted and untrusted, internal and external components. That becomes a point of failure in the eyes of infrastructure companies.

The challenge here seems to be getting the IT world to understand the focus on availability in critical infrastructure - historically, the focus has been on availability first, integrity second, confidentiality third.

You could argue this shouldn't apply to VPN interfaces in to manage systems, but given the external dependencies of these systems, they feel far less likely to remain available than the kind of solutions the sector expects.

On a separate note though, critical infrastructure need to re-think their budgets to pay for their increased availability needs, given most of the commercial solutions are third party hosted "as a service" offerings.

[1] https://status.duo.com/history?page=2

goodoldneon2• 5 years ago

Unavailability of Duo’s MFA product is extremely rare. But when it does happen, IT admins can switch to “bypass” mode to only require the 1st factor. So even though Duo doesn’t have 100% uptime, you’re no worse than no MFA

antattack• 5 years ago

They also failed to deactivate account that was no longer in use and/or have a policy for disabling stale accounts.

karmicthreat• 5 years ago

It gets to be difficult to constantly come back to management for money at a small company. The MFA/Identity services want 3-12$/user. EDR wants 4-12$/user. Phishing training, same. VPN (fortinet in my case) same.

The 57$/User of MS E5 is starting to look like a steal.

3pt141593• 5 years ago

Yes, MFA would have helped here, but I think it's besides the point. I've worked on a lot of projects, including one for a major stock exchange, and I've never seen perfect security. It's too slow. Even if you had MFA for the VPN, it doesn't stop local malware from being able to use the foothold to gain access to the network once the MFA is cleared. I'm not saying that MFA is useless, just that these types of things are very hard to get right and we only hear about the times the attackers got through. "Oh they should have done X" where X is any one of an increasingly long list of possible things they should have done ranging from security headers on their website, to MFA, to auditing their third party libraries, to locking down their development machines and networks (during Covid, ha!) it's never ending. The very best at it get by without total calamity, but the long tail of organizations just don't have the right staff, focus, or resources to do it.

Edit: I'm not saying it's right! I don't think it is right, and I've worked with governments to bring in regulations in certain industries, but to characterize this all on one failure kinda misses the point. There is a broader, harder problem that I'm not sure is even solvable.

genmud2• 5 years ago

Going to have to hard disagree there.

MFA and addressing password reuse is table stakes in 2021. Like this is literally the first item I would do as a CISO in any company.

MFA is among the core security items like running endpoint protection, updating your software or filtering spam. Bucketing it with things like CORS or auditing third party libraries is either disingenuous or extremely naive.

3pt141593• 5 years ago

Look, MFA for access to code repositories or a cloud is one thing, but if you think that MFA is "table stakes" for a VPN I don't know what to tell you other than "then very few orgs are doing table stakes with their VPNs."

These types of arguments are so hard to have over HN, because people with actual experience across all sorts of organizations can't explicitly say what they have seen and where because of NDAs and we all have our own viewpoints and individual experience.

Bucketing MFA into one broad concept is as disingenuous as calling out security headers as being as benign as CORS. Does your org have MFA on SSH? On email? On the code repo? On the Slack login? On the VPN(s)? Etc. Etc.

I have yet to see a single org that had MFA on everything and blaming a lack of MFA on the VPN misses the point. If they'd had it there, the attackers would have hit elsewhere. Most developers install random software and browser toolbars that they'll never have time to check and connect directly to their companies servers, repos, and communication tools. Pretending like MFA on a VPN would have magically solved everything is a pipedream. There are too many ways in, and every time someone gets in the response from the software community is always "oh why didn't they do X" while ignoring the larger truth: There are too many threat vectors and getting them all right, at all times, is nigh on impossible.

technion• 5 years ago

I agree with you in the sense that those are literally the first thing I would do as a CISO anywhere too.

But I've long accepted that this is why you or I will never be CISO of an organisation like this. I've worked alongside such people, and they'd be horrified that thread isn't all about a logon disclaimer on that VPN.

wepple• 5 years ago

It’s not beside the point. MFA substantially raises the effort required of the attacker.

Malware on a workstation with VPN access is a far higher bar than just being able to log directly in to a VPN with stolen/guessed creds.

dmitrygr• 5 years ago

I wonder if a useful counter-measure would be slower internet connectivity it such places. It would be very hard to exfiltrate 100GB at 8kbit speeds, while 8kbit is plenty for simple few-byte commands one would need to manage a pipeline.

goodpoint• 5 years ago

It is, and data diodes are even better.

scoopertrooper• 5 years ago

From what I understand, the hackers never got anywhere near the control systems. The systems they compromised were related to billing and probably the company's sharepoint and emails (hard to think what else the could make up the balance of the 100GB outside of the monitoring of the control systems).

You could still slow down attacks like this by limiting bandwidth (though 8kbits might be painful for document sharing). However, if you limited users a _slightly_ more reasonable 1 megabit per second, they'd be able to complete their exfiltration in within a little over 9 days. Keep in mind, Colonial only learned of the attack after the ransom demand was made!

rapjr9• 5 years ago

The article says the ransomware message appeared on a computer in the control room. That seems strange, do the pipeline operators adjust fuel flow based on how much cash is coming in? Is this a common practice? Or maybe it was not the billing system that was compromised? I guess it is possible billing would be used to predict required flow rate in some way, but that means any hack of the internet connected billing system has a direct route to controlling fuel flow in the pipeline, which seems rather dangerous. Post some fake orders and suddenly there are millions of gallons of fuel wherever you say they should be. Does the pipeline also run in reverse? If not it then that fuel could be stranded.

vmception2• 5 years ago

Password found in other leaks

No multi factor authentication

Wants government to go after people

Government should go after Colonial

ineedasername3• 5 years ago

Sure, penalties for negligent companies might help, but if someone leaves the front door open and gets robbed, you still go after the thieves too. Although the idea that the east coast pipeline infrastructure could be taken down by the rough equivalent of keeping the door open it pretty terrifying in its own right.

alexanderdmitri• 5 years ago

The door was closed and locked, but they a) didn't revoke an old key that got tied to it and b) didn't attempt to verify it was expected person using it when it was tried on the door. Making the distinction because I don't believe they were explicitly inviting people to rob them as much as they didn't take sufficient precautions in a bad neighborhood.

ClumsyPilot2• 5 years ago

"Anyone willing to break a window can break into my home."

You secure your house to your taste at your own risk - you are free to leave the door open.

What we had happen here, is the equivalent of a bank leaving the vault door unlocked and loosing other people's money. In that case you would be able to go after the bank for negligence.

xyst• 5 years ago

O&G was temporarily brought to its knees by a single password compromise. This is sad. All of their money and they can’t even implement basic edge security protocols.

gerdesj2• 5 years ago

If your budget is tight you might consider the following for MF for your VPNs:

pfSense VM at work with a single interface and an OpenVPN server listening with a server cert minted from your AD CA - this is your VPN concentrator. You get your OpenVPN server to hand off auth to RADIUS which is a DC with the NPS service installed. Mint SSL certs for all your domain joined users and computers via your AD CA. Investigate this parameter: cryptoapicert "SUBJ:" on your OpenVPN clients.

All of this stuff is documented in various places and I've dropped most of the clues!

You can actually fold in more factors with this. So the PC/user must have a SSL cert minted by the AD CA and a password, and a OTP via something like Google authenticator.

I'd start with short lived (say one month) AD minted SSL certs and then decide whether you need more factors.

Remember that a domain joined PC doesn't work very well at password change time if encryption and VPN connections rely on AD passwords.

Now, once you allow your folks on site - what do you allow them to access? Get your firewall rules sorted.

nszceta2• 5 years ago

This is hilariously complicated compared to deploying Wireguard VPN

jcrawfordor• 5 years ago

Compared to Wireguard with MFA? as far as I can tell Wireguard doesn't support MFA at all. Pure key auth is not acceptable in these contexts because of stolen keys and devices.

neeleshs• 5 years ago

If you are in gsuite, you can easily turn on 2fa for OpenVPN. OpenVPN + Gsuite LDAP + Authenticator.

gerdesj2• 5 years ago

Wireguard, OpenVPN, IPSEC etc are just ways of shuffling packets of data with encryption.

What I'm describing is actually a really easy or at least really cheap way for an organisation to grab an extra factor when authenticating their VPNs.

There are loads of howtos (most are completely wank but there are some decent ones) on minting AD SSL certs for users and computers. That's something you "have".

You could simply use SSL certs or you could also insist on a username and maybe a password ("something you know").

You use RADIUS because that is a long, long standard way of meditating auth/auth to facility.

zomgwat• 5 years ago

Duo is another option for a nice OpenVPN MFA option.

https://duo.com/docs/openvpn

ineedasername3• 5 years ago

The account was no longer in use at the time of the attack but could still be used to access Colonial’s network, he said.

Very important to de-provision accounts when an employee leaves or some other factor renders it unnecessary.

gz5• 5 years ago

Password is the cause like a gun is the cause of a shooting.

Networks are targets - both because their assets make them attractive to attack, and because a network is inherently vulnerable (since it is a set of connected nodes).

So the cause is the network and the solution is to eliminate the network. Securely identified, properly authenticated, least privileged access, app-specific, ephemeral connections. Can such a connection still be breached? Of course. But it will be more difficult and it will be isolated, micro-segmented by definition, and unable to be leveraged to attack laterally.

That said, networks makes things simple. And complexity is insecure. As soon as we can make the paradigm listed above as simple as networking, we will see a massive shift. We need it.

walrus01• 5 years ago

Password, bad.

Password, allowing access to something with a very large 'blast radius', horrible.

Systematic failure of compartmentalization and roles/access control within the organization's network.

I do not agree that eliminating the network is the solution. Properly securing the network is. I could go into a 30 page essay on what might be considered within the category of 'proper', but suffice to say that organizations of this importance should have a serious infosec/netsec department.

anonymouse008• 5 years ago

I could go into a 30 page essay on what might be considered within the category of 'proper'

I’d read that essay!

ClumsyPilot2• 5 years ago

So we are in the territory where a script kiddy could download year old data leak from some random torrent tracker and use it to log in and shut down vital national infrastructure without even completing a Hacking course on Udemi.

ed25519FUUU3• 5 years ago

Don’t worry Colonial is asking the government to go after these teenagers and that will finally fix the root of the issue.

gscott• 5 years ago

Can't be too hard just look for the only 15 year old with a new G5 jet.

arthur_sav• 5 years ago

The larger the attack surface, the easier it is.

Large corps with thousands of employees and legacy infrastructure is usually not that hard to breach.

While i haven't kept up with cyber security in a while i think the problem is largely the same. Security is not baked in, it's an afterthought at best and because of that there's shortage of talent.

loonster• 5 years ago

Not everything needs to be connected to the internet. They own right of ways for a pipeline. Go bury some fiber along the pipe. All critical company communications can now be handled internally.

ed25519FUUU3• 5 years ago

This is a good idea. Is it possible to require all critical infrastructure to be disconnected from the WAN?

bobthepanda• 5 years ago

It isn’t if you want remote monitoring, probably. So doing that could raise labor costs.

rurban2• 5 years ago

But than you have the famous office manager, who decides to shutdown all ops because his windows PC is virus infected.

All critical infrastructure depends on one overly stupid manager.

olivermarks• 5 years ago

'In the wake of the attack on his company, Blount said he would like the U.S. government to go after hackers who have found safe haven in Russia. “Ultimately the government needs to focus on the actors themselves. As a private company, we don’t have a political capability of shutting down the host countries that have these bad actors in them.”'

says it all, regardless of nation state

vijayr02• 5 years ago

Reagan gave green light to sabotage of Soviet oil pipeline (2004)

https://news.ycombinator.com/item?id=27402062

To paraphrase Clausewitz, economic war is war by other means.

"In order to disrupt the Soviet gas supply, its hard currency earnings from the West, and the internal Russian economy, the pipeline software that was to run the pumps, turbines and valves was programmed to go haywire, after a decent interval, to reset pump speeds and valve settings to produce pressures far beyond those acceptable to pipeline joints and welds," Reed writes.

"The result was the most monumental non-nuclear explosion and fire ever seen from space," he recalls, adding that US satellites picked up the explosion. Reed said that the blast occurred in the summer of 1982. "While there were no physical casualties from the pipeline explosion, there was significant damage to the Soviet economy," he writes. "Its ultimate bankruptcy, not a bloody battle or nuclear exchange, is what brought the Cold War to an end. In time the Soviets came to understand that they had been stealing bogus technology, but now what were they to do? By implication, every cell of the Soviet leviathan might be infected. They had no way of knowing which equipment was sound, which was bogus. All was suspect, which was the intended endgame for the entire operation."

Seems familiar to our predicament today, no?

jcranmer• 5 years ago

I feel like I should point out that this claim comes from an autobiographical book that has never been corroborated by any other source. When the claim involves "the most monumental non-nuclear explosion and fire ever seen from space," the lack of other sources is especially suspicious.

I personally find the claim that the Soviets were using SCADA control systems in 1982 somewhat suspicious as well.

blondie9x• 5 years ago

This should not be this easy. Really?

mac-chaffee3• 5 years ago

Soon after the attack, Colonial embarked on an exhaustive examination of the pipeline, tracking 29,000 miles on the ground and through the air to look for visible damage.

Is that an extra zero? That's more than the Earth's circumference...

mellosouls• 5 years ago

The hack that took down the largest fuel pipeline in the U.S. and led to shortages across the East Coast was the result of a single compromised password, according to a cybersecurity consultant who responded to the attack.

This is kind of like blaming unrecoverable data on the employee that accidentally deleted it - the problems are much bigger than a single compromised password (with the implied single-employee-blaming) and the credibility of reporters or security consultants proposing such causes risk being undermined with these kinds of simplifications.

goodoldneon2• 5 years ago

Depends how you read it. When I see “was the result of a compromised password” I think “they should’ve used something mitigates the risk of compromised passwords”. I don’t think it’s blaming the employee but rather blaming the company’s fragile security model. Everyone should use MFA on as many things as possible

rurban2• 5 years ago

The headline is only to drive home this bullshit narrative:

It was the first time Colonial had shut down the entirety of its gasoline pipeline system in its 57-year history, Blount said. “We had no choice at that point,” he said. “It was absolutely the right thing to do. At that time, we had no idea who was attacking us or what their motives were.”

So they saw a windows ransomware screen on their windows office computers. Even a known ransomware, with a known decrypter they could not find with a simple search on their android/IPhone.

So they decided it's a high probability that the ransomware will jump from the office PCs to the non-Windows internal pipeline control system. Foreign terrorists. Bullshit.

As we later found out, they decided to shutdown the east coast infrastructure only because they could not calculate the exact gallons delivered to each customer via their office PCs. Rounding would have lead to real money loss apparently. FYI In the real world everybody else does rounding and estimation. The sensor data is for verification and control.

spuz• 5 years ago

As we later found out, they decided to shutdown the east coast infrastructure only because they could not calculate the exact gallons delivered to each customer via their office PCs.

What is the source for this? As far as I understood, they shutdown all their computer systems to prevent the malware from spreading to them which in most cases is the correct response.

neonate• 5 years ago

https://archive.is/p2XY1

SergeAx• 5 years ago

The hack that took down the largest fuel pipeline in the U.S.

This needs to be remembered: the pipeline was okay all the time. It is accounting software that was taken down. A conspiracy theorist in me thinks that most of the PR activity around the case was (and is) intending to paint this fact over.

upofadown• 5 years ago

The big issue here is that a single access could so easily be spun up into complete access. In a large company you have to assume that even the highest wall will eventually be breached. A current employee will eventually provide access to someone up to no good. Or that current employee will themselves get up to no good.

Corporate messaging is a good example of this sort of thing. Completely unencrypted inside the wall. Tons of login information just left laying around in old plaintext messages.

Your security should not be all or nothing...

vlan0• 5 years ago

When will people realize we need to stop implementing security in this manner? The use of passwords and perimeter security should be DEAD.

https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.S...

lbriner• 5 years ago

As others have said, MFA would be a decent tool but not all attacks are against VPN so it wouldn't always be so useful. As everyone always teaches, security has to be in-depth and the one largely simple control that is very often missing is segmentation of the system.

Being able to attack a single department is annoying but unlikely to be considered fatal, however, I think that even savvy IT ops people do not really understand vlans, using switches and air-gaps to keep things apart and really concentrating on any single-points of ingress like Sharepoint sites or whatever although most problems seem to spread across the network layer rather than bridging machines over HTTP.

Also, even some simple controls like basic local backup, even to a crappy SAN box in your server room would at least enable you to stick your fingers up at ransomware.

lumost• 5 years ago

At this point I either use saved passwords or need to rotate the passwords on every request via password reset. Saved passwords are also an attack vector and as such my corporate laptop disables all mechanisms for saving passwords.

I think it's time to make pin + 2 factor the standard over passwords.

rdiddly• 5 years ago

Compromised password of an inactive account. Do regular audits and deactivate unused accounts, people! At the very least that takes care of the "disgruntled former employee" scenario. There's a lot of virtual ink in this article devoted to the "how were the credentials compromised" question. I understand the curiosity, but it's mostly unknowable and therefore not too interesting. It's like asking "Why did it rain on the day I didn't bring my umbrella?" Is that the question? "Why did it rain?" The interesting part is the point at which Colonial still had control and gave it up, which was when that employee left or whatever happened, and they didn't revoke those credentials.

I keep having a thought about how this account belonged to a fired employee who was not only bad at passwords, but bad at this job too. I have no idea if that's the case, as that's generally confidential info. Kinda funny to think of though.

Hackers breached Colonial Pipeline using compromised password

Related Stories