It's not just CrowdStrike – the cyber sector is vulnerable

https://www.ft.com/content/d2f529fe-3050-4ef5-a8c7-6487d533e57a

by jmsflknr • 2 years ago

123 127 2 years ago

ft.com

udev4096• 2 years ago

https://archive.ph/sKcp3

cs7023• 2 years ago

For most corporations, security and robustness are -- and for a long time have been -- an afterthought.

Making systems hard to hack and robust to rare events:

* is really hard,

* costs a lot of money, and

* reduces earnings in the short term.

Faced with these inconvenient facts, many executives who want to see stock prices go up prioritize... other things.

hypeatei2• 2 years ago

To them, they are thinking about it though. They installed this ultra secure thing called CrowdStrike that checked a regulatory box for cybersecurity

cs7023• 2 years ago

CrowdStrike is a textbook example of a single point of failure:

https://en.wikipedia.org/wiki/Single_point_of_failure

prasadjoglekar• 2 years ago

To be fair, most corporations signed up for Crowdstrike as a way to address some issues. I'm sure it wasn't cheap and CS was probably better at security than an IT admin at a 50 person shop.

cs7023• 2 years ago

Yeah, it wasn't cheap.

It's still not enough.

Damogran63• 2 years ago

Much like the RSA attack, now we get to see how Crowdstrike handles damage control.

croes2• 2 years ago

But what's worse, hundreds of maybe insecure companies or creating a big single point of failure?

lotsofpulp• 2 years ago

I feel like I would be doing everything in my power to de-Windows my operation.

monero-xmr• 2 years ago

The real financial problem is that cybersecurity is mostly box checking. It's an industry that is open to commoditization, as startups in lower-cost global regions manage to check the box as well as the next-most-expensive region, and cost conscious companies keep migrating. But the power of the box checking is strong.

I do not invest in cybersecurity companies, it is very risky IMO

mrkramer2• 2 years ago

The problem with cybersecurity is that there are hundreds of attack vectors; you can get pwned by supply chain attack or by some random zero-day exploit or by an insider....It is literally impossible to 100% prevent breaching of your computer network.

lucianbr5• 2 years ago

It is impossible to write bug-free/exploit-free code.

But some companies are using this as an excuse to not care about the chances of an exploit at all, and just write code in a cheaper way.

We need a middle ground, where there is at least a reasonable effort towards security.

Ekaros• 2 years ago

Also very often software quality is absolute trash... With so many issues developers spend no time on thinking about most basic things... Like applying access control on reading/editing data or what field should a request update and what not...

And these parts are the simple ones. Not even talking about operating systems, networking and so on... If even easy stuff is wrong, what hope is there for complex...

Nemo_bis3• 2 years ago

The problem with cybersecurity is that it's impossible to prove a negative. So it's easy to sell products which produce tangible downsides in return for hypothetical upsides.

sebazzz4• 2 years ago

That's because the reviewing the checklists can then be - no offence intended - offloaded to cheap workers in 2nd and 3rd world countries who are judged by the checklists they sign off. There is no room for critical thinking or adapting to the particular situation. I see this happening daily in a large company where the enforcement of infosec is offshored to east Asia.

mrkramer2• 2 years ago

Don't do automatic updates....roll updates manually. That would be a nice thing for the beginning.

ygjb2• 2 years ago

It's easy to make comments like this against automatic updates, but then you get popped because something that would have been automatically updated misses a patch because it was too critical to risk automatic updates.

In practice, failing closed (or crashing) is probably fine for most businesses, and lower cost than a breach, but the correct solution is automated testing across a broad spectrum of devices, staged and rolling updates to prevent entire fleets going down at once, and ensuring that there is an effective, tested rollback mechanism.

But that shit's expensive, so shrug :/

dataflow2• 2 years ago

but then you get popped because something that would have been automatically updated misses a patch because it was too critical to risk automatic updates.

This kind of logic only works if you ignore any kind of possible nuances in the problem and just insist on throwing the baby out with the bathwater. Just because someone let you do automatic updates (or let's be real, you probably didn't give them much of an option) that doesn't mean you should use it for everything.

Automatic update of data (like virus definitions) != automatic update of code (like kernel driver)

And really, the only time you could justify doing automatic updates on other people's machines is when have reason to believe the risk of waiting for the user to get around to it is larger than the damage you might do in the process... which doesn't seem to have been the case here.

winternett• 2 years ago

Testing in software dev is taken completely for granted by many companies on mission-critical updates... Back in the day, our deployments would get tested on configs new & old and with several different variables, we always made sure deployments went smoothly. Now it seems as if most of these companies hire junior devs and skip testing to cut cost and then just put the blame of failure all on them. Burnout levels are high in these settings.

This whole incident would have not happened if just a basic deployment test was conducted. It's so widespread, it would have been impossible to miss detecting the issue.

rs999gti• 2 years ago

Can auto updates be turned off on the crowdstrike falcon client?

Johnny555• 2 years ago

In any sizable organization, you can't get around automatic updates.

But updates should be rolled out slowly and you need enough telemetry to detect problems as it's rolled out. Reboots, crashes, cpu/memory use, end user reports, etc should all be used to detect issues and pause the rollout.

HdS84• 2 years ago

Then you end up like some of our customers with log4j. We are consultants and notice that a cave for log4j comes out. We inform our customers that we have detected an issue under active exploit, and we performed an update to non vulnerable versions and want to deploy. Customer waffles for days and gets exploited before he decides to upgrade. Threats are often only minutes away. We are currently away to slow and manual updates are slowing you down even more.

galdosdi• 2 years ago

So do automated canarying! Deploy to 1000 machines, wait 60 seconds, deploy to 10,000, etc. All done in under 5 minutes while automatically rolling back or at least halting if any metrics look bad.

Canarying is by now not a very new practice. This is like doctors not washing their hands.

lukev• 2 years ago

It's rapidly getting to the point where the cure is worse than the disease, when it comes to this kind of product.

croes2• 2 years ago

It isn't a cure in the first place it just fights the symptoms.

type03• 2 years ago

it's a kind of palliative care if you extend the analogy

Damogran63• 2 years ago

It's almost as if we're seeing the downsides to our cloud based decisions. Uncontrolled costs, lack of visibility, placing control of critical processes in the hands of other groups...that also have control of critical processes globally.

Am I bitter at losing the business decisions that push ease of management by sending control to service providers? Not really. It's been dozens of times, and I lose every time.

I can raise the concerns to make sure the decisions are educated ones, and then let the decisions be made.

altdataseller3• 2 years ago

If you managed your own servers, you would still need some sort of endpoint security solution too tho…

Damogran63• 2 years ago

Not saying it outages wouldn't happen, saying it might not happen on a _global_ scale....and it's not the only downside. There are pros and cons to each solution.

Buttons8402• 2 years ago

Cyber Security is a matter of national security, but currently we sacrifice our national security for the convenience of companies.

The disconnect is that companies are both (1) the only entity in control of their system and how it is tested and (2) not liable if a security breach does happen.

I believe we need to enable red teams (security researchers) to test the security of any system, with or without permission, so long as they report responsibly and avoid obviously destructive behavior such as sustained DDoS attacks.

A branch of the government, possibly of the military (the Space Force?) could constantly be trying to hack the most important systems in our nation (individuals and private companies too). The bad guys are doing this anyway, but hopefully the good guys could find the security holes first and report them responsibly.

Again, currently this doesn't happen because it would be embarrassing and inconvenient for powerful companies. We threaten researchers who do nothing more than press F12 (view HTML source) with jail time and then have our best surprised Pikachu faces ready for when half the nations data is stolen every week or major systems go down. Actually, we don't make faces at all, half the nation's data is stolen every week--no, actually we don't even take notice, we just accept it as the way things have to be. Because, after all, we can't expect companies to be liable, but we can trust companies to have exclusive control over the testing of their security. How convenient for them.

frenchy• 2 years ago

Isn't this what the NSA is for? Also, I think we have plenty of reason to believe they regularly try to penetrate powerful companies, they just don't necessarily tell us when they do.

Buttons8402• 2 years ago

I've never heard anything about the NSA telling a company they have a security vulnerability. Have you?

zavec• 2 years ago

They absolutely have bugs up their sleeve, but if they tell the companies to allow them to fix them then they can't use the bugs for spying (or at least, not as effectively)

silverquiet3• 2 years ago

CISA offers services to public and private providers of infrastructure deemed critical that include pen testing, but they don't have the resources to offer it to all who want it.

freitzkriesler22• 2 years ago

Cyber was a 90s buzz word that died out and became vogue when cyber security became cool. I cringe every time I hear it drop.

jabroni_salad3• 2 years ago

government and military loves that word and will probably never let it go.

whycome3• 2 years ago

Air Force. Space Force. ...Cyber Force? Inevitable.

freitzkriesler22• 2 years ago

I roasted some Booz Allen booth people when they asked me about cyber with a," the 90s called and wants its buzz word back."

The look they gave was priceless.

bluedino2• 2 years ago

I still think of how 'cyber' was used in AOL Chatrooms back in the 90's...

whycome3• 2 years ago

I forgot about that until this mention. Definitely not relegated to AOL chats.

jijijijij• 2 years ago

"Cyber sector" is reeeeally pushing it. Full body cringe.

PreInternet01• 2 years ago

The "cyber sector" is... awful? Nah... irresponsible? Nah... immature? Yeah, probably!

Right now, pretty much everyone is looking to outsource their "security" to a single vendor, disregarding the fact that security is not a product, but a process.

That... won't change! And incumbents will get less-awful about their impact on "protected" systems.

And yet, there's an opportunity here! Do you truly understand Windows? And whatever happens on that platform? And how to monitor that activity for adverse actions? Without taking down your customers on a regular/observable basis?

Step right up! There are a lot of incumbents facing imminent replacement...

silverquiet3• 2 years ago

There can't be a person alive who "truly understands" Windows; though made by humans (allegedly), any modern OS is going to be beyond the understanding of any individual. This is the fundamental problem of managing modern systems.

hypeatei2• 2 years ago

Yes, we're in a complexity crisis.

nick__m• 2 years ago

I can name two: Raymond Chen and Mark Russinovich ! I don't know if Mark is still up-to-date into the latest Windows internals now that he is the CTO of Azure but Mr Chen sure is.

bb883• 2 years ago

immature? Yeah, probably!

I find it funny that adult security researchers still get away with identifying themselves with hacking monikers in public as if they were teenagers probing the local telco back in the 1980's.

hcfman• 2 years ago

Probably get a lot more of this when the full force of the cyber resilience act kicks in.

ramesh31• 2 years ago

I remember being proud of the fact that I had an intimate knowledge and understanding of every single process running on my dev machine. Things felt sane. I could fully comprehend what was happening on my system at all times. Then the button pusher configurator class got called a new name, "DevOps" and started pushing all this crap on us. I'm ready to just start doing work on a private machine at this point.

blibble2• 2 years ago

they're correct, all the others are similarly shit

sentinelone, tanium, guardicore, defender endpoint, delina

all running as root (or worse), sucking up absurd amounts of resources, often more than the software running on the machine (but advertised as "LOW IMPACT")

they also cause reliable software to break due to bugs in e.g. their EBPF

also often serialises all network and disk on the machine through to one single thread (so much for multi-queue NVMe/NICs)

the risk and compliance attitude that results in this corporate mandated malware being required needs to go

this software creates more risk than it prevents

altdataseller3• 2 years ago

So whats the alternative? Have no endpoint protection? Have nothing in place to warn you when malware ends up in your system?

(Just playing devils advocate. I hate Crowdstrike as much as anyone here :)

iwwr• 2 years ago

Or maybe switch to an operating system that isn't a security dumpster fire?

pantalaimon• 2 years ago

Does it actually work?

oldpersonintx• 2 years ago

low permission systems

allow nothing and then gradually allow some activities that are deemed safe

do not allow software to be installed from arbitrary locations

app sandboxing and third-party vendors cannot break their sandbox

basically, iOS, Android, ChromeOS

50% of the people impacted today probably only need a browser

pkphilip• 2 years ago

One option may be to use locked read-only systems. Many of these computers at airports etc do not need a writeable local filesystem.

imtringued3• 2 years ago

The obvious alternative is to build secure systems instead of making them insecure first and then trying to fix the inevitable problems post hoc.

nimbius• 2 years ago

yes but, did it help us meet the compliance targets for this year?

keep'er running...

nightshift1• 2 years ago

also often serialises all network and disk on the machine through to one single thread

Do you have more info about this ? I am very interested. Is it impacting SAN fc storage ?

Nemo_bis3• 2 years ago

Any experience with Wazuh?

UweSchmidt2• 2 years ago

So how do you actually cybersecure a company in a compliant and practical way?

rawgabbit• 2 years ago

Today, we comply by ticking all the boxes in a checklist; it takes care of the most obvious hacks. Is it good enough? Today, we got our answer.

Practically speaking, that is all the end-user can do with Windows machines. My point is Windows is fundamentally unsecure. It is a dike with thousands of holes some of which are not even visible to Microsoft themselves. The reason of that is security has been after-thought. It is band aids/plasters put on top of other plasters.

UweSchmidt2• 2 years ago

Because security experts seem to have this slight dismissive attitude about companies' and individuals' attempts to do security, while not usually having answers or providing secure systems.

jabroni_salad3• 2 years ago

I firmly believe that most routine security issues are really just operations issues and vulns are just bugs and security largely doesnt need to be its own category at all.

I know everybody hates the C-word but if I look at 27001 requirements or the CIS benchmarks, there is nothing in there that I do not want for myself. If you can keep a list of the products and services you are running, have actually put the time into implementing it correctly, and have an ongoing maintenance plan then you are probably in the top 1% of networks.

ehPReth• 2 years ago

I've been trialling application allowlisting, but wow is it ever frustrating. So much stuff isn't signed, and when it is the accompanying DLLs aren't. or the signature is invalid. or some of Windows' own executables/dlls aren't signed (why?? you make applocker??) or the installer is, but none of the actual resultant end files

Is it just me?

Avamander• 2 years ago

It's not just you. Windows software management sucks and people just find excuses for it. WDAC is really difficult to use directly because of it.

OEM software is usually the worst offender here, all these installers and support utilities should be fully abolished. Drivers should stop loading if they aren't coming from Windows Update and haven't passed some quality control.

bluedino2• 2 years ago

Security: You must install Microsoft Defender on all Linux VM's

Devs: Ugh...why?

Security: For safety!

Devs: Fine, we won't argue. Deploy it if you may.

A few moments later...

Devs: All of our VM's are slow as crap! Defender is using 100% of the CPU!

Security: Add another core to your VM's. ticket closed

Management: Why are our developers up 30% on their cloud spend!?

blibble2• 2 years ago

and all microsoft have to do to increase cloud revenue is make defender chew up 5% more CPU every few months

directly incentivised to make shit software

whycome3• 2 years ago

Isn't this kinda the hardware model for Apple devices too? Eg batterygate

01HNNWZ0MV43FF• 2 years ago

To a point. After a while they're wasting hardware and another vendor will undercut them. After some decades people will start leaving

thdxr• 2 years ago

there was a production incident at a customer that was this exact scenario

jordanb2• 2 years ago

We had cylance take out all of our kubernetes clusters a few years ago.

The whole cybersecurity concept of installing third party mystery meat in the kernel controllable over the internet by a different company seems contrary both to good security practices and software quality assurance, immutable production architecture and repeatable builds.

pennomi• 2 years ago

Third party mystery meat is mostly intended for scapegoating if a problem does occur.

pixl97• 2 years ago

I work supporting software that processes millions of small files a day, with a lot of these scripting languages. The speed difference in total iops where AV is installed vs not installed is huge. 30-50% loss is no joke.

hulitu• 2 years ago

The problem is that the AV does not increase security.

bb883• 2 years ago

That's a management problem. IT security didn't communicate that to the finance folks. Microsoft didn't communicate that to IT security. And if they're on Azure, it's more money for Microsoft.

sebazzz4• 2 years ago

Oh finance doesn't care. Security is paramount and not something you can save money on, is the mindset. Besides, the real costs of this is allocated to many individual cost centers so in the bigger picture you won't see it.

Nemo_bis3• 2 years ago

Speaking of which, does anyone use offline Microsoft Defender updates (maybe to have direct control of canary/staggered updates)? https://learn.microsoft.com/en-us/defender-endpoint/linux-su...

lanstin• 2 years ago

Even worse, at least aesthetically, than the AV is the pacct stuff to log 1 line for every single syscall. Talk about sublinear scaling. And for what, no one can say.

Actually, even worse than that was we had to install AV on all the images that the ephemeral map-reduce/Hadoop clusters we spun up, but the way the AV stuff worked, the computes were gone by the time the registration for the new compute had gone thru whole process. And, in AWS accounts where there were maybe 63 IPs and say 600 EC2s/day they used IP as the primary key for the "list of compute in the VPC." So they stitched together totally unrelated stuff as if it was the same continuous compute. I guess it would be eventually fixed, but the bad security data was not a real concern of the devops team that was building out stuff as rapidly as possible, nor were the EC2 made in a sealed off VPC and only lived for a few tens of minutes or hours at best a serious security concern to the actual security people. Just a check list solution hitting a novel environment.

guru4consulting• 2 years ago

For a business that relies on SaaS applications over cloud and uses dumb machines (windows, iPad, whatever) as client terminals, can someone please explain what are the actual threat factors that these EDR tools like Crowdstrike Falcon address? And if SaaS applications can restrict access, detect anomalies with user behavior, have MFA for auth, etc.. will that mitigate these risks? I guess common issues like key loggers, malwares, virus attacks have much simpler solutions than a complex EDR which seems to need root access!! Someone, please educate.

mikewarot• 2 years ago

We build our "cyber fortress" out of the Turing Complete analog of Crates of C4... and wonder why things go wrong all the time.

As I say every time this happens (and it will keep happening for the next decade or so)... Ambient Authority systems can't be secured, we need to switch to Operating Systems designed around Capability Based Security.

We need at least 2 of them, from competing projects.

1vuio0pswjnm72• 2 years ago

Works where archive.ph is blocked:

https://webcache.googleusercontent.com/search?q=cache:https:...

exabrial• 2 years ago

Honestly until we can get rid of the perception that SOC2/Sables-Oxly/HiTrust provides any meaningful security we’re stuck.

1vuio0pswjnm72• 2 years ago

Works where archive.ph is blocked:

https://cc.bingj.com/cache.aspx?d=464600016483&w=i7yXBm7Gwof...

sebazzz4• 2 years ago

Centralisation is really the core of the problem here.

Take ZScaler, which is a service that proxies all network connections of a computer to a central cloud proxy server, mitms it (decrypt, inspect, log, and encrypt), and then forwards it to the target server. Imagine that this is hacked, and this isn't immediately discovered. Hackers listening in and being able to tap off cookies, bearer tokens and other confidential information for weeks. That would affect so many companies. And if they would want to cause a DoS, many computers and servers would be left without an operational internet connection.

sconi• 2 years ago

Yes, Zscaler or any other Zscaler clone (e.g., Netskope, Cato, etc) -- they're all just sitting ducks, and once they are compromised, what happens to all the customers? It doesn't make any sense and shows how much we're willing to give up for convenience.

craxdevil7• 2 years ago

thats true, it is disheartening when security controls are only seen as a checklist to comply with some framework, and not actually implemented. https://medium.com/@confusedcyberwarrior/what-is-soc2-how-to... This gives a false sense of security, which is further bad for cyber space. Crowdstrike incident on other hand shows that how we still have single points of failure on our supposedly secure and safe systems. https://medium.com/@confusedcyberwarrior/when-security-becom...

It's not just CrowdStrike – the cyber sector is vulnerable

Related Stories