CrowdStrike Update: Windows Bluescreen and Boot Loops

https://old.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_error_in_latest_crowdstrike_update/

by BLKNSLVR • 2 years ago

4489 3767 2 years ago

old.reddit.com

My workplace has a number of people reporting Windows blue-screening and going into a boot loop. The IT Department have a number of servers recently gone offline and have said there's a chance that the two issues are related, and potentially due to a Crowd Strike application update.

My laptop blue-screened and rebooted, but is working fine after the reboot.

A local radio station has also said they've got the same issues with their laptops and their phone system is down as a result.

Not seeing anything on news sites yet. Anyone else seeing similar?

Above is all based in Australia.

sammy22552• 2 years ago

Yep happened to us too. Its global. And it just started happening.

VargaLand2• 2 years ago

Same in Adelaide. Reports coming in from Gov agencies, utilities etc.

scriptsmith2• 2 years ago

It's crowdstrike: https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e...

7/18/24 10:20PT - Hello everyone - We have widespread reports of BSODs on windows hosts, occurring on multiple sensor versions. Investigating cause. TA will be published shortly. Pinned thread.
SCOPE: EU-1, US-1, US-2 and US-GOV-1
Edit 10:36PT - TA posted: https://supportportal.crowdstrike.com/s/article/Tech-Alert-W...
Edit 11:27 PM PT:
Workaround Steps:
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.

ineptech• 2 years ago

Can you post a summary? We're affected but I don't have access to that portal.

scriptsmith2• 2 years ago

Someone posted this in the thread, but I also can't log in to verify

Summary
CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.
Details
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
Current Action
Our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket.
Status updates will be posted below as we have more information to share, including when the issue is resolved.
Latest Updates
2024-07-19 05:30 AM UTC | Tech Alert Published.
Support
Find answers and contact Support with our Support Portal

nofinator• 2 years ago

They've bumped this support info to a blog post that's linked from their home page: https://www.crowdstrike.com/blog/statement-on-falcon-content...

It includes PDFs of some relevant support pages that someone printed with their browser 5 hours ago. That's probably the right thing to do in such a situation to get this kind of info publicly available ASAP, but still, oof. Looks like lots of people in the Reddit thread had trouble accessing the support info behind the login screen.

Neil44• 2 years ago

Right after you enter the bit locker recovery key.

You do have your bit locker recovery key, right? .....right?

tamimio11• 2 years ago

I doubt most of the clients who use CS know what BitLocker is, let alone how to back it up, assuming it wasn’t backed up automatically by Windows.

paganel2• 2 years ago

This was particularly interesting (from the reddit thread posted above):

A colleague is dealing with a particularly nasty case. The server storing the BitLocker recovery keys (for thousands of users) is itself BitLocker protected and running CrowdStrike (he says mandates state that all servers must have "encryption at rest").
His team believes that the recovery key for that server is stored somewhere else, and they may be able to get it back up and running, but they can't access any of the documentation to do so, because everything is down.

sebazzz• 2 years ago

BitLocker for Business stores the bitlocker key centrally. Still, it is a huge manual undertaking fixing every system.

mschuster915• 2 years ago

MS Windows Recovery screen (or the OS installer disk) might ask you for the recovery key only, but you can unlock the drive manually with the password as well! I had to do that a week ago after a disk clone gone wrong, so in case someone steps on the same issue (this here is tested with Win 10, but it should be just the same for W11 and Server):

1. Boot the affected machine from the Windows installer disk

2. Use "Repair options"

3. Click through to the option to spawn a shell

4. It will now ask you for unlocking the disk with a recovery key. SKIP THAT.

5. In the shell, type: "manage-bde -unlock C: -Password", enter the password

6. The drive is unlocked, now go and execute whatever recovery you have to do.

Good luck.

fragmede9• 2 years ago

help for if you have bitlocker turned on (because of course you do).

https://x.com/attilabubby/status/1814216589559861673

https://x.com/nathanmcnulty/status/1785094215749476722?s=46

GPO to fix:

https://gist.github.com/whichbuffer/7830c73711589dcf9e7a5217...

downrightmike4• 2 years ago

Saved to my desktop? How does that help? /s

sva_• 2 years ago

Happy weekend to everyone who works there.

2OEH8eoCRo02• 2 years ago

They had me at "crowdstrike engineering"

So engineer-like.

jamesbfb• 2 years ago

Our company is in panic mode. 15 machines blue screened for no apparent reason and stuck in boot loop. I’m a gloating Linux user :)

Also in Australia

jdleel2• 2 years ago

I think there are companies with 10000+ or probably more

dabiged• 2 years ago

Year of the Linux desktop at last!

VargaLand2• 2 years ago

According to Reddit It's hitting Croatia, Philippines, US, Germany, Mexico, India, Japan. SAP servers dropping like flies, that's Defence,Banks, Payroll all affected. Major Retail Chains like Big W down.

aenis9• 2 years ago

We have outages across whole APAC and most EMEA. Despite being a very big client of CS, we do not have an official resolution yet, an hour into the incident.

attentive11• 2 years ago

SAP isn't linux?

sgammon5• 2 years ago

Thundering herd? Idk

ajdlinux• 2 years ago

Starting to get media coverage: https://www.abc.net.au/news/2024-07-19/technology-shutdown-a...

gamma0323• 2 years ago

Australian news reporting this has hit hospitals, fire and rescue, banking, media, airlines and many other companies worldwide.

Napsty• 2 years ago

yep spreading quickly now.

https://www.geekersdigest.com/windows-blue-screen-crash-caus... https://www.timesnownews.com/technology-science/latest-crowd...

This is gonna be a tough Friday for IT departments...

yen223• 2 years ago

Newsreaders on the ABC are reading off paper notepads lmao

jaza7• 2 years ago

I was listening to Triple J (one of ABC's radio stations), they said: "welcome to our first and possibly last ever Triple J's USB Fridays, we can't play any of our usual music because the computers are all down, all we can play is the songs that happen to be on the USB stick that one of us had in our pocket". LOL!

Hawxy• 2 years ago

The impact is Australia is immense. https://downdetector.com.au/, it's almost every major org in the country. NSW Government is completely offline.

thrdbndndn• 2 years ago

I assume you have to install "CrowdStrike" yourself (i.e. not bundled with Windows by default)? I have no idea what it is before.

sammy22552• 2 years ago

Its paid antivirus software, they cater to businesses

alams• 2 years ago

Anyone found any fixes, while Crowdstrike comes up with a fix?

jimmy_lbts• 2 years ago

we will found a troubleshooting , we start in revovery mode and rename the path C:\windows\system32\drivers\Crowdstrike

h4kunamata• 2 years ago

Do this at your own risk!! Apparently there are two NON-OFFICIAL solutions:

1. Rename csagent.sys ( the file causing the BSOD ) 2. Rename c:\windows\System32\Drivers\Crowdstrike

Again, do this at your own risk. Both workaround have been reported as "working". I am Linux user so I cannot tell.

michelevr• 2 years ago

Husband is a deputy in California. His department and many others here are down as well (including PDs, jails, ambulance companies, etc.)

mcast• 2 years ago

This seems like a pretty severe point of failure.

aenis9• 2 years ago

We have ~50 thousand laptops in reboot loop and ~1.5k servers as well. No resolution yet.

kswap• 2 years ago

Faced the same issue few minutes back after few loops of reboot my system is up

aenis9• 2 years ago

We had a few machines come out of the boot loop - only to re-enter it 20 mins later. I am sure CS pulled the patch from their CDNs but ...maybe some cached versions still linger?

7am00dee• 2 years ago

A system restore helps. But obviously not when you’ve got an environment of ~500 or more clients

zcretu• 2 years ago

Quick fix that worked for us, In safe mode:

1.enter in drive C: 2.system 32 folder 3. Drivers 4. Rename crowdstrike folder to something else doesent matter what.

jimbob452• 2 years ago

Could you potentially do the same by just attaching the HDD to another computer as a secondary drive and renaming the folder if safe mode falls through?

nrr8• 2 years ago

Not likely, not unless the disk is also not BitLockered.

_kyran• 2 years ago

Was just using the energy vic website and thought I'd been rate limited when their API stopped working. Seems like it could be this.

BLKNSLVR8• 2 years ago

In terms of analysing risk factors to minimise something like this happening again, what are the factors at play here?

A Crowdstrike update being able to blue-screen Windows Desktops and Servers.

Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system. This makes me think that, maybe, Crowdstrike were unlucky enough to have accidentally discovered a bug that affects multiple versions of Windows (ie. it's a Windows bug, maybe more-so than it is a Crowdstrike bug).

There also seems to have been a ball-dropped in regards to auto-updating all the things. Yes, gotta keep your infrastructure up to date to prevent security incidents, but is this done in test environments before it's put into production?

Un-audited dependence on an increasingly long chain of third-parties.

All the answers are difficult, time consuming, and therefore expensive, and are only useful in times like now. And if everyone else is down, then there's safety in the crowd. Just point at "them too", and stay the path. This isn't a profitable differentiation. But it should be! (raised fists towards the sky).

aloha2436• 2 years ago

an application shouldn't be able to do this kind of damage to an operating system

Antivirus software by its nature probably needs the kind of access that would let it bluescreen your computer.

BLKNSLVR8• 2 years ago

Wading out my depth here, so forgive any stupidity following.

And there's a certain amount of sense to that, it has to get "under" the layer that viruses can typically get to, but I still think there should be another layer at which the OS is protected from misbehaving anti-virus software (which has been known to happen).

fulafel2• 2 years ago

This is not the case. There are many possible AV architectures, with or without kernel drivers and/or administrator level permissions.

zsims• 2 years ago

Whilst Crowdstrike are going to cop a potentially existential-threatening amount of blame, an application shouldn't be able to do this kind of damage to an operating system.

It doesn't operate in user space, they install a kernel driver.

Reason0779• 2 years ago

"they install a kernel driver"

And therein lies the problem!

gamma0323• 2 years ago

At the end of the day, if you give an application a deep set of permissions, that's on you as an administrator, not the OS. This unchecked global rollout appears to just be a violation of every good software engineering practice we know.

BLKNSLVR8• 2 years ago

This unchecked global rollout appears to just be a violation of every good software engineering practice we know.

Yeah, this is what surprises me. Corporate infrastructure policy seems to have been matched to smart phone default settings.

bzzzt• 2 years ago

Administrators are to blame because management (and a lot of 'cybersecurity policies') demand there's a virus scanner on the machines?

While virus scanners might pick up some threats not addressed by OS updates yet every one of them I've seen is a rootkit in disguise wanting full system privileges. There are numerous incidents with security holes and crashes caused by these security products. They also aren't that clever: repeatedly scanning the same files 'on access' over and over again wasting CPU and IO is not going to give you any extra security.

mrweasel4• 2 years ago

As someone who worked for a company, who's a Crowdstrike partner, I assure you that Crowdstrike does not sell to administrators. It is very much a product sold to management and company auditors.

Where you're correct is that it's on the administrators to rollout the updates, but I'm not sure that's how Crowdstrike works. It's a managed solution and updates are done for you, maybe that can be disabled, but I honestly don't know.

tomohawk• 2 years ago

This should clue you in.

CS is not sold to SA or technical types. It's sold to management as a risk reduction.

The whole point is that if you are technical, you are so untrusted that management is willing to require circumvention of known good practices and force installation of this software against technical advice.

nikau9• 2 years ago

I feel the need to make a mustang car crowdsrike meme

CoastalCoder12• 2 years ago

Could you elaborate?

ncr1002• 2 years ago

I think it was a Dodge charger. That pro-trump KKK guy in the gray car who drove through the crowd while the crowd was I think it was a George Floyd protest?

Also heard today crowd stroke.

ric2b3• 2 years ago

There's a better joke, Crowdstrike sponsors the Mercedes Formula 1 team and in 1955 Mercedes was involved in the worst motorsport accident ever, killing over 80 people watching from the stands when parts of the cars flew off and... striked the crowd...

bratwurst3000• 2 years ago

maybe they installed crowdstrike because they wanted updated without testing. and crowdstrike failed at testing them in their environment.

sounds like they didnt test all cases and stumbled on a windows bug

AmericanChopper4• 2 years ago

My experience working with Crowdstrike was that they were super arrogant about these risks. I was working on a ~50k enterprise rollout, and our CS guy was very belligerent about how long we were taking to do it, how much testing we wanted to do, the way that we were staggering roll outs and managing rollback plans. He didn’t think any of this was necessary, that we should roll it out in one fell swoop, have everything to auto-update all the time, and constantly yapped about how many bigger enterprises than ours completed their rollouts in just a couple of weeks.

He actually threatened to fire us as a client because he claimed he didn’t want the CS brand associated with an org that wasn’t “fully protected” by CS. By far the worst vendor contact I’ve ever had. I’ve had nicer meetings with Oracle lawyers than I was having with this guy. I hope this sort of thing humbles them a little.

BLKNSLVR8• 2 years ago

constantly yapped about how many bigger enterprises than ours completed their rollouts in just a couple of weeks.

Evidence is pointing towards him actually being right about this, despite likely being wrong about everything else.

It'd be worth giving him a call, just to check in how he's going, and take him up on the offer to fire you as a client.

DaoVeles12• 2 years ago

Ouch, considering the devil works under Oracles lawyers, thats bad!

shiroiushi• 2 years ago

I hope this sort of thing humbles them a little.

Hopefully not. It would be better that this company is sued into oblivion by all the customers that were affected by this huge outage.

locococo5• 2 years ago

sounds like a very inexperienced person.

If their mission is to protect businesses they should understand your concerns.

Speed is useless without control.

radiator4• 2 years ago

I hope this sort of thing humbles them a little.

What I hope, is that they stop to exist as a product and as a company. They have caused inconvenience, economic damage in global scale and probably also loss of life, given that many hospitals, ER units had outages. It has been proven that their whole way of working is wrong, from the very foundation to the top.

midtake2• 2 years ago

Maybe it's time that critical systems switch to Linux. The major public clouds are already primarily running Linux. Emergency services, booking, and traditional point-of-sale have no strong reason to run Windows. In the past 10 years, the technological capability differences between Windows and Linux have widened considerably, with Linux being the most advanced operating system in the world without question.

Concerns about usability between Windows and Linux in the modern day are disingenuous at best and malicious at worst. There is no UX concern when everything runs off a webapp these days.

Just use Linux. You will save money and time, and your system will be supported for many years, you won't be charged per E-Core, you won't suffer BSoDs in 2024. Red Hat is a trustworthy American company based out of Raleigh, NC, in case you have concerns of provenance.

Really there's no downside. If you were building your own company you would base your tech stack on Linux and not Windows.

Critical systems cannot go down; therefore they cannot run Windows. If they do, they are being mismanaged and run negligently. Management should have no issue finding Linux engineers, they are everywhere. I could make a killing right now as a consultant going from company to company and just swapping out Windows backends for Linux. And quite frankly I might just do that, literally starting right now.

DaoVeles12• 2 years ago

All good points but Windows didnt win because it had the best tech or user interface. Merely the most developer support thus user numbers. Legacy momentum is an incredibly difficult thing to sway. It has taken Apple decades an potentially hundreds of billions of dollars of marketing and good will to carve out its share of the market. Linux doesn't have that despites its clear technical advantages.

It is an incredibly frustrated battle akin to Sisyphus.

attentive11• 2 years ago

oh, you think security won't mandate to run CS on linux.

Granted it didn't down linux this time but nothing is stopping it.

dist-epoch5• 2 years ago

Linux being the most advanced operating system in the world without question.

Only if you don't need a GUI/Desktop.

AdeptusAquinas5• 2 years ago

I don't understand this sort of blindness? Linux fails all the time, with rather terrible nobody to root vulns because some idiot failed to use the right bounds check. Ye gods, XZ utils was barely a few months ago!

prmoustache11• 2 years ago

Crowdstrike has a linux version. It is mandatory in our linux servers in my company so that is not the solution.

I would say issue 1 is management/compliance forcing admins to install malwares like crowdstrike. But issue 1 is because of issue 2 which is about admins / app devs / users aren't smart enough to not have their machines compromised on a regular basis in the first place. And issue 2 is because issue 3 of the software industry not focusing on quality and making bug free software.

All in all this should be mitigated by more diversity in OS, software and "said security solution". Standardization and monopolies works well until they don't and you get this kind of shit.

NexRebular• 2 years ago

Maybe it's time that critical systems switch to Linux.

I switched critical systems to illumos and BSD years ago and it's been smooth sailing ever since. Nowadays there really is no need to contribute to linux monoculturization whatsoever.

bdd8f1df777b8• 2 years ago

I too want to see Linux more widely adopted, but it won't prevent this from happening. People will install corrupted kernel modules on Linux too for anti-virus purposes.

jsmith99• 2 years ago

Do Linux systems not crash if a third party kernel module crashes? Or was your comment sarcastic?

MatejKafka• 2 years ago

The discussed issue is not related to any meaningful difference between Windows and Linux – Crowdstrike used a kernel driver, apparently containing a serious bug, which took down the system, which is something any kernel driver can do, no matter which kernel you use. At least Windows have a well-developed framework for writing userspace drivers, unlike Linux.

Linux being the most advanced operating system in the world without question.

Very strong and mostly unfounded claim; there are specific aspects where Linux is "more advanced", and others where Windows come out ahead (e.g. almost anything related to hardware-based security and virtualization).

your system will be supported for many years

Windows Server 2008 was supported until earlier this year, longer than any RHEL release.

you won't suffer BSoDs in 2024

Until you install a shitty driver for a dubious (anti)malware service.

vbezhenar6• 2 years ago

I suggest switching to macOS. They don't allow third-party kernel drivers which is already a big advantage over Windows or Linux.

namtab00• 2 years ago

https://access.redhat.com/solutions/7068083

mhh__• 2 years ago

Other than a lack of redundant systems, it should be illegal to roll out updates like this to more than x% of any gov stuff at a time. Brute force way of avoiding correlational Armageddon.

JonChesterfield4• 2 years ago

Don't root kit yourself then cry about it when it falls over. Problem solved.

btbuildem6• 2 years ago

Crowdstrike is basically corporate malware - the failure is in large part with security dept deciders who signed off on policies that compel people to install these viruses on their work machines.

tru3_power2• 2 years ago

Crazy- wasn’t Azure having an outage earlier today? Is this related?

BLKNSLVR8• 2 years ago

I'd guess that it's related. Azure, OneDrive, and Office 365 were having issues this morning Australian time (about 6 hours ago).

jdleel2• 2 years ago

Doesn't actually look like. Noone really figured out the solution yet

tru3_power2• 2 years ago

If that’s the case I wonder why we’re only now seeing everyone’s systems BSODing. Should be an interesting write up (if they release one)

cpf_au• 2 years ago

This is effecting our company. A colleague visited her local supermarket (Woolworths) and all the self-service checkouts were effected.

Scoundreller12• 2 years ago

Silver lining?

vinura• 2 years ago

Anyone has a good news? or still it's on the BSOD loop?

tgshaik4• 2 years ago

use windows recovery to boot into safe mode and update latest crowdstrike. That is the only option at this time.

ali2724• 2 years ago

Harvey Norman’s system is down

l0g4n_me• 2 years ago

c:\system32\drivers\csagent.sys renaming this file on server with safe mode boot fixes the issue but disables agent

jmcgough26• 2 years ago

My entire emergency department got knocked offline by this. Really scary when you have ambulances coming in and are trying to stabilize a heart attack.

Update: 911 is down in Oregon too, no more ambulances at least.

sexy_seedbox• 2 years ago

But at least the best instant messaging app in the world Microsoft Teams and the best web browser in the world Microsoft Edge are working fine, right?

jmcgough26• 2 years ago

It's a bsod loop, so not really.

jeffrallen4• 2 years ago

"Always look on the bright side of life!" - M Python

spike0212• 2 years ago

Do you have offline backup processes at least? Nasty situation.

jmcgough26• 2 years ago

We're really prepared for epic to go down and have an isolated cluster that we access in emergencies. I transitioned from software engineering so I've only been in the ED for a year, but from what I could see there didn't seem to be a plan for what to do if every computer in the department bluescreened at once.

prmoustache11• 2 years ago

Aren't they doing canary release? Seems weird this would not have been detected on a smaller scale before with a good release process.

makmanalp• 2 years ago

Canary releases aren't a magical bug free fix. They might be doing it, but the conditions that trigger a problem can be sneaky and can happen outside your canary period, rendering it useless. It's a best effort method.

prmoustache11• 2 years ago

It depends on how global the issue is.

But yeah I've seen US companies for example only doing their initial releases in the USA only which has zero value for issues that might appear with different localization/language settings for example.

ikekkdcjkfke3• 2 years ago

Buying 10 computers with different configurations is not hard either. They are just lazy

sakopov• 2 years ago

If the workstations are stuck in a boot loop, how will they be able to push a hotfix out?

ernth_164• 2 years ago

what we do is safemode the PC and then open the run CMD as admin. then issue this command. sc delete csagent. reinstall crowdstrike using previous version.

ernth_164• 2 years ago

safemode PC i mean

caf• 2 years ago

Not gonna be awesome for an org that has 20,000 laptops out there at home with users who don't have admin...

kasabali5• 2 years ago

elbow grease

jaza7• 2 years ago

Get out and push?

ernth_164• 2 years ago

Uninstall the Current Version:

Open the Command Prompt as an administrator. Run the following command to uninstall the current version: shell

sc delete csagent

rubi1945• 2 years ago

temporary fix is released

https://x.com/s0lst1c3/status/1814182835269337548

dalmo3• 2 years ago

Looks like this also took down half of New Zealand's economy.

https://www.nzherald.co.nz/nz/bank-problems-reports-bnz-asb-...

Taniwha• 2 years ago

Satellite TV channels too

Reason0779• 2 years ago

Crowdstrike must have some slick salespeople in New Zealand. Seems like nearly everyone uses it. Single point of failure problem.

0x800838383• 2 years ago

CIO here. They are known to be incredibly pushy. In my company we RFP'd for our endpoint & cyber security. Found the CS salesperson went over me to approach our CEO who is completely non-technical to try and seal a contract because I was on leave for 1 week out of service (and this was known to them). When I found out by our CEO informing me of the approach we were happy to sign with SentinelOne

a4000• 2 years ago

Same in Australia https://www.abc.net.au/news/2024-07-19/global-it-outage-crow...

Banks are down so petrol stations and supermarkets are basically closed. People can't check in to airline flights, various government services including emergency telephone and police are down. Shows how vulnerable these systems are if there's just one failure point taking all those down.

afro88• 2 years ago

Banks seem fine now. At least ING and NAB.

averageRoyalty6• 2 years ago

000 was never down, and most supermarkets and servos were still up. It was bad, but ABC appear to not have the internal capacity to validate all reports.

NomDePlum2• 2 years ago

Similar being reported in Australia. Can see the effects being reported as other timezones become more active.

Scoundreller12• 2 years ago

You mean you think the update was timed and waves will hit every hour for the next nearly 24h, or it’s just 3AM in NYC right now and under the radar?

nomercy4002• 2 years ago

Netherlands is somewhat affected: Two main airports, Rotterdam harbour, a few hospitals and news reporting.

Surprisingly: banks, government, police, fire department, railways, buses are mostly unaffected.

Maybe they have a good IT department/provider.

raziel2p• 2 years ago

Probably more to do with luck than being "good".

ElCapitanMarkla• 2 years ago

I was at the supermarket here last night about the time it kicked off. It seemed payWave was down, there were a few people walking out empty handed as they only had Apple Pay, etc on them. But the vast majority of people seemed fine, my chipped credit card worked without issue.

jafru• 2 years ago

"F8 + last known good configuration" worked for us

gamma0323• 2 years ago

The biggest mistake here is running a global update on a Friday. Disrespect to every sysadmin worldwide.

usr11068• 2 years ago

Disrespect to every CIO to make their business depend on a single operating system, running automatic updates of system software without any canaries and phased deployments.

dist-epoch5• 2 years ago

You're saying I should diversify my 100% Linux operation to also use Windows?

linker3000• 2 years ago

The biggest mistake is not ringfencing this update in a test environment before sign-off for general deployment.

jaza7• 2 years ago

Agreed. What happened to Patch Tuesdays?!

chippiewill3• 2 years ago

Presumably they do test their updates, they're just maybe not good enough tests.

The ideal would be to do canary rollouts (1%, then 5%, 10% etc.) to minimise blast radius, but I guess that's incompatible with antiviruses protecting you from 0-day exploits.

prmoustache11• 2 years ago

I don't think the day matter anymore really.

The issue is update rollout process, lack of diversity of these kind of tools in the industry, and the absolute failure of the software industry to make decent software without bug and security holes.

OJFord• 2 years ago

Yeah, airlines prefer mid-week chaos & grounding.

poshlandpro• 2 years ago

Official CrowdStrike workarround: 1. Boot Windows into Safe Mode or the Windows Recovery Environment 2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory 3. Locate the file matching “C-00000291*.sys”, and delete it. 4. Boot the host normally.

Cub3• 2 years ago

Workaround + update (within a authenticated portal)

https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e...

thallaran2• 2 years ago

all Delta flights grounded

thallaran2• 2 years ago

All flights grounded. World wide airline systems outage.

jjjjjjjjjjjjjjj• 2 years ago

Just boarded a flight in Iceland, no mention of being grounded.

kalleboo• 2 years ago

It only applies to a few US carriers

https://abcnews.go.com/US/american-airlines-issues-global-gr...

hasindh09• 2 years ago

HP laptops are booting and its looping

DELL laptops are observed, after blue dump . Server is up and running fine

Temporary workaround leads compliance issue.

From India

BLKNSLVR8• 2 years ago

Flippant commentary:

Thank fuck Netflix runs on Linux. I just hope the full chain from my TV to Netflix is immune...

jmprspret• 2 years ago

It actually runs on a lot of FreeBSD iirc

BLKNSLVR8• 2 years ago

Even better!

Klonoar3• 2 years ago

Netflix uses FreeBSD for certain parts, but IIRC it's Linux outside of the streaming delivery.

(Some of the team members lurk around here, maybe we'll get lucky and one comments)

jkells2• 2 years ago

I don't understand why this outage and the Azure outage earlier don't make it to the front page.

I'm getting more up to date technical details from the regular media.

This outage looks to be huge.

wfme• 2 years ago

Agreed, I assume people are flagging this post for some reason?

speedgoose• 2 years ago

It’s most likely the flame war filtering algorithm of HN. Posts that create a lot of discussion quickly are down ranked until an admin fix the rank manually, or not.

blinkersareon2• 2 years ago

The impact is more than huge. Our whole workforce is impacted as the world's largest tech service provider.

Scoundreller12• 2 years ago

Y2K came 24+ years late!

Scoundreller12• 2 years ago

Most HNers are on the clock while browsing (for work purposes of course) and forcibly afk at the moment.

d1sxeyes3• 2 years ago

I'd guess also a significant proportion of us are involved in our company's response to the incident.

occamsrazorwit2• 2 years ago

This is literally a black swan event that'll be memorialized in textbooks, and that's not even considering the actual fallout that will follow.

DaoVeles12• 2 years ago

That is not an understatement. This is literally the largest failing of internet infrastructure to date.

Alas, using the internet has given us a lot of efficiency. The trade off is resilience. The entire global system is more brittle than ever but it what gave it such speed.

hnthrowaway03285• 2 years ago

Did you mean the Azure thing or CS thing? Just happened that the Azure thing did not impact me as I was already off...

usr11068• 2 years ago

HN has some flame war protection I remember having read, downrating some discussions where it assumes nothing useful is being discussed.

Maybe a false positive? (100% speculation. That's the problem using closed source software)

larrymcp• 2 years ago

https://archive.is/AWVuv

nallerooth• 2 years ago

Thanks!

protocolture• 2 years ago

Dont see how anyone is getting out of this without applying the workaround or reimaging their whole fleet.

kcd83• 2 years ago

Fix: Boot Windows into Safe Mode or the Windows Recovery Environment Navigate to the C:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Boot the host normally

Did it work for you?

LeoPanthera3• 2 years ago

BBC live coverage: https://www.bbc.com/news/live/cnk4jdwp49et

Looks like this is a big deal.

deutschlerner2• 2 years ago

https://www.bbc.co.uk/news/live/cnk4jdwp49et, seems to be quite a wide impact from this, e.g. Sky News in the UK is off air!

rosege2• 2 years ago

So its not all bad then!

Melatonic2• 2 years ago

US Based and got a NANOG alert email just in time. At least half our windows servers down.

I went into our crowdstrike policies and disabled auto update of the sensor. Hopefully this means it doesnt hit everything. Double check your policies!!!

Edit:

Crowdstrike has an article out on the manual fix:

https://supportportal.crowdstrike.com/s/article/Tech-Alert-W...

markus92• 2 years ago

All major US airlines have put in a total ground stop. No flights can take off anymore.

kurthr2• 2 years ago

Electronic payment at grocery stores in the US is affected.

Scoundreller12• 2 years ago

POS down? Payment processor down?

Lotta different grocery stores and processors in USA. WHO?

chgs7• 2 years ago

I’m Amazed at that. I had no idea the US had caught up and had electronic payments.

t0mas883• 2 years ago

It's morning here in Europe, departure peak time. We're still flying, but...

The problems mean the takeoff and weight & balance data is missing. It needs to be done manual by each crew. Baggage handling is also manual, so that means counting bags and the cabin crew counting people. Then manually calculating performance data before you can take off.

Big delays everywhere.

maeil7• 2 years ago

It's morning here in Europe, departure peak time. We're still flying, but...

Looks like some airlines aren't flying, KLM being one of them.

qw• 2 years ago

It's not all of Europe. The airports in Norway are operating as normal. One of the airlines reported booking issues on their website, but I haven't read about any other issues.

Some international flights have been delayed or canceled of course, depending on the route.

topspin3• 2 years ago

It needs to be done manual by each crew.

I can't imagine believing that this computation, ordinarily automated by these out of service systems, can be performed correctly by crews that probably haven't had to do this in ... years?

LeoPanthera3• 2 years ago

Not all. Only some. And many are already lifted.

usrnm• 2 years ago

All airlines in Russia are operating as usual. Sanctions work in unpredictable ways

tloriato• 2 years ago

GLOBAL OUTAGES

- Major banks, media and airlines affected by major IT outage

- Significant disruption to some Microsoft services

- 911 services disrupted in several US states

- Services at London Stock Exchange disrupted

- Sky News is off air

- Flights in Berlin grounded

- Reports the issue relates to problem at global cybersecurity firm Crowdstrike

bux93• 2 years ago

Good news for crowdstrike! It shows how critical their services are. Stock to go up! (And down a bit when they get sued, and up a bit when they don't get sued too much, etc.)

chucky_z• 2 years ago

Take a look at their stock right now.

zhiQ• 2 years ago

CrowdStrike achieved PMF - Product-Market Fiasco!

jen729w• 2 years ago

I was just at Coles, one of Australia's big 2 supermarkets.

50% of the self-serve terminals were down. Presumably the others aren't far behind.

Total chaos.

fredoralive5• 2 years ago

Sky News (UK) is back on air, but they seem to have no astons / chyrons / on screen graphics at all, and I don't think they're able to show prerecorded material either (it's just people in the studio from what I've seen), so presumably they're still having fun issues with their general production systems.

InsideOutSanta11• 2 years ago

Oh, look at that, TV has suddenly become watchable without all of the scrolling garbage all over the screen.

urbandw311er3• 2 years ago

Just got their ticker back at 09:42 !

tamimio11• 2 years ago

I have seen some pictures shared of vending machines with BSOD.. why would you have an EDR on a vending machine? To protect the valuable assets?!

fredoralive5• 2 years ago

If the vending machine handles credit cards, wouldn't Visa / Mastercard / etc. basically require it as part of their security requirements? Or it's just general CYA from someone that's backfired badly.

Lt_Riza_Hawkeye• 2 years ago

presumably they handle credit card transactions

butlike6• 2 years ago

"It's a vending machine that sells back physical stolen credit cards." Input the dollars, receive stolen credit card. Put it next to an airport terminal for maximum impact.

telesilla• 2 years ago

Schipol (Amsterdam, europe's largest) and Melbourne Airport are down

kmlx2• 2 years ago

- Services at London Stock Exchange disrupted

just their news service.

The London Stock Exchange says it's working as normal - but says there are problems with its RNS (regulatory news service).
"RNS news service is currently experiencing a third party global technical issue, preventing news from being published on www.londonstockexchange.com," the statement says.
"Technical teams are working to restore the service. Other services across the group, including London Stock Exchange, continue to operate as normal."

aenis9• 2 years ago

We are a major CS client, with 50k windows-based endpoints or so. All down.

There exists a workaround but CS does not make it clear whether this means running without protection or not. (The workaround does get the windows boxes unstuck from the boot loop, but they do appear offline in the CS host management console - which of course may have many reasons).

kurthr2• 2 years ago

Yeah, the simple renaming .sys files in safe mode does seem like it would inhibit protection.

Scoundreller12• 2 years ago

Renaming it .old or whatever would be what a sane person does.

They recommended deleting the thing: https://news.ycombinator.com/item?id=41002199

badrabbit3• 2 years ago

No, the userspace program will replace it with a good version.

blinkersareon2• 2 years ago

I think my company has more than 300k+ machines down right now :)

SLAs will be breached anyway

Reason0779• 2 years ago

Windows IT admins of the world, now is your time. This is what you've trained for. Everything else has led to this moment. Now, go and save the world!!

easytiger4• 2 years ago

Surely i'ts not normal practice to allow patches to be rolled out without a staging/testing area on an estate of that size?

usr11068• 2 years ago

It's incredibly bad practice, but it seems to be industry normal as we learned today.

InsideOutSanta11• 2 years ago

This is insane. The company I currently work for provides dinky forms for local cities and such, where the worst thing that could happen is that somebody will have to wait a day to get their license plates, and even we aren't this stupid.

I feel like people should have to go to jail for this level of negligence.

mcs5280• 2 years ago

The AI said it was ok to deploy

tomduncalf• 2 years ago

Their announcement (see Reddit for example) says it was a “content deployment” issue which could suggest it’s the AV definitions/whatever rather than the driver itself… so even if you had gradual rollout for drivers, it might not help!

aenis9• 2 years ago

True for all systems, but AV updates are exempt from such policies. When there is a 0day you want those updates landing everywhere asap.

Things like zscaler, cs, s1 are updating all the time, nearly everywhere they run.

the_gipsy2• 2 years ago

Does CS actually offer any real protection? I always thought it was just feel-good software, that Windows had caught up to separating permissions since after XP or so. Either one is lying/scamming, but which one?

padjo• 2 years ago

Every company I’ve ever worked at has wound up having to install antivirus software to pass audits. The software only ever caused problems and never caught anything. But hey, we passed the audit so we’re good right?

saagarjha16• 2 years ago

EDR vendors are generally lying unless they tell you anything but "install us if you want to pass certification".

CaliforniaKarl6• 2 years ago

Yes, from experience, I can say that CS does offer real protection.

tecleandor2• 2 years ago

AFAIK, most of the people I know that deploy CrowdStrike (including us) just do it to check a box for audits and certifications. They don't care much about protections and will happily add exceptions on places where it gives problems (and that's a lot of places)

badrabbit3• 2 years ago

Yes, it offers very real protection. Crowdstrike in particular is the best in the market, speaking from experience and having worked with their competitor's products as well and responded to real world compromises.

lucasRW8• 2 years ago

As a redteamer I guarantee you that a Windows endpoint without EDR is caviar for us...

mr_mitm11• 2 years ago

Just adding my two cents: I work as a pentester and arguably all of my colleagues agree that engagements where Crowdstrike is deployed are the worst because it's impossible to bypass.

dhjhdskj• 2 years ago

Anyone who claims CS is nothing but a compliance checkbox has never worked as an actual analyst, of course it's effective...no, dur, its worth 50bn for no reason...god some people are stupid AND loud

ta124312• 2 years ago

Does CS actually offer any real protection? I always thought it was just feel-good software, that Windows had caught up to separating permissions since after XP or so. Either one is lying/scamming, but which one?

Our ZScaler rep (basically, they technically work for us) come out with massive impressive looking numbers of the thousands of threats they detect and eliminate every month

Oddly before we had zscaler we didn't seem to have any actual problems. Now we have it and while we have lots of zscaler caused problems around performance and location, we still don't have any actual problems.

Feels very much like a tiger repelling rock. But I'm sure the corporate hospitality is fun.

cm21874• 2 years ago

Does it require to physically go to each machine to fix it? Given the huge number of machines affected, it seems to me that if this is the case, this outage could last for days.

ndsipa_pomu3• 2 years ago

The workaround involves booting into Safe mode or Recovery environment, so I'd guess that's a personal visit to most machines unless you've got remote access to the console (e.g. KVM)

The info is apparently behind here: https://supportportal.crowdstrike.com/s/article/Tech-Alert-W...

CaliforniaKarl6• 2 years ago

Our experience so far has been:

• Servers, you have to apply the workaround by hand.

• Desktops, if you reboot and get online, CrowdStrike often picks up the fix before it crashes. You might need a few reboots, but that has worked for a substantial portion of systems. Otherwise, it’ll need a workaround applied by hand.

surgical_fire2• 2 years ago

The Dildo of Consequences rarely comes lubed, it seems.

fudged713• 2 years ago

A bunch of unprotected endpoints all at once on critical systems… what could possibly go wrong?

Hope they role out a proper fix soon…

CaliforniaKarl6• 2 years ago

They did, around four hours ago.

drited• 2 years ago

Wow that's terrible. I'm curious as to whether your contract with them allows for meaningful compensation in an event like this or is it just limited to the price of the software?

m-i-l• 2 years ago

"50k windows-based endpoints or so. All down."

I'm a dev rather than infra guy, but I'm pretty sure everywhere I've worked which has a large server estate has always done rolling patch updates, i.e. over multiple days (if critical) or multiple weekends (if routine), not blast every single machine everywhere all at once.

EdwardDiego• 2 years ago

This feels like an auto-update functionality. For something that's running in kernel space (presumably, if it can BSOD you?) Which is fucking terrifying.

fredoralive5• 2 years ago

If this comment tree: https://news.ycombinator.com/item?id=41003390 is correct, someone at Crowdstrike looked at their documented update staging process, slammed their beer down, and said: "Fuck it, let's test it in production", and just pushed it to everyone.

heraldgeezer2• 2 years ago

Crowdstrike auto-updates. Please do not spread misinfo.

dschuetz3• 2 years ago

Are you rolling out CS updates as is everywhere? Are you not testing any published updates immediately at least with some N-1 staging involved?

averageRoyalty6• 2 years ago

I don't mean this to be rude or as an attack, but do you just auto update without validation?

This appears to be a clear fault from the companies where the buck stops - those who _use_ CS and should be validating patches from them and other vendors.

sunbum• 2 years ago

I'm pretty sure crowdstrike autoupdates, with 0 option to disable or manually rollout updates. Even worse people running N-1 and N-2 channels also seem to have been impacted by this.

aenis9• 2 years ago

Apparently, CS and ZScaler can apply updates on their own and thats by design, with 0day patches expected to be deployed the minute they are announced.

heraldgeezer2• 2 years ago

CS, S1, Zscaler etc auto updates and they have to. Thats the point of the product. If they dont get definitions they cannot protect.

purpleblue2• 2 years ago

Do you need to manually fix all your windows boxes? Or is there a way to update it remotely?

1oooqooq13• 2 years ago

stop the pandering. you know very well crowdstrike doesn't offer good protection to begin with!

everyone pay for legal protection. after it happens you can show you did everything, which means nothing (well now this show even worse than nothing), by showing you paid them.

if they tell you to disable everything, what does it change? they're still your blame shield. which is the reason you have cs.

... the only real feature anybody care is inventory control.

hello_moto17• 2 years ago

Quite a few people in this thread disagree with you though.

martypitt2• 2 years ago

Discussed more thoroughly here: https://news.ycombinator.com/item?id=41002195 (Not sure why that's not on the frontpage)

mirekrusin2• 2 years ago

Because ppl are locked out to say they are locked out.

pvg2• 2 years ago

Probably shoved off by the flamewar detector.

itsgrimetime2• 2 years ago

I just landed at SeaTac an hour ago and the rideshare/app pickup was absolutely nutso. Like thousands of people standing around waiting for taxis and Ubers. The one person I asked what was going on said that the computer systems at all the regional hotels are down (not sure how that makes more people need cabs). Wonder if it’s from this

itsgrimetime2• 2 years ago

Just realized this is posted on the SeaTac website now: “ SEA is experiencing temporary issues with the system that populates flight and baggage information on in terminal screens and the flySEA app/website. Travelers are recommended to check with their airlines for current gate and baggage claim information. Check With Your Airlines”

Scoundreller12• 2 years ago

I remember when my big regional train system had like 95% of its morning trains cancelled overnight due to a big snowstorm in 2014.

Of course, no news on its websites or socials because those people didn’t start until 9am.

I think they finally fixed that.

Klonoar3• 2 years ago

just landed at SeaTac an hour ago and the rideshare/app pickup was absolutely nutso. Like thousands of people standing around waiting for taxis and Ubers.

So a normal day at SeaTac?

ThePhysicist• 2 years ago

If it's true that a bad patch was the reason for this I assume someone, or multiple people, will have a really bad day today. Makes me wonder what kind of testing they have in place for patches like this, normally I wouldn't expect something to go out immediately to all clients but rather a gradual rollout. But who knows, Microsoft keeps their master keys on a USB stick while selling cloud HSM so maybe Crowdstrike just yolos their critical software updates as well while selling security software to the world.

nilsb• 2 years ago

Who needs testing when apologizing to your customers is cheaper?

falcor842• 2 years ago

I would assume that its enterprise customers have an uptime SLA as part of their contract, and that breaching it isn't very cheap for Crowdstrike.

agrajag• 2 years ago

Reputational damage from this is going to be catastrophic. Even if that’s the limit of their liability it’s hard not to see customers leaving en masse.

dclowd99016• 2 years ago

And when it’s more costly for customers to walk back the mistake of adopting your service.

Yeah, I get the impression a lot of SaaS companies operate on this model these days. We just signed with a relatively unknown CI platform, because they were available for support during our evaluation. I wonder how available they’ll be when we have a contract in place…

helsinkiandrew• 2 years ago

As at 4am NY time CRWD has lost $10Bn (~13%) in marketcap. Of course they've tested, but just not enough for this issue (as is often the case).

This is probably several seemingly non consequential issues coming together.

I'm not sure why though, when the system is this important that even successfully tested updates aren't rolled out piecemeal though (or perhaps it has and we're only seeing the result of partial failures around the world)

kjkjadksj4• 2 years ago

Exactly. They knocked half the world offline probably killed thousands in ERs and the stock is only down to about June lows.

krspnda• 2 years ago

hah that tweet was one heck of an apology. "we deployed a fix to the issue, speak with your customer rep"

jachee• 2 years ago

Here's hoping they start from the top.

They won't, but hope springs eternal.

cen4• 2 years ago

Doesn't matter what testing exists. More scale. More complexity. More Bugs.

Its like building a gigantic factory farm. And then realizing that environment itself is the birthing chamber and breeding ground of superbugs with the capacity to wipe out everything.

I used to work at a global response center for big tech once upon a time. We would get hundreds of issues, we couldn't replicate cause we literally have to set up our own govt or airline or bank or telco to test certain things.

So I used to joke with the corporate robots to just hurry up and take over govts, airlines, banks and telcos already, cause thats the only path to better control.

jonathanstrange• 2 years ago

Testing + a careful incremental rollout in stages is the solution. Don't patch all systems world-wide at once, start with a few, add a few more, etc. Choose them randomly.

roemerb• 2 years ago

Its like building a gigantic factory farm. And then realizing that environment itself is the birthing chamber and breeding ground of superbugs with the capacity to wipe out everything.

Factorio player detected

tempaway45751445• 2 years ago

Sounds like it was a 'channel file' which I think is akin to an av definition file that caused the problem rather than an actual software change. So they must have had a bug lurking in their kernel driver which was uncovered by a particular channel file. Still, seems like someone skipped some testing.

https://x.com/George_Kurtz/status/1814235001745027317

https://x.com/brody_n77/status/1814185935476863321

JonChesterfield4• 2 years ago

The parser crashing the system on a malformed input file strongly suggests their software stack in general is trash

joeldo• 2 years ago

The impact of this will be profound!

Obviously bugs are inevitable, but why this wasn't progressively rolled out is beyond me.

kondro• 2 years ago

My understanding is that multiple recent versions are affected.

davros• 2 years ago

Yeah, progressive rollout would have dramatically reduced the impact, I think that should be mandatory for any of these systems.

raverbashing3• 2 years ago

Are people counting this on the Windows TCO?

raggi• 2 years ago

they have kernel modules for macos and linux too afaik, so i wouldn't be counting those chickens too fast

crycombinator• 2 years ago

I would expect this to be a kernel specific bug. I'm on a company laptop with falcon, and we have linux systems using the same, no signs of problems so far.

Kwpolska2• 2 years ago

It's not Microsoft's fault someone installed third-party spyware/malware on their systems.

mrdeveloper16• 2 years ago

Go to advance repair option then advanced open cmd. Go to windows/system32/drivers/crowdstrike. Then list all the file and delete file name having 291 at the end using cmd "del filenameendingwith291"

RedShift14• 2 years ago

This is why you don't make changes on a Friday. Lots of weekends absolutely ruined now.

mirekrusin2• 2 years ago

Early weekend here :D

YoboDev2• 2 years ago

All US flights are grounded too. The people I was traveling with cant check into hotels

lpcvoid2• 2 years ago

Maybe the world can finally reconsider their use of software products that cater to security theater. And the politics in companies which lead to things like this being introduced ("nobody gets fired for buying IBM").

Edit: took out a bit of snark.

westpfelia• 2 years ago

I dont know how CS is considered snake oil. Or what IBM has to offer. CS and S1 are really just the best out there.

kasabali5• 2 years ago

well, apparently they aren't

lpcvoid2• 2 years ago

"Endpoint protection" is just the new, hip term for antivirus/intrusion prevention/incident logging of the past. Why not provide immutable Linux based machines (like Chromebooks, Fedora Silverblue) which are locked down outside of the browser? I am aware that this isn't possible in some areas of the industry that rely on large amounts of Windows-only desktop software, but in many cases it may be worth a thought.

If I am being naive here, happy to hear other opinions. But I hate opening my company Windows laptop and having the fans turn to 11 just because some "security" software is parsing random files for malicious signatures or running an update that BSOD loops.

hcs• 2 years ago

The IBM quip is an old saying: pick the dominant vedor because even if it's a mistake no one will blame you for it.

crazytony• 2 years ago

Have spent all my afternoon and all evening on a bridge trying to support flailing systems. Was supposed to be on a plane in 5 hours to start my vacation. Guaranteed it's not gonna happen.

With hearing 911 and other safety critical systems going down, I hope that the worst that comes out of this is a couple delayed flights and a couple missed bank payments.

baq7• 2 years ago

Good news (um, as in better than the bad news today) is the plane won't be taking off anyway, so you're golden.

tamimio11• 2 years ago

OTA update went wrong? How can an update go live without proper testing for the millions of live connected endpoints?

pageandrew3• 2 years ago

Can they recover this OTA considering the systems can’t even boot?

DaoVeles12• 2 years ago

So far requires going into recovery mode and removing/rename the cloud strike executable. Then you can boot into Windows from there, it will probably be a sys admin thing dependent on the organisation setup.

tamimio11• 2 years ago

I’m sure the patch itself can be fixed, and there will be a workaround to boot up the machine to fix it. My only concern is the BitLocker keys. If the hard drive is encrypted by Windows and assuming no backup for that key has been done, the system admins will have to activate their disaster recovery plans for these devices, and I hope they have that too, but hope isn’t a strategy!

sneak• 2 years ago

Maybe one day we will stop giving RCE to so many vendors via auto update.

attentive11• 2 years ago

I am going to stop saying this but people don't realize CS has an official RCE as a feature. As in run remote commands as root/admin on windows or linux/mac through their web.

Havoc• 2 years ago

This is why I subscribe to /r/sysadmin despite not being one ... like a canary in the coalmine for stuff like this

kitd2• 2 years ago

A reminder why switching off auto-update is a thing.

seanieb• 2 years ago

It looks like it wasn’t a software update, it was a AV definitions update, so internal to the CA application.

kitd2• 2 years ago

True, though tbf it's still part of the running system.

I read that many of those affected are global orgs. When I worked at an oil major, everything was tested to oblivion before going into production in the DCs, the reason being to avoid precisely this kind of situation where at all possible. There were clusters set aside for operational acceptance testing to ensure everything, from business application right down to kernel, ran successfully. The idea of leaving auto-update on in any production system was unthinkable. Yet here we are.

bb123• 2 years ago

There appears to be a workaround but my question is how are they going to get all of these endpoints out of a BSOD loop?

DaoVeles12• 2 years ago

Might need to manually do it. Depends on if there is any lower level admin access than windows for each system.

chad1n• 2 years ago

The antivirus did its job, now you can't get viruses. Jokes aside, I've checked their website and it was full of AI buzzwords so I guess that happens when you focus on nonsense instead of what your customers actually need (I know that all antiviruses have a machine learning component, but usually you don't advertise it as some sort of AI to get better stocks).

Kwpolska2• 2 years ago

All antivirus software is indistinguishable from malware.

chgs7• 2 years ago

I have never seen malware take half the worlds IT systems offline, even in the days of code red and slammer

meigwilym• 2 years ago

I think the AI talk is just the fashion now amongst C-level execs. Their product - no matter what it does - suddenly needs some sort of AI integration.

chall84321• 2 years ago

i work liquor distribution in the united states and our entire company is out across 44 states, allegedly due to this “crowd strike outage”

nmcveity• 2 years ago

This gem from the ABC news coverage has my mind 100% boggled:

"711 has been affected by the outage … went in to buy a sandwich and a coffee and they couldn’t even open the till. People who had filled up their cars were getting stuck in the shop because they couldn’t pay."

Can't even take CASH payment without the computer, what a world!

timthorn2• 2 years ago

At least they'd take cash if the computer wasn't broken. That's getting quite rare in the UK.

physicsguy3• 2 years ago

Not really? I've only really seen people not taking cash at trendy street food stalls and bougie coffee shops, pretty much everywhere else does.

Scoundreller12• 2 years ago

Technically a payment terminal can go into island mode and take offline credit card transactions and post them later. PIN can be verified against the card.

Depends if the retailer wants to take the chance of all that.

nielsole• 2 years ago

That is if the terminal is not dead itself

jimmydorry• 2 years ago

Good luck putting a payment terminal into island mode when it's in a bluescreen loop.

thyrsus2• 2 years ago

Not this time. Use paper, pen, and a non-electronic cash box.

Ekaros9• 2 years ago

Terminal is probably fine, the machine that tells it number to charge is dead... And it is probably not even setup to accept manual payment inputs.

skerit• 2 years ago

I'm seeing several reports of things like being unable to buy tickets for the train on-line in Belgium.

They use Windows as a part of their server infrastructure?

bamboozled• 2 years ago

The world just became a slightly better place.

rwmj7• 2 years ago

Can someone explain what Crowdstrike actually is? Reading Wikipedia it seems to be some sort of anti-virus software?

evgen• 2 years ago

It is one of the best systems available for realtime protection of windows systems against various threat actors. Prior to today you could probably have said 'no one gets fired for recommending Crowdstrike as the security tool for the company.' It is everywhere and in particular if you are a large org with a lot of Windows seats you are likely a Crowdstrike customer.

rwmj7• 2 years ago

realtime protection of windows systems against various threat actors

So it's AV + a firewall? What does it actually do?

saagarjha16• 2 years ago

Of course, this tells you a lot about the sad state of "realtime protection" software.

account428• 2 years ago

And like most AV systems it seems to be a bigger threat than what it supposedly protects against. Seriously how is it acceptable to have one corporation push a live update and take down tons of critical services all over the world. Just imagine what a malicious actor could accomplish with such a delivery vector.

attentive11• 2 years ago

realtime protection of windows systems

and mac, and linux

hdra• 2 years ago

seems to be an MDM solution thats doing tons of stuffs

rwmj7• 2 years ago

"MDM solution" leaves me even more confused than before.

t8sr• 2 years ago

It’s watching the system for events like “file was opened” and “process started”, and looking for patterns resembling hackers/malware.

It’s different from AV in that it mostly looks at runtime behavior and not signatures.

entuno3• 2 years ago

AV with shiny bits stuck on the side and a good marketing team.

zo12• 2 years ago

I see it's not just the Software development ecosystem that got affected by the cult of Hipsterism.

If I was Alex Jones, I'd go further and blame this on a decade of DEI and fluoride in the water. /s

aldanor• 2 years ago

More like enterprise-level spyware

tamimio11• 2 years ago

CS is an EDR (Endpoint Detection & Response) and it connects to other parts like XDR (Extended Detection and Response) and MDM (Mobile Device Management). They differ from the typical antivirus in how they detect threats. The AV usually checks against known threats, while EDR detects endpoint behavior anomalies. For example, if your browser spawns a shell, it will be marked and the process quarantined. Of course, they do share a lot of common domains like real-time protection, cloud analysis, etc., and some AVs have most of the EDR capabilities, and some EDRs have most of the AV capabilities. This is briefly described.

Agingcoder2• 2 years ago

We're running something similar ( not CS ) where I work.

It seems to me that these tools create lots of problems ( slows down the machine significantly in particular, gets things wrong and quarantines processes/machines when it shouldn't, injects itself into processes so changes behaviour, etc ).

The main question I have is : does anyone have an actual instance of such tools detecting something useful ? No one in the office was able to show one.

romanovcode3• 2 years ago

You also forgot the part that it is a tool to spy on everything the employees do if it is installed on their computers.

dilyevsky• 2 years ago

Yes it’s rebranded antivirus for enterprise with new fancy name - “endpoint security”. Also has remote fleet management and firewall features

midtake2• 2 years ago

Botnet that checks if your bots in the botnet act like bad bots and can be considered bad too. Also checking if some of your files match AV signature. Also reading all your logs if you really want.

chgs7• 2 years ago

Is this just a massive mistake or is it deliberate and cover for something else

Scoundreller12• 2 years ago

Good question. Weaponized/compromised update? Disgruntled employee Logic bomb?

tkubacki• 2 years ago

Industry should move to Linux on desktop - we should not rely on single vendor

WediBlino• 2 years ago

2024!!! The year of Linux on the desktop!

alex_suzuki• 2 years ago

Finally! And here I thought it would never come.

pi-rat2• 2 years ago

We got day or maybe even week of linux on the desktop at least :)

GardenLetter272• 2 years ago

At least Linux servers, this situation is crazy.

oldmanyells683• 2 years ago

Don't forget about the BSD's. We should not rely on a single Finnish man.

tgshaik4• 2 years ago

https://www.dailymail.co.uk/news/article-13650411/Microsoft-...

junto5• 2 years ago

Please at least an archive link. That website is trash.

techie128• 2 years ago

This is good and bad. This showcases the importance of CrowdStrike. This is a short term blip but in the long run they will learn from this and prevent this type of an issue in the future. On the flip side, they have a huge target on their back for the U.S. government to try and control them. They are also a huge target for malicious actors since they can clearly see that CS is part of critical US and western infra. Taking them down can cripple essential services.

On a related note, this also demonstrates the danger of centralized cloud services. I wish there were more players in this space and the governments would try their very best to prevent consolidation in this space. Alternatively, I really wish the CS did not have this centralized architecture that allows for such failure modes. Software industry should learn from great & age old engineering design principles. For example, a large ships have watertight doors that prevent compartments from flooding in case of a breach. It appears that CS didn't think the current scenario was not possible therefore didn't invest in anything meaningful to prevent this nightmare scenario.

choeger3• 2 years ago

Or, and that maybe a radical idea, YOU DON'T INSTALL THIS FUCKING SNAKE OIL IN THE FIRST PLACE.

The idea of antivirus software is laughable when Adobe cannot implement a safe and secure PDF parser then how can Crowdstrike while simultaneously supporting the parsing of a million other protocols?

Everyone involved: Vendor, operator, and auditors who mandate this shit are responsible and should be punished.

YOU HAVE TO MINIMIZE THE ATTACK SURFACE, NOT INCREASE IT.

anigbrowl4• 2 years ago

I'm not that confident that they're going to be around to recover from after their stock price falls into the toilet and they get sued out the yin-yang. I don't think 'read the EULA terms lol' is gonna cut it here.

alt2274• 2 years ago

This is a short term blip

No security engineer in the world is going to trust the words CrowdStrike after this.

kasabali5• 2 years ago

Security engineers are the ones who first came up with these crap in the first place. Sales people are not to blame, they'll sell anything.

ranjanprj• 2 years ago

High time to stop using Microsoft Windows/Azure which is full of security tech debt, that you need all these tools which themselves brick the computer

onion2k• 2 years ago

CrowdStrike Falcon has a Linux product line for 'cloud security'.

sofixa4• 2 years ago

If anyone feels like disagreeing about Azure, here's a comment of mine from a few months ago:

A random selection of serious security incidents from Azure:

just from Wiz from the past 2-3 years, and of course they aren't the only ones:

https://www.wiz.io/blog/secret-agent-exposes-azure-customers...

https://www.wiz.io/blog/storm-0558-compromised-microsoft-key...

https://www.wiz.io/blog/azure-active-directory-bing-misconfi...

https://www.wiz.io/blog/omigod-critical-vulnerabilities-in-o...

https://www.wiz.io/blog/chaosdb-explained-azures-cosmos-db-v...

Of course Microsoft AI researchers sucking at security: https://www.wiz.io/blog/38-terabytes-of-private-data-acciden...

Nice overview from Corey Quinn that predates some of those but things were already horrifically bad: https://www.lastweekinaws.com/blog/azures-terrible-security-...

Go and look for similar things for AWS and GCP, and there's nothing on this level (cross-tenant, trivial to exploit).

Oh and there's also this, them selling your usage patterns to partners (hopefully they've stopped): https://twitter.com/QuinnyPig/status/1359769481539506180

Oh and another one where they bungled the response: https://twitter.com/QuinnyPig/status/1536868170815795200

I find it impossible to believe that Azure as a whole organisation takes security seriously. There might be individuals that do, but definitely nobody with decision making power. Half of the above described exploits are trivial and should have never passed any sort of competent review process.

whoknowsidont4• 2 years ago

If anyone feels like disagreeing about Azure

Talking with people in the MSFT camp is like talking with people in a cult. I'm not being melodramatic.

Pointing out these issues is good, but to them, they'll just shrug it off.

And businesses will keep giving them money. Madness.

oldmanyells683• 2 years ago

Sounds like a good time to buy Red Hat stock

trollstrike2• 2 years ago

Sounds like a time to buy SPX (always has been).

Joke being I have given up trying to time markets :-). With some rare exceptions to the rule (once in 10 year type things).

120bits• 2 years ago

Funny how I got rejected today from crowdstrike because I couldn’t code a hard leetcode problem under 40mins. I guess leetcode isn’t true software engineering after all.

edf13• 2 years ago

This is a testing and deployment issue rather than coding... mistakes and bugs happen - but most serious businesses have routines setup to catch them before rolling them out globally!

resonious• 2 years ago

So maybe they should swap out the Leetcode for a testing and deployment test during their interview!

saagarjha16• 2 years ago

You probably weren't good enough to take down half the world's Windows systems.

hackeraccount• 2 years ago

Alternately if he had gotten the job maybe he would have take down three quarters of the world's Windows systems.

ssss114• 2 years ago

Breaking patches is? …how did this get through QA with a big enough issue that it breaks many many windows machines

monksy• 2 years ago

I'm about willing to bet they don't have a qa team.

CapeTheory• 2 years ago

Evidently you dodge bullets well, you should consider running for office.

qalmakka7• 2 years ago

When will people learn?

1. Stop putting mission critical systems on Windows, it's not the reliable OS it once was since MS has cut off most of its QA

2. AV solutions are unnecessary if you properly harden your system, AV was needed pre-Vista because Windows was literally running everything as Administrator. AV was never a necessity on UNIX, whatever MS bundles in is usually enough

3. Do not install third party software that runs in kernel mode. This is just a recipe for disaster, no matter how much auditing is done beforehand by the OEM. Linux has taught multiple times that drivers should be developed and included with the OS. Shipping random binaries that rely on a stable ABI may work for printers, not for mission critical software.

jmcgough26• 2 years ago

None of this advice is useful for massive organizations like banks and hospitals who got hit by this. They cannot switch off of windows for a number of reasons.

gloosx5• 2 years ago

They are not willing perhaps? Why cannot they?

chgs7• 2 years ago

Because they made a continuing choice over the years to write their systems to rely on windows?

the_gipsy2• 2 years ago

It's not a "3 step solution" that some IT guy can do now. It's a high level critique of using Windows and ignoring known problems.

mrweasel4• 2 years ago

There's nothing they can do right now, but my issue is that this will be forgotten when next update/purchasing round swings into action.

Take Mærsk who couldn't operate their freight terminals due to a cyber attack and had the entire operation being dependent on a hard drive in a server that happened to be offline. Have they improved network separation? Perhaps. Have they limited their critical infrastructure to only run whitelisted application? I assure you they have not. They've probably just purchased a Crowdstrike license.

Companies continuously fail to view their critical infrastructure as critical and severely underestimate risk.

qalmakka7• 2 years ago

That's the reason why I wrote, "stop putting" instead of "throw all of your PCs out of the window". Just like they migrated away from DOS they should start planning to migrate away from Windows to more modern, sandboxed solutions. There are ZERO reasons why a cash register shouldn't boot from a read-only filesystem, run AV, and so on.

bruce3434345• 2 years ago

Linux isn't exactly reliable either... I'm sorry but that OS is barely capable of outputting a stable HDMI signal, god help you if you are on a laptop with external monitor.

For 3 computers, 2 laptops, I've never _not_ had display bugs/oddities/issues. System upgrades always make me nervous because there is a very real chance of something getting fucked up and my screen staying black the next time it boots, having to go into a TTY, and manually fixing stuff up or booting the previous version that was still saved in GRUB.

We can not get computers perfect. They are too complicated. That's true for anything in life. As soon as it gets too complicated, you're left in a realm of statistics and emergent phenomena. As much as I dislike windows enough to keep using Linux, I never had display issues on windows.

To anyone compelled to reply with a text that contains "just" or "simply": simply just consider that if you are able to think of it in 10 seconds, then I have thought of it as well, and tried it too.

ben0x539• 2 years ago

Yeah I mean you probably shouldn't put mission critical systems on a laptop with an external monitor either.

KronisLV5• 2 years ago

For 3 computers, 2 laptops, I've never _not_ had display bugs/oddities/issues. System upgrades always make me nervous because there is a very real chance of something getting fucked up and my screen staying black the next time it boots, having to go into a TTY, and manually fixing stuff up or booting the previous version that was still saved in GRUB.

I've had a Debian update break GRUB itself as well: https://blog.kronis.dev/everything%20is%20broken/debian-and-...

I still use some Linux distros because when they work, they're pretty good, but when they don't, be prepared for a bunch of annoyances and debugging.

qalmakka7• 2 years ago

In my comment I was referring to mission critical systems, which most definitely you don't put on cheap commodity hardware you buy in a brick and mortar store.

Linux is used EVERYWHERE for a reason. Most car HUD now run on some form of Linux embedded, like basically all embedded and low power devices. The problem here is that people still put embedded mission critical systems on a desktop OS and slap desktop software on it, which is _a bad choice_.

Dudhbbh3343• 2 years ago

I'm sorry but that OS is barely capable of outputting a stable HDMI signal, god help you if you are on a laptop with external monitor.

You may have had particularly bad luck with poorly supported hardware, but I don't think this is a normal experience.

I've been using Linux exclusively on desktops and laptops (with various VGA, DVI, DisplayPort, HDMI, and PD-powered DisplayPort-over-USB-C monitors and TVs since 2002 without any unstable behavior or incompatibility.

whoknowsidont4• 2 years ago

Linux isn't exactly reliable either...

That's certainly a perspective.

surgical_fire2• 2 years ago

Linux isn't exactly reliable either... I'm sorry but that OS is barely capable of outputting a stable HDMI signal, god help you if you are on a laptop with external monitor.

This is demonstrably false, given the amount of people that game on Linux nowadays.

System upgrades always make me nervous because there is a very real chance of something getting fucked up and my screen staying black the next time it boots, having to go into a TTY, and manually fixing stuff up or booting the previous version that was still saved in GRUB.

I had this happen to me once. Timeshift was painless to use, and in about 15 minutes I had my machine up and running again, and could apply all updates properly afterwards. If anything it made me bolder lol.

xorcist5• 2 years ago

How is your lack of a stable HDMI signal relevant to that the world's airlines and supermarkets and banks probably shouldn't run Windows with third-party antivirus software bolted on? That is a platform originally intended for office style typewriter emulation and games.

Every engineering-first or Internet-native company that could choose chose Linux and for simple reasons. Anything not Linux in The Cloud is a rounding error. Most of the world's mobile phones is Linux. And most cloud-first desktops too. They don't seem to be particularly more troubled with HDMI signal quality or other display issues than other devices.

graemep2• 2 years ago

Linux isn't exactly reliable either... I'm sorry but that OS is barely capable of outputting a stable HDMI signal, god help you if you are on a laptop with external monitor.

It just just works for me, and has just worked with every laptop I have had in the last 15 years. My kids and I have several Linux installs and the only one with HDMI output issues is a cheap ARM tablet that is sold as a device for early adopters.

For 3 computers, 2 laptops, I've never _not_ had display bugs/oddities/issues. System upgrades always make me nervous because there is a very real chance of something getting fucked up and my screen staying black the next time it boots, having to go into a TTY, and manually fixing stuff up or booting the previous version that was still saved in GRUB.

At least that number of machines (I do not know whether you mean three or five in total) for the last 20+ years and can recall one such issue.

rpdillon• 2 years ago

I mean, this isn't a support forum.

But working with hundreds of computers, running many different distributions of Linux for decades, they just haven't ever seen what you're describing. It's really hard to reconcile what I read here with my hands-on experience.

baq7• 2 years ago

This issue could easily happen on any other OS - Linux, macOS, BSDs - because it's a third party kernel driver which would be installed by the corporate IT regardless of anyone's opinion for compliance reasons. Your advice is incompatible with how the real world operates.

htrp• 2 years ago

corporate IT heads need to roll for that to ever change.

The romans used to make the architects stand under the arches they built, to enforce the idea of consequences for bad work.

monocasa6• 2 years ago

I've seen orgs get through soc2 and pci-dss without kernel anti virus.

It's all about compensating controls.

chippiewill3• 2 years ago

Yeah, my work requires me to run an antivirus kernel module on my Ubuntu laptop.

Corporate IT is always going to lean towards the "safe" compliance option.

prmoustache11• 2 years ago

Corporate ITs need to stop mandating security malware tools on their systems just because someone showed them some nice powerpoints.

KronisLV5• 2 years ago

AV was never a necessity on UNIX, whatever MS bundles in is usually enough

What prevents someone pushing a malicious package that takes my user data (that is accessible from a logged in session directly) and sends it somewhere? Especially in non-system repos, like Maven/NuGet/npm/pip/RubyGems and so on? What about the too widespread practice of piping shell scripts from the web, or applications with custom update mechanisms that might be compromised and pull in malicious code?

I'm not saying that AV software would protect against all of these, but even if users don't do stupid things (which they absolutely will anyways, sooner or later), then there are still vectors of attack against any system.

As for why *nix systems don't see that much malware, I've no idea, maybe because it's not as juicy of a target because of the lower count of desktop software installations (though the stuff that is on the systems might be more interesting to some, given the more tech savvy userbase), or maybe because a lot of the exploits focus on server software, like most CVEs.

On Windows, I guess the built in AV software is okay, maybe with occasional additional scans by something like Malwarebytes, but that's situational.

bugbuddy3• 2 years ago

I would like to inform you that none of the AV product on the market will be able to protect you from piping a bad script from the web. Case closed.

bruce3434345• 2 years ago

Nothing, in fact there have been many cases where python's and nodejs's package systems were exploited to achieve arbitrary code execution (because that's a feature, not a bug, to allow "complicated installation processes to just work").

https://arstechnica.com/information-technology/2021/12/malic...

AVs are the wrong way to go about security anyway, it's a reactionary strategy in a cat and mouse game by definition. For prevention, I think the BSDs are doing some promising work with the "pledge" mechanism. And as much hate as they get, I like appimages and snap et al for forcing people to consider a better segmentation model and permission system for installed software.

eecc• 2 years ago

Meh, without proper MAC with process namespaces, I guess nothing.

SELinux and whatever Apple is doing looks right IMHO

xorcist5• 2 years ago

What prevents someone pushing a malicious package that takes my user data

That's not an argument in good faith. If you install unvetted packages in your airline control system, bank, or supermarket, the kind of systems that we're talking about here, you have much bigger problems to worry about.

I'm not saying that AV software would protect against all of these,

Or indeed any of these. Highly privileged users piping shell scripts from untrusted sources is out of scope for any antivirus system, on any platform.

That doesn't mean all platforms are identical, or share the same attack vectors. It is much more accepted to install kernel mode drivers on the Windows platform, where it is not only accepted but have established quality control programs to manage it, than on Linux, where the major vendor will very literally show you the middle finger on video for everyone to see for doing so.

The Linux community is more for doing that kind of work upstream. If some type of new access control or binary integrity checking is required, that work goes upstream for everyone to use. It is not bolted on running systems with kernel mode drivers. That is because Linux is more like a shared platform, and less like a "product". That culture goes way beyond mere technical differences between the systems.

WatchDog3• 2 years ago

This isn't a windows issue. For what it's worth, I've had plenty of problems in the past with kernel panics from crowdstrike's macos system extension, although it was fairly random, nothing like today's issue.

AdeptusAquinas5• 2 years ago

1. this is a crowdstrike issue not windows

2. plenty of malware and c2 systems happily operate off all systems, regardless of how hardened (or how unix) they are - IDS/IPS is a reactive way to try and mitigate this

3. you don't need third party software to compromise the unix kernel, you just need to wait a week or two until someone finds a bug in the kernel itself

all that being said, this has solarwinds vibes. the push for these enterprise IDS systems needs to be weighted, the approach adjusted

tamimio11• 2 years ago

Windows Defender is more than sufficient for most of these companies, but they need that false sense of security, or maybe they have excess budget to spare, or they are transferring the risk per their risk management plan.

bugbuddy3• 2 years ago

Bingo

benfortuna2• 2 years ago

Transferring the risk of malicious actors, but creating a resilience risk they are not owning.

nox1013• 2 years ago

Do not install third party software that runs in kernel mode. T

You mean don't install Steam nor the Epic Store, nor many of the games.

Note: I'm agreeing with you except that pretty much the only reason I have a Windows machine is for games. I do have Steam installed. I also have the Oculus software installed. I suspect both run in kernel mode. I have to cross my fingers that Valve and Facebook don't do bad things to me and don't leave too many holes.

I don't install games that require admin.

Oh, and I have Photoshop and I'm pretty sure Adobe effs with the system too >:(

Zr402• 2 years ago

Steam does not have any kernel-mode components.

qalmakka7• 2 years ago

You mean don't install Steam nor the Epic Store, nor many of the games.

...would you install Steam on a POS machine? Is your gaming PC a "mission critical system"?

ivxvm2• 2 years ago

I've never seen a program running in kernel mode other than AV software. Pretty sure all stuff you listed doesn't. Asking admin permissions doesn't mean it's kernel mode software.

punkrox999• 2 years ago

this "kernel level = invasive" paranoia that's been going on lately is complete FUD at its core and screams tech illiteracy

no software vendor needs to or wants to write a driver to spy on you or steal your data when they can do all of that with user-level permissions without triggering any AV.

3rd party drivers are completely fine, and its normal that advanced peripherals like an Oculus uses them

2rsf• 2 years ago

CrowdStrike Falcon is not an AV, Windows can be decently hardened and Microsoft did not "cut off most of its QA"

qalmakka7• 2 years ago

Microsoft did not "cut off most of its QA"

Windows RTMs used to be shipped in a usable state (albeit buggy) for more than a decade. You installed it from a CD and it worked fine, you installed patches every once in a while from a random Service Pack CD you got from somewhere.

Modern Windows has had the habit of being so buggy after release in such horrendous ways that I can't imagine being able to use the same install CD for years. This is definitely putting less attention to detail in my view.

alt2274• 2 years ago

Microsoft did not "cut off most of its QA"

Yes they did. Hear about it first hand from an ex windows software tester:

https://www.youtube.com/watch?v=S9kn8_oztsA

https://www.youtube.com/watch?v=lRV6PXB6QLk

tomashubelbauer2• 2 years ago

The slice of Microsoft stuff I worked at certainly did not have dedicated QA at the time I was there and used to have a QA team before, so there is some degree of truth to the statement. I can't speak for other Microsoft teams and offices. It was very disappointing for me, because I have had the opportunity to work with great QA staff before and in my current job and there is no way a developer dedicating 25 % of their time (which is what was suggested as a replacement for having dedicated QA) can do a job anywhere near as good.

szszrk• 2 years ago

I have a feeling most commenters (not just here) don't really know what Falcon is and does, if EDR (and more?) keeps getting compared to a plain antivirus.

cm21874• 2 years ago

Not saying the microsoft QA is stellar but I also remember heartbleed.

deutschlerner2• 2 years ago

Stop putting mission critical systems on Windows, it's not the reliable OS it once was since MS has cut off most of its QA

Dream on, that ship sailed long, long, long ago.

It’s always funny to me reading comments on this site from users like this who have no idea how the real world operates

scheeseman486• 2 years ago

Or doesn't operate, as it currently stands.

systemvoltage2• 2 years ago

Why is this type of a comment upvoted to the top?

It's a knee jerk reaction to the midwit zeitgeist. No real understanding of how the real world operates. No maturity or thoughtfulness.

Totally misleading prognosis. Misleading advice.

maeil7• 2 years ago

You missed the most important one:

Have some kind of soaking/testing environment for production critical systems, especially if you're a big business. If you're hip, something like a proper blue/green setup (please chime in with best practices!). If you're legacy, do it all by hand if you must.

Blindly enabling immediate internet-delivered auto-update on production systems will always allow a bad update to cause chaos. It doesn't matter how well you permission things off on your favourite Linux flavor. If an update is to be meaningful, the update can break the software. And clearly you're relying on the software, otherwise you wouldn't be using it.

hulitu7• 2 years ago

Windows, it's not the reliable OS it once was

And when exactly was Windows a reliable OS ?

When it was turned off ? /s

steven43126• 2 years ago

Crowdstrike is not just "Antivirus" capability.

Depending on the threats pertinent to the org they may require deep observability and the ability to perform threat hunting for new and emerging threats, or detect behaviour based signals, or move to block a new emerging threat. Not all threats require Administrator privileges!

Not installing AV might be fine for a small number of assets in a low risk industry, but is bad advice for a larger more complex environment.

If were unbiased here the apparent crowdstrike problem could occur on any OS and with any vendor where you have updates or configuration changes automatically deployed at scale.

steve19775• 2 years ago

Linux has taught multiple times that drivers should be developed and included with the OS.

I've had Linux GPU drivers fail multiple times due to system updates, to the point were I needed to roll back. I've had RHEL updates break systems in a way were even Red Hat support couldn't help me (I had to fix them myself).

I don't see how Linux is any better in this regard than Windows to be honest.

Also: > AV was never a necessity on UNIX

Sure, why write a virus when you can just deploy your malware via official supply chains?

orwin• 2 years ago

Do you have/need GPUs on your 'mission critical systems'? I would bet most of us don't.

I quite agree with OP here. VMs are now quite lightweight (compared to available resources on machines at least) and I would rather use a light, hardened Linux as my base OS that will run windows VM and do snapshots for quick rollbacks. Actually, that's what I run on my own PC, and I think it would be the sanest way to operate.

rhim2• 2 years ago

Imagine a supply chain attack on a closed system and nobody finding out about it.

jmcgough26• 2 years ago

Took down our entire emergency department as we were treating a heart attack. 911 down for our state too. Nowhere for people to be diverted to because the other nearby hospitals are down. Hard to imagine how many millions of not billions of dollars this one bad update caused.

epolanski2• 2 years ago

May I say that starting from "treating a heart attack" and ending up worrying about millions lost in productivity sounds a bit "wrong"?

plonk5• 2 years ago

Billions in losses means a somewhat worse life for a huge number of people and potentially much worse healthcare problems down the line, the NHS was affected

jmcgough26• 2 years ago

I just had a ten hour hospital shift from hell, apologies if my writing is lacking. I can't think of a better way to try to measure the scope of the damage caused by this.

jojobas5• 2 years ago

Millions lost is sizeable parts of people's lives they won't get back.

kube-system5• 2 years ago

When it comes to measuring the impact to society at scale, dollars is really the only useful common proxy. One can't enumerate every impact this is going to have on the world today -- there's too many.

plonk5• 2 years ago

Did the person survive?

jmcgough26• 2 years ago

We have limited visibility into this in the emergency department. You stabilize the patient and admit them to the hospital, then they become internal medicine or ICU's patient. Thankfully most of the work was done and consults were called prior to the outage, but they were in critical condition.

swyx• 2 years ago

i mean not just dollars but lives also right? do we have a way to track that?

aydyn• 2 years ago

Yup through electronic medical records... o wait

lupire4• 2 years ago

What's the NASDAQ ticker for lives?

beeboobaa35• 2 years ago

And how many lifes lost?

kyledrake• 2 years ago

I remember a fed speaker in the 90s at Alexis hotel Defcon trying to rationalize their weirdly over-aggressive approach to enforcement by mentioning how hackers would potentially kill people in hospitals, fast forward to today and it's literally the "security" software vendor that's causing it.

emodendroket• 2 years ago

Not like hackers haven’t done the same.

BuckRogers2• 2 years ago

I can't imagine why any critical system is connected to the internet at all. It never made sense to me. Wifi should not be present on any critical system board and ethernet plugged in only when needed for maintenance.

This should be the standard for any life sustaining or surgical systems, and any critical weapons systems.

zitterbewegung• 2 years ago

Well cryptolockers have actually compromised various hospitals and I remember the first one was in the United Kingdom .

kspacewalk22• 2 years ago

It's not like hackers haven't killed people in hospitals with e.g. ransomware. Our local dinky hospital system was hit by ransomware twice, which at the very least delayed some important surgeries.

jodrellblank7• 2 years ago

"Took down our entire emergency department as we were treating a heart attack."

Not questioning that it happened, but this was a boot loop after a content update. So if the computers were off and didn't get the update, and you booted them, they would be fine. And if they were on and you were using them, they wouldn't be rebooting, and it would be fine.

How did it happen that you were rebooting in the middle of treating a heart attack? [Edit: BSOD -> auto reboot]

sleepydog• 2 years ago

Not OP, but some (most? many?) machines receiving the update crashed with a BSOD. So that's how they could enter the boot loop.

lupire4• 2 years ago

What happens when a computer gets rebooted as part of daily practice or because of the update, and then it becomes unusable, and then the treatment team needs to use it hours later?

TeMPOraL14• 2 years ago

Beyond the BSOD that happened in this case, in general this is not true with Windows:

And if they were on and you were using them, they wouldn't be rebooting, and it would be fine.

Windows has been notorious for forcing updates down your throat, and rebooting at the least appropriate moments (like during time-sensitive presentations, because that's when you stepped away from the keyboard for 5 minutes to set up the projector). And that's in private setting. Corporate setting, the IT department is likely setting up even more aggressive and less workaround-able reboot schedule.

Things like this is exactly why people hate auto-updates.

newhotelowner• 2 years ago

Half of the hotels (Choice) computers were down. We never reboot the computer, unless it's not working or working slowly or Windows update.

a0123• 2 years ago

idk, a lot of system are never meant to be rebooted outside of the update schedule, so they wouldn't have been off in the first place. And if those systems control others, then there is a domino effect.

I can see very well how one computer could have screwed all others. It's really not hard to imagine.

davycro3• 2 years ago

The system crashed while my coworker was running a code (aka doing CPR) in the ER last night. Healthcare IT is so bad at baseline that we are somewhat prepared for an outage while resuscitating a critical patient.

smsm429• 2 years ago

Now this is an unusual meeting of two meanings of "running a code".

dwatson923• 2 years ago

Excuse my ignorance, but what systems are needed for CPR?

peterleiser3• 2 years ago

The second largest hospital group in Nashville experienced a ransomware attack about two months ago. Nurses told me they were using manual processes for three weeks.

SarahWSJ3• 2 years ago

Hello, I'm a journalist looking to reach people impacted by the outage and wondering if you could kindly connect with your ER colleague. My email is sarah.needleman@wsj.com. Thanks!

FireBeyond• 2 years ago

As a paramedic, there is very little about running a code that requires IT. You have the crash cart, so not even stuck trying to get meds out of the Pyxis. The biggest challenge is charting / scribing the encounter.

ww5203• 2 years ago

I saw one of the surgery videos recently. The doctor was saying, "Alexa, turn on suction." It boggled my mind. There could be so many points of failure.

senortumnus• 2 years ago

Fwiw this is not typical, we don’t have alexa/siri type smart devices in any OR I work in, and suction is turned on off with a button and a dial.

nullbyte• 2 years ago

ALEXA, TURN OFF THE SUCTION! ALEXA!!

“Loive from NPR news in Washington“

CamperBob22• 2 years ago

I don't suppose there was a doctor or nurse named Alexa involved?

importantbrian2• 2 years ago

Wife is a nurse. They eventually go 2 computers working for her unit. I don't think it impacted patients already being treated, but they couldn't get surgeries scheduled and no charting was being done. Some of the other floors were in complete shambles.

SarahWSJ3• 2 years ago

Hi, as I noted to another commenter, I'm a journalist looking to speak with people who've been impacted by the outage. I'm wondering if I could speak with your wife. My email is sarah.needleman@wsj.com. Thanks.

willmadden2• 2 years ago

I'm guessing hundreds of billions if you could somehow add it all up.

I can't believe they pushed updates to 100% of Windows machines and somehow didn't notice a reboot loop. Epic gross negligence. Are their employees really this incompetent? It's unbelievable.

I wonder where MSFT and Crowdstrike are most vulnerable to lawsuits?

lanstin4• 2 years ago

This outage seems to be the natural result of removing QA by a different team than the (always optimistic) dev team as a mandatory step for extremely important changes. And neglecting canary type validations. The big question is will businesses migrate away from such a visibly incompetent organization. (Note I blame the overall org; I am sure talented individuals tried their best inside a set of procedures that asked for trouble.)

SoftTalker4• 2 years ago

Their terms of use undoubtedly disclaim any warranty, fitness for purpose, or liability for any direct or incidental consequences of using their product.

I am LMFAO at the entire situation. Somewhere, George Carlin is smiling.

fsloth11• 2 years ago

If you’ve ever been forced to use a PC with Crowdstrike it’s not amazing at all. I’m amazed incident of this scale didn’t happen earlier.

Everything about it reeks of incompetence and gross negligence.

It’s the old story of the user and purchaser being different parties-the software needs to be only good enough to be sold to third parties who never neeed to use it.

It’s a half-baked rootkit part of performative cyberdefence theatrics.

Drygord3• 2 years ago

Same people who destroyed a US bridge recently.

This is the result of giving away US jobs overseas at 1/10th the salary

fortran775• 2 years ago

MSFT doesn’t recommend or specify CrowdStrike

cma• 2 years ago

This will go on for multiple days, but hundreds of billions would be >$36 trillion annualized if it was that much damage for one day. World annual GDP is $100 trillion.

HaZeust2• 2 years ago

Yup - my mom went into the ER for stroke symptoms last night and was put under MRI. The MRI imaging could NOT be sent to the off-site radiologist and they had to come in -- turned out the MRI outputs weren't working at all.

We were discharged at midnight by the doctor, the nurse didn't come into our exam room to tell us until 4am. I can't imagine the mess this has caused.

davidw• 2 years ago

I hope she's ok.

olwmc• 2 years ago

I wish your mother the best <3

jmcgough26• 2 years ago

Hope she's okay. For better or worse, our entire emergency department flow is orchestrated around epic. If we can't even see the board, nurses don't know what orders to perform, etc.

jessechang• 2 years ago

Wishing you and your mom the best

failbuffer• 2 years ago

Honestly, that sounds like a typical ER visit.

rbanffy16• 2 years ago

I wish your mother recovers promptly. And I’m glad she doesn’t run on Windows. ;-)

nirav72• 2 years ago

A relative of mine had back surgery late yesterday. Today the hospital nursing staff couldn’t proceed with the pain medication process for patients recovering from surgery because they didn’t have access to the hospital systems.

theGnuMe2• 2 years ago

Idk… critical hospital systems should be air gapped.

diebeforei4852• 2 years ago

We were discharged at midnight by the doctor, the nurse didn't come into our exam room to tell us until 4am. I can't imagine the mess this has caused.

That's an extra 4 hours of emergency room fees you ideally wouldn't have to pay for.

rsync3• 2 years ago

"Took down our entire emergency department as we were treating a heart attack. 911 down for our state too."

Why would Windows systems be anywhere near critical infra ?

Heart attacks and 911 are not things you build with Windows based systems.

We understood this 25 years ago.

Wytwwww5• 2 years ago

Well you can use stupid broken software with any OS, not just Windows. Isn't CrowdStrike Falcon available on Linux, is there any reason why couldn't they have introduced a similar bug and similar consequences there?

Nathanael_M3• 2 years ago

Because Windows is accessible and Linux requires uncommon expertise and short term cost that is just not practical for lots of places.

Goodluck teaching administrators an entirely new ecosystem, goodluck finding software off the shelf for Linux.

Bespoke is expensive, expertise is rare, Linux is sadly niche.

sentientslug2• 2 years ago

It seems like you’ve never worked with critical infra. Most of it runs on 6 to 10 year old unpatched versions of Windows…

charles_f3• 2 years ago

Why would Windows systems be anywhere near critical infra ?

Why would computers be anywhere near critical infra? This sounds like something that should failsafe, the control system goes down but the thing keeps running. If power goes down, hospitals have generator backups, it seems weird that computers would not be in the same situation

orbillius• 2 years ago

Why would Windows systems be anywhere near critical infra ?

This is just a guess, but maybe the client machines are windows. So maybe there are servers connected to phone lines or medical equipment, but the doctors and EMS are looking at the data on windows machines.

freehorse8• 2 years ago

I do not think windows is the problem here. The problem is that equipment that is critical infrastructure being connected to the internet, imo. There is little reason for a lot of computers in some settings to be connected to the internet, except for convenience or negligence. If data transfer needs to be done, it can happen through another computer. Some systems should exist on a (more or less) isolated network at best. Too often we do not really understand the risk of a device being connected to the internet, until something like this happens.

nO0b• 2 years ago

Why would Windows systems be anywhere near critical infra ?

maybe Heartbleed or the xzUtils debacles convinced them to switch.

drhelix• 2 years ago

Why is the emergency department using windows?

bell-cot2• 2 years ago

High-end hospital-management software is not simple stuff, to roll your own. And the (very few) specialty companies which produce such software may see no reason to support a variety of OS's.

johncessna2• 2 years ago

Why did they update everything all at once?

et2o2• 2 years ago

Because essentially every large hospital in the USA does?

charles_f3• 2 years ago

Hard to imagine how many millions of not billions of dollars this one bad update caused.

I mean, if the problem is that hospitals can't function anymore, money is hardly the biggest problem

charles_f3• 2 years ago

Took down our entire emergency department

What does this mean? Did the power go down? Is all the equipment connected? Or is it the insurance software that can't run do nothing gets done? Maybe you can't access patient files anymore but is that taking down the whole thing?

jmcgough26• 2 years ago

Every computer entered a bluescreen loop. We are dependent on Epic for placing orders, for nursing staff to know what needs to be done, for viewing records, for transmitting and interpreting radiology machines. It's how we know the current state of the department and where each patient (out of 50+ people we are simultaneously treating) is at. Our equipment still works but we're flying blind and having to shout orders at each other and have no way to send radiology images to other doctors for consultation.

atoav5• 2 years ago

Not to be that guy, but I often say software engineering as a field should have harsher standards of quality and certainly liability for things like these. You know like civil engineers, electrical engineers and most people whose work could kill people if done wrongly.

Usually when I write this devs get all defensive and ask me what the worst thing is that could happen.. I don't know.. Could you guarantee it doesn't involve people dying?

Dear colleagues, software is great because one persons work multiplies. But it is also a damn fucking huge responsibility to ensure you are not inserting bullshit into the multiplication.

whydoyoucare4• 2 years ago

I believe instances like this will push people to reconsider the lax stance. Humans in general have a hard time regulating something abstract. The fact that people can be killed is well-known since the 80s', see https://en.wikipedia.org/wiki/Therac-25

roeles• 2 years ago

Related talk by Alan Kay: https://youtu.be/D43PlUr1x_E

Tao3300• 2 years ago

I'd endorse this. That way when my hypothetical PHB wants to know why something is taking so long I can say "See this part? Someone could die if we don't refactor it."

oriel2• 2 years ago

I'd go a step farther and say software engineering as a field is not respected at the same levels as such certified/credentialed engineers, because of these lacks of standards and liabilities. Leading to common occurrences of systemic destructive failures such as this, due to organization level direction being very lax in dealing with software failure potential.

hnthrow2895702• 2 years ago

"If only we asked harder problems for our leetcode interview!"

sigseg1v• 2 years ago

Some countries such as Canada have taken minor steps towards this, for example making it illegal to call oneself a software engineer unless you are certified by the provinces professional engineering body, however this is still missing a lot. I also don't wish to be "that guy" but I'll go further and say that the US is really holding this back by not making using Software Engineer as a title (without holding a PEng) illegal in a similar fashion.

If we can at least get that basis then we can start to define more things such as jobs that non Engineers can not legally do, and legal ramifications for things such as software bugs. If someone will lose their professional license and potentially their career over shipping a large enough bug, suddenly the problem of having 25,000 npm dependences and continuous deployment breaking things at any moment will magically cease to exist quite quickly.

sunnybeetroot2• 2 years ago

It’s important not to disregard that software engineers are often overruled by management or product when strict deadlines and targets exist.

sizzle2• 2 years ago

Contact a lawyer if this affected her health please. Any delays in receiving Stroke care can have injured her more I imagine. Any docs here?

bookofjoe2• 2 years ago

Yes. Reading and learning.

psychlops• 2 years ago

I can't even begin to imagine the cost of proving the health effects and attempting to win the case.

jmcgough26• 2 years ago

ER worker here. It really depends on the details. If she was C-STAT positive with last known normal within three hours, you assume stroke, activate the stroke team, and everything moves very quickly. This is where every minute counts, because you can do clot busting to recover brain function.

The fact that she was discharged without an overnight admit suggests to me that the MRI did not show a stroke, or perhaps she was outside the treatment window when she went to the hospital.

WhyNotHugo• 2 years ago

It's honestly terrifying that someone would opt for Windows in systems critical to medical emergencies.

I hope organisations start revisiting some of these insane decisions.

afavour7• 2 years ago

Eh. There are a great many problems that could befall a medical emergency systems that are unrelated to OS. Like power loss. I think the core problem here really is a lack of redundancy.

wannacboatmovie• 2 years ago

I've had updates break Linux machines.

Just a few weeks ago I had an OpenBSD box render itself completely unbootable after nothing more than a routine clean shutdown. Turns out their paranoid-idiotic "we re-link the kernel on every boot" coupled with their house-of-cards file system corrupted the kernel, then overwrote the backup copy when I booted from emergency media - which doesn't create device nodes by default so can't even mount the internal disks without more cryptic commands.

Give me the Windows box, please.

TheCondor• 2 years ago

Not my story to tell, so I'm relaying it. Childhood friend works for a big company, you've heard their name, they make nuclear control systems for nuclear reactors; they have products out in the field they support and there are new reactors in parts of the world from time to time. We were scheduled to have lunch a couple years back and he bailed, we rescheduled, he bailed because that was the day you couldn't defer XP updates anymore, they came in and some XP systems became Windows 10. XP was "nuclear reactor approved" by someone, they had a tool chain that didn't work right on other versions of windows, it all gave me chills.

They ended up giving MS a substantial amount of money to extend support for their use case for some number of years. I can't remember the number he told me but it was extremely large.

type04• 2 years ago

some critical software has DRM that only works in Windows

ayakang314153• 2 years ago

If true, this is insane that critical facilities like hospital do not have decentralized security system.

mandevil8• 2 years ago

Crowdstrike is on every machine in the hospital because hospitals and medical centers became a big target for ransomware a few years ago. This forced medical centers to get insured against loss of business and getting their data back. The insurance companies that insure companies against ransomware insist on putting host based security systems onto every machine or they won't cover losses. So Crowdstrike (or one of their competitors) has to run on every machine.

niutech2• 2 years ago

It's insane why critical facilities use Windows OS rather than Linux/*BSD, which is rock-solid.

hassiktir2• 2 years ago

I dont understand how this isnt bigger news?

Local emergency services were basically nonfunctioning for better part of the day along with the heat wave and various events, seems like a number of deaths (locally at least, specific to what I know for my mid sized US city) will be indirectly attributable to this.

jmcgough26• 2 years ago

It's entirely possible (likely, even) that someone died from this, but it's hard to know with critically ill patients whether they would have survived without the added delays.

jjav9• 2 years ago

Took down our entire emergency department as we were treating a heart attack.

It makes my blood boil to be honest that there is no liability for what software has become. It's just not acceptable.

Companies that produce software with the level of access that Crowdstrike has (for all effective purposes a remote root exploit vector) must be liable for the damages that this access can cause.

This would radically change how much attention they pay to quality control. Today they can just YOLO-push barely tested code that bricks large parts of the economy and face no consequences. (Oh, I'm sure there will be some congress testimony and associated circus, but they will not ever pay for the damages they caused today.)

If a person caused the level and quantity of damage Crowdstrike caused today they would be in jail for life. But a company like Crowdstrike will merrily go on doing more damage without paying any consequence.

throwaway73562• 2 years ago

Companies that produce software

What about companies that deploy software with the level of quality that Crowdstrike has? Or Microsoft 365 for that matter.

That seems to be the bigger issue here; after all Crowdstrike probably says it is not suitable for any critical systems in their terms of use. You shouldn't be able to just decide to deploy anything not running away fast enough on critical infrastructure.

On the other hand, Crowdstrike Falcon Sensor might be totally suitable for a non-critical systems, say entertainment systems like the Xbox One.

fransje266• 2 years ago

Hard to imagine how many millions of not billions of dollars this one bad update caused.

And even worse, possibly quite a few deaths as well.

I hope (although I will not be holding my breath), that this is the wake-up call we need to realise that we cannot have so much of our critical infrastructure rely on the bloated OS of company known for its buggy, privacy-intruding, crapware riddled software.

I'm old enough to remember the infamous blue-screen-of-death Windows 98 presentation. Bugs exist but that was hardly a glowing endorsement of high-quality software.. This was long ago, yet it is nigh on impossible to believe that the internal company culture has drastically improved since then, with regular high-profile screw-ups reminding us of what is hiding under the thin veneer of corporate of respectability.

Our emergency systems don't need windows, our telephone systems don't need windows, our flight management systems don't need windows, our shop equipment systems don't need windows, our HVAC systems don't need windows, and the list goes on, and on, and on.

Specialized, high-quality OSes with low attack surfaces are what we need to run our systems. Not a generic OS stuffed with legacy code from a time when those applications were not even envisaged.

Keep-it-simple-stupid -KISS-is what we need to go back to, our lives literally depend on it.

With the mutli-billion dollars screw-up that happened yesterday, and an as-of-yet unknown number of deaths, it's impossible to argue that the funds are unavailable to develop such systems. Plurality is what we need, built on top of strong standards for compatibility and interoperability.

fluoridation2• 2 years ago

OK, but this was a bug in an update of a kernel module that just happened to be deployed on Windows machines. How many OSs are there that can gracefully recover from an error in kernel space? If every machine that crashed had been running, say, Linux and the update had been coded equivalently, nothing would've changed.

Perhaps rather than an indictment on Windows, this is a call to re-evaluate microkernels, at least for critical systems and infrastructure.

tamimio11• 2 years ago

I just skimmed through the news. A lot of airports, hospitals, and even governments are down! It's ironic how people are putting their eggs in one basket, trying to avoid downtime caused by malware by relying on a company that put their system down. A lot of lessons will be learned after this for sure.

phatfish2• 2 years ago

Unless you run half your devices on one security vendor and half on another surely there is no way round it? Companies install this stuff over "Windows Defender" so they can point fingers at the security vendor when they get hacked, this is the other side of the coin.

It has happened before where security software has unwanted effects, can't say i remember anyone else managing to blue screen Windows and require a safe mode boot to fix the endpoints though.

whyoh2• 2 years ago

Relying on easy-install "security vendors" is the problem. It's one thing to run an antivirus on a general purpose PC that doesn't have a qualified human admin. But many of the computers affected here are single-purpose devices, which should operate with a different approach to security.

pageandrew3• 2 years ago

I don’t know Windows systems. I’ve read it’s causing Blue Screen of Death.

I take that to mean that systems can’t even boot. Right?

Can this be fixed over the air?

monocasa6• 2 years ago

Right now the workaround is doing brain surgery on the system in safe mode, so probably no ota fix.

saagarjha16• 2 years ago

== kernel panic if that clears it up

surfingdino• 2 years ago

Back in the 1990s when Microsoft wanted to enter the embedded systems market there was a saying "You don't want Windows controlling your car's breaks". We now let them control a huge part of our lives. Should we let them add AI to the already unpalatable cocktail?

novaRom• 2 years ago

Chances if Microsoft or Crowdstrike will be held liable for financial losses caused by this outage?

rwmj7• 2 years ago

None whatsoever, their contracts with customers will limit liability to the price paid for the software/subscription. If there was open-ended liability for software failures then very little software would get written.

account428• 2 years ago

This is an insane take. Do you think other industries get away with limiting their liability to the product cost? No, because that doesn't provide adequate incentives for making a safe product. The amount of software that gets written depends mostly on the demand for that software. Even if Micrososft would not be willing to up their game to make the risk viable then someone else would.

Reason0779• 2 years ago

It doesn’t really matter what the contract says. Laws take precedence over contracts. For example, Boeing’s liability for 737 airliners that crash due to faulty software certainly isn’t limited to the price of the planes.

anigbrowl4• 2 years ago

Crowdstrike's stock closed at $343 yesterday, I imagine that and MSFT are going to be cratering later this morning.

ssss114• 2 years ago

That’s what the license agreement says. Wait till every man and his dog sues them.

beejiu• 2 years ago

Caveat to this: In the UK and many other countries, you cannot limit liabilities that cause death or personal injury arising from negligence.

citrin_ru2• 2 years ago

Yes, software industry as we know would not exists if companies where held liable for all damages. But in the current state of affairs they have little incentive to improve software quality - when incident like this happens they can suffer an insignificant short term valuation loss but unless it happens too often they can continue businesses as usual.

Many companies paying lip service to quality/reliability but internal incentives almost always go against maintenance and quality of service work (and instead reward new projects, features e. t. c.).

0x1ceb00da3• 2 years ago

From windows tos:

Disclaimer. Neither Microsoft, nor the device manufacturer or installer, gives any other express warranties, guarantees, or conditions. Microsoft and the devicemanufacturer and installerexclude all implied warranties and conditions, including those of merchantability, fitness for a particular purpose, and non-infringement. If your local law does not allow the exclusion of implied warranties, then any implied warranties, guarantees, or conditions last only during the term of the limited warranty and are limited as much as your local law allows. If your local law requires a longer limited warranty term, despite this agreement, then that longer term will apply, but you can recover only the remedies this agreement allows.

DaoVeles12• 2 years ago

Hypothetically even if they were liable they would bankrupt before even a few percent of damages is recovered. You cannot pluck a bald chicken.

InsideOutSanta11• 2 years ago

they would bankrupt before even a few percent of damages is recovered

Wouldn't that be the desirable outcome, though? Given the amount of damage they have caused, they should cease to exist.

llm_trw4• 2 years ago

This is where public executions of executives help.

interludead5• 2 years ago

Financial liability often doesn't equate to actual recovery of damages

jmarbert• 2 years ago

You cannot pluck a bald chicken.

Haven't heard that one before but I love everything about that!

maeil7• 2 years ago

Financial losses? The comment you're replying to is mentioning heart attack treatment here. We're talking about deaths. Most of us won't like to hear this but for all of us who work at SaaS that is deployed on servers around the worlds, our bugs cause people to die. It's a given that at least a dozen people will die directly (medical flights, hospitals both being hit) due to this broken update, let alone indirectly.

frereubu• 2 years ago

I don't think the parent comment was ignoring that. The penalty for a company who does this can't be to bring someone back from the dead, it's likely to be financial, which is the aspect they're talking about.

bugbuddy3• 2 years ago

We must demand both financial and criminal liabilities against the perpetrators! Get the torches and pitchforks out! We need to teach them a lesson!

immibis6• 2 years ago

Since we live in a capitalism, financial losses are the only one anyone cares about at scale. What's a human life worth nowadays? About 10 million for a healthy prime age adult? Negative for elderly?

HenryBemis3• 2 years ago

I made a separate (longer) comment about this..

Why don't orgs test their updates? Every decent IT management/governance under the sun demands that you test your updates. How the hell did so many orgs that are ISO 2700x, COBIT, PCI-DSS, NIST CSF, etc. certified failed so hard??

(ToS/contracts will probably get you out of any damages.)

LinXitoW• 2 years ago

If companies want the nice parts of being "a person", they should also deal with the bad parts of being a person. Financial fines are not enough. Though I'm not sure how we'd build a jail cell for an entire company.

kaliqt• 2 years ago

Yes, SLA. No one gets held liable if the legal is done correctly and there were no guarantees, but on cloud there is 100% SLA so they will pay out.

Keyframe2• 2 years ago

How about people in charge of choosing these clown solutions - both crowdstrike and windows?

swarnie3• 2 years ago

In your world i should switch my modest 1000 seats over to Linux desktops?

I'm not sure how i'm going to explain the productivity loss and retraining costs to the board if im honest.

high_na_euv2• 2 years ago

Windows being clown solution? Out of touch with reality is huge here

traceroute66• 2 years ago

Chances if Microsoft or Crowdstrike will be held liable for financial losses caused by this outage?

Zero. Exactly Zero.

Clearly you have never been involved in buying insurance or writing contracts for IT products/services.

Loss of contracts, profits, goodwill, economic loss, loss of data and all that jazz is excluded in whole or limited to a fixed monetary value.

It is known as indirect, consequential or special loss, damage or liability.

No lawyer worth their salt will let an IT product/service company draft a contract that does not have the above type of clause..

And good luck finding an insurance contract that will pay out for such losses, indeed most of them have conditions that state your contracts with customers must exclude or limit such losses.

Most software also has clauses excluding use in safety critical environments.

kvgr• 2 years ago

There will be no Crowdstrike left after this. I am just upset I cant short it…

odieldomanie• 2 years ago

Some people in the comments claim CS was used for compliance reasons. Some others claim Windows & CS do not offer warranties. How can a product satisfy the compliance check-box, if it does not offer the warranty and not accept liability for the related features?

Closi• 2 years ago

While software is often warranted, contracts won't often accept liability in terms of business damages etc, and that's not usually a requirement for compliance.

If it was, it would also make it impractical for a small business to contract with a large one because of risk.

PretzelPirate3• 2 years ago

I dont think you can hold Microsoft liable for 3rd party software pushing its own update. Microsoft didn't make anyone install Crowdstrike or it's update files.

btbuildem6• 2 years ago

Prison time for the CEO and board of directors would be nice.

Enough of this limited liability nonsense, there need to be serious, severe, life-changing consequences.

lenerdenator• 2 years ago

Depends. I'm at an EMR maker; our Windows machines (as well of those of our clients - read: hospitals and doctors offices) are down. That is, of course, bad for the patients under their care.

Do these clients have SLAs? If so, they're definitely on the hook for something. You could probably get a few businesses together for a decent class-action against Crowdstrike. You're then expecting a lawyer to be able to convince a dozen semi-random people with varying degrees of computer knowledge that Crowdstrike's software was negligently designed, developed, and deployed in a way that caused financial or life losses for customers.

So, really, it's a coin flip.

nubinetwork2• 2 years ago

What if your company mandated your customers run crowdstrike in order to run your software? What are the legal implications of that? Wouldn't that also put your contracts on the hook?

ohwat• 2 years ago

Negligence at Crowdstrike is not covered by any SLA. Even if insured, Crowdstrike could be fucked. Let alone, companies going to try and how much cost this has. Long term, their fucked.

TheOne1001• 2 years ago

Do we not remember "Ma" Bell?This should perhaps be a wakeup call in regards to Microsoft and other large tech having concentrated fingers in too many pies. This appears to be an anti-trust issue at its core.

Was it really a botched update? Or was it a test run for holding the world hostage prior to a coup?

iamkneel• 2 years ago

Can't wait for the Kevin Fang video about this.

techbrovanguard6• 2 years ago

i've seen photos of the bsod from an affected machine, the error code is `PAGE_FAULT_IN_NONPAGED_AREA`. here's some helpful takeaways from this incident:

1) mistakes in kernel-level drivers can and will crash the entire os

2) do not write kernel-level drivers

3) do not write kernel-level drivers

4) do not write kernel-level drivers

5) if you really need a kernel-level driver, do not write it in a memory unsafe language

dboreham8• 2 years ago

0) don't load a new driver into your working kernel.

YawningAngel• 2 years ago

Basically all AV either runs as root or uses a kernel driver. I guess the former is preferable

happyweasel• 2 years ago

5) Well how much of those kernel-level drivers we rely upon ARE written in a memory unsafe language ??? Like 99% ?

And we are not crashing and dying every day?

Sure, Rust is the way to go. it just took Rust 18 years to mature to that level.

Also, quite frankly, if your unwrap() makes your program terminate because an array out of bounds isn't that exactly the same thing ? (program terminates)

But IMHO if we are hopping along a minefield at this moment every second of every day, well... If this is the worst case scenario, yeah it's not that worse after all.

techbrovanguard6• 2 years ago

Well how much of those kernel-level drivers we rely upon ARE written in a memory unsafe language ??? Like 99% ? And we are not crashing and dying every day?

we shouldn't discount the consequences of memory safety vulnerabilities just because flights haven't physically been grounded.

Also, quite frankly, if your unwrap() makes your program terminate because an array out of bounds isn't that exactly the same thing ? (program terminates)

this is a strawman, if you were writing a kernel-level driver in rust you'd configure the linter to deny code which can cause panics.

here's a subset:

- https://rust-lang.github.io/rust-clippy/master/index.html#/u...

- https://rust-lang.github.io/rust-clippy/master/index.html#in...

maeil7• 2 years ago

I've said this elsewhere but the enabling of instant auto-updates on software relied on by a mission critical system is a much bigger problem than kernel drivers.

Just imagine that there's a proprietary firewall that everyone uses on their production servers. No kernel-level drivers necessary. A broken update causes the firewall to blindly reject any kind of incoming or outgoing request.

Easier to rollback because the system didn't break? Not really, you can't even get into the system anymore without physical access. The chaos would be just as bad.

A firewall is an easy example, but it can be any kind of application. A broken update can effectively bring the system down.

nomercy4002• 2 years ago

There sure are a lot of mission-critical systems and companies hit by this. I am surprised that auto-updates are enabled. I read about some large companies/services in my country being affected, but also a few which are unaffected. Maybe they have hired a good IT provider.

gukov2• 2 years ago

One thing to consider with security software, though, is that time is of essence when it comes to getting protection again 0day vulnerabilities.

Gotta think that the pendulum might swing into the other direction now and enterprises will value gradual, canary deployments over instant 100% coverage.

AndriyKunitsyn4• 2 years ago

Not a helpful takeaway, I've yet to see a Java kernel driver.

techbrovanguard6• 2 years ago

be the change you wish to see

saagarjha16• 2 years ago

Nobody is telling you to use Java. Although, if you want to revive Singularity that would be pretty neat.

rfoo4• 2 years ago

Memory safe language does not prevent crash.

In case of potential UB (and then memory corruption), you get a guaranteed crash.

Wait, crash? :wink:

techbrovanguard6• 2 years ago

did you have a crowdstroke while writing this reply?

bdd8f1df777b8• 2 years ago

The problem is that some viruses may run in the kernel mode, so an AV has to do the same, or it will be powerless against such viruses.

skydhash3• 2 years ago

If a virus got that far, you're already in trouble. What stops them from attacking the anti-virus?

thwarted• 2 years ago

I'm not a Windows programmer so the exact meaning of PAGE_FAULT_IN_NONPAGED_AREA is not clear to me. I am familiar with UNIX style terminology here.

Is this just a regular "dereferencing a bad pointer", what would be a "segmentation violation" (SEGV) on UNIX, a pointer that falls outside the mapped virtual address space?

As this is in ring 0 and potentially has direct access to raw, non-virtual physical addressing, is there a distinction between "paged memory" (virtual address space) and "nonpaged memory" (physical address) with this error?

Is it possible to have a page fault failure in a paged area (PAGE_FAULT_IN_PAGED_AREA?), or would that be non-fatal and would be like "minor page fault" (writing to a shared page, COW) or "major page fault" (having to hit disk/swap to bring the page into physical memory)?

Are there other PAGE_FAULT_ errors on Windows?

Searching for this is difficult, as all the results are for random spammy user-centric tech sites with "how do I solve PAGE_FAULT_IN_PAGED_AREA blue screen?" content, not for a programmer audience.

matltc• 2 years ago

https://wiki.osdev.org/Exceptions#Page_Fault

jasomill• 2 years ago

https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

7373737373• 2 years ago

an audio driver once blue screen of death'd my windows whenever i started Discord.

i'm surprised i'm not hearing a stronger call for microkernels yet

rustcleaner3• 2 years ago

5) if you really need a kernel-level driver, do not write it in a memory unsafe language

I C what you're doing... >_>

techbrovanguard6• 2 years ago

pointing out the obvious? why are you upset i’m stating mixing hot oil and water will make a mess?

p_l20• 2 years ago

Rust's memory safety does not prevent category errors like using nonpaged memory for things supposed to be paged and vice versa

techbrovanguard6• 2 years ago

this all-or-nothing mindset is is reductive and defeatist—harm reduction is valuable. sure, rust won’t magically make your kernel driver bug free, but will reduce the surface area for bugs, which will likely make it more stable.

thih92• 2 years ago

Other takeaways:

- do not put critical infrastructure online

- do not push updates that work around the update schedule

- do not push such updates to all machines at once

- do not skip testing and QA, relevant to the number and kind of the machines affected

Even one of these would have massively improved the situation, even with a kernel-level driver written in an unsafe language.

radiator4• 2 years ago

So Crowdstrike protects your computers from cyber attacks. But who is going to protect you from Crowdstrike?

rochak2• 2 years ago

Not Microsoft, that’s for sure

_Tomas• 2 years ago

I dunno, Coast Guard?

dimgl• 2 years ago

Underrated joke, thanks

choeger3• 2 years ago

It's eye-opening how bad our crucial IT infra is nowadays. Running in-kernel third-party tools (AV) on critical infrastructure on Windows? Central banks? Control towers? Seriously? We should fire everyone involved and start IT from scratch. This level of negligence cannot be fixed.

attentive11• 2 years ago

I am most annoyed by built-in RCE. Who thought that to be a good idea?

ryandv3• 2 years ago

Absolutely shameful display of how the cure can be worse than the disease. It's nonsense snake oil and security theater such as this that throws the cyber"security" industry into disrepute. One may as well have just installed McAfee Anti Virus.

stefan_2• 2 years ago

This has been the story of the antivirus "industry" all along. They simultaneously seem to employ actual bona-fide security researchers while also making sure none of their software products are ever touched by people you could even refer to as "developers". I can't even imagine the noise at Microsoft from all the crash reports solely caused by antivirus software written by utter clowns injecting into other programs and, as here, into the kernel.

Previous: https://infosec.exchange/@wdormann/112530285189478825

Previous: https://thehackernews.com/2022/05/chinese-hackers-caught-exp...

Previous: https://www.ftc.gov/news-events/news/press-releases/2024/02/...

Previous: https://www.fortiguard.com/psirt/FG-IR-24-015

...

ryandv3• 2 years ago

"We're only backdooring your machines for your own good! We are the good guy experts, we promise."

aa-jv2• 2 years ago

snake oil

So much software - especially in the 'operating system' sphere of things - really is just snake oil.

Its just, very functional oil, in many cases - and highly toxic and slippery in many, many other cases.

cyber "security" industry

Yes, I agree this is a market of smoke and mirrors, lies and propaganda.

The reason is, operating systems are broken. Pretty much all of them. Only, some of them work well enough to get a lot of work done, most of the time. Of the 99.9995% of the time it works, its great.

But, here's a thing I feel needs broader attention and discussion - It is my firm opinion that "Operating Systems Vendors" are a very poor, ragged class of professionals these days.

The decisions made at Microsoft - and other OS vendor corporations - have really lost the plot.

I can prove this by asking the golden question among the general public, and categorically get a standard response: "does this feature benefit the user, or does it benefit an advertiser?"

"No, this all seems to be some sort of setup. Windows doesn't feel like its for us, any more."

I mean, how many 3rd-party vendors do I need, secretly installing crippling 'updates' in my production systems, before I realize that there is no security, so write better software that doesn't need all this utter junk.

I mean this sincerely, operating systems vendors are treasonous to the user if 3rd parties are of more relevance to production runtime, than the thing the user very definitely needs to be operating.

The cloud is for backups, encrypted. It is made of snake oil.

Always run your own machines.

cromka7• 2 years ago

This is a sample of what Y2K would look like if not for the countermeasures.

mrinfinitiesx2• 2 years ago

Actually really scary to see/read the comments on. Like Die Hard's 'Fire Sail"

qeternity• 2 years ago

*sale

Eji17003• 2 years ago

Welp this fucked my night. A toast to the rest of you who are waaaaaay more screwed than me

gloosx5• 2 years ago

Thanks god all the critical infrastructure in my country is still on MS DOS!

LeoPanthera3• 2 years ago

That school running their HVAC infra on an Amiga must be pretty happy.

jaza7• 2 years ago

Hahaha, you mean all the CRITIC~1.INF ?

hulitu7• 2 years ago

Hahaha, you mean all the CRITIC~1.INF ?

Kids those days. It shall be CRITIC~1.COM

woodylondon• 2 years ago

CyberStrike offers a temporary solution for crashed systems Cyberstike has given users a potential way to fix their systems.

Boot Windows into Safe Mode or the Windows Recovery Environment (you can do that by holding down the F8 key before the Windows logo flashes on screen) Navigate to the C:WindowsSystem32driversCrowdstrike directory Locate the file matching “C-00000291.sys” file, right click and rename it to “C-00000291.renamed” Boot the host normally.

oldmanyells683• 2 years ago

Great. now I'll do this 10,000 times.

0x1ceb00da3• 2 years ago

Practice makes a man perfect.

rosege2• 2 years ago

fun doing that on thousands of machines

steve19775• 2 years ago

It's easy, the users can do it themselves. Just send them an e-mail with the inst... oh wait...

moxli• 2 years ago

Source for me was a reddit comment: https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/commen...

nickdothutton2• 2 years ago

The only surprising thing is that this doesn't happen every month. Nobody understands their runtime environment. Most IT org's long ago "surrendered" control and understanding of it, and now even the "management" of it (I use the term loosely) is outsourced.

ssss114• 2 years ago

“Cloud” is so great huh

nickdothutton2• 2 years ago

"I'm going to install an agent from Company X, on this machine, which it is essential that they update regularly, and which has the potential to both increase your attack surface and prevent not just normal booting but also successful operation of the OS kernel too". I am not going to provide you with a site specific test suite, you're going to just have to trust me that it wont interrupt your particular machine".

swozey2• 2 years ago

This is mostly physical machines in person, kiosks and pos terminals, office desktops and things like that. Windows is a tiny portion of GCP and AWS and the web in general.

I'm 100% "cloud" with tens of thousands of linux containers running and haven't been affected at all.

patates4• 2 years ago

Nowadays it seems like everyone is running stuff inside VMs because IT removes all the rights in the host system.

jkells2• 2 years ago

Maybe they do perform canary deployments and Australia was the canary?

Certainly feels like it's disproportionately affecting us down under.

bobnamob2• 2 years ago

We’re just the only ones awake to feel it

bobnamob2• 2 years ago

Anecdotal evidence: the global mega corp I unfortunately work for is definitely feeling this globally

tomashubelbauer2• 2 years ago

It looks like the whole world is the canary, but they will have the release ready and in top shape for the Mars deployment.

rochak2• 2 years ago

I haven’t seen a simultaneous outage as big as this in my entire life. I’m just hoping this gets enterprises to move off of Windows.

BartjeD3• 2 years ago

Except it hasn't got much to do with Windows... its a faulty kernel software package from a commercial vendor unrelated to the OS.

Philosophically it's always good to have diversity, precisely to avoid such disruptions. But the real issue here is: A) Apparently half the world runs CloudStrike... So everything is disrupted. B) Apparently CloudStrike didn't test their update properly.

I'm very curious what will happen to CloudStrike. This seems like a huge liability?

sgammon5• 2 years ago

We don't know that yet

lachlanj• 2 years ago

I'm confused, is this an issue with Windows or with Crowdstrike software installed on Windows?

pi-rat2• 2 years ago

With a crowdstrike kernel driver, so technically not a microsoft/windows issue.

dboreham8• 2 years ago

For a while I've joked with family and colleagues that software is so shitty on a widespread basis these days that it won't be long before something breaks so badly that the planet stops working. Looks like it happened.

drooopy2• 2 years ago

I've picked the perfect day to return from vacation. Being greeted by thousands of users being mad at you and people asking for your head on a plate makes me reconsider my career choice. Here's to 12 hours of task force meetings...

jeffrallen4• 2 years ago

"Gentlemen! You can't fight in here, this is the War Room!"

Have a nice day, anyway.

urbandw311er3• 2 years ago

Huge sympathies to you. If it's any consolation, because the scale of the outage is SO massive and widely reported, it will quickly become apparent that this was beyond your control, and those demanding your 'head on a plate' are likely to appear rather foolish. Hang in there my friend.

drooopy2• 2 years ago

To their credit, the stakeholder that asked for my head personally came to me and apologised once they realised that entire airports have been shut down worldwide. But yeah, not a Friday/funday hahaha

cm20123• 2 years ago

Unless he works high up at CrowdStrike lol

swarnie3• 2 years ago

Its not just us, form an orderly queue and you'll been seen soon.

Do you really all have these mentally unstable userbases?

Eji17003• 2 years ago

Ohhh absolutely. And it's not just users, it's also management. "How does this affect us? Are we compromised? What are our options? Why didn't we prevent this? How do you prevent this going forward? How soon can you have it back up? What was affected? Why isn't it everyone? Why are things still down? Why didn't X or Y unrelated vendor schlock prevent this?..."

And on and on and on. Just the amount of time spent unproductively discussing this nightmare is going to cost billions.

CoastalCoder12• 2 years ago

Maybe you picked the right week to start sniffing glue.

kelsey987654313• 2 years ago

Your head belongs on the plate for not being able to point back to your recommendation for failover posture enhancement such as identifying core business systems, core function roles, having fully offline emergency systems, warning of the dangers of making cloud services your only services, and then pointing to the proposed costs to implement these systems being lower than the damages caused by outage to core business services.

Move to a new career if you feel you don't have the ability to push right back against this.

raverbashing3• 2 years ago

and people asking for your head on a plate

I'd say I'd given what they want and left to get an icecream at the part

Let then try to fix it themselves

1oooqooq13• 2 years ago

well, you did agree to go in business with crowdstrike, and base your company IT on windows, so...

adzm• 2 years ago

Workaround fixed it for me, thankfully I had access to the bitlocker recovery keys. This will be a bad day for IT people worldwide.

cromka7• 2 years ago

BBC reports: “ The cause is not known - but Microsoft says it's taking mitigation action”.

Most of the media I found say it’s because “cloud infrastructure”. I am yet to see any major source actually factually report this is caused by a bad patch in Crowdstrike software installed on top of Windows.

Gets to show how little competency there is in journalism nowadays. And begs a question how often they misinterpret and misreport things in other fields?

spuz• 2 years ago

On BBC news a few minutes ago, an expert did describe the problem as affecting Microsoft Azure cloud systems as well as Windows systems running Crowdstrike due to an "update gone wrong".

cromka7• 2 years ago

Well, in BBC’s live coverage, just minutes ago, their technology editor said:

“ There have been reports suggesting that a cybersecurity company called Crowdstrike, which produces antivirus software, issued a software update that has gone horribly wrong and is bricking Windows devices - prompting the so-called "blue screen of death" on PCs. Now, whether these two issues are the same thing, or whether it's a perfect storm of two big things happening simultaneously - I don't yet know. It certainly sounds like it's going to be causing a lot of havoc.”

What two issues? Two major independent outages? This is seriously bad and purely speculative.

esskay4• 2 years ago

BBC are doing a rare awful job. I've got BBC News on here in the UK and they just keep saying a "Microsoft IT outage".

cromka7• 2 years ago

Or maybe they usually do an awful job, except this time this is our field, so we know for certain that they do.

benfortuna2• 2 years ago

https://www.abc.net.au/news/2024-07-19/what-is-crowdstrike-o...

d1sxeyes3• 2 years ago

The BBC are starting to say that 'tech people are saying this is Crowdstrike', so I guess it's just a question of being certain? Perhaps we'd have similar concerns about rigour in journalism if it were to turn out that it's actually not Crowdstrike specifically, it's caused by the interplay of Crowdstrike and some other currently unknown thing, and actually it's not Crowdstrike that's behaving improperly, but this other currently unknown thing.

It's looking more and more like Crowdstrike screwed up, but I appreciate rigour and accuracy more than FRISTTT!!! type announcements.

handity• 2 years ago

BFMTV, French broadcaster, reports:

"Selon le quotidien The Australian, qui relaie les déclarations du ministère australien des Affaires intérieures, l'entreprise Crowdstrike pourrait être en cause, après avoir été victime d'une brèche au sein de sa plateforme."

Translated/summarized: "According to the publication The Australian, Crowdstrike may be the cause of the outage after having suffered a security breach"

I like how it redirects blame away from those responsible and perpetuates the idea that "hackers" are the real threat.

Source: https://www.bfmtv.com/tech/direct-une-panne-informatique-mon...

gedw992• 2 years ago

Ironically SolarWinds court case happened yesterday. SEC won. SolarWindows was fraudulent to say their software way “secure”. They should rename a side channel attack a “Tom and Jerry”, because its getting like a game of Cat and Mouse

gedw992• 2 years ago

Rock me Amadeus.

At least the central flight booking system is up I guess. Google brought it years ago and it's a mainframe.

Hence why google flights is so tapped in :)

reegnz• 2 years ago

Cybersecurity company secures computers worldwide by not allowing them to be turned on. - not the onion

mro_name2• 2 years ago

What do card houses do for a living?

vlugovsky• 2 years ago

Crowdstrike is a perfect name for a company that could cause a worldwide outage.

lizknope8• 2 years ago

Yeah, I've always thought it was a bad name. I see them during Formula 1 advertisements because they sponsor the Mercedes team.

They might as well have named themselves "cluster bomb" as they have done a huge amount of damage today and for the next few days.

Lammy• 2 years ago

Makes me think of flystrike which is also a perfect analogy https://en.wikipedia.org/wiki/Myiasis

jiehong• 2 years ago

IMO, having a mix of servers would help in mitigating issues like that.

Like run stuff on Linux, windows and freebsd servers, so that you have OS redundancy should an issue affect one in particular (kernel or app).

Just like you want more than a single server handling your traffic, you’d want 2 different base for those servers to avoid impacting them both with an update.

romanovcode3• 2 years ago

Not using this crap security software would mitigate this issue.

tanelpoder• 2 years ago

I'm curious why this post is still not the 1st (but 2nd after an ebook reader announcement), despite all the upvotes.

saagarjha16• 2 years ago

Lots of people commenting drags down the position on the front page.

silamay2• 2 years ago

I’m trying to refresh to get the latest update …

bgnn4• 2 years ago

This is a manifestation of almost everything wrong about software development and marketing practices.

I work in hardware development and such a failure is almost impossible to imagine. It has to work, always. It puzzles me why this isn't the casebfor software. My SWE colleagues often get mad at us HW guys because we want to see their test coverage for the firmware/drivers etc.. The focus is having something which compiles and pushing the code to production as fast as possible and then regressing in production. Most of HW problems are a result of this. I found it's often better to go over the firmware myself and read line by line to understand what the code does. It saves so much time from endless debugging sessions later. It pisses of firmware guys, but hey, you have to break some eggs to make an omelette.

aa-jv2• 2 years ago

It puzzles me why this isn't the case for software

In my anecdotal experience, its because corporate software projects are not typically run by people who are good at building safe things - but rather, just building things quickly.

There's a huge issue with the mentality of "it works, ship it" being propagated.

I build systems software for safety-critical and mission-critical markets, and I can say without a doubt that if there aren't at least two quality stages in your process workflow (and your workflow isn't waterfall), then you're going to be in for a rough time, rookies.

Always, always delay your releases, and always, always, eat your own dog food by testing your delayed releases in your own customer-like environment, which is to say, never release a developers build ..

bgnn4• 2 years ago

This is also my experience. And the worst was always being lectured by SW project managers about being agile and having to move quick and release early. I won't release anything without making sure everything works in every possible condition. This is why it takes years to build a complex chip (CPU, Fpga, any SoC really). Their firmware is often squeezed into months, and often the developers are handling like 10 different projects. So, no focus, no time to understand the details of the design. At the end it's common to have firmware issues in the first year after release. It's kind of expected even.

doug_durham• 2 years ago

Complexity. As you get further from the driver and the kernel software complexity expands massively. It gets to a point where it is beyond the abilities of humans and processes to manage it in a cost effective manner.

bgnn4• 2 years ago

I understand that might be case for a lot of SW development. But in the context I was talking about, the HW is so much more complex than the SW. Valid for a lot of cases too. But then, why? If we know that we cannot build a 100km long bridge, nobody attempts to build that and waste resources. Why does software development lack this?

silamay2• 2 years ago

I’m trying to refresh to get latest update… let’s keep posting

mehh• 2 years ago

Don’t be fooled, it’s Skynet, head to the bunkers!

steve19775• 2 years ago

Username "mehh" has been noted.

openrisk2• 2 years ago

The details (the particular companies / systems etc) of this global incident don't really matter.

When the entire society and economy are being digitized AND that digitisation is controlled and passes through a handful of choke points its an invitation to major disaster.

It is risk management 101, never put all your digital eggs in one (or even a few) baskets.

The love affair with oligopoly, cornered markets and power concentration (which creates abnormal returns for a select few) is priming the rest of us for major disasters.

As a rule of thumb there should be at least ten alternatives in any diversified set of critical infrastructure service providers, all of them instantly replaceable / forced to provide interoperability...

Some truths will hit you in the face again and again until you acknowledge the nature of reality.

pcblues• 2 years ago

Centralisation vs decentralisation. Cost-savings vs localisation of disaster.

It's a swinging pendulum of decisions. And developers know that software/hardware provision is a house of cards. The more levels of dependency, the more fragile the system is.

Absolutely horrible when lives will be lost, but determining the way our global systems are engineered and paid for will always be a moving target based on policy and incentive.

My heart goes out to life and death results of this. There are no perfect tech solutions.

asib• 2 years ago

It is risk management 101, never put all your digital eggs in one (or even a few) baskets.

The fact it's widespread is because so many individual organisations individually chose to use CrowdStrike, not because they all got together and decided to crown CrowdStrike as king, surely?

I agree with you in principle, but the only solution I can think of would be to split up a company with reach like CrowdStrike's. The consequences of having to do that are up for debate.

alex-nt• 2 years ago

It's never that simple. There is a strong herd mentality in the business space. Just yesterday I've been in a presentation from the risk department and they described the motives around choosing a specific security product as `safe choice, because a lot of other companies use it in our space, so regulator can't complain`...the whole decision structure boiled down to: `I don't want to do extra work to check the other options, we go with whatever the herd chooses`. Its terrifying to hear this...

7952• 2 years ago

Perhaps the organisations have a similar security posture. And that creates a market that will eventually result in a few large providers who have the resources to service larger corporations. You see something similar in VPN software where Fortinet and Palo become the linchpin of security. The deeper question is to wonder at the soundness of the security posture itself.

jaggederest4• 2 years ago

It's not just Crowdstrike, it's all up and down the software and hardware supply chain.

It's that so many people are on Azure - which is a defacto monopoly for people using Microsoft stack - which is a defacto monopoly for people using .Net

And if they're doing that, the clients are on Windows as well, and probably also running Crowdstrike. The AD servers that you need to get around Bitlocker to automatically restore a machine are on Azure, running Windows, running Crowdstrike. The VM image storage? Same. This is basically a "rebuild the world from scratch" exercise to some greater or lesser degree. I hope some of the admins have non-windows machines.

izacus3• 2 years ago

I wonder how many of those orgs were "independently" audited by security firms which made passing audit without Crowdstrike specifically a hell.

Most of crap security I met in big organisations was driven by checklist audits and compliance audits by a few "security" firms. Either you did it the dumb way or good luck fighting your org and their org to pass the audit.

lucasRW8• 2 years ago

When it comes to security products, the size of the customer base matters. More customers means more telemetry. More telemetry means better awareness of IOCs, better training sets to determine what's good and what's bad.

spacebanana7• 2 years ago

There's a strong drive for everyone to do things the same way in IT. Some of the same pressure that drives us towards open standards can also drive us towards using a standard vendor.

I agree with you in principle, but the only solution I can think of would be to split up a company with reach like CrowdStrike's.

Changing corporate structures doesn't necessarily help. It's possible that if CrowdStrike were split up into to smaller companies, all the customers would go to the one with the "better" product and we'd be in a similar position.

chiefalchemist3• 2 years ago

They choose because other have. "Look how many others choose us" is a common marketing cry. Perhaps instead too popular is a reason not to choose? Perhaps not parroting your competitors and industry is a reason not to choose?

anonylizard• 2 years ago

Society was able to move to mass WFH on a global scale in a single month during Covid, thanks to the highly centralized and efficient cloud infrastructure. That could have easily saved tens of millions of lives (Imagine the Spanish flu with mass air travel, no vaccines, no strain-weakening)

These small 'downages' basically never cause serious issue. Your solutions are just alarmist and extremely costly (though they will provide developer employment...).

jaggederest4• 2 years ago

These small 'downages' basically never cause serious issue.

Hospitals, airlines, 911, grocery stores, electric companies, gas companies, all offline. There will be more than a few people dead as an indirect result of this outage, depending on how long it lasts.

randomcarbloke• 2 years ago

true, but incentives should be in place to encourage a more diverse array of products, at the moment with many solutions (especially security) it is a choice between that one popular known product (Okta, CrowdStrike, et al, $$$) and bespoke ($$$$$$$$$$).

If only because we can then move away from one-size-fits-all, while mitigating the short-term impact of events like the above.

shakna• 2 years ago

These small 'downages' basically never cause serious issue.

Emergency Departments and 911 were knocked offline. People will indirectly die because of this, just like the last time 911 went down, and just like the last time EDs went down.

It's not alarmist, it's realist.

TestingWithEdd• 2 years ago

If CrowdStrike can cause this with a faulty update (allegedly), what do you think could happen to Western infrastructure from a full blown cyberwar? It's a valid risk.

Society was able to move to mass WFH on a global scale in a single month during Covid

I don't know how much WFH saved lives, seeing as ordered isolation and social distancing was a thing during the Spanish Flu too (you just take the economic hit). But yes it allowed companies to keep maintaining profits. Those that couldn't WFH got paid in most countries anyway (furlough in England, etc).

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2862334/

dagaci4• 2 years ago

Be aware that enterprise firms actively choose and "asses" who their AV suppliers are on-premis and in the cloud not imposed by msft. Googling it does seem that CrowdStrike, does have a history of Kernel Panics. Perhaps such interesting things as Kernel panic should be part of compliance checklist.

Googling it seems crowdstrike has a history of causing kernel panics.

https://www.google.com/search?q=crowdstrike+kernel+panic

konschubert• 2 years ago

They have a Gartner Magic Quadrant on the landing page.

This is checkbox software, selling to decision makers in enterprises who will never see nor touch the software they are buying.

This kind of software where the user isn’t the customer is always terrible.

Y_Y3• 2 years ago

Maybe if they got off their "asses" and did some more comprehensive assessments we wouldnt be in theis mess.

AshamedCaptain• 2 years ago

What are they going to do, go back to Tanium?

Everytime there was a mysterious performance problem affecting a random subset of machines, it was Tanium. I know how difficult it is for anyone to just get rid of this type of software, but frankly it has been proven over and over that antivirus are just more surface attack, not less.

vb-8448• 2 years ago

The details (the particular companies / systems etc) of this global incident don't really matter.

It definitively matters. The main issue he is that Crowdstrike was able to push and update on all server around the world where all their agent is installed ... it looks like an enormous botnet ...

We need a detailed post-mortem on what happened here.

lupusreal• 2 years ago

The name sounds like something security researchers would name a botnet.

d--b• 2 years ago

Yet, catastrophic failures like this happen, and people move on. Sure, there is that one guy who spent 10 years building a 10-fold redundancy plan, and his service didn't go down when the whole planet went down, but do people really care?

alt2274• 2 years ago

His customers do

JackMorgan2• 2 years ago

If he provides emergency services like fire and ambulance people care a lot.

nyx_land2• 2 years ago

this isn't even the first time something like this has happened. it's literally a running joke in programmer circles that AWS East going down will take down half the internet, and yet there's absolutely zero initiative being taken by anyone who makes these sorts of decisions to maybe not have every major service on the internet be put into the same handful of points of failure. nothing will change, no one will learn anything, and this will happen again.

bilbo0s2• 2 years ago

That’s very different though. That’s avoidable. We all can easily have our services running in different data centers around the world. Heck, the non-amateurs out there all have their services running in different Amazon data centers around the world. So you can get that even from a single provider. Hardware redundancy is just that cheap nowadays.

This CS thing, there’s no way around. You use it and they screw up, you get hit. Period. You don’t failover to another data center in Europe or Asia. You just go down.

Hardware, even cloud hardware, is rarely the issue. Probably especially cloud hardware is not an issue because failover is so inexpensive relative to software.

Software is a different issue entirely. How many of us will develop, shadow run, and maintain a parallel service written on a separate OS? My guess is “not many”. That’s the redundancy we’re talking about to avoid something like this. You’d have to be using a different OS and not using CS anywhere in that new software stack. (Though not using CS wouldn’t be much of a problem if the OS is different but I think you see what I mean.)

Amazon, implementing failover for your hardware is a few clicks. But if you want to implement an identical service with different software, you better have a spare dev team somewhere.

higeorge135• 2 years ago

Or perhaps it's just because companies release features, planes, devices, etc. without any form of QA, aiming just to increase their profits?

In this case, has CS done any QA on this release? Have they tested it for months on all the variations of the devices that they claim to support? It seems not.

_cenw• 2 years ago

Considering CS Falcon causes your performance to drop by about half and does the same to your battery life, I doubt they have any sort of QA that cares about anything but hitting stakeholder goals.

Frieren• 2 years ago

Efficiency goes against resilience.

Can you imagine having just one road connecting two big cities to cut costs? No alternative roads, nor big nor small.

That will be really cheap to maintain, and you can charge as much as you want in tolls as there are no alternatives. And you can add ads all over the road as people has to watch them to move from one city to the other.

And if the road breaks, the goverment needs to pay for the cost as they cannot allow the cities to go unconnected.

We live in the middle-ages of technology.

nicbou• 2 years ago

We have multiple exits to avoid trapping people in a fire. Redundancy is important when the cost of failure is high.

seanp2k2• 2 years ago

Or like...one power company in Texas?

funcDropShadow2• 2 years ago

Efficiency goes against resilience.

Only when you are focusing on short-term effects. If you thinking long-term it is always better to not be out of business.

28304283409234• 2 years ago

Can you imagine having just one road connecting two big cities to cut costs?

Sure! But now imagine those roads private property.

baxtr3• 2 years ago

It is also risk management 101 that managing (i.e. avoiding or insuring) risks doesn't come for free.

The cost and benefits of managing risks need to be balanced.

So I am not convinced that there need to be "at least ten alternatives" to be fail safe as society.

Where I agree with you is that these decisions should be deliberate and done after a cost / benefit analysis.

openrisk2• 2 years ago

So I am not convinced that there need to be "at least ten alternatives" to be fail safe as society.

The required number "N for safety" is a good discussion to have. Risk-Return, Cost-Benefit etc are essential considerations. We live in the real world with finite resources and stark choices. But I would argue (without trying to be facetious) that they are risk management 102 type considerations.

Why? Because they must rely on the pretense of knowledge [1]. As digitization keeps expanding to encompass basically everything we do, the system becomes exceedingly complex, nobody has a good picture of all internal or external vulnerabilities and how much they might cascade inside an interconnected system.

Assessing cost versus benefit implies one can reasonably quantify all sides of the equation. In the absence of a demonstrably valid model of the "system" the prudent thing is to favor detail-agnostic rules of thumb. If these rules suggest that reducing unsafe levels of concentration is not economically viable there must be something wrong with the conceptual business model of digitization as it is now pursued.

[1] https://www.nobelprize.org/prizes/economic-sciences/1974/hay...

teeheelol14• 2 years ago

That's not really the problem here.

The problem is allowing a single vendor, with a reputation of fucking up over and over again, to push code into your production systems at will with no testing on your part.

AndriyKunitsyn4• 2 years ago

Right. I thought the "big guys" know better and they have some processes to vet Crowdstrike updates. Maybe even if they don't get its source code, they at least have a separate server that manages the updates, like Microsoft's WSUS.

But no, they are okay with a black box that calls home and they give it kernel access to their machines. What?

aboringusername• 2 years ago

there should be at least ten alternatives in any diversified set of critical infrastructure service providers, all of them instantly replaceable / forced to provide interoperability...

And does anyone actually know how to actually implement this, at the scale required (dealing with billions of transactions daily) in a way that would resolve the problems we are seeing?

It very much seems like a data access problem; places can't access/modify data. The physical disks themselves are most likely fine, but the 'interfaces' are having troubles (assuming that the data isn't stored on the devices having the issue).

But in any case how do you design a system that, if the 'main' interface is troubled, you can switch over, instantly, seemlessly, duplicating access controls, permissions, data validation, logic etc.

There is a reason everything is centralised because it makes no financial sense to duplicate for an extremely unlikely and rare chance. The world is random and these things will happen, but a global outage on this type of scale is not a daily occurance.

We'll look back in a few years and think "those were a crazy few hours" and move on...

aredox• 2 years ago

The oligopoly is not a "love affair", that's how IT works: prime mover advantage, "move fast and break things" (the first of them being interoperability), moats, the brittleness of programming...

The whole startup/unicorns ecosystem exists only because there is the possibility of becoming the dominant player in a field within a few years (or being bought out by one of the big players). This "love affair with oligopoly" is the reason why Ycombinator/HN exists.

noobermin• 2 years ago

I feel like posting this as if it's a given means you are unaware of history. Just because something exists now doesn't mean it is always determined.

immibis6• 2 years ago

It works this way because we, as a society, decided we wanted it to.

Retr0id4• 2 years ago

It's not an intrinsic property of IT, it's a property of how we've built it and allowed it to be built.

ropable• 2 years ago

Easy to state; non-trivial to implement.

gmerc• 2 years ago

Hey hey, Silicon Valley just bought themselves a VP to ensure no regulation.

adityaathalye• 2 years ago

Sadly, I'm going to have to update examples in my blog post... https://www.evalapply.org/posts/software-debt/index.html#sof....

Software debt is networked.

I'm writing this in the wake of the aftermath of the disclosure of the log4j zero-day vulnerability. But this is only a recent example of just one kind of networked risk.

With managed services we effectively add one more level to the Inception world of our software organisation. We outsource nice big chunks of supply chain risk management, but we in-source a different risk of depending critically on entities that we do not control and cannot fix if they fail.

Not to mention the fact that change ripples through the parallel yet deeply enmeshed dimensions of cyberspace and meatspace. Code running on hardware is inexorably tied to concepts running in wetware. Of course, at this level of abstraction, the notion applies to any field of human endeavour. Yet, it is so much more true of software. Because software is essentially the thoughts of people being played on repeat.

ryandv3• 2 years ago

Setting aside the utter fecklessness if not outright perniciousness of cybersecurity products such as this, I hope this incident (re-)triggers a discussion of our increasing dependence on computing technology in our lives, its utter inescapability, and our ever-growing inability to function without it in modern society.

Not everything needs to be done through a computer, and we are seeing the effects now of organizing our systems such that the only way to interface with them is through a digital device or a smartphone, with no alternative. Such are the consequences of moving everything "into the cloud" and onto digital devices as a result of easy monetary policy and the concomitant digital gold rush where everyone and their dog scrambled to turn everything into a smartphone app.

chiefalchemist3• 2 years ago

This past week I purchased a thermostat. There were "high-end" touch only models, models that were app-assisted also with analog controls, and then finally old school analog only. I went with the middle / combo so that I have analog as a call back if the pure tech mode fails.

Being prepared can cost more and/or be less flashy (read: I didn't get touch-only) but it's only peace of mind, at least for critical components. I want a thermostat that works, I don't get no satisfaction from any bragging rights. Nod to the Rolling Stones.

stavros• 2 years ago

This isn't some global conspiracy, it's just incentives and economies of scale. When it's cheaper to pay a hyperexpert to handle your security, why wouldn't you?

The fact that physical distance is no longer a limit to who you do business with means that you can select the cheapest vendor globally, but then that vendor has an incentive to hyperspecialize (because everyone goes to them for this one thing), which means that even more people go to them.

Avoiding once-in-a-century events just isn't something we're willing to pay the extra cost for, except now we have around twenty places where these once-in-a-century events can happen, which kind of makes them more frequent.

How much stuff do you host on Hetzner instead of AWS?

peanut-walrus2• 2 years ago

I think the enterprise software ecosystem currently is not really "all eggs in one basket", but rather you have a whole bunch of baskets, some of them you are not even aware of, some are full of eggs, some have grenades in them instead, some are buckets instead. All baskets are being constantly bombarded with a barrage of eggs from unknown sources, sometimes the eggs explode for inexplicable reasons. Oh yeah and sometimes the baskets themselves disintegrate all at once for no apparent reason.

Y_Y3• 2 years ago

You seem to have put an awful lot of eggs in the basket of that metaphor.

roenxi• 2 years ago

The other aspect of risk management is an acceptance that something going wrong isn't necessarily a reason to change what you are doing. If the plan was tacitly to run something at a 99% uptime, then incidents causing 1% downtime can be ignored.

We are going to get hit by some terrible outage eventually (I hope someone is tracking things like what happens if a big war breaks out and the GPS constellations all go down together). But having 10x providers won't help against the big IT-related threats which are things like grid outages and suchlike having cascading effects into food supplies.

NDizzle4• 2 years ago

While this is a great theory, how would you actually accomplish this with antivirus software?

Multiple machines, each one using different vendor software? What other software needs to be partitioned this way? What about combinations of this software?

I’m just barely awake but don’t know if I’m affected yet. One of my devs is, our client support staff is, and I have no idea how our servers are doing just yet.

lofaszvanitt• 2 years ago

Now they know the state of each of the affected companies systems. How adept their sysops guys are, a birds eye view of their security practices. Nice move and plausible deniable too :D.

I mean how did this happen at all? Are there no checks in place @ crowdstrike? Like deploying the new update to a selected machines and check whether everything is ok, and then releasing it to the wild incrementally?

Mind boggling.

ojhughes• 2 years ago

I suspect the `assertNoBSOD()` test was marked as flakey

epolanski2• 2 years ago

While I agree 100% with what you say in principal, stats show that these occurrences are increasingly rare.

DrBazza• 2 years ago

As a rule of thumb there should be at least ten alternatives

I think you mean 14. https://xkcd.com/927/

When the entire society and economy are being digitized AND that digitisation is controlled and passes through a handful of choke points its an invitation to major disaster.

Once again, it's Microsoft, directly, or indirectly, choosing a strategy of eventually getting all worldwide Windows desktops online, and connected via their systems.

Which is why I installed Fedora after Windows 7 and never looked back. 100% local, 100% offline if needed.

My company is looking to a non-Microsoft desktop. We're not affected by this, but it will certainly encourage us to move sooner rather than later.

arp242• 2 years ago

It is risk management 101, never put all your digital eggs in one (or even a few) baskets.

I mean, plenty of businesses only have penguin eggs in their basket, and some sort of penguin problem would cause major problems for them. I believe that last time this happened was with the leap second thing around 2005 or thereabouts.

"Don't put all your eggs in one basket" sounds nice, but it would mean a completely different independent service all through your stack. That's not really realistic, IMHO.

The bigger issue here is that: 1) some driver update "just" gets pushed (or how does this work?), and 2) the inability to easily do "this is broken, restore to last version". That is even something that could be automatic.

HenryBemis3• 2 years ago

The details (the particular companies / systems etc) of this global incident don't really matter.

But they do matter. This is elementary. It's like saying "playing with matches doesn't matter". This is a problem that has happened before, albeit on smaller scale, and the solution/cure is well known and imho it should be established 2 decades ago to every org on the planet.

This is basic COBIT (or BYOFramework) stuff from 10-15-20 years ago.

How can you push a patch/update without testing it fist? I get it if you are a tiny company with 1 IT person an 20 local PCs. Stuff like that cripples you for a couple of days. But when you are an org, with 10k+ laptops, 500+ servers (half of them MS Win), how can you NOT test each and every update?

If you don't want to have the test/staging environments, then at least wait 1-3-5 days to see what the updates will do to others/the news.

Sorry not sorry guys and gals. I've been auditing systems and procedures for so many years, that this is a basic failure. "One cannot just push an update without testing it first" any update, no matter how small/innocent.

nsagent• 2 years ago

Monocultures are known to be points of failure, but people keep going down that path because they optimize for efficiency (heck, most modern economics is premised on the market being efficient).

This problem is pervasive and effects everything from food supply (planting genetically identical seeds rather than diversified "heirloom" crops) to businesses across the board buying and gutting their competitors thus reducing consumer choice.

It's a tough problem akin to a multi-armed bandit: exploit a known strategy or "waste" some effort exploring alternatives in the hopes of better returns. The more efficient you are (exploitation), the higher the likelihood of catastrophic failure in weird edge cases.

ssss114• 2 years ago

I believe that today they struck the entire crowd… (or should that be cloud)

chucke19923• 2 years ago

I guess Microsoft can now offer some similar to a Crowstrike solution for Azure users.

teeheelol14• 2 years ago

Throwaway account...

CrowdStrike in this context is a NT kernel loadable module (a .sys file) which does syscall level interception and logs then to a separate process on the machine. It can also STOP syscalls from working if they are trying to connect out to other nodes and accessing files they shouldn't be (using some drunk ass heuristics).

What happened here was they pushed a new kernel driver out to every client without authorization to fix an issue with slowness and latency that was in the previous Falcon sensor product. They have a staging system which is supposed to give clients control over this but they pissed over everyone's staging and rules and just pushed this to production.

This has taken us out and we have 30 people currently doing recovery and DR. Most of our nodes are boot looping with blue screens which in the cloud is not something you can just hit F8 and remove the driver. We have to literally take each node down, attach the disk to a working node, delete the .sys file and bring it up. Either that or bring up a new node entirely from a snapshot.

This is fine but EC2 is rammed with people doing this now so it's taking forever. Storage latency is through the roof.

I fought for months to keep this shit out of production because of this reason. I am now busy but vindicated.

Edit: to all the people moaning about windows, we've had no problems with Windows. This is not a windows issue. This is a third party security vendor shitting in the kernel.

tamimio11• 2 years ago

give clients control over this but they pissed over everyone's staging and rules and just pushed this to production.

Interesting..

We have to literally take each node down, attach the disk to a working node..

Probably the easiest solution for you is to go back in time to a previous scheduled snapshot, if you have that setup already.

teeheelol14• 2 years ago

That would make sense but it appears everyone is doing EBS snapshots in our regions like mad so they aren't restoring. Spoke to our AWS account manager (we are a big big big org) and they have contention issues everywhere.

I really want our cages, C7000's and VMware back at this point.

amluto3• 2 years ago

I did approximately this recently, but on a Linux machine on GCP. It sucked far worse than it should have: apparently GCP cannot reliably “stop” a VM in a timely manner. And you can’t detach a boot disk from a VM that isn’t “stopped”, nor can you multi-attach it, nor can you (AFAICT) convince a VM to boot off an alternate disk.

I used to have this crazy idea that fancy cloud vendors had competent management tools. Like maybe I could issue an API call to boot an existing instance from an alternate disk or HTTPS netboot URL. Or to insta-stop a VM and get block-level access to its disk via API, even if I had to pay for the instance while doing this.

And I’m not sure that it’s possible to do this sort of recovery at all without blowing away local SSD. There’s a “preview” feature for this on GCP, which seems to be barely supported, and I bet it adds massive latency to the process. Throwing away one’s local SSD on every single machine in a deployment sounds like a great way to cause potentially catastrophic resource usage when everything starts back up.

Hmm, I wonder if you’re even guaranteed to be able to get your instance back after stopping it.

WTF. Why can’t I have any means to access the boot disk of an instance, in a timely manner? Or any better means to recover an instance?

Is AWS any better?

teeheelol14• 2 years ago

AWS is not any better really on this. In fact 2 years ago (to the day!) we had a complete AZ outage in our local AWS region. This resulted in their control plane going nuts and being unable to shut down or start new instances. Then capacity problems.

khrystoph6• 2 years ago

There is a way with AWS, but it carries risk. You can force detach an instance's volume while it's in the shutting down state, but if you re-attach it to another machine, you risk the possibility of a double-write/data corruption while the instance is still shutting down.

As for "throwing away local SSD", that only happens on AWS with instance store volumes which used to be called ephemeral volumes as the storage was directly attached to the host you were running on and if you did a stop/start of an ebs-backed instance, you were likely to get sent to a different host (vs. a restart API call, which would make an ACPI soft command and after a duration...I think it was 5 minutes, iirc, the hypervisor would kill the instance and restart it on the same host).

When the instance would get sent to a different host, it would get different instance storage and the old instance storage would be wiped from the previous host and you'd be provisioned new instance storage on the new host.

However, with EBS-volumes, those travel from host to host across stop/start cycles and they're attached via very low latency across the network from EBS servers and presented as a local block device to the instance. It's not quite as fast as local instance store, but it's fast enough for almost every use case if you get enough IOPS provisioned either through direct provisioning + correct instance size OR through a large enough drive + large enough instance to maximjze the connection to EBS (there's a table and stuff detailing IOPs, throughput, and instance size in the docs).

Also, support can detach the volume as well if the instance is stuck shutting down and doesn't get manually shut down by the API after a timeout.

None of this is by any means "ideal", but the complexity of these systems is immense and what they're capable of at the scale they operate is actually pretty impressive.

The key is...lots of the things you talk about are do-able at small scale, but when you add more and more operations and complexity to the tool stack on interacting with systems, you add a lot of back-end network overhead, which leads to extreme congestion, even in very high speed networks (it's an exponential scaling problem).

The "ideal" way to deal with these systems is to do regular interval backups off-host (ie. object/blob storage or NFS/NAS/similar) and then just blow away anything that breaks and do a quick restore to the new, fixed instance.

It's obviously easier said than done and most shops still on some level think about VMs/instances as pets, rather than cattle or have hurdles that make treating them as cattle much more challenging, but manual recovery in the cloud, in general, should just be avoided in favor of spinning up something new and re-deploying to it.

Twirrim3• 2 years ago

apparently GCP cannot reliably “stop” a VM in a timely manner.

In OCI we made a decision years ago that after 15 minutes from sending an ACPI shutdown signal, the instance should be hard powered off. We do the same for VM or BM. If you really want to, we take an optional parameter on the shutdown and reboot commands to bypass this and do an immediate hard power off.

So worst case scenario here, 15 minutes to get it shut down and be able to detach the boot volume to attach to another instance.

pizza234• 2 years ago

Based on your description, AWS has another level of stop, the "force stop", which one can use in such cases. I don't have statistics on the time, so I don't know if that meets your criteria of "timely", but I believe it's quick enough (sub-minute, I think).

HelloNurse• 2 years ago

Before reaching the "pushed out to every client without authorization" stage, a kernel driver/module should have been tested. Tested by Microsoft, not by "a third party security vendor shitting in the kernel" that some criminally negligent manager decided to trust.

teeheelol14• 2 years ago

Yeah we have a staging and test process where we run their updated Falcon sensor releasees.

They shit all over our controls and went to production.

This says we don't control it and should not trust it. It is being removed.

Shorn• 2 years ago

Tested by Microsoft

MS don't have testers any more. Where do you think CS learned their radically effective test-in-prod approach?

nox1013• 2 years ago

why would Microsoft be required to test some 3rd party software? Maybe I mis-understood.

spectrumero2• 2 years ago

I always said if you want to create real chaos, don't write malware. Get on the inside of a security product like this, and push out a bad update, and you can take most of the world down.

valval• 2 years ago

So… Write malware?

Reason0779• 2 years ago

”This is not a windows issue. This is a third party security vendor shitting in the kernel.“

Sure, but Windows shares some portion of the blame for allowing third-party security vendors to “shit in the kernel”.

Compare to macOS which has banned third-party kernel extensions on Apple Silicon. Things that once ran as kernel extensions, including CrowdStrike, now run in userspace as “system extensions”.

deepsummer2• 2 years ago

...but still, if the user space process is broken, MacOS will fail as well. Maybe it's a bit easier to recover, but any broken process with non-trivial privileges can interrupt the whole system.

phendrenad22• 2 years ago

Yes... in the same sense that if a user bricks their own system by deleting system32 then Windows shares some small sliver of the blame. In other words, not much.

naib0930• 2 years ago

Back in 2006 the Microsoft agreed to allow kernel level access for Security companies due to an EU anti trust investigation. They were being sued by anti virus companies because they were blocking kernel access in the soon to be released Vista.

https://arstechnica.com/information-technology/2006/10/7998/

numbsafari3• 2 years ago

This is not a windows issue.

Honest question, I've seen comments in these various threads about people having similar issues (from a few months/weeks back) with kernel extension based deployments of CrowdStrike on Debian/Ubuntu systems.

I haven't seen anything similar regarding Mac OS, which no longer allows kernel extensions.

Is Mac OS not impacted by these kinds of issues with CrowdStrike's product, or have we just not heard about it due to the small scale?

Personally, it's a shared responsibility issue. MS should build a product that is "open to extension but closed for modification".

they pissed over everyone's staging and rules and just pushed this to production.

I am guessing that act alone is going to create a massive liability for CrowdStrike over this issue. You've made other comments that your organization is actively removing CrowdStrike. I'm curious how this plays out. Did CrowdStrike just SolarWind themselves? Will we see their CISO/CTO/CEO do time? This is just the first part of this saga.

teeheelol14• 2 years ago

The issue is where it is integrated. You could arguably implement CrowdStrike in BPF on Linux. On NT they literally hook NT syscalls in the kernel from a driver they inject into kernel space which is much bad juju. As for macOS, you have no access to the kernel.

There is no shared responsibility. CrowdStrike pushed a broken driver out, then triggered the breakage, overriding customer requirement and configuration for staging. It is a faulty product with no viable security controls or testing.

Twirrim3• 2 years ago

Most of our nodes are boot looping with blue screens which in the cloud is not something you can just hit F8 and remove the driver.

It took a bit to figure out with some customers, but we provide optional VNC access to instances at OCI, and with VNC the trick seems to be to hit esc and then F8, at the right stage in the boot process. Timing seems to be the devil in the details there, though. Getting that timing right is frustrating. People seem to be developing a knack for it though.

sphar19702• 2 years ago

But it is Windows because the kernel should be able to roll back a bad update, there should NEVER be BSODs.

teeheelol14• 2 years ago

Windows does do that. Crowdstrike sticks it back in at the UEFI level by the looks, because you know, "security".

OhMeadhbh5• 2 years ago

pish! this isn't VM/SP! commodity OSes and hardware took over because customers didn't want to pay firms to staff people who grokked risk management. linux supplanted mature OSes because some dork implied even security bugs were shallow with all those billions of eyes. It's a weird world when MSFT does a security stand down in 2003 and in 2008 starts widening security holes because the new "secure" OS they wrote was a no-go for third parties who didn't want to pay $100 to hire someone who knew how to rub two primes together.

I miss my AS/400.

This might be a decent place to recount the experience I had when interviewing for office security architect in 2003. my background is mainframe VM system design and large system risk management modeling which I had been doing since the late 80s at IBM, DEC, then Digital Switch and Bell Canada. My resume was pretty decent at the time. I don't like Python and tell VP/Eng's they have a problem when they can't identify benefits from JIRA/SCRUM, so I don't get a lot of job offers these days. Just a crusty greybeard bitching...

But anyway... so I'm up in Redmond and I have a decent couple of interviews with people and then the 3rd most senior dev in all of MSFT comes in and asks "how's your QA skills?" and I start to answer about how QA and Safety/Security/Risk Management are different things. QA is about insuring the code does what it's supposed to, software security, et al is about making sure the code doesn't do what it's not supposed to and the philosophic sticky wicket you enter when trying to prove a negative (worth a google deep dive if you're unfamiliar.) Dude cuts me off and says "meh. security is stupid. in a month, Bill will end this stupid security stand down and we'll get back to writing code and I need to put you somewhere and I figured QA is the right place."

When I hear that MSFT has systems that expose inadequate risk management abstractions, I think of the culture that promoted that guy to his senior position... I'm sure he was a capable engineer, but the culture in Redmond discounts the business benefits of risk management (to the point they outsource critical system infrastructure to third parties) because senior engineers don't want to be bothered to learn new tricks.

Culture eats strategy for breakfast, and MSFT has been fed on a cultural diet of junk food for almost half a century. At least from the perspective of doing business in the modern world.

crustycoder2• 2 years ago

Although it's a .sys file, it's not a device driver.

"Although Channel Files end with the SYS extension, they are not kernel drivers."

https://www.crowdstrike.com/blog/technical-details-on-todays...

teeheelol14• 2 years ago

Yeah it's a way of delivering a payload to the driver, which promptly crashed.

Which is horrible!

willmadden2• 2 years ago

It's a shitty C++ hack job within CloudStrike with a null pointer. Because the software has root access, Windows shuts it down as a security precaution. A simple unit test would have caught this, or any number of tools that look for null pointers in C++, not even full QA. It's unbelievable incompetence.

jakebleiberg• 2 years ago

Hello:

I'm a reporter with Bloomberg News covering cybersecurity. I'm trying to learn more about this Crowdstrike update potentially bypassing staging rules and would love to hear about your experience. Would you be open to a coversation?

I'm reachable by email at jbleiberg2@bloomberg.net or on Signal at JakeBleiberg.24. Here's my Bloomberg author page: https://www.bloomberg.com/authors/AWuCZUVX-Pc/jake-bleiberg.

Thank you.

Jake

jpl563• 2 years ago

Obligatory XKCD : dependency [0]

[0] https://xkcd.com/2347/

saagarjha16• 2 years ago

I assure you that Crowdstrike is being paid very well for their software.

jpl563• 2 years ago

For sure, but Crowdstrike eventually hosts a subproject some random person in nebraska has been thanklessly maintaining since 2003 :p

esskay4• 2 years ago

Sitting in our work slack feeling pretty smug that I forced the migration to only Linux servers and Linux or macOS work computers now.

altdataseller3• 2 years ago

I dont understand? Linux servers dont need endpoint protection?

I venture the vast majority of servers with crowdstrike are linux

kozziollek• 2 years ago

I'm assuming Linux implementation is different and the bug is not present there.

drpossum• 2 years ago

The bug only affects Windows machines

kasabali5• 2 years ago

endpoint protection

That's a polite way to say malware

Medox• 2 years ago

Until the VPN is affected by the Domain Controller that is on Windows.

esskay4• 2 years ago

100% nix based here so thankfully zero systems affected. Everything from routers to devices, we have a total blanket ban on any Windows based software.

pulkitsh1234• 2 years ago

Workaround steps: https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/commen...

ndsipa_pomu3• 2 years ago

CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
Workaround Steps:
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.

tjoff2• 2 years ago

Was thinking about a bootable usb-stick that would do that automagically. But I guess it is harder to boot from a usb-stick in these environments than the actual fix.

I guess more feasible and even neater to do it if you have network boot or similar.

entuno3• 2 years ago

There's potentially a huge issue here for people using BitLocker with on-prem AD, because they'll need the BitLocker recovery keys for each endpoint to go in an fix it.

And if all those recovery keys are stored in AD (as they usually are), and the Domain Controllers all had Crowdstrike on them...

mschuster915• 2 years ago

Assuming that they also have a regular Bitlocker password, there's hope with a bit manual effort. https://news.ycombinator.com/item?id=41003893

hulitu7• 2 years ago

This is the best definition of "Single point of failure" i have ever seen.

depr• 2 years ago

Bitlocker keys are apparently not necessary: https://x.com/AttilaBubby/status/1814216589559861673

mejs• 2 years ago

I got around BitLocker and booted into safe mode by setting automatic boot to safe mode via bcdedit https://blog.vladovince.com/mitigating-the-crowdstrike-outag...

commandersaki7• 2 years ago

So booting into safe mode should do the trick right, even if Bitlocker is enabled?

higeorge135• 2 years ago

2000s vibes.

sschueller• 2 years ago

What if you have 50k workstations? Can you even do this remotely?

The problem may be fixed but I can see some companies having a really shit weekend.

jpl563• 2 years ago

Was watching TV this morning in France (TF1, 8:00 CET), the weather forecast map system was out. The journalist just gave us the information as if he was on the radio, telling he was sorry for the system to be failing.

netsharc3• 2 years ago

They sponsor the Mercedes F1 team https://crowdstrikeracing.com/f1/about-partnership/ , who have a race this weekend and practice sessions today. It'd be funny if their cars can't go on track because their computers are down...

netsharc3• 2 years ago

They did go down! https://www.reddit.com/r/formula1/comments/1e71dtn/mercedes_...

But someone probably fixed it, and the cars were able to go out on track for the first practice session.

shubhamjain• 2 years ago

Naive question, if it’s a blue screen of death with a boot loop, how are they going to restore things? Don’t tell me the answer is going to every system manually.

chgs7• 2 years ago

Go to every system manually, boot to safe mode, rename the sys files, run a fix.

Easier to just rebuild from the image. For every windows machine your company has. lol.

CaliforniaKarl6• 2 years ago

Lots of systems (not all) are able to reboot, and have CrowdStrike download the fix before the bad code is able to crash things. But otherwise, yes, you have to go to systems manually.

ogurechny2• 2 years ago

Well, it seems that Windows is not yet accessible remotely when it crashes.

If system administrator had too much free time, and configured every system to probe network on booting, and there is no encryption, it is possible to boot from a minimal Linux image with a script that automatically renames the driver and restarts.

The corporate version of the same approach uses Intel AMT (or however else it is called), but it is only available on licensed hardware from big suppliers.

Otherwise, you can distribute flash drives with the same auto-executing fix to everyone who is able to enter firmware setup, and boot from USB. If it's not available for security reasons, more manual work is required.

But what happens next? If Crowdstrike handled all the security measures, and there was no additional firewall rules, address checks, and so on, your network is now as open as it can be. I suppose certain groups have been celebrating, and uploading gigabytes of data from networks whose detection systems became severed.

misswaterfairy2• 2 years ago

Going to every system manually, then delete a file via command line in Windows' recovery environment.

muixoozie• 2 years ago

Remote access control (e.g. iDRAC) or physical access.

mro_name2• 2 years ago

heavy clouds this morning.

Maybe time to reconsider how solid a ground clouds are.

ch_123• 2 years ago

Would this issue not affect bare metal as well?

berkaydumaner• 2 years ago

Hi guys, what is the KB code of this update?

uitgewis• 2 years ago

https://archive.is/AWVuv

r2vcap• 2 years ago

This is why I don't use Windows and refuse any SWE jobs that require Windows machines. Additionally, I believe kernel-level game anti-cheat software should be banned.

clydethefrog• 2 years ago

10 hours ago someone posted a critical post about CrowdStrike on the "wallstreetbets" subreddit.

https://old.reddit.com/r/wallstreetbets/comments/1e6ms9z/cro...

elorant2• 2 years ago

How can an antivirus update affect Azure's servers?

dboreham8• 2 years ago

TBF although I worried about this possibility the first time the IT dude wandered into my office in 1989 holding a floppy he said he wanted to put into all the PCs we had (we had no PCs), it has actually taken a very long time for the shit to hit the fan.

nullify88• 2 years ago

The sheer coverage of this outage across multiple businesses and industries, the impact must be greater than some of the malicious cyber attacks from ransomware, worms etc.

fudged713• 2 years ago

This title doesn’t nearly describe the breadth and severity of the problem…

solidninja• 2 years ago

Ah the "move fast and break things" philosophy gets a demonstration.

topspin3• 2 years ago

That's misplaced. Windows is an ancient platform. CrowdStrike is ubiquitous and routinely updated. There was no "move fast" here, at least on the part of the people operating these systems.

account428• 2 years ago

Pushing an update to all clients worldwide simultaneously isn't "move fast"?

endstart• 2 years ago

The workaround suggests removing a file with .sys extension. What does the file do normally? If removed, what happens to the state of security on that system?

saagarjha16• 2 years ago

.sys files on Windows are typically drivers and driver-related files

codeulike• 2 years ago

Microsoft are going to be pissed that this is widely being discussed as a Microsoft outage. Do AV vendors like Crowdstrike need a license or something from Microsoft to push these kernal driver based things? Or is it just like anyone can make one?

1oooqooq13• 2 years ago

yes, and they have two. it is a windows problem.

scopeh• 2 years ago

How many people still believe the "cloud" was worth it? Maybe we should go back to the days of buying software and running it ourselves with our own infrastructure.

I know, I'm dreaming.

ogurechny2• 2 years ago

They all should have used some expensive corporate-and-government-level product that promises protection against exactly that kind of large scale attack on infrastructure.

junto5• 2 years ago

I assume this is irony, since isn’t Crowdstrike exactly that these days?

illiac7863• 2 years ago

Is it believed to be an attack? I only saw mention of a bug so far.

fx1994• 2 years ago

That piece of "AV software" slowed down my brand new corporate i7 Lenovo to shit so I switched to M2 Pro. Best decision ever.

sensanaty• 2 years ago

Until corporate decides to install new MDM software on the usually-blazingly-fast apple silicon chips :(

I can't even open files larger than 500 lines without my whole system slowing to a crawl because of the insanely aggressive and slow "antivirus" bloatware the MDM forces on me.

qxmat• 2 years ago

Their Windows sensor has made development almost unworkable. Not sure why but I haven't noticed the OSX sensor slow things down appreciably. I suspect my Windows profile is configured to be more aggressive?

WatchDog3• 2 years ago

This was apparently caused by a faulty "channel file"[0], which is presumably some kind of configuration database that the software uses to identify malware.

So there wasn't any new kernel driver deployed, the existing kernel driver just doesn't fail gracefully.

[0] https://x.com/brody_n77/status/1814185935476863321

vesinisa• 2 years ago

Why on earth don't they have staged rollouts for updates?

tehlike5• 2 years ago

Everytime i look into such catastrophic issues, it always boils down to lack of robust canarying mechanisms.

They have enough client base that they can even run an A/B test on the whole binary level, but no.

malkia3• 2 years ago

Also, why not have some sort of graceful degradation (well kind of), like: OS Boots, loads CS driver, the driver loads some new feature/config, and before/after new recent thing ("runtime flag") marked whether it successfully worked, and if not on the next reboot that thing gets either disabled, or the previous known good config (obviously some combination of things might cause another issue), but instead of blindly rebooting to the same state....

averageRoyalty6• 2 years ago

More importantly, why are CS customers not validating? Upstream patches should be treated as faulty/malicious if not tested to show otherwise, especially if they're kernel level.

callalex3• 2 years ago

Why put effort in engineering when you can just fear monger in marketing and buy politicians in sales?

roschdal• 2 years ago

USA can no longer be trusted to supply import technology for the world.

pjmlp2• 2 years ago

Yet another good example why liability in software should already be a common thing.

patates4• 2 years ago

This company has post-apocalyptic style photos to make you panic-buy their solution.

https://ibb.co/Bc6n527

"62 minutes could bring your business down"

I guess they could bring all the businesses down much quicker.

edit: link https://www.crowdstrike.com/en-us/#teaser-79minutes-adversar...

hulitu7• 2 years ago

"62 minutes could bring your business down"
I guess they could bring all the businesses down much quicker.

It is because the buyer does not get the message. And, when they get it, it is too late.

sofixa4• 2 years ago

And for much longer.

briankelly2• 2 years ago

Boy, that is some corny branding.

bob10292• 2 years ago

I agree but I've also personally witnessed how effective this crap is on a certain cohort of IT managers. You can see the 3 or 4 gears grinding together in their head... something like "oh my goodness look at all the things I get for one purchase order!".

joenot443• 2 years ago

Their "Statement" is remarkably aloof for having brought down flights, hospitals, and 911 services.

"The issue has been identified, isolated and a fix has been deployed."

Maybe I'm misunderstanding what I read elsewhere, but is the machine not BSODing upon boot, prior to a Windows Update service being able to run? The "fix" I see on reddit is roughly:

Workaround Steps:

1. Boot Windows into Safe Mode or the Windows Recovery Environment

2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

3. Locate the file matching “C-00000291*.sys”, and delete it.

I'm horrified at the thought of tens of thousands of novice Windows users digging through System32 to delete driver files; can someone set my mind at ease and assure me this will eventually be fixed in an automated fashion?

https://www.crowdstrike.com/blog/statement-on-windows-sensor...

gulbanana• 2 years ago

It cannot and will not be fixed in an automated fashion.

wesmin• 2 years ago

Good luck to Joe Schmoe in the IT dept who has to do this over and over flawlessly

andrybak2• 2 years ago

Their "Statement" is remarkably aloof for having brought down flights, hospitals, and 911 services.

Their lawyers certainly won't allow mentioning such dramatic (is "dramatic" appropriate here?) consequences.

hskalin• 2 years ago

And I guess if they use bitlocker then they need to enter the key as well? Imagine doing that to thousands of computers

throwawaysleep• 2 years ago

I'm horrified at the thought of tens of thousands of novice Windows users digging through System32 to delete driver files; can someone set my mind at ease and assure me this will eventually be fixed in an automated fashion?

Nope. Both my orgs (+2000 each) have sent out a Google doc to personal emails on using CMD Prompt to delete that file.

Anyone with technical experience is being drafted to get on calls and help people manually delete this file.

Jyaif• 2 years ago

My employers pays Crowdstrike to double my build times. Quite astounding really.

guenthert• 2 years ago

afaiu google (and I presume other operators of large number of computers) deploy updates to their software first to a small set of nodes and only if after a given time the update has been deemed successful, continue to update an increasingly larger set til complete.

Isn't this done as well with automatic updates of end user software or embedded systems and if not, why not?

ivxvm2• 2 years ago

Things like hospitals, airlines, 911, should have multiple systems with different software stacks and independent backends running in-parallel, so that when one infra goes down they can switch to another.

judgementalfish• 2 years ago

There is an enormous cost associated with the kind of redundancy you're talking about. Capitalism prevents us from being set up in the way you're describing. Why invest in company A if company B can run the same business with half the operational expenses? Shareholder profit above all.

SkyBelow• 2 years ago

Is company B allowed to take the full brunt of all the problems when there is a failure, or does government protect it by limiting damages? If company B's cheaper choice leads to harm and lets people and estates sue company B into the ground, then company A is a safer investment even if it has lower returns. If government interaction limits such recovery options, then that is what leads to company B's higher returns not also having higher risks, so they'll be the better investment. But that is a result of government intervention, not the economic system in play.

tflinton• 2 years ago

For some areas of our critical systems we have three independent software groups program the same exact system on different infrastructure. Just for moments like these...

lucasRW8• 2 years ago

Good day for OSINTers, APTs, redteamers, to find out who uses Crowdstrike on their endpoints.

gnuser• 2 years ago

dumb techbro c-suites: what, why would you have an issue with a proprietary closed source app that frequently self updates and sends tons of data to a third party while essentially being a backdoor? We said we wanted security and this has Security(tm) all over the literature! Look we even have dashboards for the gui-ninjas like the security team!

attentive11• 2 years ago

It's backdoor as a feature. It even has a cute name - "CrowdStrike Real Time Response".

AndyMcConachie• 2 years ago

Finally the crowd has struck!

ba-dum ching!

commercialnix• 2 years ago

All my customers endpoints are on Linux based endpoints. Because our users' Windows apps run in vdi with disposable instances based off snapshots and highly restrictive networking on the Linux endpoints, none of our users are effected.

Running Windows on bare-metal was always obviously very stupid. The consequences of such stupidity are just being felt now.

dmarble• 2 years ago

Plot twist: The * in C-00000291*.sys is "-block-ultron"

Premature deployment of Crowdstrike AGI disaster response plan.

KingOfCoders2• 2 years ago

All down had no backup plan.

pharos92• 2 years ago

1. This is why kernel modules are a bad idea 2. This is why centralism is a bad idea 3. This is why sacrificing stability for security is a bad idea 4. Security still needs to factor in security of supply - not just data safety

imdsm• 2 years ago

Centralisation in a nutshell. Monopolies so big that they become globally fragile. CloudFlare outages break a lot of the internet, and now we can see, Windows-based updates bricking machines across the world.

We've all pushed bad updates but how was this not tested?

2-3-7-43-18074• 2 years ago

i've never heard of crowdstrike ever but it (co-)runs half of the essential IT infrastructure, worldwide?

(also, great choice of name i must say)

BoredPositron• 2 years ago

I only know them because their CEO is a relatively good amateur racing driver lol.

2-3-7-43-18074• 2 years ago

seems like he's also a relatively good amateur kernel driver developer - or at least his team is

and he's the former CTO of mcafee?

badrabbit3• 2 years ago

Their stock price will suffer but they can waive license fees for a year or so for every endpoint affected (~$50).

They better pin this on a rogue employee, but even then, force pushing updates shouldn't be in their capability at all! They must guarantee removal of that capability.

Lawsuits should be interesting. They offer(ed?) $1 mil breach insurance to their customers, so if they were to pay only that much per customer this might be compensation north of $10B. But to be honest, wouldn't surprise me if they can pay up without going bankrupt.

The sad situation is, as twitter people were pointing out, IT teams will use this to push back against more agents for a long time to come. But in reality, these agents are very important.

Crowdstrike Falcon alone is probably the single biggest security improvement any company can make and there is hardly any competition. This could have been any security vendor, the impact is so widespread because of how widely used they are, but there is a reason why they are so widely used to begin with.

Oh and just fyi, the mitigation won't leave you unprotected, when you boot normal, the userspace exe's will replace it with a fixed version.

InsideOutSanta11• 2 years ago

the single biggest security improvement

Clearly not, unless you don't count a world-wide economic and societal disruption of unprecedented scale a security incident.

This could have been any security vendor

...that apparently deploys Kernel Extensions to millions of Windows devices at once, without any staggering.

there is a reason why they are so widely used to begin with

Because companies need to check a box, and purchasing CrowdStrike checks that box.

akho• 2 years ago

Crowdstrike Falcon

“Cybersecurity’s AI-native platform for the XDR era”.

I hope there’s a blockchain somewhere in it.

_kb2• 2 years ago

Assuming this event itself isn't malicious, what an excellent POC for something that is. I sure hope every org out there with this level of market reach has good security in place. It's certainly going to be getting some probing after this.

nyx_land2• 2 years ago

worded badly whatever

rfoo4• 2 years ago

Because it's not a Windows update.

dbbk• 2 years ago

Because it's not a Windows update perhaps

smokel• 2 years ago

worded badly whatever

I feel your pain. Perhaps it is time to increase the karma level required for downvoting.

dark-star• 2 years ago

That's what you get for letting a company install a root kit on your servers and desktops ;-)

I mean, don't they do canary updates on CrowdStrike too? Every Windows admin has done this for the last 5+ years, test Windows updates on a small number of systems to see if they are stable. Why not do the same for 3rd party software?

kingkongjaffa• 2 years ago

It's kind of surprising so much infra was using windows servers or windows cloud VMs for these things. I assumed these systems would all be Linux VMS in Azure/AWS/GCP at this point.

on https://azure.status.microsoft/en-gb/status the message is currently:

We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD) and get stuck in a restarting state.

junto5• 2 years ago

Welcome to the enterprise. Where “lift and shift” was sold to corporate CTO’s as better than maintaining their own IT infrastructure.

patates4• 2 years ago

This company has post-apocalyptic style photos to make you panic-buy their solution.

https://ibb.co/Bc6n527

"62 minutes could bring your business down"

I guess they could bring all the businesses down much quicker.

https://www.crowdstrike.com/en-us/#teaser-79minutes-adversar...

(Repeating my comment because other story is duped)

dang3• 2 years ago

(Repeating my comment because other story is duped)

Please don't do this! It makes merging threads a pain. It's better to let us know at hn@ycombinator.com and then we'll merge the threads so your comment shows up in the main one.

patates4• 2 years ago

I'm so sorry, TIL.

Thorentis• 2 years ago

Perversely, this may make many companies no longer invest in this type of cyber security software. Which may lead to a whole host of other problems...

mnau• 2 years ago

Yeah, these events will be fun once new product liability directive (that includes sw) comes into force.

nynyny7• 2 years ago

Crowdstrike marketing slogan on their website: "A radical new approach proven to stop breaches". I'll give them that: Putting all Windows computers within a company into an endless BSOD loop is a very radical approach to stop breaches. :)

CoastalCoder12• 2 years ago

"We breach your systems to hackers can't!"

tm-guimaraes• 2 years ago

How does such a huge company do “full deploys” like this? At this number of endpoints, only a few % should have been updated (and faced the problems) before a full rolout

This is not a small startup with some SaaS, these guys are in most computers of too many huge companies. Not rolling out the updates to everyone at the same time seems just too obvious

ndsipa_pomu3• 2 years ago

This incident definitely makes a good case for staggered deploys of patches.

dorkwood• 2 years ago

Is this something that could be solved by building AI code review directly into git clients? I can't help thinking Claude 3 would have caught this.

machinekob• 2 years ago

Yes AI already solved it, my god why people are so high on LLM's solving everything

redrove• 2 years ago

LMAO

Yeah let's throw an LLM at the C++ kernel driver and auto-push to prod

tslocum• 2 years ago

Poe's law in action.

tehlike5• 2 years ago

Rolling out updates in an A/B test slowly is the only way to reduce the occurrence of such issues _significantly_. There's no other way, literally, nothing.

alibarber3• 2 years ago

I have been told 'not to worry' because it isn't a cyber attack. Yet the outcomes we are seeing feel a lot like the doomsday predictions of what a cyberattack would do. It is almost as if we are experiencing the cybersecurity/warfare equivalent of 'friendly fire'.

elorant2• 2 years ago

For years now antivirus solutions have ridiculous amount of control over the OS. I accidentally installed an adware antivirus the other day that was bundled-up with a third party software, and I had to boot to Linux to manage to completely remove the damn thing from Windows. The uninstall option left a process running that couldn’t be forcefully killed.

Microsoft needs to take control and forbid anyone and anything from running software with that kind of behavior.

simonjgreen2• 2 years ago

CrowdStrike should have learned the lesson from the more seasoned players in the industry to slow roll their updates and observe.

echoangle3• 2 years ago

Why would they roll out this update globally to all users immediately? Isn’t it normal to do gradual rollouts? Or did this update contain some critical security fix they wanted everyone to have as fast as possible?

madisp• 2 years ago

if I'm reading this correctly the short interest for the stock doubled over June? :)

https://www.nasdaq.com/market-activity/stocks/crwd/short-int...

nicholasbraker• 2 years ago

This article seems more relevant than ever and was posted a few days ago: https://ea.rna.nl/2024/07/12/no-it-really-no-i-t/

pxc• 2 years ago

Vendors of tools like this drive the cybersecurity industry discourse, so 'defense in depth' often practically sorta means 'add more software that does more things'.

But maybe this kind of thing can actually impart the lesson that loading your OS up with always-on, internet-connected agents that include kernel components in order to instrument every little thing any program does on the system is, uh, kinda risky.

But maybe not. I wonder if we'll just see companies flock to alternative vendors of the exact same type of product.

zmmmmm5• 2 years ago

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it.

My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer starting up and then in the critical path of every network connection the computer makes. The risk of ZScaler being a central point of failure is not considered. But - the risk of failing the compliance checkbox it satisfies is paramount.

All over the place I'm seeing checkbox compliance being prioritised above actual real risks from how the compliance is implemented. Orgs are doing this because they are more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting. So we need to hold regulatory bodies accountable as well - when they frame regulation such that organisations are cornered into this they get to be part of the culpability here too.

SuperSandro2000• 2 years ago

Sounds familiar

RedShift14• 2 years ago

Don't put your eggs in one basket, I use multiple anti-virus products so that if one blows up at least not all computers are affected. Looks like my old wisdom is still new wisdom.

Clarification: I mean that every computer has one anti-virus product, but not every computer has the same anti-virus product. I'm not installing multiple anti-virus products on the same computer.

cqqxo4zV46cp3• 2 years ago

This is much easier applied personally than it is to a 30k person organisation. No need to be condescending.

NDizzle4• 2 years ago

You use multiple anti-virus products. Let's assume you use 3. Do you have multiple clusters of machines, each running their own AV product, so in case one has this problem the other two are unaffected?

How much overhead are we talking about here? Because if you're just using multiple AV software installed on one machine, 1) holy shit, the performance penalty, 2) you'd still be impacted by this, as CS would have taken it down.

xorcist5• 2 years ago

Not sure that's a great idea. This stuff tends to have very high privilege access.

It's enough that one of your anti virus vendors get hacked for your whole organization to get owned...

fire_lake• 2 years ago

If you have all of them on a critical path then your risk of blow up increases!

Lovesong• 2 years ago

Was there any situation where having 3 anti-virus was more beneficial than having only 2?

interludead5• 2 years ago

Indeed a wise strategy

baxtr3• 2 years ago

hey, but at least (a) we have a process (b) we documented it and (c) we review it regularly!

what can go wrong?!

rubatuga• 2 years ago

There should be a new term called compliance hell.

theallan• 2 years ago

The flip side is, if you don't do auto updates and an exploit is published and used against you and you haven't yet tested / pushed the patch, that you would have been protected against if it had auto updated, you are up the creak without a paddle in that situation as well.

To some degree you have to trust the software you are using not to mess things up.

cqqxo4zV46cp3• 2 years ago

Yes. And the vast majority of the time, it doesn’t mess things up.

The notion that you may take on risk to net alleviate risk is somehow lost on a lot of people in these conversations.

zmmmmm5• 2 years ago

So since I do mission critical healthcare I do run into this concept. But it's not as unresolvable as you portray. Consider for example HIPAA "break the glass" requirement. It says that whatever else you implement in terms of security you must implement a bypass that can be activated by routinely non-authorised staff to access health information if someone's life is in danger.

Similarly, when I questioned, "why can't users turn off ZScaler in an emergency" we were told that it wouldn't be compliant. But it's completely implementable at a technical level (Zscaler even supports this). You give users a code to use in an emergency and they can activate it and it will be logged and reviewed after use. But the org is too scared of compliance failure to let users do it.

tempaway45751445• 2 years ago

You are completely right. BTW It wasn't a software update, it was a content update, a 'channel file'. Someone didn't do enough testing. edit: or any testing at all?

https://x.com/George_Kurtz/status/1814235001745027317

https://x.com/brody_n77/status/1814185935476863321

Shank2• 2 years ago

It's an automatic update of the product. Semantic "channel vs. binary" doesn't indicate anything. If your software's definition files can cause a kernel mode driver to crash in a bootloop you have bigger problems, but the outcome is the same as if the driver itself was updated.

morpheuskafka2• 2 years ago

That’s even worse—-they should be fuzz testing with bad definitions files to make sure this is safe. Inevitably the definitions updates will be rushed out to address zero days and the work should be done ahead of time to make them safe.

bravetraveler4• 2 years ago

That makes me even more unsettled! Shouldn't this be closer to metadata than operational/mechanical?

Feels like they made unsafe data for the format they created. Untrustworthy. To your point, they aren't testing.

weinzierl4• 2 years ago

As I understand, this only manifests after a reboot and if the 'content update' is tested at all it is probably in a VM that just gets thrown away after the test and is never rebooted.

Also, this makes me think:

How hard would it be to craft a malware that has the same signature as an important system file?

Preferably one that doesn't cause immediate havoc when quarantined, just a BSOD after reboot, so it slips through QA.

I don't believe this is what's happened, but I think it is an interesting threat.

glimshe• 2 years ago

It passed all unit tests!

account428• 2 years ago

President & CEO CrowdStrike, Former CTO of McAfee

Well that's certainly a track record.

andyjohnson0• 2 years ago

It wasn't a software update, it was a content update, a 'channel file'

Because I know nothing about Crowdstrike... what is a "channel file"? Some kind of config file?

chrisjj14• 2 years ago

Since when is this content not software, just because it is.not an .exe?

drivebycomment• 2 years ago

This will go down as one of the worst examples of communication during an outage.

philipwhiuk4• 2 years ago

It's possible there's no human involvement from detection to deployment.

bonoboTP4• 2 years ago

more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting.

Duh, else there would be no need to audit them to force compliance, they'd just do it by themselves. The only reason it needs forcing is that they otherwise aren't motivated enough.

lucianbr4• 2 years ago

Good point. But the audit seems useless now. It's supposed to prevent the carelessness from causing... this thing that happened anyway.

Sure, maybe it prevented even more events like this from happening. But still.

interludead5• 2 years ago

The dynamic between compliance and operational integrity

lmm6• 2 years ago

Audit is papering over the problem rather than fixing it. The only way to make them responsible is to put real liability on them.

stefan_2• 2 years ago

When we think "security" on HN we think about the people who escalate wiggling voltages at just the right time into a hypervisor shell on XBox, but I've had to recognize that my learned bias is not correct in the real world. In the real world, "computer security" is a profession full of hucksters that can't tell post-quantum from heap and whose daily work of telling people repeatedly to not click links in Outlook and filling out checklists made by people exactly like them has essentially no bearing on actual security of any sort.

Shank2• 2 years ago

It's driven by a lot of things. Part of it is driven by rising cyber liability insurance rates, for one. A lot of organizations would rather not pay for CrowdStrike, but the premiums for not having an "EDR/XDR/NGAV" solution can be astoundingly high at-scale.

Fundamentally there's a lot of factors in this ecosystem. It's really wild how incentives that seem unrelated end up with crazy "security" products or practices deployed.

l33t73322736• 2 years ago

I’ve met highly paid “security engineers” that talked about not really being into programming or being okay with python but everything else is too complicated.

It shocks me that such a low level of technical competence is required.

tristor• 2 years ago

In the real world, "computer security" is a profession full of hucksters

Always has been. The information security model is about analogizing digital systems as physical systems, and employing the analogues of those physical controls that date back hundreds of years on those digital systems. At no point, in my relatively long career, have I ever met anyone in Information Security who actually understands at depth anything about how to secure digital systems. I say this as someone who has spent a lot of my career trying to do information security correctly, but from the perspective of operations and software engineering, which is where it must start.

The entire information security model the world works with is tacking on security after the fact, thinking you need to builds walls and a vault door to protect the room after the house has already been built, when in fact you need to build the house to be secure from the start because attacks don't go through doors, attacks are airborne (I recognize the irony of my analogizing digital concepts to physical concepts surrounding security, but I do it because of any infosec people that may read my comment so they can understand my point).

Because of this model, we have gone from buying "boxes" to buying "services", but it has never matured away from the box-checking exercise it's been since day one. In fact, many information security people have /no training or education/ in security, it's entirely in regulatory compliance.

2-3-7-43-18074• 2 years ago

I don't even want to know how many mission critical systems automatically deploy open source software downloaded from github or (effectively random) public repositories.

rini17• 2 years ago

Unlike Windows, there is at least the option to use curated software distributions such as Debian or RH that won't apply random stuff from upstream repositories.

interludead5• 2 years ago

These risks must be carefully managed that's it I think

higeorge135• 2 years ago

So true! It seems like all of these were invented to create another market for b2b saas security, audit, monitoring, etc. companies. Nobody cares about actual security or infrastructure anymore. Everything is just buying some subscription for random saas companies, not checking their permissions and grant policies and ticking boxes because... compliance.

zh3• 2 years ago

"The metric becomes the measure", i.e. Goodhart's Law.

https://en.wikipedia.org/wiki/Goodhart%27s_law

kmlx2• 2 years ago

the risk of failing the compliance checkbox it satisfies is paramount.

i'm curios as to what compliance is there to be satisfied to necessitate such a hardcore measure?

rcxdude• 2 years ago

There's a reasonable number of circumstances where there are cybersecurity standards that get imposed on organisations: insurance, from a customer, or from the government (especially if they are a customer). These standards are usually fairly reasonably written, but they are also necessarily vague and say stuff like "have a risk assessment", and "take industry-standard precautions". This vagueness can create a kind of escalation ratchet: when people tasked with (or responsible for) compliance are risk-averse and/or lazy, they will essentially just try to find as many precautions as they can find and throw them all in blanket-style, because it's the easiest and safest way to say that you're in compliance. This is especiallly true when you can more or less just buy one or two products which promise to basically tick every possible box. And if something else pops up as a suggestion, they'll throw that in too. Which then becomes the new 'industry standard', and it becomes harder to justify not doing it, and so on.

victorbjorklund• 2 years ago

I worked in orgs where customers put a certain security standard in the contract. So if you fail that you are kind of in breach of contract

izacus3• 2 years ago

Of course, disabling those auto updates will have you fail the external security audit and now your security team needs to fight with the rest of the leadership in the company explaining why you're generating needless delays, costs against the "state of the art in security industry" and why your security guys are smarter than the people who have the power to approve or deny your security certification.

chii4• 2 years ago

you fail the external security audit

aka, you fail the cover-your-ass security, rather than actual security.

lysp• 2 years ago

Or lose IT/security insurance for not installing or disabling it.

guax5• 2 years ago

Well. Now you have something to point to. Next RFO you can ignore the blameless part and point to a executive override of a technical decision.

throwaway7ahgb3• 2 years ago

I have been in these audits and nowhere does it say that software has to be 'auto updated', this is a ridiculous statement and requirement.

What a proper audit will look for is a update and testing control with supporting evidence.

anchochilis2• 2 years ago

What I don't understand is why they don't have a canary update process. Server side deployments do this all the time. You would think Windows would offer that to their institutional customers, for all types of updates including (especially) 3rd party.

btbuildem6• 2 years ago

I've taken part in some security audits where I work. They're not a joke only because they're a tragic story of incompetence, hubris, and rubberstamping. They 100% focus on checking boxes and cargo-culting, while leaving enormous vulnerabilities wide open.

remram5• 2 years ago

Why do we call managers "leaders" now? That's not what they are.

belter4• 2 years ago

So we need to hold regulatory bodies accountable as well - when they frame regulation such that organisations are cornered into this they get to be part of the culpability here too.

No, we need to hold Architects accountable, and this is the core of the issue. Creating systems with single, outsourced responsibility, in the critical path.

ta124312• 2 years ago

As a CTO, when your company goes down you get fired.

When every company goes down you get let off.

The sensible thing is follow the herd and centralise. You're outsourcing the risk to your own job.

capybara_2020• 2 years ago

I find that in today's world it is no longer about one person being "accountable". There is always an interplay of factors, like others have pointed out cyber security has a compliance angle. Other times it is a cost factor, redundancy costs money. Then there is the whole revolving door of employees coming and going, so institutional knowledge about why a decision was made lost with them.

That is hard to do for even a small company. How do you balance all that out for critical infrastructure at a much larger scale?

citrin_ru2• 2 years ago

The problem is that even knowing that this likely to happen many companies would still put CrowdStrike into a critical system for the sake of security compliance / audit. And it's not even prioritization of security over reliability because incentives are to care more about check-boxes in the audit report than about the actual security. Looks like almost no party in this tragic incident had a strong incentive to prevent it so it's likely to happen again.

lelanthran3• 2 years ago

The risk of ZScaler being a central point of failure is not considered. But - the risk of failing the compliance checkbox it satisfies is paramount.

You're conflating Risk and Impact, and you're not considering the target of that Risk and that Impact.

Failing an audit:

1. Risk: high (audits happen all the time)

2. Impact to business: minimal (audits are failed all the time and then rectified)

3. Impact to manager: high (manager gets dinged for a failing audit).

Compare with failing an actual threat/intrusion:

1. Risk: low (so few companies get hacked)

2. Impact to business: extremely high

3. Impact to manager: minimal, if audits were all passed.

Now, with that perspective, how do you expect a rational person to behave?

[EDIT: as some replies pointed out, I stupidly wrote "Risk" instead of "Odds" (or "Chance"). Risk is, of course, the expected value, which is probability X impact. My post would make a lot more sense if you mentally replace "Risk" with "probability".]

Freak_NL• 2 years ago

Almost correct, but you mean 'chance' where you write 'risk':

    Risk = Chance × Impact

The chance of failing an audit initially are high (or medium, present at least). The impact is usually low-ish. It means a bunch of people need to fix policy and set out improvement plans in a rush. It won't cost you your certification if the rectification is handled properly.

It's actually possible that both of your examples are awarded the same level of risk, but in practice the latter example will have its chance minimized to make the risk look acceptable.

noahmasur• 2 years ago

I think the parent was correct in the use of the word "Risk"; it's different than your definition, which appears to be closer to "likelihood".

Risk is a combination of likelihood and impact. If "risk" were just equivalent to "likelihood" then leaving without an umbrella on a cloudy day would be a "high-risk situation".

A rational person needs to weigh both the likelihood and impact of a threat in order to properly evaluate its risk. In many cases, the impact is high enough that even a low likelihood needs to be addressed.

rakoo2• 2 years ago

Now, with that perspective, how do you expect a rational person to behave?

What is a business to do, maximize business or manager contentment ?

vasco3• 2 years ago

Moreover no manager gets dinged for "internet-wide" outages unfortunately, so the compliance department keeps calling the shots. The amount of times I've had to explain there's no added security in adding an "antivirus" to our linux servers as we already have proper monitoring at eBPF level is annoying.

collinc777• 2 years ago

You’re conflating risk and frequency

heisenbit• 2 years ago

ZScaler and similar software also has some hidden costs: Performance and all the other fun that comes with a proxy between you and the server you connect to.

maxerickson• 2 years ago

If you read their comment holistically, they obviously agree with you and think that the outcome of the audit should be more meaningful.

danaris3• 2 years ago

But...surely you're also missing another point of consideration:

Single point of failure fails, taking down all your systems for an indeterminate length of time:

1. Risk: moderate (an auto-updating piece of software without adequate checks? yeah, that's gonna fail sooner or later)

2. Impact to business: high

3. Impact to manager: varies (depending on just how easy it is to spin the decision to go with a single point of failure rather than a more robust solution to the compliance mandate)

makeitdouble2• 2 years ago

1. Risk: low (so few companies get hacked)

I wonder how much that's still true now that ransomware has apparently become viable.

Finding an insecure target, setup the data hostage situation, have the victim come to pay is scalable and could work in volume. If getting small money from a range of small targets becomes profitable, small fishes will bear sinilar risks to juicier targets.

beeboobaa35• 2 years ago

3. Impact to manager: minimal, if audits were all passed.

I don't know about you, but I'll be making sure everyone knows that the manager signed off on the spectacularly stupid idea to push through an update on a friday without testing.

cvoss• 2 years ago

Now, with that perspective, how do you expect a rational person to behave?

They'd deploy the software on the critical path. That's exactly GP's point, isn't it? That's why GP explicitly wants us to shift some of the blame from the business to the regulators. GP advocates for different regulatory incentives so that a rational person would then do the right thing instead of the wrong thing.

greggsy2• 2 years ago

Risk: low (so few companies get hacked)

I’m at risk of sounding like chicken little, the reality is companies are getting popped all the time - you just don’t hear about them very often. The bar for media reporting is constantly being raised to the point where you only hear about the really big ones.

If you read through any of the weekly Risky Biz News posts [1] you’ll often see a five or more highly impactful incidents affecting government and industry, and they’re just the reported ones.

[1] https://news.risky.biz/

ThePowerOfFuet• 2 years ago

Risk: low (so few companies get hacked)

Come on.

p0seidon• 2 years ago

This is 100% the reality

monkeydust2• 2 years ago

"All over the place I'm seeing checkbox compliance being prioritised above actual real risks from how the compliance is implemented."

Great statement and one that needs to be seriously considered - would DORA regulation in the EU address this I wonder? Its a monster piece of tech legislation that SHOULD target this but WILL it - someone should use todays disaster and apply it to the regs to see if its fit for purpose.

piker3• 2 years ago

Broken regulations? Fix by adding more!

emdowling• 2 years ago

EU tech regulation actually addressing an issue effectively? I wouldn't hold my breath, but there is a first time for everything.

wjnc• 2 years ago

Emphatically NO. Involved in (IT) Risk and DORA in a firm that actually does IT risk scenario planning (the sort opposite of checkbox compliance). DORA is rubber stamping al the way round. One caveat is that we are way ahead of DORA, so treating DORA as a checkbox exercise might be situational. But I haven’t noticed a place where the rubber hits the road regulatory wise. It’s too easy to stay in checkbox compliance if the board doesn’t see IT-risk as a major concern. I’m happy one of our board members does. We’ve gone so far as to introduce a person and paper based credit line, so we can continue an outgoing cashflow if most of our processes fail (for an insurer).

dijit4• 2 years ago

It's possible that CrowdStrike heavily incentivises being left to update itself.

Removing the features that would allow sysadmins to actually do it automatically, even via the installer itself- would definitely be one way, but another one could be aggressive focus-stealing nags (similar to Windows' own nags) which in a server environment can actually cause some major issues, especially when automating processes in Windows (as you need to close the program when updating).

I think it's easy to blame the sysadmins, but I would also be remiss if I didn't point out that in the Windows world we have been slowly accepting these automatic dark patterns and alternative (more controlled) mechanisms have been removed over time.

I almost don't recognise the deployment environment today as to what it was in 2004; and yes, 20 years is a long time, but the total loss of control over what a computer is doing is only going to make issues like this significantly more common.

mr_mitm11• 2 years ago

They say it was caused by a faulty channel file. I don't know what a channel file is, and they claim to not rely on virus signatures, but typically anti virus product need the latest signatures all the time and poll them probably once an hour or so. So I'm not surprised that an anti virus product wants to stay hyper updated and updates are rolled out immediately to everyone globally.

bandrami2• 2 years ago

In any kind of serious environment the admin should not have any interaction with any system's screen when performing any kind of configuration change. If it can't be applied in a GPO without any interaction it has no business being in a datacenter.

lucasRW8• 2 years ago

Electricity too is a SPOF.

RobotToaster4• 2 years ago

Most mission critical medical systems have a backup generator.

jrockway• 2 years ago

Zscaler is truly amazing. It can't do HTTP/2. Our product is HTTP/2-only. So we can't use our own product at work.

Avamander4• 2 years ago

Their visibility and process in general for handling abuse of their services is also abysmal.

mrweasel4• 2 years ago

I know a few people who have Zscaler deployed at work. It will routinely kick them of the internet, like multiple times a day. It has gotten to the point where they can sort of tell in advance that it's about to happen.

The theory so far it that it's related to their activities, working in DevOps they will sometimes generate "suspicious" traffic patterns which will then trigger someone policy in Zscaler, but they're not actually sure.

supertrope• 2 years ago

ZScaler itself uses port 443 UDP, but blocks QUIC. The last time I checked it didn't support IPv6 so they told customers to disable IPv6. Security software is legacy software out of the box and cuts the performance of computers in half.

pwarner• 2 years ago

What Zscaler can and will do though is break your network randomly and in strange ways. They don't even seem to charge for that feature!

grishka3• 2 years ago

If I were running an organization that needs these audits, I'd always have fallback procedures in place that would keep everything running even if all computers suddenly stop working, like they did today. General-purpose software is too fragile to be fully relied upon, IMO.

If a general-purpose computer must be used for something mission-critical, it should not have an internet connection and it should definitely not allow an outside organization to remotely push arbitrary kernel-mode code to it. It should probably also boot from a read-only OS image so that it could always be restored to a known-good state by just rebooting.

sys_647382• 2 years ago

Organizations don't want to increase risk by listening to an employee with their personal opinion. Orgs want an outside vendor who they can point at and say "it's their fault", and await a solution. Employees going rogue and not following the vendor defined SW updates is a much higher risk than this particular crisis.

sobkas2• 2 years ago

It depends on what your position is. Are you there to actually provide security to your org or to tick a in an audit. If both which is more important. Because failing an audit have real consequences, while having breaches in security have almost none. Just look at credit score companies.

the84723• 2 years ago

Regulation or auditors rarely require specific solutions. It's the companies themselves that choose to achieve the goals by applying security like tinctures: "security solutions". The issue is that the tinctures are an approved remedy.

lqet3• 2 years ago

Can anyone explain how CrowdStrike could possibly fix this now? If affected machines are stuck in an endless BSOD cycle, is it even possible to remotely roll out a fix? My understanding is that the machines will never come to the point where a CS update would be automatically installed. Is the only feasible option the official workaround of manually deleting system files after booting into the recovery environment? How could this possibly be done on scale in organizations with tens of thousands of machines?

ExoticPearTree7• 2 years ago

CrowdStrike itself will not fix anything. They published a guide on how to workaround the problem and that's it. Most likely a lot of sales reps and VPs will be fielding calls all over the weekend explaining large customers how did they manage to screw up and how much discount will they offer on the next renewal cycle.

Legally, I think somewhere in their license it says is that they're not responsible in any way or form if their software malfunctions in any way.

m348e912• 2 years ago

There are orgs out there right now with 50,000+ systems in a reboot loop. Each one needs to me manually configured to disable CS via safe mode so that the agent version can be updated to the fixed version. Throw bitlocker in the mix which makes this process even longer, we're talking about weeks of work to recover all systems.

chrisjj14• 2 years ago

But they've already fixed it.

"CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes."

How they've reverted changes on non-booting PCs, goodness only knows... ;)

pelasaco2• 2 years ago

Imagine if North Korea comes with a statement, that they did it.. It would spawn such amount of work internally at CS to proof if it was intentional or a simple mistake.

justinclift• 2 years ago

Sounds like people in some of these environments will be doing their level best to automate an appropriate fix.

Hopefully they have IPMI and remote booting of some form available for the majority of the affected boxes/VMs, as that could likely fix a large chunk of the problem.

jojobas5• 2 years ago

The question is, how did they manage to not crash everything for so long without a staged/rolling update deployment strategy?

chrisjj14• 2 years ago

Perhaps it took this long for the offending file to pass 64K Excel rows... :)

ExoticPearTree7• 2 years ago

All over the place I'm seeing checkbox compliance being prioritized above actual real risks from how the compliance is implemented.

Because if everyone is doing their job and checks their box, they're not gonna get fired. Might be out of a job because the company goes under, but hey, it was no one's fault, they just did their job.

dagaci4• 2 years ago

Most enterprises will have teams of risk and security people. They will be asking who authorized deployment of an untested update into production. If CrowdStrike deployments cannot be managed, then they will switch to a product which can be managed.

basisword2• 2 years ago

> It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it.

While orgs using auto update should reconsider, the fact that CrowdStrike don't test these updates on a small amount of live traffic (e.g. 1%) is a huge failure on their part. If they released to 1% of customers and waited even 24 hours before rolling out further this seems like it would have been caught and had minimal impact. You have to be pretty arrogant to just roll out updates to millions of customers devices in one fell swoop.

lqet3• 2 years ago

Why even test the updates on a small amount of live customers first? Wouldn't this issue already have surfaced if they tested the update on a handful of their own machines?

fireflash382• 2 years ago

That's some very twisted logic. If I expect someone to clean the kitchen as part of restaurant closeup checklist, and they fuck it all up, would I blame the checklist, or the person doing the work?

You blame the person fucking it up. In this case, it's someone who only cares about checking a box. Or someone who pushes broken shit.

lqet3• 2 years ago

If this person simultaneously fucks up millions of kitchens around the world, you do not blame that person. You blame the checklist which encouraged giving a single person global interlocked control over millions of kitchens, without any compartmentalization.

shadowgovt• 2 years ago

Consolidation / optimization of labor.

When Crowdstrike messes up and BSODs thousands of machines, they have a dedicated team of engineers working the problem and can deliver a solution.

When your company gets owned because you didn't check a compliance checkbox, it's on you to fix it (and you may not even currently have the talent to do so).

We see similar risk tradeoffs in cloud computing in general; yes, hosting your stuff on AWS leaves you vulnerable to AWS outages, but it's not like outages don't happen if you run your own iron. You're just going to have to dispatch someone a three hour drive away to the datacenter to fix it when they do.

chrisjj14• 2 years ago

they have a dedicated team of engineers working the problem and can deliver a solution.

No. Can merely facilitate the customer's on-site admin to deliver a solution.

afh1• 2 years ago

Isn't there a way to schedule the updates? With Windows updates, when I used to work at a firm with a critical system running on Windows, we had main and DR servers and the updates were scheduled to first rollout on the main server and a day after I think at the DR, which has saved us at least once in the past from a bad Windows update...

vladvasiliu4• 2 years ago

More or less. You can set up some update policies which and apply those to subsets of your machines. You can disable updates during time blocks, or block them altogether. There's also the option of automatically installing the "n-1" update.

We run auto n-1 at work, but this also happened at the same time on my test machine with runs "auto n". It never happened before, so this looks like something different than the actual installed sensor version, especially since the latest version was released something like a week ago.

jimnotgym6• 2 years ago

So we need to hold regulatory bodies accountable as well...

My bank, my insurer, my payment card processor, my accounting auditor and probably others may all insist I have anti-virus and insist that it is up to date. That is why we have to have these systems. However, I used to prefer systems that allowed me to control the update cycle and push it to smaller groups.

bhbh• 2 years ago

CrowdStrike has various auto update policies, including not to automatically update to the latest version, but to the latest version -1 or even -2. Customers with those two policies are also impacted.

beeboobaa35• 2 years ago

It's what you get when you let luddites, also known as managers, make the rules.

lenkite• 2 years ago

It's a big stretch to call this the regulator's fault when its basic lack of testing by Microsoft and/or Crowdstrike. If a car manufacturer made safety belts that broke, you don't blame the regulators.

The root cause is automatic, mindless software update without proper testing - nothing to do with regulators.

number65259• 2 years ago

My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer starting up

Hi fellow CVS employee. Are you enjoying your zscaler induced SSO outages every week that torpedo access to email and every internal application? Well now your VMs can bluescreen too. A few more vendor parasites and we'll be completely nonfunctional. Sit tight!

Puts3• 2 years ago

This is an oversimplification. IF we are talking about compliance to ISO 27001 you are supposed to do your own risk assessment and implement necessary controls. The auditor will basically just check that you done the risk assessment, and that you have done the controls you said yourself you need to do.

I'd say this has nothing with regulatory compliance to do at all. The real truth is that modern organizations are way too attached to cloud solutions. And this runs across all parts of the organization with Saas and PaaS whether it's email (imagine Google Workspace having a major issue), AWS, Azure, Okta…

I've had the discussions so many times and the answer is always – the risks doesn't matter because the future is cloud and even talking about self hosting anything is naive and honestly we need to evaluate your competence for even suggesting it.

(Also the cloud would maybe not be this fragile if it wasn't for lock-in with different vendors. If you read the TOS it says basically on all cloud services that you are responsible for the backup – but getting your data out of the service is still pain in the ass – if possible at all)

crazygringo3• 2 years ago

The real truth is that modern organizations are way too attached to cloud solutions.

I'm confused. This is a security product for your local machine. Not the cloud.

Unless you call software auto-update "the cloud", but that's not what people usually mean. The cloud isn't about downloading files, it's about running programs and storage remotely.

I mean, if CloudStrike were running entirely on the cloud, it seems like the problem would be vastly easier to catch immediately and fix. Cloud engineers can roll back software versions a lot easier than millions of end users can figure out how to safe boot and follow a bunch of instructions.

coliveira• 2 years ago

This is all well deserved for executives who trust MS to run their businesses. If you have the resources, like a bank, it is crime to put your company in the hands of MS.

alecco• 2 years ago

IME, Boomer managers refuse to use anything but Windows. We have a few more years of this.

metalliqaz• 2 years ago

this is the world that lawyers gave us

pookha• 2 years ago

It's easy to blame CrodStrike because they're the ones to blame here. They lit a billion system32 folders on fire with an untested product and push out fear mongering corny marketing material. Turns out you should be afraid.

MetaWhirledPeas4• 2 years ago

Orgs are doing this because they are more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting.

If a failed audit is the big scary monster in their closet, then it sounds like the senior leadership is not intimately familiar with the technology and software in general, and is unable to properly weigh the risks of their decisions.

More and more companies are becoming software companies whether they like it or not. The software is essential to the product. And just like you would never want a non-lawyer running your law firm, you don't want a non-software person running your software company.

temporarely• 2 years ago

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself.

TIL that US government has pressured foreign nations to install a mystery blob in the kernel of machines that run critical software "for compliance".

If this wasn't a providential goof on the part of Crowdstrike -- the entire planet is now aware of this little known fact -- then some helpful soul in Crowdstrike has given us a heads-up.

narrator• 2 years ago

No doubt there will be another piece of software mandated to check up on the compliance software. When that causes a global IT outage, software that checks up on the software that checks up on the compliance software will be mandated.

radiator4• 2 years ago

Very sharp and to the point, this comment. I would like to add that in large companies the audit will, in my experience, very often examine documents only -- not actual configuration or code.

phone8675309• 2 years ago

Orgs are doing this because they are more scared of failing an audit than they are of the consequences failure of the underlying systems the audits are supposed to be protecting.

I've been someone in one of those audit meetings defending decisions made and defending things based on the records we keep and I understand this because it is both a deeply unpleasant and expensive affair to pull people from current projects and place them before auditors for several hours to debate what compliance actually means.

maxrecursion• 2 years ago

I work for government organization that is constantly audited and I've seen this play out over and over.

An important aspect I never see mentioned is most Cyber Security personnel don't have the technical experience to truly understand the systems they are assessing, they are, like you said, just pushing to check those compliance boxes.

I say this as someone who is currently in a Cyber Security role, unfortunately, as I'm coming to learn cyber roles suck. But this isn't a jab at those Cyber Security personnel's intelligence. It's literally impossible to understand multiple systems at a deep level, it takes employees working on those systems weeks to months to understand this stuff, and that's with them being in the loop. Cyber is always on the outside looking in, trying like hell to piece it all together.

Sorry for the rant. I just wanted to add on with my personal opinions on the cyber security framework being severely broken because I deal with it on a daily basis.

lokar• 2 years ago

Our compliance and security people turned up with an urgent request to patch out Linux kernels in AWS.

The pcmcia driver had a vuln

I don’t listen to them much anymore

Hizonner• 2 years ago

It's literally impossible to understand multiple systems at a deep level,\

No, it's not. It takes above average intelligence, and major investment in actual education (not just "training"), and actual depth of experience, but it's not impossible.

unethical_ban• 2 years ago

I know of an important company currently pushing to implement a redundant network data loss prevention solution, while they don't have persistent VPN enabled and multiple known misconfigurations of things that prevent web decryption working properly.

Because someone needs a checkbox.

zmmmmm5• 2 years ago

Do you think it comes from a fundamental misconception of how these roles should be structured? My take is that you just can't fundamentally assess technical elements from the outside unless they have been designed that way in the first place (for assessability). For example I educate my team that they have structure their git commits in a way that demonstrates their safety for audit / compliance purposes (never ever combine a high risk change with a low risk one, for example). That should go all the way up the chain. Failure to produce an auditable output is failure to produce an output that can be deployed.

smsm429• 2 years ago

Well, if you fail at compliance, you can be fired and sometimes even sued. If your compliance efforts cause system wide outage - nobody's to blame, shit happens. I predict this screwup will end up with zero consequences for anyone who took the decisions that led to it too. So how else do you expect this system to evolve, given this incentive structure?

bargle0• 2 years ago

CrowdStrike, ZScaler, and the rest of these people surely have lobbyists that ensure their software is compelled by regulators.

23orj2o3• 2 years ago

When a metric becomes a target ...

xphilter• 2 years ago

It’s even worse. The consultants who run the audits (usually business school recent grads) work with other consultants who shill the third party software and implementation work.

wyager• 2 years ago

Zscaler is such insane garbage. Legitimately one of the worst pieces of software I have ever used. If your organization is structurally competent, it will never use Zscaler and will just use wireguard or something.

ShuffleBoard• 2 years ago

Picking up pennies in front of a steamroller...

Gormo• 2 years ago

So we need to hold regulatory bodies accountable as well - when they frame regulation such that organisations are cornered into this they get to be part of the culpability here too.

Replacing common-law liability with prescriptive regulation is one of the main drivers of this problem today. Instead of holding people accountable for the actual consequences of their decisions, we increasingly attempt to preempt their decisions, which is the very thing that incentivizes cargo-cult "checkbox compliance".

It motivates people who otherwise have skin in the game and immediate situational awareness to outsource their responsibility to systems of generalized rules, which by definition are incapable of dealing effectively with outliers.

casey22• 2 years ago

It's VERY easy to blame CrowdStrike and companies like them as they are the one LOBBYING for those checkboxes. Both zscaler and Crowdstrike spent 500K last year lobbying.

wparad• 2 years ago

That for sure is a great point, it's actually something Dorota brought up in her talk this year https://vimeo.com/940487124

YoboDev2• 2 years ago

My team and I have begun to refer to this issue as CrowdStroke

naizarak• 2 years ago

this is really microsoft's fault for handing out kernel access to random 3rd parties, none of which are doing anything special that microsoft couldn't implement themselves (AV, anti-cheat, security)

immibis6• 2 years ago

Yes, Microsoft should just grant itself a monopoly on all those markets "for your own safety" and see what happens with their lawyers.

commandersaki7• 2 years ago

Or do what Apple does, disallow kernel extensions, and provide rigid kernel faciltiies for VPN clients, EDR agents, etc. to use, so they don't have to implement custom code resident in the kernel.

dschuetz3• 2 years ago

I wonder who exactly messed up the update, microsoft or crowdstrike. Usually, there is pre-rollout update testing AND some companies use N-1 version staging for critical/production systems. For me it feels much more complex a failure than just "it's crowdstrike's fault". Everybody involved must have done something wrong.

dschuetz3• 2 years ago

That was quick: Microsoft is blaming "3rd party" and announced that "a fix is forthcoming". Very curious indeed. https://techcrunch.com/2024/07/19/banks-airlines-brokerage-h...

runningmike• 2 years ago

Guess we will never read the real facts. Truth is RMS was right. Again. Closed source security software is too often malware by design. We need open solutions we can truly trust.

wcski4• 2 years ago

Closed source security software is too often malware by design.

Can you be more specific? Genuinely curious what you mean here.

lizknope8• 2 years ago

Crowdstrike is closed source security software.

What's the difference between malware and what Crowdstrike has done to the world today?

We might as well reclassify Crowdstrike as malware and remove it from all computers to avoid this situation in the future.

AndrewDucker2• 2 years ago

This is, of course, why they should be doing phased rollouts. 1% of their customers, then 10%, then all the rest.

0x1ceb00da3• 2 years ago

Or just test your code?

AndrewDucker2• 2 years ago

Both. No testing is going to catch everything. And if it only hits 1 computer in 100 then your local testing will probably miss it

Dentrax• 2 years ago

I'm just curious, don't they have something like "gradual rollout" to update their app? They just bulk-update simultaneously across entire agents? No way. Something is a bit off for me. But there are good lessons to learn for sure.

Jamie99122• 2 years ago

My company stays multiple versions behind latest for this exact reason, but we were still affected

bravetraveler4• 2 years ago

I'm a bit unfamiliar with this stuff anymore... supposedly it was a content update, not the agent itself :/

modestygrime• 2 years ago

I read that they pushed a new configuration file, so possibly they don't consider that a "software update" and pushed it to everyone. Which is obviously insane. If I am publishing software, it doesn't matter if I've changed a .py file or a .yaml file. A change is a change and it's going to be tagged with a new version.

Capricorn24812• 2 years ago

Wondering, how were you affected if you didn't update?

felix_kirkine• 2 years ago

Just gonna leave this here: https://news.ycombinator.com/item?id=32548671

grahar642• 2 years ago

I wonder what the rollout procedure is for CrowdStrike. I put $100 down that this was a minor update they decided was so minimal it didn't need extensive testing.

So many places use the "emergency break glass rollout procedure" on every deploy because it doesn't require all the hassle

jaza7• 2 years ago

Considering what CrowdStrike's software does, I'd say the majority of their updates could be quite easily argued as being "emergency" updates, so yeah, quite possibly they've gotten into the habit of "omg URGENT must break glass" way too often.

vinay_ys• 2 years ago

If you are IT team for a large impactful organization, you have to control updates to your organization's fleet. You cannot let vendors push updates directly. You have to stage those updates and test them and then do a gradual rollout to your whole organization.

Plus, for your critical communication systems, you must have a disaster recovery plan that actually helps you recover quickly in minutes, not hours or days. And you have to exercise this plan regularly.

If you are crowd strike, shame on you for not testing your product better. You failed to meet a very low bar. You just shipped a 100% reproducible widely impactful bug. Your customers must leave you for a more diligent vendor.

And I really hope the leadership teams in every software engineering organization learn a valuable lesson from this – listen to that lone senior engineer in your leadership team who pushes for better craft and operational rigor in your engineering culture; take it seriously - it has real business impact.

latexr• 2 years ago

There’s already a Wikipedia page on the outage.

https://en.wikipedia.org/wiki/July_2024_global_cyber_outages

omoikane• 2 years ago

It now redirects to a new page that names CrowdStrike specifically.

https://en.wikipedia.org/wiki/2024_CrowdStrike_incident

agilob• 2 years ago

Do we have any estimates how many machines are affected?

bidikburger• 2 years ago

how can billion dollar company push update before testing?

crypt1d• 2 years ago

This never would have happened if all these orgs used a blockchain.

/sarcasm

/but is it really?

hellajack3d• 2 years ago

I guess this article might need some updating soon:

https://www.crowdstrike.com/resources/reports/total-economic...

jeffrallen4• 2 years ago

...delivers -407% ROI...

FTFY.

JonChesterfield4• 2 years ago

"Falcon Complete managed detection and response (MDR) delivers 403% ROI, zero breaches and zero hidden costs"

clwg• 2 years ago

I'm always curious on how security software can provide a ROI.

I had McAfee tell me one time that the hackersafe logo on our website would increase sales by 10%, this was at a Fortune 50 doing billions in sales online every year.

I was pretty hyped because it would have done wonders for my career, but then they walked it back and wouldn't explain it to me. I wasn't mad, I was disappointed.

w4rh4wk5• 2 years ago

I love the name! Really tells you what's going on ^^

ghoshbishakh• 2 years ago

No worries for us. https://pinggy.io/ is working like a charm :)

StrLght• 2 years ago

From the guidelines:

Don't solicit upvotes, comments, or submissions. Users should vote and comment when they run across something they personally find interesting—not for promotion.

nvarsj• 2 years ago

Was involved in a "security mandated" mandatory rollout of Crowdstrike at my prior company.

This software was utter shit, and broke stuff all over the place. And installs itself as basically malware into critical paths everywhere. We objected to ever using it as a SPOF, but was overruled.

So yeah, not remotely surprised this happened.

Any kind of middleware/dynamic agent is highly suspect in my experience and to be avoided.

mrinfinitiesx2• 2 years ago

Half of the world's computers are down. The biggest tech failure of our time. Airports. Banks. NYSE. 298 of the fortune 500 companies. RIP.

mulmen9• 2 years ago

The biggest tech failure of our time.

Is it? Is this even the first time this year that all the airlines are down? How many times a day is a hospital a victim of ransomware?

mads_quist• 2 years ago

I'm actually very fond of "fail fast" and "no blame" culture, but someone needs to get fired for this!

mrkramer• 2 years ago

Windows breaking computers since 1985.

RadixDLT• 2 years ago

does Russia has something to do with this?

bsodfriday• 2 years ago

https://isitbsodfriday.com/

birracerveza• 2 years ago

Feels like what people imagined the millennium bug would have been like, just short of PCs catching on fire.

simonjgreen2• 2 years ago

CrowdStrike have finally posted publicly on it: https://www.crowdstrike.com/blog/statement-on-windows-sensor...

alibarber3• 2 years ago

Brilliant press release:

- Not apologetic

- An 'issue' in a 'single' release

I bet they were originally planning on starting with a '[reported by a] small number of customers' too.

user3939382• 2 years ago

“We’ve encountered a small single issue in which.. cough we’ve crashed all the computers in the world…”

Daedren• 2 years ago

Their legal team would definitely not allow nothing else.

DebtDeflation3• 2 years ago

This is pretty wild. I woke up to a news alert on my phone stating a "global IT outage" took down banks, airlines (who were calling for a global ground stop for all flights), hospitals, emergency services, etc. Expected it to be some sort of Tier 1 Network issue. Nope, a failed update for some third party Windows security app.

xorcist5• 2 years ago

We often read about how organizations are so bad because they don't spend enough on security. That slope is particularly slippy.

Crowdstrike is very expensive.

jaza7• 2 years ago

Today's incident shows that the real problem is actually that organisations spend too much (money, but too little time / manpower) on security.

Hey, third-party vendor, I'll give you all the money you want, I'll let you pwn all my systems, I'll be your little bitch, just make me secure, I don't have time for all that security shit, kthxbye.

piva00• 2 years ago

It's bizarre reading all the headlines about companies offline, flights canceled, banks not working because of a piece of antivirus software in 2024.

Mostly because I lived through Y2K and every fear about Y2K just materialised but because of Crowdstrike instead.

I can't imagine the amount of wasted work this will create, not only the lost of operations across many industries but recovery will be absolute hell with Bitlocker. How many corporate users have access to their encryption keys? And when stored centrally, how many of the servers have Crowdstrike running and just got stuck in a boot loop now?

I don't envy the next days/weeks for Windows IT admins of the world...

sidmkp96• 2 years ago

The thing that amazes me is how they've rolled out such a buggy change at such a scale. I would assume that for such critical systems, there would be a gradual rollout policy, so that not everything goes down at once.

mavamaarten• 2 years ago

This. I can see such an update shipping out for a few users. I mean I've shipped app updates that failed spectacularly in production due to a silly oversight (specifically: broken on a specific Android version), but those were all caught before shipping the app out to literally everybody around the world at the same time.

CubsFan10602• 2 years ago

The only thing I can think of is they were trying to defend from a very severe threat very quickly. But... it seems like if they tested this on one machine they'd have found it.

escapecharacter• 2 years ago

That’s the most charitable hypothesis, and I agree could be possible!

I myself have ninja-shipped a fix a minor problem, but then caused a worse problem since I rushed it.

trusche• 2 years ago

"works on my machine" at Internet scale. What a scary thought

altdataseller3• 2 years ago

Unless that threat was a 0day bug that allows anyone to SSH to any machine with any public key, it was not worth pushing it out in haste. Full stop. No excuses.

__MatrixMan__21• 2 years ago

I'd love to know what the original threat was. I hope it was something dumb like applying new branding colors to the systray indicator.

peheje2• 2 years ago

With fear of sounding like a douche-bag, I honestly believe there's A LOT of incompetence in the tech-world, which permeates all layers, security companies, AV companies, OS companies etc.

I really blame the whole power-structure, it looked like the engineers had the power, but last 10 years tech has been turned upside-down and exploited as any other industry, controlled by the opportunistic and greedy people. Everything is about making money, shipping features, the engineering is lost.

Would you rather tick compliance boxes easily or think deep about your critical path? Would you rather pay 100k for a skilled engineer or 5 cheaper (new) ones? Would you rather sell your HW now despite pushing feature-incomplete buggy app ruining the experience for many many customers? Will you listen to your engineers?

I also blame us, the SWE engineers, we are waay to easily busied around by these types of people who have no clue. Have professional integrity, tests is not optional or something that can be cut, it's part of SWE. Gradual rollout, feature-toggles, fall-backs/watchdogs etc. basic tools everyone should know.

commandersaki7• 2 years ago

I know people really dislike how Apple restricts your freedom to use their software in any way they don't intend. But this is one of the times where they shine.

Apple recognised kernel extension brought all sorts of trouble for users such as instability, crashing, etc. and presented a juicy attack surface. They deprecated and eventually disallowed kernel extensions supplanting them with a system extensions framework to provide interfaces for VPN functionality, EDR agents, etc.

A Crowdstrike agent couldn't panic or boot loop macOS due to a bug in the code when using this interface.

donatj• 2 years ago

Yes. I have been working in the tech industry since the early aughts and I never seen the industry so weak on engineer lead firms. Something really happened and the industry flipped.

williamvds• 2 years ago

Far from being douchey, I think you've hit the nail on the head. No one is perfect, we're all incompetent to some extent. You've written shitty code, I've definitely written shitty code. There's little time or consideration given to going back and improving things. Unless you're lucky enough to have financial support while working on a FOSS project where writing quality software is actually prioritized.

I get the appeal software developers have to start from scratch and write their own kernel, or OS, etc. And then you realize that working with modern hardware is just as messy.

We all stack our own house of cards upon another. Unless we tear it all down and start again with a sane stable structure, events like this will keep happening.

l33t73322736• 2 years ago

I know 100k+ engineers (some in security) that definitely should not be described as skilled.

hnthrowaway03285• 2 years ago

I think you are correct on that many SWEs are incompetent. I definitely am. I wish I had the time and passion to go through a complete self-training of CS fundamentals using Open Course resources.

locococo5• 2 years ago

I don't know that incompetence is the best way to describe the forces at play but I agree with your sentiment.

There is always tension between business people and engineering. Where the engineers want things to be perfect and safe, because we need to fix the arising issues during nights and weekends. The business people are interested in getting features released, and don't always understand the risks by pushing arbitrary dates.

It's a tradeoff which in healthy organizations where the two sides and leadership communicate effectively is well managed.

yard20108• 2 years ago

Pardon my snarkiness but this is what you get when imbeciles MBAs and marketers run every company, not engineers.

MetaWhirledPeas4• 2 years ago

I honestly believe there's A LOT of incompetence in the tech-world

I can understand why. An engineer with expertise in one area can be a dunce in another; the line between concerns can be blurry; and expectations continue to change. Finding the right people with the right expertise is hard.

jimbokun3• 2 years ago

it looked like the engineers

Engineers can be lazy and greedy, too. But at least they should better understand the risks of cutting corners.

Have professional integrity, tests is not optional or something that can be cut, it's part of SWE. Gradual rollout, feature-toggles, fall-backs/watchdogs etc. basic tools everyone should know.

In my career, my solution for this has been to just include doing things "the right way" as part of the estimate, and not give management the option to select a "cutting corners" option. The "cutting corners" option not only adds more risk, but rarely saves time anyway when you inevitably have to manually roll things back or do it over.

dclowd99016• 2 years ago

We’re definitely in a moment. I’ve seen a large shift away from discipline in the field. People don’t seem to care about professionalism or “good work”.

I mean back in the mid teens we had the whole “move fast and break things” motif. I think that quickly morphed into “be agile” because no one actually felt good about breaking things.

We don’t really have any software engineering leaders these days. It would be nice if one stood up and said “stop being awful. Let’s be professionals and earn our money.” Like, let’s create our own oath.

ThrowawayR2• 2 years ago

"Listen to the engineers?" The problem is that there are no engineers, in the proper sense of the term. What there are is tons and tons of software developers who are all too happy to be lax about security and safe designs for their own convenience and fight back hard against security analysts and QA when called out on it.

citizen_friend• 2 years ago

Nobody did gradual rollout in 1992

altdataseller3• 2 years ago

I would add that a lot of ppl in this industry also just blindly follow the herd too, without any independent thinking.

Oh, everyone is using Crowdstrike? I guess i have to do so too!

Oh, everyone is using Kubernetes? I guess i better start migrating our services to it too!

Oh, everyone is using this fancy Vercel stuff? We better use it too!

Oh everyone is migrating their workloads to the cloud even tho we dont need to and it costs 5x more?!! We better do so too!!

nyarlathotep_2• 2 years ago

I also blame us, the SWE engineers, we are waay to easily busied around by these types of people who have no clue. Have professional integrity, tests is not optional or something that can be cut, it's part of SWE.

Then maybe most of what's done in the "tech-industry" isn't, in any real sense, "engineering"?

I'd argue the areas where there's actual "engineering" in software are the least discussed---example being hard real-time systems for Engine Control Units/ABS systems etc.

That _has_ to work, unlike the latest CRUD/React thingy that had "engineering" processes of cargo-culting whatever framework is cool now and subjective nonsense like "code smells" and whatever design pattern is "needed" for "scale" or some such crap.

Perhaps actual engineering approaches could be applied to software development at large, but it wouldn't look like what most programmers do, day to day, now.

How is mission-critical software designed, tested, and QA'd? Why not try those approaches?

rustcleaner3• 2 years ago

Just wait until the PE licensing requirements come to legally charge money for code.

ken473• 2 years ago

In most companies, businesspeople without any real software dev experience control the purse strings. Such people should never run companies that sell life-or-death software.

The reality is there is plenty of space in the software industry to trade off velocity against "competent" software engineering. Take Instagram as an example. No one is going to die if e.g. a bug causes someone's IG photo upload to only appear in a proper subset of the feeds where it should appear.

There's a lot of incompetence by choice.

cyanydeez5• 2 years ago

100% what we seen in the last couple of decades is the march of normies into the techno sphere to the detriment of the prior natives.

We've essentially watched digital colonialism, and it certainly peaks with Elon musk wealth and ego, attempting to buy up the digital market place of ideas.

GordonS• 2 years ago

This is the thing that gets me most about this. Any Windows systems developer knows that a bug in a kernel driver can cause BSODs - why on earth would you push out such changes en-masse like this?!

Strenuous9770• 2 years ago

Lack of gradual, health mediated rollout is absolutely the core issue here. False positive signatures, crash inducing blocks, etc will always slip through testing at some % no matter how good testing is. The necessary defense in depth here is to roll out ALL changes (binaries, policies, etc) in a staggered fashion with some kind of health checks in between (did > 10% of endpoints the change went to go down and stay down right after the change was pushed?).

Crowdstrike bit my company with a false positive that severely broke the entire production fleet because they pushed the change everywhere all at once instead of staggering it out. We pushed them hard in the RCA to implement staggered deployments of their changes. They sent back a 50 page document explaining why they couldn't which basically came down to "that would slow down blocks of true positives" - which is technically true but from followup conversations quite clear that is was not the real reason. The real reason is that they weren't ready to invest the engineering effort into doing this.

You can stagger changes out within a reasonable timeframe - the blocks already take hours/days/weeks to come up with, taking an extra hour or two to trickle the change out gradually with some basic sanity checks between staggers is a tradeoff everyone would embrace in order to avoid the disaster we're living through today.

Need a reset on their balance point of security:uptime.

legohead• 2 years ago

The core issue? I'd say it's QA.

Deploy to a QA server fleet first. Stuff is broken. 100% prevention.

LrnByTeach3• 2 years ago

Wow !! good to know real reason for non-staggered release of the software ...

Crowdstrike bit my company with a false positive that severely broke the entire production fleet because they pushed the change everywhere all at once instead of staggering it out. We pushed them hard in the RCA to implement staggered deployments of their changes. They sent back a 50 page document explaining why they couldn't which basically came down to "that would slow down blocks of true positives" - which is technically true but from followup conversations quite clear that is was not the real reason. The real reason is that they weren't ready to invest the engineering effort into doing this.

justspamjustin• 2 years ago

Yes, one of the first steps of this gradual rollout should be rolling out to your own company in the classic, "eat your own dogfood" style.

jorblumesea• 2 years ago

CS recently went through cost cutting measures, which is likely why there's not a QA fleet to deploy to or improving their engineering processes.

2o4j2o3543o• 2 years ago

Sure, gradual rollout seems obviously desirable, but think of it from a liability perspective.

You roll out a patch to 1% of systems, and then a few of the remaining 99% get attacked and they sue you for having a solution but not making it available to them. It won't matter that your sales contract explains that this is how it works and the rollout is gradual and random.

Just a thought.

zmmmmm5• 2 years ago

There's some irony there in that the whole point of CrowdStrike itself is that it does behavioural based interventions. ie: it notices "unusual" activity over time and then can react to that autonomously. So them telling you they can't engineer it is kind of like them telling you they do don't know how to do a core feature they actually sell and market the product itself doing.

cyanydeez5• 2 years ago

In modern terms, you mean they simply weren't willing to babysit longer install frames.

this_steve_j• 2 years ago

If and when there is a US Cyber Safety Review Board investigation of this incident, documents like that are going to be considered with great interest by the parties involved.

Often it is the engineers working for a heavily invested customer at the sharp end of the coal face who get a glimpse underneath the layers of BS and stare into the abyss.

This doesn’t look good, they say. It looks fine from up top! Keep shoveling! Comes the reply.

nebulous1• 2 years ago

In 2012 a local bank rolled out an update that basically took all of their customer services offline. Couldn't access your money. Took them a month to get things working again.

1024core2• 2 years ago

No concept of "canarying", eh?

chronid3• 2 years ago

I also blame the customers here to be completely honest.

The fact the software does not allow for progressive rollout of a version in your own fleet should be an instantaneous "pass". It's unacceptable for a vendor to decide when updates are applied to my systems.

totallywrong3• 2 years ago

Well said. I can't fathom companies being fine with some 3rd party pushing arbitrary changes to their critical production systems.

jcgrillo2• 2 years ago

Absolutely. I may be speaking from ignorance here, as I don't know much about Windows, but isn't it also a big security red flag that this thing is reaching out to the Internet during boot?

I understand the need for updating these files, they're essentially what encodes the stuff the kernel agent (they call it a "sensor"?) is looking for. I also get why a known valid file needs to be loaded by the kernel module in the boot process--otherwise something could sneak by. What I don't understand is why downloading and validating these files needs to be a privileged process, let alone something in the actual kernel. And to top it all off, they're doing it at boot time. Why?

I hope there's an industry wide safety and reliability lesson learned here. And I hope computer operators (IT departments, etc) realize that they are responsible for making sure the things running on their machines are safe and reliable.

roca3• 2 years ago

What I'm curious about: other than checkbox compliance, how does Crowdstrike convince companies to buy their product? Do they present evidence that their product is effective at protecting customers? Because certainly Crowdstrike customers still get hacked.

bdd8f1df777b8• 2 years ago

I would imagine that their best weapon is that so many other big organizations are using CS, so choosing CS gives the decision maker the best shield from responsibilities, similar to "nobody gets fire by choosing IBM".

Of course, how they started from small was completely different.

bob10292• 2 years ago

I've watched it occur countless times. Often times the people making the purchase decision are largely incompetent.

They usually come out and take your team to a nice lunch. Then they run you through a fancy slide deck and convince you to let them run some scaremongering reporting tool over your infra. By the end of the day, most of your leadership is convinced they need the solution.

Rinse and repeat hundreds of times and you have the 3rd party vendor hodgepodge hellscape that constitutes most large corporations' IT infrastructure.

DebtDeflation3• 2 years ago

At one point overnight airlines were calling for an "international ground stop for all flights globally". Planes in the air were unable to get clearance to land or divert. I don't believe such a thing has ever happened before except in the immediate aftermath of 9/11.

acomjean• 2 years ago

Flights have been delayed/canceled. Not as critical as the OP hospital, but not good.

Effecting airlines/brodcasters and banks.

https://www.nytimes.com/live/2024/07/19/business/global-tech...

DebtDeflation3• 2 years ago

If a plane is in the air, and can't get permission to land anywhere, well, they only have a finite amount of fuel onboard.

shuntress2• 2 years ago

I would also point out that the backup plan (Radio and Binoculars) are not only effective but also extremely cheap & easy to keep ready in the control tower at all times.

The same cannot be said for medical records.

jojobas5• 2 years ago

Unbelievable these systems run on Windows.

locococo5• 2 years ago

I feel Windows is wrongly blamed here it's just an OS.

If you rely on your applications to be available you should have disaster recovery plans for scenarios like this.

muzani4• 2 years ago

Do you expect them to run on Android?

1oooqooq13• 2 years ago

the same time new showed up here, on wechat tiktok clone (moments i think, in English) was showing animations of the usa air traffic maps and how the tech blackout affected it. from those images i that it was huge.

FullMetalBitch• 2 years ago

So what are going to be the consequences of this? In my country some healthcare institutions and emergency systems are working.

artk42• 2 years ago

I guess all the blamed EuroCommission will again have to do their job to bring anti-oligo/monopoly regulations, which everyone will hate but still slightly work.

Architecting technical systems is MUCH WAY easier than architecting social-economical systems. I hope one day all those tech-savvy web3 wannabe revolutionaries will start to do the real job a designing socially working systems, not only technically barely working cryptographically strong hamster-tapping scams

Renaud• 2 years ago

When you see the size if the impact across the world, the number of people who will die because hospital, emergency and logistics systems are down…

You don’t need conventional war any more. State actors can just focus on targeting widely deployed “security systems” that will bring down whole economies and bring as much death and financial damage as a missile, while denying any involvement…

netsharc3• 2 years ago

Seeing all the cancelled and delayed flights, it makes me think a hacking kind of climate activism/radicalism would be more useful than gluing hands to roads, or throwing paint on art.

mrguyorama2• 2 years ago

Activism is mostly about awareness, because generally you believe your position to be the one a logical person will accept if they learn about it, so doing things that get in the news but only gets you a small fine or month in jail are preferred.

Taking destructive action is usually called "ecoterrorism" and isn't really done much anymore.

treesknees2• 2 years ago

muzani4• 2 years ago

I'm not so sure it's hyperbole: https://news.ycombinator.com/item?id=41002977

btbuildem6• 2 years ago

We are far past that point. So many critical systems are running on autopilot, with people who built and understood them retiring, and a new batch of unaware, aloof, apathetic people at the helm.

There's no real need for some Bad Actor -- at some point, entropy will take care of it. Some trivial thing somewhere will fail, and create a cascade of failures that will be cataclysmic in its consequences.

It's not fear-mongering, it's kind of a logical conclusion to decades of outsourcing, chasing profit above and over anything else, and sheer ignorance borne of privilege. We forgot what it took to build the foundations that keep us alive.

tonymet6• 2 years ago

I was in my 20s during the peak hysteria of post-9/11 and GWOT. I had to cope with the hysteria hyped 24/7 by media and DHS of a constant terror threat to determine if it was real.

The fact that global infra is so flimsy and vulnerable brought me tremendous relief. If the terror threats were real, we would have been experiencing infrastructure attacks daily.

I remember driving through rural California thinking if the terrorist cells were everywhere, they could trivially <attack critical infra that I don't want to be flagged by the FBI for>

I've read a lot of cyber security books like Countdown to Zero Day, Sandworm, Ghost in the Wires and each one brings me relief. Many of our industrial systems have the most flimsy, pathetic , unencrypted & uncredentialed wireless control protocols that are vulnerable to remote attack.

The fact that we rarely see incidents like this, and when they do happen, they are due to gross negligence rather than malice, is a tremendous relief.

kjkjadksj4• 2 years ago

This is the silver lining of global capitalism. When every power on earth is invested in the same assets there is little interest in rocking the boat unless the financial justification to do so is sufficiently massive.

ulises314• 2 years ago

it turns out, civilization works because most of us are civilized.

kjkjadksj4• 2 years ago

Given how obvious the vector is for targeting after its so widespread, stands reason to believe the same state actors would push phishing schemes and other such efforts in order to justify having a tool like crowdsrike used everywhere. We are focusing on the bear trap snapping shut here, but someone took the time to set up that trap right where we'd be stepping in the first place.

hnthrowaway03285• 2 years ago

I always think it's easy for state actors to pull out this trick.

Considering PR review is usually done within the team. A state actor can simply insert a manager, a couple of senior developers and maybe a couple of junior developers into a large team to do the job. Push something in Friday so few people bother to check, gets approved by another implant and here you go.

All people can then leave leisurely.

yard20108• 2 years ago

This happened with intelij a while back didn't it? A spy pushed a code that caused a suplly chain outage somewhere, I can't remember the details.

Anyway, I believe this is what happened here in this case.

gregw22• 2 years ago

This is, in a way, why Kaspersky was banned in the US... "who scans the scanners?". Kaspersky is not that different from a Cloudstrike EDR product.

   https://news.ycombinator.com/item?id=4092187

GardenLetter272• 2 years ago

But will Europe ban CrowdStrike?

yard20108• 2 years ago

This clusterfuck is a dress rehearsal if you ask my honest opinion.

crooked-v• 2 years ago

I was going to buy some put options against CRWD with spare pocket money, but it turns out that the service I have my investment money is in broken right now. I wonder if that's because of Crowdstrike.

alkhimey• 2 years ago

An update to the internal database. It still did not sunk to developers that data has equivalent risk as code. A400 crashed because of an XML file update. I have witnessed my share of critical bugs caused by "innocent? updates to "data" which were treated less seriously because of this. Management and devs alike should change their conception about this.

avidphantasm• 2 years ago

How long before companies start consciously de-risking by replacing general-purpose systems like Windows with newer systems with smaller attack surfaces? Why does an airline need to use Windows at all for operations? From what I’ve seen, their backend systems are still running on mainframes. The terminals are accessed on PCs running Windows, but those could trivially be replaced with iPadOS devices that are more locked down than Windows and generally more secure by design.

ricardobayes2• 2 years ago

Often, the cost of switching is too high or too complex to justify. On top of that, many applications commonly run in manufacturing etc., simply does not run on any other OS.

fransje266• 2 years ago

It's true that a multi-billion dollars screw-up, with possible deaths, is a cost that is much easier to justify..

lobsterthief• 2 years ago

They likely run software written for windows, patched together over decades, that wouldn’t port easily to an iPad.

andrybak2• 2 years ago

One of the problems possibly preventing this is that budgets for buying software aren't controlled by people administering the software. Definitely not by people using it.

krembo4• 2 years ago

“Nobody ever got fired for Buying IBM”

jeffhuys4• 2 years ago

They should’ve been.

nullindividual5• 2 years ago

Certain backend systems run on mainframes, yes. But the airline's website? No (only the booking portion interacts with a mainframe via API calls). Identity management system? No. Etc.

nprateem• 2 years ago

Never if they can help it and have heard of Santander

samcat1163• 2 years ago

Embedded Windows has always seemed like an oxymoron to me.

mr90210• 2 years ago

You lost me at iPadOS.

DuckHunt• 2 years ago

So apparently "The issue has been identified, isolated and a fix has been deployed" https://x.com/George_Kurtz/status/1814235001745027317

Yet the chaos seems to continue. Could it be that this fix can't be rolled out automatically to affected machines because they crash during boot - before the Crowdstrike Updater runs?

CubsFan10602• 2 years ago

Correct. Many just end up in an endless loop and never actually boot.

It's about as bad as it gets.

jwitthuhn• 2 years ago

Yeah, you need to manually fix each affected system by booting in safe mode. Not possible to do remotely.

rudasn2• 2 years ago

Ouch!

martyvis3• 2 years ago

And you will need your bitlocker recovery key to access your encrypted drive in safe mode. I luckily had mine available offline

There's going be a lot of handholding to get end users through this.

martypitt2• 2 years ago

That update is so tone-deaf and half-assed. There's no apology.

If you go to the website, there's nothing on their front-page. The post on their blog (https://www.crowdstrike.com/blog/statement-on-windows-sensor...) doesn't even link to the solution. There's no link to "Support Portal" anywhere to be seen on their front-page. So, you have to go digging to find the update.

And the "Fix" that they've "Deployed" requires someone to go to Every. Single. Machine. Companies with fleets of 50k machines are on this HN thread - how are they supposed to visit every machine?!?!

mycall3• 2 years ago

They won't apologize for legal reasons. Also, it will only make their stock fall further.

_heimdall6• 2 years ago

Any response they make in the middle of a global outage will be half-assed. They have all available resources figuring out what the hell just happened and how to fix it.

An apology this early is a lose-lose. If they do apologize they'll piss off people dealing with it and want a fix not an apology. If they do t apologize they're tone deaf and don't seem to care.

senectus12• 2 years ago

lol sounds good, but how the hell do they deploy a fix to a machine that has crash and is looping BSOD with no internet or netwrok connectivity...

You do what I've been doing for the last 10 hours or so. you walk to each and every desktop and manually type in the bitlocker key so you can remove the offending update.

at least the virtual devices can be fixed sitting at a desk while suckling at a comfort coffee..

jeffchien• 2 years ago

Phenomenal timing: https://old.reddit.com/r/wallstreetbets/comments/1e6ms9z/cro...

krembo4• 2 years ago

Coincidence?

bryancoxwell• 2 years ago

Probably.

shostack• 2 years ago

There's a lot of comments knocking the due diligence, but the call out of the threat vector and timing of this make it a bit hard to brush off as coincidence.

andrelaszlo• 2 years ago

I'm curious about investing and economy, and I always wonder about P/E ratios like Crowdstrike's (currently 450-something, was over 500 last week).

Some P/E ratios for today, for some companies I find interesting:

- Shopify: 615.12

- Crowdstrike: 455.70

- Datadog: 341.98

- Palantir: 212.34

- Pinterest: 187.67

- Uber: 99.0

- Broadcom: 77.68

- Tesla: 58.33

- Autodesk: 52.36

- Adobe: 49.23

- Microsoft: 37.97

What's going on here? Do investors expect Shopify, for example, to increase their earnings by an order of magnitude despite already having done extraordinarily well in a very competitive market? Can anyone ELI5?

blackbear_• 2 years ago

The greater fool theory: https://en.m.wikipedia.org/wiki/Greater_fool_theory

Essentially, investors buy as long as they think they will be able to sell at a higher price in the future, regardless of economic fundamentals.

ab_goat• 2 years ago

Capture the market by not making customers pay full costs => low profits.

Grow revenues without substantially increasing costs (i.e running a loss)

Hope you can turn up the profit dial later.

Seems like the modern way?

aimazon• 2 years ago

The only common belief held by investors in a stock is that the price is going to go up. You may have value investors with a belief that Shopify is undervalued based on earnings, you may have investors betting that the rest of the market will buy Shopify, you may have people who’ve seen the line go up and decided to buy…

Stock prices have been decoupled from earnings or “value” for a long time now and that’s toothpaste we will never get back in the tube. We are in the Robinhood age where you can buy and sell a stock in seconds with no effort.

jhallenworld• 2 years ago

Stock buybacks help push these up. Buybacks are a way to pay investors at capital gain tax rates instead of normal income (dividend) rates.

xyzzy4747• 2 years ago

The enterprise value is 80.58B. The gross profit is 2.5B. 80/2.5 is 32, similar to Tesla stock.

The earnings are affected by how much the company reinvests (which shows up as a cost) before it becomes earnings on the accounting sheet.

aveao3• 2 years ago

arm is also surprisingly high at 557.95x according to my broker btw. yet it's the only stock in my portfolio that reliably goes up.

DonsDiscountGas2• 2 years ago

With numbers like that, either the market is crazy or the market believes the actual meaningful earnings are substantially higher than the GAAP reported numbers. Although even there the difference would have to be pretty big.

eschery• 2 years ago

Former equity analyst here. Nobody on "The Street" is actually valuing these companies on PE ratios. Tech companies often intentionally re-invest earnings back into the business in real time and so their reported EPS is often quite low and a poor metric to evaluate the underlying business on. So instead, analysts typically use other metrics like EV/EBITDA or even P/Sales ratios in their valuation models.

Very generally speaking, trading these companies is kind of more of like placing a bet on whether or not their future top-line growth will be dramatically different than the market's current expectations.

markus_zhang8• 2 years ago

TBH I don't think many figures here make any financial sense -- but I gotta hold it if my friends all hold it. And once everyone holds it no one is allowed to mass sell it because it's going to hurt your friends, and in finance that's a sin.

huhtenberg• 2 years ago

30-50 is a reasonable PE range for larger companies.

ibejoeb4• 2 years ago

That's bananas. Bro is about to get a knock-knock.

SilasX2• 2 years ago

He says he bought seven put contracts for $7.30 at the $185 strike. Absolute max profit, from CS going to -0, would be (185-7.3) x 7 X 100 = ~$125k.

I don’t know if the absolute amount of profit affects decisions here. It seems if he were more certain of what’s going on he would have bet a lot more.

spacephysics• 2 years ago

This smells of some inside trading. Someone internal at crowdstrike (or their relative/friend) got wind of this and is trying to save face if they get investigated.

Reading the post its obvious they don’t have a deep understanding of tech, while having that be core to their thesis.

It’s prohibitively hard to hack into a “cloud system” due to few possible entry points - as a reddit commenter said, open S3 buckets are tough to crack!

throwaway7ahgb3• 2 years ago

It wouldn't work, the SEC has incredible tools for finding these things.

Especially for the mom/pop investors.

higeorge135• 2 years ago

Most timed post ever?

mFixman• 2 years ago

How is Microsoft stock down less than a percent?

The problem was Windows giving arbitrary access to the kernel to software that can be updated OTA without user intervention and allowing that to crash the kernel, right? Wouldn't this mean that Windows is considerably less secure and stable than assumed?

didntcheck3• 2 years ago

Not really. These were kernel modules authorized and installed by the system admin. Of course kernel code runs the risk of crashing your system. The same is true on Linux, and according to another commenter it already has happened with Crowdstrike for Linux

Vilian2• 2 years ago

No one assumed Windows were stable and secure that's why they install crowdstrike

9999000009992• 2 years ago

I'd bet my career CS isn't spending enough on QA. It's always the first thing to be cut, no one cares about QA when everything is going well, but when things go wrong...

alphabetting• 2 years ago

Google spending a boatload for Wiz looks smarter now

bkj5122• 2 years ago

Wow

bkj5122• 2 years ago

Lol we were using Symantec software so thankfully no affect.

bandrami2• 2 years ago

What a fun time to be less than 48 hours out from a transcontinental flight

monkeydust2• 2 years ago

So where can I buy an ETF of companies specializing in software Quality Assurance?

higeorge135• 2 years ago

Nobody does that anymore. :)

cja• 2 years ago

Sorry to be dense, but what is CrowdStrike and do I have it on my computer?

vtemian• 2 years ago

What's the actual magnitude of this outage? Is there a way to estimate how many machines were down?

zteppenwolf• 2 years ago

I guess people who continue to use Windows in 2024 arguably deserve this, particularly those utilizing it in a production environment.

cedws• 2 years ago

It's truly horrifying how many critical systems run Windows.

dopylitty3• 2 years ago

It was Windows in this case but nothing is stopping it from happening with any other widely used system that gets online updates. CrowdStrike has root on Linux/MacOS as well after all.

The problem is relying on networked computers for critical infrastructure with no contingency plan. This sort of thing will happen whether because of a bug or because of ransomware. The software and hardware industries are incapable of producing reliable and safe products in our economic system.

Important services such as hospitals, groceries, water treatment plants, and electric grids should be able to operate in offline mode when this sort of thing inevitably happens.

x86x87• 2 years ago

what about all the people that use services provided by people that use windows? should there be some sort of pushback here?

personalityson• 2 years ago

This is what AI's first strike will look like

butler14• 2 years ago

I’m guessing it’s completely incidental that the CEO of crowdstrike was critical of China earlier this year, and that China is somehow unaffected by this ‘global’ issue!

scrollaway2• 2 years ago

I mean… yeah? almost everyone in the west except for some highly corrupt people is critical of china, and china doesn’t deploy crowdstrike.

scrollaway2• 2 years ago

Those focusing on QA and staged rollouts are misguided. Yes of course a serious company should do it but CrowdStrike is a compliance checkbox ticker.

They exist solely to tick the box. That’s it. Nobody who pushes for them gives a shit about security or anything that isn’t “our clients / regulators are asking for this box to be ticked”.

The box is the problem. Especially when it’s affecting safety critical and national security systems. The box should not be tickable by such awful, high risk software. The fact that it is reflects poorly on the cybersecurity industry (no news to those on this forum of course, but news to the rest of the world).

I hope the company gets buried into the ground because of it. It’s time regulators take a long hard look at the dangers of these pretend turnkey solutions to compliance and we seriously evaluate whether they follow through on the intent of the specs. (Spoiler: they don’t)

eitland3• 2 years ago

Some Canonical guy I think many years ago mentioned this as their sales strategy a few year ago after a particularly nasty Windows outage:

We don't ask customers to switch all systems from Windows to Ubuntu, but to consider moving maybe a third to Ubuntu so they won't sit completely helpless next time Windows fail spectacularly.

While I see more and more Ubuntu systems, and recently have even spotted Landscape in the wild I don't think they were as successful as they hoped with that strategy.

That said, maybe there is a silver lining on todays clouds both WRT Ubuntu and Linux in general, and also WRT IT departments stopping to reconsider some security best practices.

51Cards• 2 years ago

Except further up this thread another poster mentions that CrowdStrike took down their Debian servers back in April as well. As soon as you're injecting third party software into your critical path with self-triggered updates you're vulnerable to the quality (or lack of) that software despite platform.

Honestly your comment highlights one of the few defenses... don't sit all on one platform.

noveltyaccount• 2 years ago

Hopefully they won't botch the update for two operating systems at the same time. But yeah. Hope.

potatolicious2• 2 years ago

Yeah, I see a lot of noise on social media blaming this on Microsoft/Windows... but AFAIK if you install a bad kernel driver into any major OS the result would be the same.

The specific of this CrowdStrike kernel driver (which AFAIK is intended to intercept and log/deny syscalls depending on threat assessment?) means that this is badnewsbears no matter which platform you're on.

Like sure, if an OS is vulnerable to kernel panics from code in userland, that's on the OS vendor, but this level of danger is intrinsic to kernel drivers!

II2II• 2 years ago

Sure, but note the sales pitch was to encourage resiliency through diversity. While that may not be helpful in cases where one vendor may push the same breaking change through to multiple platforms, it also may be helpful. I remember doing some work with a mathematics package under Solaris while in university, while my peers were using the same package under Windows. Both had the same issue, but the behaviour was different. Under Solaris, it was possible to diagnose since the application crashed with useful diagnostic information. Under Windows, it was impossible to diagnose since it took out the operating system and (because of that) it was unable to provide diagnostic information. (It's worth noting that I've seen the opposite happen as well, so this isn't meant to belittle Windows.)

__MatrixMan__21• 2 years ago

You can also make rollback easy. Just load the config before the one where you took the bad update.

Of course that means putting the user in control of when they apply updates, but maybe that would be a good thing anyway.

lolinder• 2 years ago

Yep, these episodes are the banana monoculture [0] applied to IT. The solution isn't to use this vendor or avoid that vendor, it's to diversify your systems such that you can have partial operability even if one major component is down.

[0] https://en.m.wikipedia.org/wiki/Gros_Michel_banana

emporas2• 2 years ago

don't sit all on one platform.

Debian has automatic updates but they can be manual as well. That's not the case in Windows.

The best practice for security critical infrastructure in which peoples lives are at stake, is to install some version of BSD stripped down to it's bare minimum. But then the company has to pay for much more expensive admins. Windows admins are much cheaper and plentiful.

Also as a user of Ubuntu and Debian for more than a decade, i have a hunch that this will not happen in India [1].

[1] https://news.itsfoss.com/indian-govt-linux-windows/

baggy_trough• 2 years ago

Still haven't forgiven Ubuntu for pushing a bad kernel of their own that caused a boot loop if you used containers...

j33zusjuice2• 2 years ago

I’ll never forgive them for the spyware they defaulted to on in their desktop stuff. It wasn’t the worst thing in the world, but they’re also the only major distro to ever do it, so Ubuntu (and Canonical as a whole) can get fucked, imo.

aiauthoritydev• 2 years ago

Seems like a reasonable strategy. Not just Ubuntu but some redundancy in some systems.

samcat1163• 2 years ago

Sure but if that Canonical sales person was successful in that, I'd almost guarantee that after they switched the first third they'd be in there arguing to switch out the rest.

eitland3• 2 years ago

Absolutely.

I'm just saying what they said their strategy was, not judging their sales people.

make3• 2 years ago

things are so interdependent that in this scenario you might now just end up crashing the system if either Windows or Ubuntu are down instead of just the one of them you chose

secondcoming2• 2 years ago

Ubuntu has unattended-upgrades enabled by default

sgarland9• 2 years ago

Yes, but by default the only repo enabled for it is $(cat /etc/os-release)-security.

mrintegrity• 2 years ago

Many years ago an Ubuntu tech sales guy demoed their (openstack?) Self hosted cloud offering, his laptop was running windows..

grumpyprole3• 2 years ago

Linux and open source also have the potential to be far more modular than Windows is. At the moment we have airport display boards running a full windows stack including anti-virus/spyware/audit etc, just to display a table ... madness

pbhjpbhj3• 2 years ago

I'm a Kubuntu user that, seemingly due to Canonical's decision to ship untested software regularly, has been repeatedly hit by problems with snaps. What were initially basic, obvious, and widespread issues with major software.

Yes, distribute your eggs, but check the handles on the baskets being sold to you by the guy pointing out bad handles.

eitland3• 2 years ago

FWIW, while some people like Kubuntu, I have had much better results with KDE Neon.

Stable Ubuntu core under the surface, and everything desktop related delivered by the KDE team.

totallywrong3• 2 years ago

Canonical in particular are no better, they do the exact same thing with that aberration called snap. They have brought entire clusters down before with automatic updates.

lopkeny12ko• 2 years ago

Isn't a Windows BSOD the equivalent of a kernel panic? I don't understand how this is CrowdStrike's fault. Vanilla userspace operations shouldn't cause a kernel panic--that's a bug in the OS, not a bug in some user software. If anything, we should be blaming Windows here?

jhfdbkofdchk• 2 years ago

CrowdStrike is not a vanilla userspace program, it hooks deeply into the operating system.

EvanAnderson8• 2 years ago

Vanilla userspace operations shouldn't cause a kernel panic...

The component Crowdstrike says you need to remove to restore functionality is a ".sys" file. That's a kernel-mode driver. The fault is happening on the kernel side.

kidbomb• 2 years ago

Lessons learned from this:

- CS: Have a staging (production-like) environment for proper validation. It looks like CS has one of these bu they have just skipped it - IT Admins: Have controlled roll-outs, instead of doing everything in a single swoop. - CS: Fuzz test your configuration

Anything I have missed?

krembo4• 2 years ago

Don't. Deploy. On. Fridays.

mycall3• 2 years ago

It is possible Cloudflare did a timepointed release on this. Controlled roll-outs wouldn't work if all the daily chunked updates didn't activate the kernel driver until some point in the future.

totaldude87• 2 years ago

I hope the narrative of , install crowd strike and pass the audit or else changes after this .

but being in the industry for so long , I don't expect any changes whatsoever, it's either CS or some other tool

mattbrewsbytes• 2 years ago

Does crowdstrike work similarly on MacOS? I have to imagine the "walled garden" doesn't allow for 3rd parties to insert themselves into the OS kernel but I could be wrong.

checker659• 2 years ago

I mean, you can always disable SIP and install your own extensions.

frankohn2• 2 years ago

The Windows ecosystem typically deployed in corporate PCs or workstations is often insecure, slow, and poorly implemented, resulting in ongoing issues visible to everyone. Examples include problems with malware, ransomware, and Windows botnets.

In corporate environments, IT staff struggle to contain these issues using antivirus software, firewalls, and proxies. These security measures often slow down PCs significantly, even on recent multi-core systems that should be responsive.

Microsoft is responsible for providing an operating system that is inherently insecure and vulnerable. They have prioritized user lock-in, dark patterns, and ease of use over security.

Apple has done a much better job with macOS in terms of security and performance.

The corporate world is now divided into two categories: 1. Software-savvy companies that run on Linux or BSD variants, occasionally providing macOS to their employees. These include companies like Google, Amazon, Netflix, and many others. 2. Companies that are not software-focused, as it's not their primary business. These organizations are left with Microsoft's offerings, paying for licenses and dealing with slow and insecure software.

The main advantage of Microsoft's products is the Office suite: Excel, Word and Powerpoint but even Word is actually mediocre.

EDIT: improve expression and fix errors:

Darvon• 2 years ago

If you were ready to ditch corpomicrosoft why would you go to corpoapple instead of something foss like debian tho

nextos4• 2 years ago

I'd say something implementing the ideas of NixOS, i.e. immutable versioned systems and declarative system definitions, is poised to replace the current deployment mess, which is extremely fragile.

With NixOS, you can upgrade without fear, as you can always roll back to a previous version of your system. Regular Linux distributions, macOS, and Windows make me very nervous because that is not the case.

1over1372• 2 years ago

Because there is software that runs only on certain OSes, and not others.

danaris3• 2 years ago

Because for some people (certainly not all), their objection is not to a "corporate" OS, but to the specific things Microsoft does that Apple does not.

harimau7774• 2 years ago

When I took a Linux course in college I had an old laptop that I installed Linux on. However, for some reason my wireless card wouldn't work. I mentioned it to my professor and the next day he told me "It's actually quite simple, you just have to open up the source code for the wireless driver and make a one line change."

Maybe things have gotten better, but I think that's why people use Mac. It's POSIX but without having to jump through arcane hoops.

Wytwwww5• 2 years ago

foss

Because you just want stuff to work and couldn't care less about the ideology part?

Also no feature parity (it's not about Windows being "better" than Linux or the other way around, none of that matters) there are not out of the box solutions to replace some of the stuff enterprise IT relies in Windows/etc. which would mean they'd have to hire expensive vendors to recreate/migrate their workflows. The costs of figuring out how to run all of your legacy Windows software, retraining staff etc. etc. would be very significant. Why spend so much money with no clear benefits?

To be fair I'm not sure how Apple figures into this. They don't really cater to the enterprise market at al..

quotemstr4• 2 years ago

The Windows ecosystem typically deployed in corporate PCs or workstations is often insecure, slow, and poorly implemented

Yes, but that's not because of Windows itself (which is fast and secure out of the box) but because of an decades-old "security product" culture that insists on adding negative-value garbage like Crowdstrike and various anti-virus systems on the critical path, killing performance and harming real security.

It's a hard problem. No matter how good Windows itself gets and no matter how bad these "security products" become, Windows administrators are stuck in the same system of crappy incentives.

Decades of myth and superstition demand they perform rituals and make incantations they know harm system security, but they do them anyway, because fear and tradition.

It's no wonder that they see Linux and macOS as a way out. It's not that they're any better -- but they're different, and the difference gives IT people air cover for escaping from this suffocating "you must add security products" culture.

rlanday4• 2 years ago

> The Windows ecosystem typically deployed in corporate PCs or workstations is often insecure, slow, and poorly implemented
Yes, but that's not because of Windows itself

Come on. There’s a reason Windows users all want to install crappy security products: they’ve been routinely having their files encrypted and held for ransom for the last decade.

Avamander4• 2 years ago

Windows itself (which is fast and secure out of the box)

That's a really bold claim. I'd say Windows comes with a lot of unsafe defaults OOB.

arzig• 2 years ago

Fun fact, these negative value garbage offerings are often “required” by box checking certifications like SOC2. Sure, if you have massive staffing to handle compliance you might be able to argue you’ve achieved the objective without this trash. The rest of us are just shrug and do it.

Some of the “compliance managers as a service” push you in this direction as well.

lbadmin3• 2 years ago

did you really just say windows is secure out of the box?

sys_647382• 2 years ago

Yes, but that's not because of Windows itself (which is fast and secure out of the box)

I think what you're really saying is that a Windows system is secure until you apply power to the computer.

mr_mitm11• 2 years ago

which is fast and secure out of the box

Disagree. At least in the context of business networks.

My favorite example is the SMB service, which is enabled by default.

In the Linux world, people preach:

- disabling SSH unless necessary

- use at least public key-based auth

- better both public key and password

- don't allow root login

In Windows, the SMB service:

- is enabled by default

- allows command execution as local admin via PsExec, so it's essentially like SSH except done poorly

- is only password-based

- doesn't even support MFA

- is not even encrypted by default

It's a huge issue why everyone gets encrypted by ransomware.

I always recommend disabling it using the Windows firewall unless it is actually used, and if it is necessary define a whitelist of address ranges, but apparently it is too hard to figure out who needs access to what, and much easier to deploy products like Crowdstrike which admittedly strongly mitigate the issue.

The next thing is that Windows still allows the NTLM authentication protocol by default (now finally about to be deprecated), which is a laughably bad authentication protocol. If you manage to steal the hash of the local admin on one machine, you can simply use it to authenticate to the next machine. Before LAPS gained traction, the local admin account password was the same on all machines in basically every organization. NT hashes are neither salted nor do they have a cost factor.

I could go on, but Microsoft made some very questionable security decisions that still haunt them to this day because of their strong commitment to backwards compatibility.

oytis2• 2 years ago

Companies that are not software-focused, as it's not their primary business. These organizations are left with Microsoft's offerings

I wonder why is it the case. These companies still have IT departments, someone has to manage these huge fleets of Windows machines. So nothing would prevent them from hiring Linux admins instead of Windows admins. What makes the management of these companies consider Windows to be the default choice?

gvurrdon• 2 years ago

I don't know, but I would guess that Microsoft Office is what retains people; personal anectodal experience suggests that anything else (Apple's offerings, Google Docs, LibreOffice &c.) is not acceptable to the average user. My suspicion is that Microsoft would be very unhappy to have MS Office running successfully on Linux systems.

dariosalvi782• 2 years ago

Office. The entire world runs on Excel, Word and Powerpoint. Unfortunately.

conception• 2 years ago

Excel. There is no other software that can currently fill excel’s role in business. It’s the best at what it does and what it does is usually very important. Unfortunately.

hnlmorg• 2 years ago

It's because of two things:

1. Users are more comfortable running Windows and Office because it's Windows they likely used in school and on personal laptops.

2. This is the biggie: Microsoft's enterprise services for managing fleets of workstations are actually really good -- or at least a massive step up from the competition. Linux (and it's ilk) is much better for managing fleets of servers, but workstations require a whole different type of tooling. And once you have AD and it's ilk running and thus Windows administrators hired, it's often easier to run other services from Windows too, rather than having to spin up another cluster of management services.

Software focused businesses generally start out with engineers running macOS or Linux, so they wouldn't have Windows management services pre-provisioned. And that's why you generally see them utilising stuff like Okta or Google Workspace

Asmod4n• 2 years ago

The business world runs on Windows, no way around that unless you only need a simple cash register and inventory software.

TeMPOraL14• 2 years ago

Inertia, plus integration - AFAIK Exchange and SharePoint don't run on Linux, so if the company buys into that, then it's Windows all the way down.

Still, all this is a red herring. Using Linux instead of Windows on workstations won't change anything, because it's not the OS that's the problem. A typical IT department is locked in a war on three fronts - defending against security threats, pushing back on unreasonable demands from the top, and fighting the company employees who want to do their jobs. Linux may or may not help against external attackers, but the fight against employees (which IT does both to fulfill mandates from the top and to minimize their own workload) requires tools for totalitarian control over computing devices.

Windows actually is better suited for that, because it's designed to constrain and control users. Linux is designed for the smart user to be able to do whatever they want, which includes working around stupid IT policies and corporate malware. So it shouldn't be surprising corporate IT favors Windows workstations too - it puts IT at an advantage over the users, and minimizes IT workload.

afavour7• 2 years ago

These companies still have IT departments

A lot actually don’t, in any meaningful sense. My partner’s company has a skeleton IT staff with all support requests being sent offshore. An issue with your laptop? A new one gets dispatched from ??? and mailed to you, you mail the old one back, presumably to get wiped and redispatched to the new person that has a problem.

chucke19923• 2 years ago

Tooling, infra, knowledge? The only reason why people are talking about "issues in Windows" because people are widely using it.

If linux had software anywhere close to the amount that windows has, it would have experienced the same issues too. After all it is not just about running a server and tinkering with config files. It is about ability to manage the devices, rolling out updates and so on.

gadders• 2 years ago

Word, Excel, Powerpoint and all the other windows software. Plus all the people that know how to use the windows software vs Linux equivalents (if they exist).

bregma• 2 years ago

Purchasing decisions are made by purchasing managers. Purchasing managers spend their time torturing numbers in spreadsheets, writing reports, and getting free lunches from channel sales reps. Microsoft is just a sales organization with some technical prowess, and their channel reps are very effective.

Technical arguments, logic, and sense do not contribute much to purchasing decisions in the corporate world.

arccy2• 2 years ago

Apple has done a much better job with macOS in terms of security and performance.

Do not underestimate corporate IT's ability to slow down Macs with endpoint security software.

ChrisMarshallNY• 2 years ago

This has been my experience.

I used to run a C++ shop, writing heavy-duty image processing pipeline software.

It did a lot, and it needed to do it in realtime, so we were constantly busting our asses to profile and optimize the software.

Our IT department insisted that we install 'orrible, 'orrible Java-based sneakware onto all of our machines, including the ones we were profiling.

We ended up having "rogue" machines, that would have gotten us in trouble, if IT found out (and I learned that senior management will always side with IT, regardless of whether or not that makes sense. It resulted in the IT department acting like that little sneak that sticks his tongue out at you, while hiding behind Sister Mary Elephant's habit).

But, to give them credit, they did have a tough job, and the risks were very real. Many baddies would have been thrilled to get their claws on our software.

balder1991• 2 years ago

Yeah, idk what they do, but in my company some new MacBook Pros with M3 are taking 15 minutes to login after typing the user password.

ta124312• 2 years ago

Had a problem with a "slow network" from a mac to a nas drive, was capping about 800mbit a second, despite having a 10g link.

As I looked through I killed sophos. Suddenly speeds shot up above 7gbit. A few seconds later they dropped back down, sophos has retured.

A "while (true) pkill sophos" later and the malware was sedated.

Having proved it wasn't a network problem I left it with the engineer to determine the best long term solution.

mattmcknight3• 2 years ago

Mosyle is doing their best to make Macs unusable.

dagaci4• 2 years ago

I think you represent the schism in your own post. Retail is hyper focused on the name Microsoft and Windows. But the enterprise and technical people are focused on rolling back a bad CrowdStrike bad update. They will spend hours and even days focusing on doing that, asking why they were vulnerable to such an update and what they should have done to avert being vulnerable to a bad update.

And for them it will be a bit of a stretch to say Microsoft should have stopped us deploying CrowdStrike. I’m sure Microsoft would love to do just that and sell its own Microsoft Solution.

Now if enterprises decide to run only Linux, BSD, or MacOS would they have been invulnerable to a bad CrowdStrike update: https://www.google.com/search?q=crowdstrike+kernel+panic

No so your entire premis is fully invalidated by a single google search.

On the other had I do feel Microsoft does have life far too easy in so many enterprises, but the fault here lies as much with the competition.

gred• 2 years ago

it will be a bit of a stretch to say Microsoft should have stopped us deploying CrowdStrike

I read GP's post to mean that if you take a step back, Windows' history of (in)security is what has led us to an environment where CrowdStrike is used / needed.

AgentME2• 2 years ago

MacOS has been phasing out support for third-party kernel extensions and CrowdStrike doesn't use a kernel extension there according to some other posts.

graemep2• 2 years ago

The issue with Crowdstrike on Linux did not cause widespread failures, so its clear that the majority of enterprises that do run their servers on Linux were not affected. They were invulnerable because they do not need Crowdstrike or similar.

Linux (or BSD) servers do not usually require third party kernel modules. Linux desktops might have the odd video driver or similar.

Vilian2• 2 years ago

The difference is that i van easily rollback a linux system, a complete update too, nota on windows

miah_• 2 years ago

Crowdstrike on Linux is only useful for appeasing corporate auditors, and making Crowdstrike money.

lizknope8• 2 years ago

In the case of a bad Linux kernel update I would just reboot and pick the previous kernel from the boot menu. By default most Linux distributions keep the last 3. I'm not an IPMI remote management expert but it may be possible to script this.

All my machines at home run Linux except for my work laptop. It is stuck in this infinite blue screen reboot loop. Because we use Bitlocker I can't even get it into safe mode or whatever to delete the bad file. I think IT will have to manually go around to literally 8,000 work laptops and fix them individually.

pepa65• 2 years ago

If you ran "only Linux, BSD, or MacOS" on a Microsoft hypervisor, yes. I would never recommend that, and your link exemplifies one reason why.

slumberlust• 2 years ago

What makes you think the FAANG companies don't use windows? Spent four years at Amazon recently and unless you were a dev, you were more likely to have a windows PC than Mac. Saw zero Linux laptops.

marcyb5st• 2 years ago

Depend on which FAANG I guess. Approaching now 10y at Google and I saw Windows laptops only used by very few sales people. Everyone else is either using Macs or Chromebook.

mdip3• 2 years ago

It's funny how that works.

Leave FAANG and most internal developers at large corporations are running Windows. It wasn't until I started at a smaller shop that I found people regularly using Linux to do their jobs, usually in a dual-boot or with a virtual Windows install "just in case" but most never touched it.

I'm presently working supporting a .NET web app (some of which is "old .NET Framework) but my work machine runs OpenSUSE Tumbleweed. I can't see that flying at the larger shops I have previously worked at. I'll admit, that might be different -- today -- I haven't worked at a large shop in more than a decade.

briandear4• 2 years ago

At Apple nobody uses Windows.

jimnotgym6• 2 years ago

Apple has done a much better job with macOS in terms of security and performance.

I really like their corporate IT products that are going to push MS out as you say. I particularly love iActive Directory, iExchange, iSQLserver, iDynamics ERP, iTeams. Apples office products are the reason noone uses Excel any more. Their integration with their corporate cloud, iAzure is amazing. I love their server products in particular, it being so easy to spin up an ios server and have dfs filesharing, dns etc is great. MS must be quaking in their shoes

frankohn2• 2 years ago

I particularly love iActive Directory, iExchange, iSQLserver, iDynamics ERP, iTeams. Apples office products are the reason noone uses Excel any more.

I see your sarcasm backfire as most you are listing is just Microsoft dog-food with no real usefulness. The only good thing in your list is Excel, all the rest is bloatware. Teams is a resource hog that serve no useful purpose. Skype was perfectly fine to send messages or have some video call.

I admit I don't have experience as an IT administator but things like managing emails, accounts, database, manage remote computers can be done with well estalished tools from the linux/BSD world.

indymike• 2 years ago

Most of the software you list either has a Mac version or will interop well with Apple's ecosystem and has for a decade.

PedroBatista2• 2 years ago

You got to admire Apple fanboy's nerve to say Apple is a better company when it comes to IT in a professional setting.

It appears whatever their basic and narrow use-case is becomes what the whole "corporate IT" is.

Windows sucks and recently Microsoft has been on a path to make it suck more, but saying Apple is better for this part of the IT universe is.. hilarious.

fortran775• 2 years ago

I think their "product" is getting people to astroturf on forums like this!

Apple always does just as bad, if not worse, on pwn2own https://www.bleepingcomputer.com/news/apple/apple-fixes-safa... as everyone else. And there are several companies that make a lot of money installing spyware on iPhones.

Stranger436• 2 years ago

All of those are product that creates huge risks when deployed to mission critical environments and this is exactly the problem.

The entire wintel ecosystem depends on people putting their heads in the sand and repeating "nobody ever got fired for buying Microsoft/crowdstrike/IBM" and neglecting to run even the most trivial simulation of what happens when the very well understood design flaws of those platforms gets triggered by a QA department you have no control over drops the ball.

The problem is that as long as nobody dares recognizing that the current mono culture around the "market leading providers" this kind of event will remain really likely even if nobody is trying to break it and and extremely likely once you insert well funded malicious actors(ranging from bored teenagers to criminal gangs and geopolitical rivals).

The problem is that adding fair weather product that gives the illusion of control though fancy dashboards on the days they work is not really an substitute for proper reliance testing and security hardening but far less disruptive to companies that don't really want to leave the 90ies PC metaphor behind.

b3lvedere• 2 years ago

Azure status/support page is amazingly amazing. Their current advice regarding virtual machines with the Crowdstrike problem? Keep rebooting!

https://azure.status.microsoft/en-us/status

briandear4• 2 years ago

Dynamics, Teams, Exchange, Active Directory all suck. There are better alternatives but CIOs are stuck in 1996. Apple themselves in their corporate IT environment use none of those things yet somehow are one of the biggest and most profitable companies in the world. Azure is garbage compared to AWS. Using Azure Blob vs S3 is a nightmare. MSSQL is garbage compared to PostgreSQL. Slack is vastly better than Teams in literally every aspect. I just did a project moving a company from AWS to Azure and it was simply atrocious. Nobody at the user level likes using MS products if they have experience using non-MS products. It’s like Bitbucket — nobody uses that by choice.

lostlogin2• 2 years ago

I really like their corporate IT products that are going to push MS out as you say. I particularly love iActive Directory, iExchange, iSQLserver, iDynamics ERP, iTeams.

You’re being sarcastic, but do you like those MS products, specifically Teams?

I genuinely believe that any business that doesn’t make Teams is doing the lords work.

philistine3• 2 years ago

The fact Apple is not trying to be a tentacular behemoth syphoning profits in every enterprise environment does not invalidate the fact macOS is secure and performant.

Apple is a tentacular behemoth in the consumer space.

MetaMalone• 2 years ago

lol. i’ll dunk on Apple as much as i’ll dunk on any other OS, but they wouldn’t be as praised for security if they had to manage the infrastructure and users that Windows supports

Intermernet• 2 years ago

Honestly, windows out of the box is pretty secure. I don't want to defend Microsoft here, but adding third party security to Windows hasn't been anything but regulatory compliance at best and cargo culting at worst for over a decade now. If you actually look at core windows exploits compared to market share, they're comparable to Apple. Enterprises insist on adding extra attack surface area in the name of security.

I agree that people who actually know what they're doing are generally running Linux backends, but Microsoft have enterprise sewn up, and this attack is not their fault.

patmorgan232• 2 years ago

A lot of active directory defaults are wildly insecure, even on a newly built domain, and there are a lot of active directory admins out there that don't know how to properly delegate as permissions.

mattmcknight3• 2 years ago

Apple on the desktop/laptop, Google in the cloud for email, collaboration, file sharing, office suite. I ran a substantial sized company this way for a decade. Then we did a merger and had to migrate to Microsoft- massive step backwards, quintupling of IT problems and staff.

gortok• 2 years ago

downvoted, because in your response you conflate two issues:

1. The problem with using Microsoft 2. The lack of institutional knowledge of securing BSD and MacOS and running either of those at the scale Microsoft systems are being run at.

The vast majority of corporate computer endpoints are running windows. The vast majority of corporate line-of-business systems are running Windows Server (or alternatively Microsoft 365).

That means a whole lot of people have knowledge on how to administer windows machines and servers. That means the cost of knowledge to adminster those systems is going down as more people know how to do it.

Contra that with MacOS Server administration, endpoint administration, or BSD Administration. Far fewer people know how to do that. Far fewer examples of documentation and fixing issues administrators have are on the internet, waiting to help the hapless system administrator who has a problem.

It's not just about better vs. worse from your perspective; it's about the cost of change and the cost of acquiring the knowledge necessary to run these corporate systems at scale -- not to mention the cost of converting any applications running on these Windows machines to run on BSD or MacOS -- both from an endpoint perspective and a corporate IT system perspective.

It's really not even feasible to suggest alternatives to any of the corporations using Microsoft that are impacted by this outage.

If you want to create an alternative to Microsoft's Corporate IT Administration you're gonna need to do a lot more than point to MacOS or BSD being "better".

__MatrixMan__21• 2 years ago

The poor quality of Windows and associated software is not the problem here. The problem is that Microsoft especially, but software vendors generally, encourage users to blindly accept updates which they do not understand or know how to roll back. And by "encourage" I mean that they've removed the "no thanks" and "undo" buttons.

Here on Linux (NixOS), I am prompted at boot time:

which system config should be used?

If I applied a bad update today, I can just select the config that worked yesterday while I fix it. This is not a power that software vendors want users to have, and thus the users are powerless to fix problems of this sort that the vendors introduce.

It's not faulty software, it's a problematic philosophy of responsibility. Faulty software is the wake-up call.

glitchc• 2 years ago

Windows is leagues ahead of MacOS in terms of granularity of permissions and remote management tools. It's not even close. That's mainly why enterprise IT prefers it to alternatives.

gchamonlive• 2 years ago

I know I have the benefit of hindsight in this regard, but how isn't there redundant checks and tests that would prevent a mishap of this magnitude?

I mean, there should be extensive automated testing using many different platforms and hardware combinations as a prerequisite for any rollout.

I guess this is what we get when everything is opaque, not only the product and the code, but also the processes involved in maintaining and evolving the solution. They would think twice about not investing heavily in testing their deployment pipelines if everyone could inspect their processes.

It might also be the case that they indeed have a thorough production and testing process deployed to support the maintenance of crowdstrike solutions, but we are only left to wonder and to trust whatever their PR will eventually throw at us, since they are a closed company.

fnord77• 2 years ago

I take it patching remote machines is going to be difficult or impossible?

I haven't used windows in years, but from what I read you need to be in safe mode to delete a crowdstrike file in a system directory, but you need some 48 char key to get into safe mode now if it is locked down?

red_admiral2• 2 years ago

I can't wait for rachelbythebay's comments on this.

gz5• 2 years ago

Seems CS themselves may have been hacked? For example, seems unlikely that both:

1. CS normally pushes global updates to entire user base simultaneously?

2. This made it through their testing. Not only 'just' QA but likely CS employees internally run a version or two ahead of their customer base?

Just speculation - folks who know either answer can validate or debunk.

mmiyer• 2 years ago

(they confirmed there was no hack) I think you have too much faith in minimal software development practices being applied at companies.

Kye9• 2 years ago

Crowdstrike seems like the kind of thing that's sold to CEOs at conferences, forced on IT against objections, and the subject of a lot of discussion at Defcon.

EvanAnderson8• 2 years ago

I wonder what Crowdstrike's opsec is like re: malicious actors gaining control of their automated update servers. This incident certainly highlights the power of that type of attack, even if this one just ends up being typical human incompetence-based.

luismedel• 2 years ago

Probably a stupid question but, how can the Windows kernel recover so well after a graphics driver crash and at the same time being unable to do the same for other kind of drivers.

red_admiral2• 2 years ago

From reddit:

I'm in Australia. All our banks are down and all supermarkets as well so even if you have cash you can't buy anything.

I hope the national security/defense people are looking at this closely. Because you can bet the bad guys are. What's the saying, civilisation is only ever three days away from collapse or something?

I am pretty convinced this is a fuckup not an attack, but if Iran or someone managed something like this, there would be hell to pay.

misswaterfairy2• 2 years ago

You can bet a substantial amount of money on CloudStrike inadvertently painting a huuuuge hacker target on their back over this...

jimberlage• 2 years ago

For $150/hour, I will spend today consulting for businesses who need someone in the St Louis area to go reboot a remote workers’ machine.

lencastre• 2 years ago

Lots of issues in Spain and Germany.

anchochilis2• 2 years ago

We routinely implement phased / canary deployments in server-side systems to prevent faults from rolling out globally. How is it possible that CrowdStrike and/or Windows does not have a similar system built in for large, institutional customers? This is outrageous.

Tylast2• 2 years ago

Oh, the foresight of the 1st episode of Connections. https://www.youtube.com/watch?v=XetplHcM7aQ

pelasaco2• 2 years ago

If i was North Korea, I would say that was me. That would be however a crazy story if Russia and China had done anything about it.

whoknowsidont4• 2 years ago

Somewhere out there, there is an engineer with the biggest "I told you so" shit eating grin scrolling through every social media site and basking in the glory.

gquere• 2 years ago

There's supposedly a fix being deployed (https://x.com/George_Kurtz/status/1814235001745027317). Since it's a channel update I'm assuming that it would be downloaded automatically? Has anyone received it yet? Does the garbage driver disappear or is it replaced?

Edit: got in touch with an admin:

C-00000291-00000000-00000029.sys SHA256 1A30..4B60 is the bad file (timestamp 0409 UTC)

C-00000291-00000000-00000030.sys SHA256 E693..6FAE is the fix (timestamp >= 0527 UTC)

Do not rely on the hashes too much as these might vary from org to org I've read.

cryptica2• 2 years ago

I've been warning about the coming software apocalypse for years. This isn't a one-off, this is the beginning of a pattern. Tech recruitment is broken, software is more complex than ever, more and more people are turning to hacking, people are growing increasingly dissatisfied with the status quo...

cryptica2• 2 years ago

This is what happens when you entrust software security to ex-hackers. Hackers love complexity because that's the kind of environment they thrive in; yet when they start working for the other side as security consultants, they still love complexity. Complexity ought to be the security consultant's worst enemy.

Ex-hackers often talk about security as if it's something you need to add to your systems... Security is achieved through good software development practices and it's about minimalism. You can't take intrinsically crappy, over-engineered, complex software and make it more secure by adding layers upon layer of complex security software on top.

lobochrome• 2 years ago

This is how I would start a war… surreal.

I hope it’s just a bug.

Melatonic2• 2 years ago

Good luck everyone. I just spent all night fixing my shit and we caught it early

lifeisstillgood3• 2 years ago

What do we do next week?

So assuming everyone uses sneaker-net to restart what’s looking like millions of windows boxes, there comes recriminations but then … what?

I think we need to look at minimum viable PC - certain things are protected more than others. Phones are a surprisingly good example - there is a core set of APIs and no fucker is ever allowed to do anything except through those. No matter how painful. At some point MSFT is going to enforce this the way Apple does. The EU court cases be damned.

For most tasks for most things it’s hard to suggest that an OS and a webbrowser are not the maximum needed.

We have been saying it for years - what I think we need is a manifesto for much smaller usable surface areas

andrewia• 2 years ago

Isn't that basically the point of WinRT and Windows 10 S Mode? The problem is getting developers to adopt the new more secure APIs.

9999000009992• 2 years ago

In this case even dockerized environments would allow you to redeploy with ease.

But that's too much work, many of these systems are running docker resistant software. Management doesn't want to invest in modernization - it works this quarter, it's someone else's problem next quarterly.

You're basically proposing Windows 12 to radically limit what software and drivers can do. Even then eventually someone will probably still break it with weird code.

I'm actually amazed these updates are being tested in prod. Do they have no QA environments ?

Do I personally need to create a startup company called Paranoia... We actually run a clone of your prod environment minus any sensitive data, then we install all the weird and strange updates before they hit your production servers...

As an upsell we'll test out privileges, to take sure your junior engineers can't break prod.

Someone raise a seed round, I'm down to get started this week.

quasse• 2 years ago

In this case even dockerized environments would allow you to redeploy with ease.

Not if the CIO mandated that your bare-metal OS hosting Docker has to run a rootkit developed by bozos.

lifeisstillgood3• 2 years ago

I think this is existential for Windows, and by extension MSFT. Something like 95% of corporate IT activity is either over http (ie every saas and web app) or is over the serial port (controlling that HVAC, that window blind, that garage lifter)

So what we need in 95% of boxes is not a fully capable PC - we need a really locked down OS. Or rather we can get by with a locked down OS.

I would put good money on there already being a tiny OS from the ground up in MSFT that could be relabelled windows-locked-Down(13) and sold exclusively to large corporates (and maybe small ones who sign a special piece of marketing paper)

The thing is once you do that you are breaking the idea that windows can run everywhere (or rather we claim Linux runs everywhere but the thing that’s on my default unbuntu install and the thing on my router are different

harimau7774• 2 years ago

It seems like this would indirectly tell us what systems use Cloudstrike. Could that in of itself be information that could help an attacker? I know the security team at work is adamant about not leaking details of our system.

smithington• 2 years ago

There's already somebody trying to cash in on this problem:

https://fix-crowdstrike-apocalypse.com

steveBK1232• 2 years ago

Wild that a piece of software so integral to basic function has such bad release discipline. A/B, Blue/Green, Canary, Rolling, etc..

I've worked on 4 person software teams that at least followed basic user group rolling release system.

guax5• 2 years ago

Integral? I would argue that is wild that a piece of software so useless for the basic and correct function gets so much privilege.

steveBK1232• 2 years ago

Well I mean integral here in that your PC can't boot and do anything useful if the software breaks.

It's not some solitaire app or saas website.

Completely nuked 1000s of companies ability to operate for a day, and the thing is auto-updating with apparently global big bang release push method..

JackC2• 2 years ago

Crowdstrike did this to our production linux fleet back on April 19th, and I've been dying to rant about it.

The short version was: we're a civic tech lab, so we have a bunch of different production websites made at different times on different infrastructure. We run Crowdstrike provided by our enterprise. Crowdstrike pushed an update on a Friday evening that was incompatible with up-to-date Debian stable. So we patched Debian as usual, everything was fine for a week, and then all of our servers across multiple websites and cloud hosts simultaneously hard crashed and refused to boot.

When we connected one of the disks to a new machine and checked the logs, Crowdstrike looked like a culprit, so we manually deleted it, the machine booted, tried reinstalling it and the machine immediately crashes again. OK, let's file a support ticket and get an engineer on the line.

Crowdstrike took a day to respond, and then asked for a bunch more proof (beyond the above) that it was their fault. They acknowledged the bug a day later, and weeks later had a root cause analysis that they didn't cover our scenario (Debian stable running version n-1, I think, which is a supported configuration) in their test matrix. In our own post mortem there was no real ability to prevent the same thing from happening again -- "we push software to your machines any time we want, whether or not it's urgent, without testing it" seems to be core to the model, particularly if you're a small IT part of a large enterprise. What they're selling to the enterprise is exactly that they'll do that.

kachapopopow• 2 years ago

This is gold. My friend and me were joking around that they probably did this to macos and linux before, but nobody gave a shit since it's... macos and linux.

(re: people blaming it on windows and macos/linux people being happy they have macos/linux)

zarzavat2• 2 years ago

I don’t think people are saying that causing a boot loop is impossible on Linux, anyone who knows anything about the Linux kernel knows that it’s very possible.

Rather it’s that on Linux using such an invasive antiviral technique in Ring 0 is not necessary.

On Mac I’m fairly sure it is impossible for a third party to cause such a boot loop due to SIP and the deprecation of kexts.

jimbobjim• 2 years ago

I’ve had an issue with it before in my work MacBook. It would just keep causing the system to hang, making the computer unusable. Had to get IT to remove it.

WatchDog3• 2 years ago

I’ve had sporadic kernel panics with the macos version.

Kye9• 2 years ago

I wonder if the changes they put in behind the scenes for your incident on Linux saved Linux systems in this situation and no one thought to see if Windows was also at risk.

trueismywork• 2 years ago

I think a big thing probably is that few linux systems are centrally upgraded.

HTG43• 2 years ago

Interesting that they push updates on a Friday when support profile will be way different across companies and organizations during that time.

joezydeco• 2 years ago

It makes you wonder if there was some critical vulnerability that forced them to deploy to everyone simultaneously at an awkward time.

JackC2• 2 years ago

Oh, if you are also running Crowdstrike on linux, here are some things we identified that you _can_ do:

- Make sure you're running in user mode (eBPF) instead of kernel mode (kernel module), since it has less ability to crash the kernel. This became the default in the latest versions and they say it now offers equivalent protection.

- If your enterprise allows, you can have a test fleet running version n and the main fleet run n-1.

- Make sure you know in advance who to cc on a support ticket so Crowdstrike pays attention.

I know some of this sounds obvious, but it's easy to screw up organizationally when EDR software is used by centralized CISOs to try to manage distributed enterprise risk -- like, how do you detect intrusions early in a big organization with lots of people running servers for lots of reasons? There's real reasons Crowdstrike is appealing in that situation. But if you're the sysadmin getting "make sure to run this thing on your 10 boxes out of our 10,000" or whatever, then you're the one who cares about uptime and you need to advocate a bit.

guax5• 2 years ago

Im suspicious that turning it off entirely would also provide equivalent protection as kernel and user space mode. If not more more.

umanwizard8• 2 years ago

Just a nit, I don't think it's correct to call eBPF "user mode". It's just a different, much more sandboxed, way of running kernel-mode code.

MrDrMcCoy2• 2 years ago

Depending on what kernel I'm running, CrowdStrike Falcon's eBPF will fail to compile and execute, then fail to fall back to their janky kernel driver, then inform IT that I'm out of compliance. Even LTS kernels in their support matrix sometimes do this to me. I'm thoroughly unimpressed with their code quality.

1oooqooq13• 2 years ago

do you work for them? if not what took you to do free tech support? ... honestly this looks a little insane

MetaWhirledPeas4• 2 years ago

we push software to your machines any time we want, whether or not it's urgent, without testing it

Do they allow you to control updates? It sounds like what you want is for a small subset of your machines using the latest, while the rest wait for stability to be proven.

bink2• 2 years ago

This is what happened to us. We had a small fraction of the fleet upgraded at the same time and they all crashed. We found the cause and set a flag to not install CS on servers with the latest kernel version until they fixed it.

jeeva• 2 years ago

It's worth noting that this feature is available, but the recent Windows issue [reportedly] ignored it.

not_wyoming• 2 years ago

we're a civic tech lab

Obviously not the point of your post, but say more? This sounds like it could be pretty cool!

philipwhiuk4• 2 years ago

You should send this to every tech reporter you like.

Nemo_bis4• 2 years ago

Interesting. How was the faulty upgrade distributed? Not from Debian archives I assume.

MrDrMcCoy2• 2 years ago

CrowdStrike Falcon may ship as a native package, but after that it completely self-updates to whatever they think you should be running. Often, I have to ask IT to ask CS to revert my version because the "current" one doesn't work on my up-to-date kernel/glibc/etc. The quality of code that they ship is pretty appalling.

OskarS3• 2 years ago

Please tell me you've ended the contract with CrowdStrike after this?

hughesjj• 2 years ago

we push software to your machines any time we want, whether or not it's urgent, without testing it

... Just, no... HOW does a vendor get away with that? No rolling releases allowed? No hostclasses or placement groups? No local cache or proxy?

The more I've learned over the past day or so, the more crowdstrike keeps edging towards the malware side of the malware/anti-malware spectrum for me

oklischat• 2 years ago

Every real malware programmer would be damn proud of this blast radius.

vfclists• 2 years ago

So in a nutshell it is about corporations pushing for legislation which compels usage of their questionable products, because such products enable management to claim compliance when things go wrong, even when the things that go wrong is are the compliance ensuring products.

Sounds like a good gig to me.

nicklad• 2 years ago

This seems to be misinformation? The CrowdStrike KB says this was due to a Linux kernel bug.

---

Linux Sensor operating in user mode will be blocked from loading on specific 6.x kernel versions

Published Date: Apr 11, 2024 Symptoms

In order to not trigger a kernel bug, the Linux Sensor operating in user mode will be prevented from loading on specific 6.x kernel versions with 7.11 and later sensor versions.

Applies To Linux sensor 7.11 in user mode will be prevented from loading:

For Ubuntu/Debian kernel versions:

6.5 or 6.6

For all distributions except Ubuntu/Debian, kernel versions:

6.5 to 6.5.12

6.6 to 6.6.2

Linux sensor 7.13 in user mode will be prevented from loading:

For Ubuntu 22.04 running the 6.5 kernel:

6.5.0 - 6.5.0-1014-aws 6.5.0- 6.5.0-1015-azure 6.5.0 - 6.5.0-1014-gcp 6.5.0 - 6.5.0-24-generic 6.5.0 - 6.5.0-1015-oem Ubuntu kernel version:

6.6 to 6.6.2 For Debian kernel version:

6.5 to 6.5.12

6.6 to 6.6.2

For all distributions except Ubuntu/Debian, kernel versions:

6.5 to 6.5.12

6.6 to 6.6.2

Linux Sensors running in kernel mode are not affected.

Resolution CrowdStrike Engineering identified a bug in the Linux kernel BPF verifier, resulting in unexpected operation or instability of the Linux environment.

In detail, as part of its tasks, the verifier backtracks BPF instructions from subprograms to each program loaded by a user-space application, like the sensor. In the bugged kernel versions, this mechanism could lead to an out-of-bounds array access in the verifier code, causing a kernel oops.

This issue affects a specific range of Linux kernel versions, that CrowdStrike Engineering identified through detailed analysis of the kernel commits log. It is possible for this issue to affect other kernels if the distribution vendor chooses to utilize the problem commit.

The commit where the kernel bug was introduced is seen at https://github.com/torvalds/linux/commit/fde2a3882bd07876c14... and the commit that resolves the issue is seen at https://github.com/torvalds/linux/commit/4bb7ea946a370707315...

To avoid triggering a bug within the Linux kernel, the sensor is intentionally prevented from running in user mode for the specific distributions and kernel versions shown in the above section

These kernel versions are intentionally blocked to avoid triggering a bug within the Linux kernel. It is not a bug with the Falcon sensor. Sensors running in kernel mode are not affected.

No action required, the sensor will not load into user mode for affected kernel versions and will stay on kernel mode.

For Ubuntu 22.04 the following 6.5 kernels will load in user mode with Falcon Linux Sensor 7.13 and higher:

6.5.0-1015-aws and later 6.5.0-1016-azure and later 6.5.0-1015-gcp and later 6.5.0-25-generic and later 6.5.0-1016-oem and later

If for some reason the sensor needs to be switched back to kernel mode: Switch the Linux sensor backend to kernel mode sudo /opt/CrowdStrike/falconctl -s --backend=kernel

tedajax• 2 years ago

The first time I experienced crowdstrike in a corporate environment it seemed obvious that something like this would eventually happen.

ezoe• 2 years ago

Those EDR software is implemented as a kernel driver.

A third party closed source Windows kernel driver that can't be audited. It gathers massive amount of activities and send back to the central server(which can be sold) as well as execute arbitrary payload from the central server.

It became single point of failure to your whole system.

If an attacker gain control of the sysadmin PC, it's over.

If an attacker gain administrator privilege on EDR-installed system, it run the same privilege with EDR so attacker can hide their activities from EDR. There aren't many EDR products in the world it can be done.

I'd like to call it "full trust security model".

ngneer9• 2 years ago

Security technology harming security? Shocker. We need less monoculture. Trouble is monoculture pays. Write the software once, deploy it everywhere - free money.

I manage a simple Tier-4 cloud application on Azure, involving both Windows and Linux machines. Crowdstrike, OMI, McAfee and endpoint protection in general has been the biggest thorn in my side.

snailb• 2 years ago

On a positive note, I'm in morocco and getting money from ATM wasn't working for the whole day I believe because of this outage. I was at the till in a supermarket and people started asking if they can chip in to pay for some food I bought because I didn't have the cash.

Humanity 1 - Technology 0

Edit: Outage of all ATM's in Morocco was yesterday not today. so not sure how the two are related.

hughw2• 2 years ago

So, why did our little company's (little used) two Windows machines not BSOD overnight? They were just sitting idle. They run CS Falcon sensor. Did the update force a restart? Didn't seem to happen here.

Kye9• 2 years ago

It looks like a configuration file update is the culprit. The software presumably picks up the update, then BSODs.

egberts14• 2 years ago

I am quite sure that they have had three precious timezone hours to detect a total failure of telemetry after their fateful midnight upgrade.

Like the most useful Canary Island in the Coal Mine.

KingOfCoders2• 2 years ago

2024 years after 2k we have 2k.

Kye9• 2 years ago

Maybe a silly question, but: why hasn't this affected Linux? I assume it uses a proprietary kernel module just like it does on Windows. I guess this will come out in a post-mortem if they publish one, but it's been on my mind.

edit: aha https://news.ycombinator.com/item?id=41005936

They did do this to Linux, but in the past. Maybe whatever they did to deal with it saved Linux this time around

ngneer9• 2 years ago

"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. It demands the same skill, devotion, insight, and even inspiration as the discovery of the simple physical laws which underlie the complex phenomena of nature."

"The most important property of a program is whether it accomplishes the intention of its user."

C.A.R. Hoare

uticus2• 2 years ago

Agreed but have you been in the industry lately? Nobody hires assembly programmers anymore. Want money you must work at wobbly top of abstraction mountain.

ngneer9• 2 years ago

I am well aware, but the quotes are timeless for a reason. Not to be cheeky, but "Want money" is exactly how you get to the many routinely broken endpoint solutions that wind up reducing reliability and at times increasing the attack surface. Wherever you are in the stack, please make it more robust and easier to reason about. No matter how far from the assembly.

thih92• 2 years ago

It’s not just about the tech abstraction mountain, it’s about the app logic and dev process too.

A react native JS app with a clear spec and a solid release process can be more reliable than bloated software that receives an untested hotfix, even if the latter was handwritten in assembly.

sans_souse• 2 years ago

Why would you name your company "CrowdStrike" anyway? What does Crowd Strike even mean?

josephd792• 2 years ago

Year of Linux

markus_zhang8• 2 years ago

In pre-market, CRWD is 14% down. I think investors are a bit scared that THIS time there is going to be some consequences.

rjmunro3• 2 years ago

I'm amazed it's just 14%, not more like 75%-80%. Surely a lot of customers are going to uninstall and move to competitors. The remainders are at least going to demand much cheaper service with better guarantees going forward.

markus_zhang8• 2 years ago

Yeah, and now recovered to -9.39%. Let's see what happens. I guess CrowdStrike is backed by enough powerful people to NOT lose too much business.

ddgflorida• 2 years ago

Do you suppose they test before pushing updates out?

thomasjudge• 2 years ago

Why are "security" patches not tested before they are deployed?

egberts14• 2 years ago

Yet Lennart Pottering and Redhat (spelled that way as I am one of the original pre-IPO investor of RedHat via Alex Brown/Deutsche Bank) wants to put networking of Linux into UEFI this quarter, inside the most sacrosanct PID 1.

They still won’t learning anything from Crowdstrike’s mistakeS!

Maybe it is time for me to ditch that stock.

btreecat• 2 years ago

Source of claim?

egberts14• 2 years ago

The part about UEFI?

https://www.redhat.com/sysadmin/bios-uefi

egberts14• 2 years ago

Network sockets are in the systemd code repository.

integricho3• 2 years ago

Ironic that the software intended to prevent exactly these kinds of outages ends up causing it.

integricho3• 2 years ago

This should at the very least put them out of business by causing each and every client to abandon them as their security solution.

sytelus3• 2 years ago

Genuine question: How the heck crapeware like CloudStrike got into all critical systems from 911 to hospitals to airlines? My understanding was that all these critical systems are just super lazy to upgrade or install anything at all. I would love to know all the sales tactics CS used to get into millions of systems for money!

ncr1002• 2 years ago

Reading other comments here, sorry I don't have the link, one crowd strike salesperson threatened to cancel them as a Client, yes you read that right, if the client wasn't easier to work with. So they're bullies or at least that one salesperson in crowd strike is a bully.

Another article talked about crowd strike being required for compliance, people here talking about checkbox compliance. So there's a systemic requirement from perhaps insurers for there to be some kind of comprehensive near real-time updated antivirus solution.

Furthermore, the haste makes waste philosophy seems to not be honored, in my opining mind, by the minds who drive The impacted sectors of our economy. Hospitals, Banks, airlines. This kind of vulnerability should not have been accepted. It's a single point of failure. Even on crowdstrike's website they have this kind of like radar ring hotspot Target kind of graphic, where they show at the very center one single client app .. theirs, as if that one single client is the thing that's going to save us?

sytelus3• 2 years ago

This is amazing sales tactics! So, you buddy up with insurance, they create a checkbox and recommend you for a revenue cut! Now you suddenly have millions of customers out of nowhere and your product gets installed on billions computers before you even know it. I have seen this tactic get used for many mediocre products. For example, 3rd party dishwasher soap recommended by dishwasher company. Amazingly powerful. I don’t think most of CrowdStrike employees even knew they were in more than billion computers with paid service. The CEO was just busy doing brutal marketing of this pointless product.

convivialdingo2• 2 years ago

Here’s my take as a security software dev for 15 years.

We put too much code in kernel simply because it’s considered more elite than other software. It’s just dumb.

Also - if a driver is causing a crash MSFT should boot from the last known-good driver set so the install can be backed out later. Reboot loops are still the standard failure mode in driver development…

erichocean• 2 years ago

Not possible in this situation, the "driver" is fine, it's a file the driver loads during startup that is bad, causing the otherwise "good" driver to crash.

Going back to an earlier version—since the driver is "good—would just re-load the same driver, loading the updated file, and then crashing again.

convivialdingo2• 2 years ago

A driver that crashes with bad input is not “fine.” Bad design, bad configuration loading and crap input validation. Did they even fuzz the code?

We’d spend 20x development time on kernel code because BSOD is never an option.

I get that this was a bad release - but IMHO it’s incredible that they pushed this out to a billion devices before the red flags went up.

amai2• 2 years ago

It seems monocultures are not only bad for resilience in agriculture, but also in IT.

jacobgorm• 2 years ago

The great clownstrike.

whoisstan2• 2 years ago

Can someone with experience explain how integration tests did not detect that?

guax5• 2 years ago

Why are you assuming there were tests?

whoisstan2• 2 years ago

Right.

I just can‘t imagine how it passed tests for a common configuration that is exhibited by large number of windows machines. Stuff always can go wrong, but OS is not booting should be caught?

amai2• 2 years ago

Did Crowdstrike forget the rule, that one does not simply deploy on Friday?

https://www.reddit.com/r/ProgrammerHumor/comments/f79iag/don...

dev1ycan• 2 years ago

Crazy isn't it, I had no issues because my group policy updates have been off since last year, guess the "everyone must forcefully update" for "security reasons" ended up backfiring, who could've thought

daemonologist• 2 years ago

My company has some bios bitlocker extension installed which prompts for a password on boot, so automatic updates (one of which tried to install last night) just get stuck there in jet engine mode. Normally this is extremely annoying but today I count myself lucky - aside from a couple of people with Chromebook thin clients I am the only person showing as online in Teams right now.

kaladin-jasnah• 2 years ago

Anecdote: my first job was IT at a small org. We had somehow gotten a 15 minute remote meeting with Kevin Mitnick, and asked him several questions about security best practices and software recommendations. I don't remember a lot about that meeting, but I do remember his strong recommendation of Crowdstrike. Interesting to see it brought up again in this context.

farceSpherule• 2 years ago

I absolutely abhor these end point solutions that "auto update for your convenience and safety."

I can control and manage my own systems. I do not need nanny state auto updating for me.

Crowdstrike should be held liable for financial losses associated with this nonsense.

axelthegerman• 2 years ago

Looks like crowdstrike are just delivering what their name promised, striking crowds around the world

aktuel• 2 years ago

Germany is not affected since it's Krautstrike only.

GrumpyNl• 2 years ago

Ho do they test this before they roll it out? Looks like a bug thats easy to spot. I would presume they test it at several configurations and when it passes the test ( a reboot), they roll it out. Has this been tested?

xyst4• 2 years ago

CRWD dropped $50/share at market open. Wild.

Is this specific to only Windows machines “protected” with CS or is this impacting Linux/macOS as well?

remram5• 2 years ago

I can't wait to see the CloudFlare traffic report after this. All those computers going down must have affected traffic worldwide. Even from Linux systems as their owners couldn't run jobs from their bricked Windows laptops.

aeyes2• 2 years ago

It looks quite normal so far: https://radar.cloudflare.com/traffic

DE-CIX traffic is also often a good indicator during global events, looks normal: https://www.de-cix.net/en/locations/frankfurt/statistics

remram5• 2 years ago

Interesting! Thanks for that. I guess most servers and consumer endpoints are fine, and those are driving all the traffic.

charlie0• 2 years ago

Next up: CrowdStrike Sued for Alleged Negligence.

I hate lawyers, but this is the reason why companies outsource. Why take the blame (and spend the money) when you can blame the vendor?

Geezus_42• 2 years ago

"Incidents of this nature do occur in a connected world that is reliant on technology." - Mike Maddison, CEO, NCC Group

Until I see an explanation of how this got past testing, I will assume negligence. I wasn't directly affected, but it seems every single Windows machine running their software in my org was affected. With a hit rate that high I struggle to believe any testing was done.

zzhelezc• 2 years ago

From the BBC's cyber correspondent Joe Tidy [1]:

A "content update" is how it was described. So, it wasn’t a major refresh of the cyber security software. It could have been something as innocuous as the changing of a font or logo on the software design.

He can't be serious, right? Right?

[1] https://www.bbc.co.uk/news/live/cnk4jdwp49et?post=asset%3Abd...

Beijinger• 2 years ago

Speaking of security. I got an email yesterday that I need a different system now to log into my social security account. This one:

https://www.id.me/government

It is for social security, taxes, unemployment benefits, whatever. And running under a foreign TLD, .ME for Montenegro. I am not a security specialist. But I think this is asking for trouble.

By the way, do you remember when fuck.yu became fuck.me ?

rajeshivivek2• 2 years ago

DO NOT REDEEM SAARRRRRRRRSSSSS! BLODDY BASTARDS INVALID FORMATING SARRRRRRSSSSS!!

stevetron• 2 years ago

Working late Thursday night in Florida, USA. I have someone in Australia wanting me to write a quick script in LSL for an object in Second Life. We were interrupted: Second Life kept running, but Discord went down, telling me to 'try another server' which doesn't make sence when you are 1-on-1 with someone. All my typing in Discord turned red. Additionally, I couldn't log into the email portal for outlook.com: I got a screen of tiny-fonted text all clinging to the left edge of the display, unreadable, unusable. Second Life, though, stayed online and kept working for me, but then I'm on Windows 7. My friend who had requested the collaboration froze in Second Life on his Windows 10 system, and I don't know what his Discord was doing. I ended the session since I couldn't get a no/no-go out of him for the latest script version.

_def• 2 years ago

Wow I didn't know second life was still a thing. Literally yesterday I looked at a 20 year old archived version of a freeware portal which also listed a version of second life.

dboreham8• 2 years ago

Looks like it affected the Crowdstrike stock, but not Microsoft.

upofadown• 2 years ago

Perhaps a dumb question for someone who actually knows how Microsoft stuff works...

Why would an anti-malware program be allowed to install a driver automatically ... or ever for that matter?

Added: OK, from another post I now know Crowdstrike has some sort of kernel mode that allows this sort of catastrophe on Linux. So I guess there is a bigger question here...

atoav5• 2 years ago

Maybe I am in the minority, but it always puzzled me that anybody in IT would think a mega-priviledged piece of software that looks into all files was a good idea.

If there is any place that historically was exploited more than all other things it was broken parsers. Congratulations if such an exploited file is now read by your AV-software it now sits now at a position where it is allowed (expected) to read all files and it would not surprise me if it could write them as well.

And you just doubled the number of places in which things can go wrong. Your system/software that reads a PNG image might do everything right, but do you know how well your AV-software parses PNGs?

This is just an example, but the question we really should ask ourselves is: why do we have systems where we expect malicous files to just show up in random places? The problem with IT security is not that people don't use AV software, it is that they run systems that are so broken by design that they are sprinkled on top.

This is like installing a sprinkler system in a house full of gasoline. Imagine gasoline everywhere including in some of the water piping — in the best case your sprinkler system reacts in time and kills the fire, in the worst case it sprays a combustive mix into it.

The solution is of course not to build houses filled with gasoline. Meanwhile AV-world wants to sell you ever more elaborate, AI-driven sprinkler systems. They are not the ones profiting from secure systems, just saying..

Sohcahtoa824• 2 years ago

but it always puzzled me that anybody in IT would think a mega-priviledged piece of software that looks into all files was a good idea.

Because otherwise, a piece of malware that installs itself at a "mega-privileged" level can easily make itself completely invisible to a scanner running as a low-priv user.

Heck, just placing itself in /root and hooking a few system calls would likely be enough to prevent a low-priv process from seeing it.

hnthrowaway03285• 2 years ago

I wonder why and how does security software read a PNG file. Sure it's not tough to parse a PNG file, but what does it look for exactly?

TiredOfLife• 2 years ago

As I understand it was a definition update that caused a crash inside already installed driver.

gusfoo• 2 years ago

Why would an anti-malware program be allowed to install a driver automatically ... or ever for that matter?

While the files are named XXX.SYS they are apparently not drivers. The issue is that a corrupted XXX.SYS was loaded by the already-installed driver which promptly crashes.

red-iron-pine2• 2 years ago

Why would an anti-malware program be allowed to install a driver automatically ... or ever for that matter?

Because malware that gets into a system will do just that -- install its own backdoor drivers -- and will then erect defense to protect itself from future updates or security actions. e.g. change the path that Windows Updater uses to download new updates, etc.

Having a kernel module that answers to CloudStrike makes it harder for that to happen, since CS has their own (non-malicious) backdoor to confirm that the rest of the stack is behaving as expected. And it's at the kernel level, so it has visibility into deeper processes that a user-space program might not (or that is easy to spoof).

sudosysgen4• 2 years ago

Or, much more likely, the malware will use a memory access bug in an existing, poorly written kernel module (say, CrowdStrike?) to load itself at the kernel level without anyone knowing, perhaps then flashing an older version of the BIOS/EFI and nestle there, or finding it's way into a management interface. Hell, it might even go ahead and install an existing buggy driver by itself it's not already there.

All of these invasive techniques end up making security even worse in the long term. Forget malware - there's freely available cheating software that does this. You can play around with it, it still works.

tonymet6• 2 years ago

Vendors are allowed to install drivers , even via Windows update. Many vendors like HP, install functionality like telemetry as drivers to make it more difficult for the users to remove the software.

So next time you think you are doing a "clean install", you are likely just re-installing the same software that came with the machine.

chuckadams3• 2 years ago

It doesn't install the driver, it is the driver. As for the Linux version, it uses eBPF which has a sandbox designed to never crash the kernel. Windows does have something similar nowadays, but Crowdstrike's code probably predates it and was likely just rawdogging the kernel.

rajeshivivek2• 2 years ago

SARRRRRRRSSSSS!

purpleblue2• 2 years ago

Do all the machines need to be manually fixed? It doesn't seem like an automatica update will work here...

snappr021• 2 years ago

“To err is human, but to really fuck things up requires a computer.” ~ Len Beattie

resters• 2 years ago

Any company that inserts itself so heavily into US politics cannot be counted on as a solid engineering organization.

localfirst3• 2 years ago

This is the first time I'm hearing about crowdstrike, what is it and why is this such a big deal?

casey22• 2 years ago

Isn't Crowdstrike the same company the heavily lobbied to get make all their features a requirement for government computers? https://www.opensecrets.org/federal-lobbying/clients/summary... They have plenty of money for congress, but it seem little for any kind of reasonable software development practices. This isn't the first time crowdstrike has pushed system breaking changes.

WillPostForFood• 2 years ago

Since we are in political season here in the US, they are also well known as the company that investigated the Russian hack of the DNC.

https://www.crowdstrike.com/blog/bears-midst-intrusion-democ...

pjot2• 2 years ago

The DNC has since has implemented many layers of protection including crowdstrike, hardware keys, as well as special auth software from Google. They learned many lessons from 2016.

lawlessone3• 2 years ago

Afaik didn't they hack republicans too? They only released democrat emails though.

brookst• 2 years ago

On the bright side, they are living up to their aptronym.

tyingq• 2 years ago

I wonder if it might starting being a common turn of phrase. "Crowdstrike that directory", etc.

chambored• 2 years ago

According to that link the most money they contributed to lobbying in the past 5 years was $600,000 most years around $200,000. That’s barely the cost of a senior engineer.

113• 2 years ago

You'd be surprised how cheap politicians are.

bogzz• 2 years ago

Corporate brainrot strikes again.

yurlungur• 2 years ago

Given its origin and involvement in these high profile cases I always thought Crowdstrike is a government subsidized company which barely has any real function or real product. I stand corrected I guess.

Aperocky2• 2 years ago

This still doesn't demonstrate that it has any real function tbf.

hello_moto17• 2 years ago

There's something missing here... You know nothing about Crowdstrike (as per your own statement) and critical infrastructure depends on them.

That two things tell us something about your knowledge;)

Aperocky2• 2 years ago

This demonstrated that Crowdstrike lacks the most basic of tests and staging environments.

andrepd• 2 years ago

Seems to be a perfectly rational decision to maximise short term returns for the owners of the company.

Now make of that what you will.

squigz• 2 years ago

Isn't Crowdstrike the same company the heavily lobbied to get make all their features a requirement for government computers?

Do you have any more sources on this specifically? The link you gave doesn't seem to reference anything specific.

steelframe5• 2 years ago

Wow, this hits close to home. Doing a page fault where you can't in the kernel is exactly what I did with my very first patch I submitted after I joined the Microsoft BitLocker team in 2009. I added a check on the driver initialization path and didn't annotate the code as non-paged because frankly I didn't know at the time that the Windows kernel was paged. All my kernel development experience up to that point was with Linux, which isn't paged.

BitLocker is a storage driver, so that code turned into a circular dependency. The attempt to page in the code resulted a call to that not-yet-paged-in code.

The reason I didn't catch it with local testing was because I never tried rebooting with BitLocker enabled on my dev box when I was working on that code. For everyone on the team that did have BitLocker enabled they got the BSOD when they rebooted. Even then the "blast radius" was only the BitLocker team with about 8 devs, since local changes were qualified at the team level before they were merged up the chain.

The controls in place not only protected Windows more generally, but they even protected the majority of the Windows development group. It blows my mind that a kernel driver with the level of proliferation in industry could make it out the door apparently without even the most basic level of qualification.

bonestamp22• 2 years ago

without even the most basic level of qualification

That was my first thought too. Our company does firmware updates to hundreds of thousands of devices every month and those updates always go through 3 rounds of internal testing, then to a couple dozen real world users who we have a close relationship with (and we supply them with spare hardware that is not on the early update path in case there is a problem with an early rollout). Then the update goes to a small subset of users who opt in to those updates, then they get rolled out in batches to the regular users in case we still somehow missed something along the way. Nothing has ever gotten past our two dozen real world users.

rvnx4• 2 years ago

Or it could be made that Windows stops loading drivers that are crashing.

Third-party driver/module crashed more than 3 times in a row -> Third-party driver/module is punished and has to be manually re-enabled.

yibg• 2 years ago

This was my first thought too. I'm not that familiar with the space, but I would think for something this sensitive the rollout would be staggered at least instead of what looks like globally all at the same time.

mihaaly5• 2 years ago

Exactly this what I was missing in the story. Like why not to have a limited set of users have it before going live for the whole user base at a mission critical product like this is beyond comprehension of everyone ever came across software bugs (so billions of people). And then we already overcame the part of not testing internally well, or at all? Something clusteruck must have happened there which is still better than imagining that this is the normal way the organization operates. Which is a very scary vision. Serious rethinking of trusting this organization is due everywhere!

cush• 2 years ago

It's baffling how fast and wide the blast radius was for this Crowdstrike update. Quite impressive actually, if you think about it - updating billions of systems that quickly.

rkagerer7• 2 years ago

That is the right way to do it.

throwaway73562• 2 years ago

But do you ever get free world-wide advertisement that everyone uses your product? Crowdstrike sure did and I'm sure they'll use that to sell it to more people.

brightlancer3• 2 years ago

Even then the "blast radius" was only the BitLocker team with about 8 devs, since local changes were qualified at the team level before they were merged up the chain.

Up the chain to automated test machines, right?

nick73761822• 2 years ago

You would think automated test would come before your teammates work stations / commit to head.

dralley• 2 years ago

https://www.usenix.org/system/files/1311_05-08_mickens.pdf

"Perhaps the worst thing about being a systems person is that other, non-systems people think that they understand the daily tragedies that compose your life. For example, a few weeks ago, I was debugging a new network file system that my research group created. The bug was inside a kernel-mode component, so my machines were crashing in spectacular and vindic- tive ways. After a few days of manually rebooting servers, I had transformed into a shambling, broken man, kind of like a computer scientist version of Saddam Hussein when he was pulled from his bunker, all scraggly beard and dead eyes and florid, nonsensical ramblings about semi-imagined enemies. As I paced the hallways, muttering Nixonian rants about my code, one of my colleagues from the HCI group asked me what my problem was. I described the bug, which involved concur- rent threads and corrupted state and asynchronous message delivery across multiple machines, and my coworker said, “Yeah, that sounds bad. Have you checked the log files for errors?” I said, “Indeed, I would do that if I hadn’t broken every component that a logging system needs to log data. I have a network file system, and I have broken the network, and I have broken the file system, and my machines crash when I make eye contact with them. I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS. My only logging option is to hire monks to transcribe the subjective experience of watching my machines die as I weep tears of blood.”

qingcharles2• 2 years ago

Ah, the joys of trying to come up with creative ways to get feedback from your code when literally nothing is available. Can I make the beeper beep in morse code? Can I just put a variable delay in the code and time it with a stopwatch to know which value was returned from that function? Ughh.

Arrath3• 2 years ago

That's beautiful.

killerstorm2• 2 years ago

This is an interesting piece of creative writing, but virtual machines already existed in 2013. There are very few reasons to experiment on your dev machine.

dwattttt3• 2 years ago

Quoting James Mickens is always the winning move. I recommend the entire collection of his wisdom, https://mickens.seas.harvard.edu/wisdom-james-mickens

rkagerer7• 2 years ago

Somebody get this man a serial port, or maybe a PC Speaker to Morse out diagnostics signals.

brcmthrowaway• 2 years ago

What does this mean?

Windows kernel paged, linux non paged?

ec109685• 2 years ago

Linux kernel memory isn’t paged out to disk, while Windows kernel memory can be: https://knowledge.broadcom.com/external/article/32146/third-...

ww5203• 2 years ago

The memory used by the Windows kernel is either Paged or Non-Paged. Non-Paged means pinning the memory in physical RAM. Paged means it might be swapped out to disk and paged back in when needed. OP was working on BitLocker a file system driver, which handles disk IO. It must be pinned in physical RAM to be available all the times; otherwise, if it's paged out, an IO request coming would find the driver code missing in memory and try to page in the driver code, which triggers another IO request, creating an infinite loop. The Windows kernel usually would crash at that point to prevent a runway system and stops at the point of failure to let you fix the problem.

p_l20• 2 years ago

Linux is a bit unusual in that kernel memory is generally physically mapped and unless you use vmalloc any memory you allocate has to correspond to pages backed by RAM. This also ties into how file IO happens, swapping, and how Linux approach to IO is actually closer to Multics and OS/400 than OG Unix.

Many other systems instead default to using full power of virtual memory including swapping kernel space to disk, with only things explicitly need to be kept in ram being allocated from "non-paged" or "wired" memory.

EDIT: fixed spelling thanks to writing on phone.

temac• 2 years ago

"It blows my mind that a kernel driver with the level of proliferation in industry could make it out the door apparently without even the most basic level of qualification."

It was my understanding that MS now sign 3rd party kernel mode code, with quality requirements. In which case why did they fail to prevent this?

einpoklum3• 2 years ago

In which case why did they fail to prevent this?

"Oh, crowdstrike? Yeah, yeah, here's that Winodws kernel code signing key you paid for."

YZF2• 2 years ago

Drivers have had to be signed forever and pass pretty rigorous test suites and static analysis.

The problem here is obviously this other file the driver sucks in. Just because the driver didn't crash for Microsoft in their lab doesn't mean a different file can't crash it...

password43212• 2 years ago

https://news.ycombinator.com/item?id=41006104#41006555

the flawed data was added in a post-processing step of the configuration update, which is after it's been tested internally but before it's copied to their update servers

per a new/green account

cududa• 2 years ago

So have we decided to stop using checksums or something?

spaceywilly2• 2 years ago

“And so that’s why we recommend using phased rollouts” -Every DevOps engineer from now on

mihaaly5• 2 years ago

when something is changed, we usually re-test. that's the whole point of testing anyway. :)

jboy552• 2 years ago

I was thinking, this doesn't seem like its a case of all these machines still on an old version of windows, or some specific version, that is having issues. Therefore QA just missed one particular variant in their smoke testing. It seems like its every windows instance with that software, so either they don't have basic automated testing, or someone pushed this outside of a normal process.

EasyMark• 2 years ago

This is what I don’t get, it’s extremely hard for me to believe this didn’t get caught in CI when things started blue screening. Every place I ever did test rebooting/powercycling was part of CI, with various hardware configs. This was before even our lighthouse customers even saw it.

tomrod5• 2 years ago

Disgruntled employee trying to use Crowd Strike to start a General Strike?

Fire-Dragon-DoL3• 2 years ago

What makes you think they have CI after what happened?

mtlynch• 2 years ago

Doing a page fault where you can't in the kernel is exactly what I did with my very first patch I submitted after I joined the Microsoft BitLocker team in 2009.

Hello from a fellow BitLocker dev from this time! I think I know who this is, but I'm not sure and don't want to say your name if you want it private. Was one of your Win10 features implementing passphrase support for the OS drive? In any case, feel free to reach out and catch up. My contact info is in my profile.

steelframe5• 2 years ago

Win8. I've been seeing your blog posts show up here and there on HN over the years, so I was half expecting you to pick up on my self-doxx. I'll ping you offline.

Fire-Dragon-DoL3• 2 years ago

I'm completely ignorant on the topic but isn't rebooting a default test for kernel code, given how sensitive it is?

steelframe5• 2 years ago

Oh I rebooted, I just didn't happen to have the right configuration options to invoke the failure when I rebooted. Not every dev workstation was bluescreening, just the ones with the particular feature enabled.

isatty• 2 years ago

I do not mean this to be blamey in any way shape or form and am asking only about the process:

Shouldn’t that have been caught in code review?

steelframe5• 2 years ago

My manager actually blamed the more senior developer who reviewed my code for that one.

usr11068• 2 years ago

I didn't know at the time that the Windows kernel was paged.

At uni I had a professor in database systems, who did not like written exams, but mostly did oral exams. Obviously for DBMSes the page buffer is very relevant, so we chatted about virtual memory and paging. So in my explanation I made the difference for kernel space and user space. I am pretty sure I had read that in a book describing VAX/VMS internals. However, the professor claimed that a kernel never does paging for its own memory. I did not argue on that and passed the exam with the best grade. Did not check that book again to verify my claim. I have never done any kernel space development even vaguely close to memory management, so still today I don't know the exact details.

However, what strikes me here: When that exam happened in 1985ish the NT kernel did not exist yet, I'd believe. However, IIRC a significant part of the DEC VMS kernel team went to Microsoft to work on the NT kernel. So the concept of paging (a part of) kernel memory went with them? Whether VMS --> WNT, every letter increased by one is just a coincidence or intentionally the next baby of those developers I have never understood. As Linux has shown us today much bigger systems can be successfully handled without the extra complications for paging kernel memory. Whether it's a good idea I don't know, at least not a necessary one.

nullindividual5• 2 years ago

If you want to hear the history of [DEC/VMS] NT from the horses mouth:

https://www.youtube.com/watch?v=xi1Lq79mLeE

usr11068• 2 years ago

It blows my mind that a kernel driver with the level of proliferation in industry could make it out the door apparently without even the most basic level of qualification.

Discussed elsewhere it is claimed that the file causing the crash was a data file that has been corrupted in the delivery process. So the development team and their CI have probably tested a good version, but the customer received a bad one.

If that is true to problem is that the driver first uses an unsigned file at all, so all customer machines are continuously at risk for local attacks. And then it does not do any integrity check on the data it contains, which is a big no no for all untrusted data, whether user space or kernel.

vlod2• 2 years ago

If the file was signed, wouldn't that have prevented the corrupted transmission file from being loaded.

I assume if the signed file was hacked (or parts missing), then it wouldn't pass verification.

wedesoft• 2 years ago

Still a staggered roll-out would have reduced the impact.

mbreese7• 2 years ago

And then it does not do any integrity check on the data it contains, which is a big no no for all untrusted data, whether user space or kernel.

To me, this is the inexcusable sin. These updates should be signed and signatures validated before the file is read. Ideally the signing/validating would be handled before distribution so that when this file was corrupted, the validation would have failed here.

But even with a good signature, when a file is read and the values don’t make sense, it should be treated as a bad input. From what I’ve seen, even a magic bytes header here would have helped.

sandworm101• 2 years ago

So the key test, the test that was not run, was to turn the machine off and on again? Classic windows.

hevisko• 2 years ago

Must have been DNS... when they did the deployment run and the necessary code was pulled and the DNS failed and then the wrong code got compiled...</sarcasm>

that they don't even do staged/A-B pushes was also <mind-blown-away>

But the most.... ironical was: https://www.theregister.com/2024/07/18/security_review_failu...

CKMo• 2 years ago

This is a good example of why you don't want ring0 level access for clients. Or just, you don't want client-based solutions. The provider just becomes another threat vector.

swozey2• 2 years ago

I know there's a better word to be used here, but what initially looked like a massive cyberattack turning out to be a massive defender foot-broom is chefs kiss.

I saw it was Windows and went to bed. What a great feeling.

I'm sorry to those of you dealing with this. I've had to wipe 1200 computers over a weekend in a past life when a virus got in.

Did I receive any appreciation? Nope. I was literally sleeping under cubicle desks bringing up isolated rows one by one. I switched everything in that call center to linux after that. Ironically it turns out it was a senior engineers ssh key that got leaked somehow and was used to get in and dig around servers in our datacenter outside of my network. My filesystem logging (in Windows, coincidentally) alerted me.

IT is fun.

cloin• 2 years ago

I'm confused as to how this issue is so widespread in the first place. I'm unfamiliar with how Crowdstrike works, do organizations really have no control over when these updates occur? Why can't these airlines just apply the updates in dev first? Is it the organizations fault or does Crowdstrike just deliver updates like this and there's no control? If that's just how they do it, how do they get away with this?

rboyd2• 2 years ago

Companies operate on a high level of fear and trust. This is the security vendor, so in theory they want those updates rolled out as quickly as possible so that they don't get hacked. Heh.

prpl• 2 years ago

I mean, they pay a lot of money to crowdstrike. A failure this widespread is a Crowdstrike dev issue.

commandlinefan• 2 years ago

Can somebody summarize what CrowdStrike actually is/does? I can't figure it out from their web page (they're an "enterprise" "security" "provider", apparently). Is this just some virus scanning software? Or is it some bossware/spyware thing?

kube-system5• 2 years ago

Is this just some virus scanning software?

Essentially, yes. It is fancy endpoint protection.

noduerme• 2 years ago

It's both. Antivirus along with spyware to also watch for anything the user is doing that could introduce a threat, such as opening a phishing email, posting on HN, etc.

matthew-wegner• 2 years ago

The thing people are paying for is regulatory compliance. The actual product is anti-virus software.

mym19905• 2 years ago

These updates happen automatically and as far as I can tell, there is no option to turn this feature off. From a security perspective, the vendor will always want you to be on the most recent software to protect from attack holes that may open up by operating on an older version. Your IT department will likely want this as well to avoid culpability. Just my 2 observations, whether it is the right away or if CS is effective at what it does, no idea.

apitman4• 2 years ago

CrowdStrike is an endpoint detection and response (EDR) system. It is deeply integrated into the operating system. This type of security software is very common on company-owned computers, and often have essentially root privileges.

rty32• 2 years ago

Well, actually more than root. Even for an administrator user on Windows, it’s pretty hard to mess with things and get into BSOD. CrowdStrike has these files as drivers (as indicated by .sys file extension) which run in the kernel mode.

jtdev• 2 years ago

It's a Mossad/CIA sponsored spyware agent.

yard20108• 2 years ago

Source: bro trust me

jmsgwd• 2 years ago

Presumably endpoint detection & response (EDR) agents need to do things like dynamically fetch new malware signatures at runtime, which is understandable. But you'd think that would be treated as new "content", something they're designed to handle in day-to-day operation, hence very low risk.

That's totally different to deploying new "code", i.e. new versions of the agent itself. You'd expect that to be treated as a software update like any other, so their customers can control the roll out as part of their own change management processes, with separate environments, extensive testing, staggered deployments, etc.

I wonder if such a content vs. code distinction exists? Or has EDR software gotten so complex (e.g. with malware sandboxing) that such a distinction can't easily be made any more?

In any case, vendors shouldn't be able to push out software updates that circumvent everyone's change management processes! Looking forward to the postmortem.

nudgeee• 2 years ago

My guess is it probably was a content update that tickled some lesser trodden path in the parser/loader code, or created a race condition in the code which lead to the BSOD.

Even if it’s ‘just’ a content update, it probably should follow the rules of a code update (canaries, pre-release channels, staged rollouts, etc).

stainablesteel• 2 years ago

its strange how people who work in professions that are considered crucial infrastructure are held to such a high standard but there's always some tech problem that cripples them the hardest

queuebert2• 2 years ago

And they all invariably use Windows instead of a high-reliability OS.

cypress662• 2 years ago

Windows is high reliability. The problem here is what's basically a third party backdoor.

basch3• 2 years ago

There are sooo many companies in the world, when snowflake or crowdstrike or solarwinds has an issue, it’s going to touch every industry.

TeMPOraL14• 2 years ago

The people working in those professions are; their bosses and their IT departments are not. IT security is treated as solved problem - if you deploy enough well-known solutions that prevent your employers from working, everything will be Safe from CyberAttacks. There's an assumption of quality like you'd normally have with drugs or food in the store. But this isn't the case in this industry, doubly so in security. Quality solutions are almost non-existent, so companies should learn to operate under the principle of caveat emptor.

rboyd2• 2 years ago

Seems like a modern operating system would have an automatic rollback mechanism for cases like this.

Kye9• 2 years ago

Windows has restore points that do this in the event of a failed update, but this wasn't Windows.

tgtaptarget• 2 years ago

In my org, none of the essential systems went down (those used by labor). However all of management's individual PCs went down which got me wondering... Is this the beginning (or continuation) of whittling down what is "essential" human labor versus what could be done remotely (or eliminated completely)?

Or perhaps Microsoft is just garbage and soon will be as irrelevant as commercial real estate office parks and mega-call centers

kubov• 2 years ago

Were cloud providers (AWS and azure) so heavily impacted because they use CS internally or because so many users use CS?

raphar• 2 years ago

I want to see the internal postmortem of why this happened to CrowdStrike (if they are still in business)

gonzo41• 2 years ago

Do people not have test environments?

tbatchelli3• 2 years ago

This event is predicted in Sydney Dekker’s book “Drift into Failure”, which basically postulates that in order to prevent local failure we setup failure prevention systems that increase the complexity beyond our ability to handle, and introduce systemic failures that are global. It’s a sobering book to read if you ever thought we could make systems fault tolerant.

COGlory• 2 years ago

We need more local expertise is really the only answer. Any organization that just outsources everything is prone to this. Not that organizations that don't outsource aren't prone to other things, but at least their failures will be asynchronous.

bjelkeman-again• 2 years ago

Funny thing is that for decades there were predictions about how there was a need for millions of more IT workers. It was assumed one needed local knowledge in companies. Instead what we got was more and more outsourced systems and centralized services. This today is one of the many downsides.

smsm429• 2 years ago

The problem here would be that there's not enough people who can provide the level of protection a third-party vendor claims to provide, and a person (or persons) with comparable level of expertise would be much more expensive likely. So companies who do their own IT would be routinely outcompeted by ones that outsource, only for the latter to get into trouble when the black swan swoops in. The problem is all other kinds of companies are mostly extinct by then unless their investors had some super-human foresight and discipline to invest for years into something that year after year looks like losing money.

mym19905• 2 years ago

Many systems are fault tolerant, and many systems can be made fault tolerant. But once you drift into a level of complexity spawned by many levels of dependencies, it definitely becomes more difficult for system A to understand the threats from system B and so on.

tbatchelli3• 2 years ago

Do you know of any fault tolerant system? Asking because in all the cases I know, when we make a system "fault tolerant" we increase the complexity and we introduce new systemic failure modes related to our fault-tolerant-making-system, making them effectively non fault tolerant.

In all the cases I know, we traded frequent and localized failure for infrequent but globalized catastrophic failures. Like in this case.

notNNT• 2 years ago

Also a major point in the Black Swan. In the Black Swan, Taleb describes that it is better for banks to fail more often than for them to be protected from any adversity. Eventually they will become "too big to fail". If something is too big to fail, you are fragile to a catastrophic failure.

UniverseHacker3• 2 years ago

I was wondering when someone would bring up Taleb RE: this incident.

I know you aren't saying it is, but I think Taleb would argue that this incident, as he did with the coronavirus pandemic for example, isn't even a Black Swan event. It was extremely easy to predict, and you had a large number of experts warning people about it for years but being ignored. A Black Swan is unpredictable and unexpected, not something totally predictable that you decided not to prepare for anyways.

nurbl• 2 years ago

"Antifragile" is even more focused around this.

jessriedel• 2 years ago

It's also in line with arguments made by Ted Kaczynski (the Unabomber)

Why must everything collapse? Because, [Kaczynski] says, natural-selection-like competition only works when competing entities have scales of transport and talk that are much less than the scale of the entire system within which they compete. That is, things can work fine when bacteria who each move and talk across only meters compete across an entire planet. The failure of one bacteria doesn’t then threaten the planet. But when competing systems become complex and coupled on global scales, then there are always only a few such systems that matter, and breakdowns often have global scopes.

https://www.overcomingbias.com/p/kaczynskis-collapse-theoryh...

https://en.wikipedia.org/wiki/Anti-Tech_Revolution

localfirst3• 2 years ago

crazy how much he was right. if he hadn't gone down the path of violence out of self-loathing and anger he might have lived to see a huge audience and following.

ricardo81• 2 years ago

I haven't read it, but I'd take a leap to presume it's somewhere between the people that say "C is unsafe" and "some other language takes care of all of things".

Basically delegation.

smsm429• 2 years ago

Just yesterday listened to a lecture by Moshe Vardi which covers adjacent topics:

https://simons.berkeley.edu/events/lessons-texas-covid-19-73...

joe_the_user• 2 years ago

I think it was "predicted" by Sunburst, the Solarwinds hack.

I don't think centrally distributed anti-virus software is the only way to maintain reliability. Instead, I'd say companies to centralize anything like administration since it's cost effective and because they actually aren't concerned about global outage like this.

JM Keynes said "A ‘sound’ banker, alas! is not one who foresees danger and avoids it, but one who, when he is ruined, is ruined in a conventional and orthodox way along with his fellows, so that no one can really blame him." and the same goes for corporate IT.

akira25016• 2 years ago

we setup failure prevention systems

You can't prevent failure. You can only mitigate the impact. Biology has pretty good answers as to how to achieve this without having to increase complexity as a result, in fact, it often shows that simpler systems increase resilliency.

Something we used to understand until OS vendors became publicly traded companies and "important to national security" somehow.

jjav9• 2 years ago

if you ever thought we could make systems fault tolerant

The only possible way to fault tolerancy is simplicity and then more simplicity.

Things like crowsdtrike have the opposite approach. Add a lot of fragile complexity attempting to catch problems, but introducing more attack surfaces than they can remove. This will never succeed.

ryukoposting• 2 years ago

As an architect of secure, real-time systems, the hardest lesson I had to learn is there's no such thing as a secure, real-time system in the absolute sense. Don't tell my boss.

oneepic• 2 years ago

I think a surprising amount of people already share this view, even if they don't go into extensive treatment with references like Dekker presumably does (I haven't read it).

I suspect most people in power just don't subscribe to that. which is precisely why it's systemic to see the engineer shouting "no!" when John CEO says "we're doing it anyway." I'm not sure this is something you can just teach, because the audience definitely has reservations about adopting it.

cjbgkagh13• 2 years ago

Due to the scale I think it’s reasonable to state that in all likelihood many people have died because of this. Sure it might be hard to attribute single cases but statistically I would expect to see a general increase in probability.

I used to work at MS and didn’t like their 2:1 test to dev ratio or their 0:1 ratio either and wish they spent more work on verification and improved processes instead of relying on testing - especially their current test in production approach. They got sloppy and this was just a matter of time. And god I hate their forced updates, it’s a huge hole in the threat model, basically letting in children who like to play with matches.

My important stuff is basically air-gapped. There is a gateway but it’ll only accept incoming secure sockets with a pinned certificate and only a predefined in-house protocol on that socket. No other traffic allowed. The thing is designed to gracefully degrade with the idea that it’ll keep working unattended for decades, the software should basically work forever so long as equivalent replacement hardware could be found.

Supermancho• 2 years ago

And god I hate their forced updates,

My windows machine notified me of the update, asked me to restart. I was busy, so I didn't. Then the news broke, then the update was rolled back.

vel0city4• 2 years ago

It wasn't a Windows update. If you got a notification for an update, it wasn't the update that did this.

dtech• 2 years ago

I don't see what this has to much do with MS. A bad proprietary kernel module can crash any OS.

cjbgkagh13• 2 years ago

I don’t know the specifics of this case, but formal verification of machine code is an option. Sure it’s hard and doesn’t scale well but if it’s required then vendors will learn to make smaller kernel modules.

If something cannot be formally verified at the machine code level there should be a controls level verification where vendors demonstrate they have a process in place to achieving correctness by construction.

Driver devs can be quite sloppy and copy paste bad code from the internet, in the machine code Microsoft can detect specific instances of known copy and pasted code and knows how to patch it. I know they did this for at least one common error. But if I was in the business of delivering an OS I want people to rely on my OS this stuff formal verification at some level would be table stakes.

falcor842• 2 years ago

No other OS forces an auto-restart.

philistine3• 2 years ago

I blame Microsoft in the larger sense; they still allow kernel extensions for use cases that Apple has shown could be moved outside the kernel.

SAI_Peregrinus2• 2 years ago

An immutable OS can be set up to revert to the previous version if a change causes a boot failure. Or even a COW filesystem with snapshots when changes are applied. Hell, Microsoft's own "System Restore" capability could do this, if MS provided default-on support for creating system restore points automatically when system files are changed & restoring after boot failures.

ltadeut• 2 years ago

MS could've leaned more towards user-space kernel drivers though. Apple has been going in that direction for a while and I haven't seem much of that (if anything) coming from MS.

That would have prevented a bad driver from taking down a device.

Salgat7• 2 years ago

I love their forced updates, because if you know what you're doing you can disable them, and if you don't know what you're doing, well you shouldn't be disabling updates to begin with. I think people forget how virus infested and bug addled Windows used to be before they enforced updates. People wouldn't update for years and then bitch how bad Windows was, when obviously the issue wasn't Windows at that point.

cjbgkagh13• 2 years ago

Ignoring all of the other approaches to that problem I wonder if this update will take the record for most damage done by a single virus/update. At some point the ‘cure’ might be worse than the disease. If it were up to me I would be suggesting different cures.

__MatrixMan__21• 2 years ago

If the user wants to boot an older, known-insecure, version so that they can continue taking 911 calls or scheduling surgeries... I say let 'em. Whether to exercise this capability should be a decision for each IT department, not imposed by Microsoft on to their whole swarm.

zzyzxd• 2 years ago

At one company I used to work for, we had boring, airgapped systems that just worked all the time, until one day security team demanded that we must install this endpoint security software. Usually, they would fight tooth and nail to prevent devs from giving any in-house program any network access, but they didn't even blink once to give internet access to those airgapped systems because CrowdStrike agents need to talk to their mothership in AWS. It's all good, it's for better security!

It never caught any legit threat, but constantly flagged our own code. Our devs talked to security every other week to explain why this new line of code is not a threat. It generated a lot of work and security team's headcount just exploded. The software checked a lot of security checkboxes, and our CISO can sleep better at night, so I guess end of day it's all worth it.

seniorThrowaway• 2 years ago

It never caught any legit threat, but constantly flagged our own code

When I worked in large enterprise it got to the point that if a piece of my app infrastructure started acting weird the blackbox security agents on the machines were the first thing I suspected. Can't tell you how many times they've blocked legit traffic or blown up a host by failing to install an update or logging it to death. Best part is when I would reach out to the teams responsible for the agents they would always blame us, saying we didn't update, or weren't managing logs etc. Mind you these agents were not installed or managed by us in any way, were supposed to auto update, and nothing else on the system outran the logrotate utility. Large enterprise IT security is all about checking boxes and generating paperwork and jobs. Most of the people I've interacted with on it have never even logged into a system or cloud console. By the end I took to openly calling them the compliance team instead of the security team.

cjbgkagh13• 2 years ago

I know I've lost tenders due to not using a pre-approved anti-virus vendors which really does suck and has impinged the growth of my company, but since I'm responsible for the security it helps me sleep at night. This morning I woke up to a bunch of emails and texts asking me if my systems have been impacted by this and it was nice to be able to confidently write back that we're completely unaffected.

I day-dream about being able to use immutable unikernels running on hypervisors so that even if something was to get past a gateway there would be no way to modify the system to work in a way that was not intended.

Air-gapping with a super locked down gateway was already getting more popular precisely due to the forced updates threat surface area, and after today I expect it to be even more popular. At the very least I’ll be able to point to this instance when explaining the rational behind the architecture which could help in getting exemptions from the antivirus box ticking exercise.

RajT883• 2 years ago

This is almost definitely on Crowdstrike.

There is a windows release preview channel that exists for finding issues like this ahead of time.

To be fair - it is possible the conflicting OS update did not make it to that channel. It is also possible it is due to an embarassing bug from MSFT (uknown as yet).

Until I hear that this is the case - I am pinning this on Crowdstrike. This should have been caught before prod.

cjbgkagh13• 2 years ago

Even if this is entirely due to Crowdstrike I see it as Microsofts failure to properly police their market.

There is the correctness by testing vs correctness by construction dynamic and in my view given the scale of interactions between an OS and the kernel modules trying to achieve correctness by testing is negligent. Even at the market scale Microsoft has there are not enough Windows computers to preview test every combination. Especially when taking into account the people on the preview ring have different behaviors to those on the mainline so many combinations simply won't appear in the preview.

I see it as Microsoft owning the Windows kernel module space and has allowed sloppiness by third parties and themselves, I don't know the specifics but I could easily believe that this is a due to a bug from Microsoft. The problem with allowing such sloppiness is that the slopy operators out compete the responsible operators, the bad pushes out the good until only the bad remains. A sloppy developer can push more code and gets promoted while the careful developer gets fired.

satisfice2• 2 years ago

As a tester, I'm frustrated by how little support testing gets in this industry. You can't blame bad testing if it's impossible to get reasonable time and cooperation to do more than a perfunctory job.

nimbius• 2 years ago

I work for a diesel truck repair facility and just locked up the doors after a 40 minute day :( .

- lifts wont operate.

- cant disarm the building alarms. (have been blaring nonstop...)

- cranes are all locked in standby/return/err.

- laser aligners are all offline.

- lathe hardware runs but controllers are all down.

- cant email suppliers.

- phones are all down.

- HVAC is also down for some reason (its getting hot in here.)

the police drove by and told us to close up for the day since we dont have 911 either.

alarms for the building are all offline/error so we chained things as best we could (might drive by a few times today.)

we dont know how many orders we have, we dont even know whos on schedule or if we will get paid.

shoebham• 2 years ago

wow, why do lifts require an OS?

warkdarrior• 2 years ago

How else are you going to update your grocery list while operating the lift?

kulikalov• 2 years ago

the question is - why lifts require windows?

thedrbrian• 2 years ago

Why do lathes , cranes and laser alignment systems need a new copy of windows?

morning-coffee3• 2 years ago

I think the same question can be asked for why lots of equipment seemingly requires an OS. My take is that these products went through a phase of trying to differentiate themselves from competitors and so added convenience features that were easier to implement with a general purpose computer and some VB script rather than focusing on the simplest most reliable way to implement their required state machines. It's essentially convenience to the implementors at the expense of reliability of the end result.

freehorse8• 2 years ago

Because we live deep into the internet of shit era.

saganus4• 2 years ago

How come lifts and cranes are affected by this?

Are they somehow controlled remotely? or do they need to ping a central server to be able to operate?

I can see how alarms, email and phones are affected but the heavy machinery?

(Clearly not familiar with any of these things so I am genuinely curious)

lima• 2 years ago

Lots and lots of heavy machinery uses Windows computers even for local control panels.

ddulaney• 2 years ago

Probably a Windows-based HMI (“human-machine interface”).

I used to build sorting machines that use variants of the typical “industrial” tech stack, and the actual controllers are rarely (but not never!) Windows. But it’s common for the HMI to be a Windows box connected into the rest of the network, as well as any server.

davepm• 2 years ago

I'm still running multiple CNC/Industrial equipment with win3.1/98/xp. Only just retired one running Dos 6.2

__MatrixMan__21• 2 years ago

Oh man, you work with some cool (and dangerous) stuff.

Outage aside, do you feel safe using it while knowing that it accepts updates based on the whims of far away people that you don't know?

NegativeK5• 2 years ago

I hate to be that person, but things have moved to automatic updates because security was even shittier when the user was expected to do it.

I can't even imagine how much worse ransomware would be if, for example, Windows and browsers weren't updating themselves.

xyst4• 2 years ago

what are the brands of these systems?

Saris2• 2 years ago

I'm just impressed that the lifts, alarms, cranes, phones, etc all run on Windows somehow.

a10c• 2 years ago

In a lot of cases you find tangential dependencies on Windows in ways you don't expect. For example a deployment pipeline entirely linux-based deploying to linux-based systems that relies on Active Directory for authentication.

fransje266• 2 years ago

I don't know if "impressed" is the right word..

"Appalled", "bewildered" and "horrified" and also comes to mind..

moritzwarhier2• 2 years ago

we dont have 911 either

Holy cow...

Who on earth requires a Windows-based backend (or whatever else had CrowdStrike, in the shop or outside) for regular (VoIP) phone calls.

This should really lead to some learnings for anyone providing any kind of phone infrastructure.

jajko2• 2 years ago

Or lathe, or cranes, or alarms, or hvac... what the actual fuck.

Next move should be some artisanal as mechanical-as-possible quality products, or at least Linux(TM) certified product or similar (or Windows-free (TM)). The opportunity is here, everybody noticed this clusterfuck, and smart folks don't like ignoring threats that are in your face.

But I suppose in 2 weeks some other bombastic news will roll over this and most will forget. But there is always some hope

Filligree• 2 years ago

That’s not it. 911 itself was down.

GirishSharma643• 2 years ago

Who is responsible for this billion dollar mistake?

vlan0• 2 years ago

Anything that has root/kernel access is a risk. It always has been. When will we learn. Probably never. Because money runs this world. So sad. Time to open a bakery and move on from this world.

Sohcahtoa824• 2 years ago

Considering what Crowdstrike is intended to do, it's not really possible for it to work without running at the kernel level.

steine65• 2 years ago

Here's a visual representation of flight cancellations and delays at major US airports https://www.flightaware.com/miserymap/

Tylast2• 2 years ago

Oh wow! Thanks for that. I use flight aware and twas not award of that particular page.

1oooqooq13• 2 years ago

this animations were all the rage in Chinese wechat while this was going on

_yb2s• 2 years ago

Why are so many mission critical hardware connected systems connected to the internet at all or getting automatic updates?

This is just basic IT common sense. You only do updates during a planned outage, after doing an easily reversible backup, or you have two redundant systems in rotation and update and test the spare first. Critical systems connected to things like medical equipment should have no internet connectivity, and need no security updates.

I follow all of this in my own home so a bad update doesn’t ruin my work day… how do big companies with professional IT not know this stuff?

gedy2• 2 years ago

Why are so many mission critical hardware connected systems connected to the internet at all or getting automatic updates?

Because it lets them "scale" by having fewer and cheaper offsite IT and contractors to manage vs hiring pesky onsite employees.

basch3• 2 years ago

You do that for antivirus definition updates?

mckn1ght2• 2 years ago

Probably, implicitly. Have automated regular backups, and don’t let your AV automatically update, or even if it does, don’t log into all your computers simultaneously. If you update/login serially, then the first BSOD would maybe prevent you from doing the same thing on the other (or possibly, send you running to the other to accomplish your task, and BSODing that one too!)

But yeah this is one reason why I don’t have automatic updates enabled for anything, the other major one being that companies just can’t resist screwing with their UIs.

UniverseHacker3• 2 years ago

I’m not an IT professional, but I don’t use antivirus software on my personal macs and linux machines- I do regular rotated physical backups, and only install software digitally signed by trusted sources and well reviewed Pirate Bay accounts (that's a joke :-).

My only windows machine is what I would classify as a mission critical hardware connected/control device, an old Windows 8 tablet I use for car diagnostics- I do not connect it to the internet, and never perform updates on it.

I am an academic and use a lot of old multi-million dollar scientific instruments which have old versions of windows controlling them. They work forever if you don't network them, but the first time you do, someone opens up a browser to check their social media, and the entire system will fail quickly.

barryrandall2• 2 years ago

Yes. In an environment where you have so many clients that they can DDoS the antivirus management server, you have to stagger the update schedule anyway. The way we set it up, sysadmins/help desk/dev deployments updated on day 1, all IT workstations/test deployments updated on day 2, and all workstations/staging/production deployments on day 3.

vel0city4• 2 years ago

CrowdStrike let's you create update strategies and rollout groups.

This update bypassed all of those settings.

UniverseHacker3• 2 years ago

Well that context makes it make a little more sense... I still wouldn't be trusting a service like that for mission critical hardware that shouldn't be connected to the internet in the first place.

The question with these types of services is: is your goal to keep the system as reliable as possible, or to be able to place the blame on a 3rd party when it goes down? If it's a critical safety system that human lives depend on, the answer better be the former.

asynchronous2• 2 years ago

What people aren’t understanding is MOST of the outage isn’t caused by a crowdstrike install itself, it’s caused because something upstream of it (a critical application server) is what got borked, and that’s having a domino effect on everything else.

irusensei2• 2 years ago

Can we end the whole “loading a kernel rootkit” thing? AFAIK Apple already shuns kernel extensions. What’s preventing Microsoft to do the same? As a bonus shit like anti cheat will go away too.

franczesko2• 2 years ago

I just wanted to mention that Microsoft has 3 tiers of Windows beta releases before changes are pushed to production. I can't comprehend how this wasn't noticed before.

bloopernova4• 2 years ago

It didn't come from Microsoft or Windows Update. It was pushed by Crowdstrike to their corporate security kernel extension.

franczesko2• 2 years ago

3rd party got unsupervised access to kernel? I'd say it's even worse then

jimt1234• 2 years ago

I watched a presentation by someone representing "I Am The Cavalry" at B-Sides, Las Vegas, a few years ago. Very interesting stuff, gave me a whole new perspective on "cyber security".

https://iamthecavalry.org

PaulHoule• 2 years ago

People at my workplace were affected but I dodged the bullet because I left my computer turned on overnight because I always want to be able to RDP in the next morning in case I decide to stay home.

rootforce• 2 years ago

AWS has posted some instructions for those affected by the issue using EC2.

[AWS Health Dashboard](https://health.aws.amazon.com/health/status)

"First, in some cases, a reboot of the instance may allow for the CrowdStrike Falcon agent to be updated to a previously healthy version, resolving the issue.

Second, the following steps can be followed to delete the CrowdStrike Falcon agent file on the affected instance:

1. Create a snapshot of the EBS root volume of the affected instance

2. Create a new EBS volume from the snapshot in the same Availability Zone

3. Launch a new instance in that Availability Zone using a different version of Windows

4. Attach the EBS volume from step (2) to the new instance as a data volume

5. Navigate to the \windows\system32\drivers\CrowdStrike\ folder on the attached volume and delete "C-00000291*.sys"

6. Detach the EBS volume from the new instance

7. Create a snapshot of the detached EBS volume

8. Create an AMI from the snapshot by selecting the same volume type as the affected instance

9. Call replace root volume on the original EC2 Instance specifying the AMI just created"

rahkiin• 2 years ago

That is a lot of steps. Can this not be scripted?

timoteostewart• 2 years ago

FWIW, I find the high-level overview more useful, because then I can write a script tailored to my situation. Between `bash`, `aws` CLI tool, and Powershell, it would be straightforward to programmatically apply this remedy.

Here's something quick that ChatGPT ginned up: https://chatgpt.com/share/293ea9d5-b7ac-4064-b870-45f8266aea...

CalumSult• 2 years ago

Yes it can, that's what I ended up writing at 4am this morning, lol. We manage way more instances then is feasible to do anything by hand. This is probably too late to help anyone, but you can also just stop instance, detach root, attach it to another instance, delete file(s), offline drive, detach, reattach to original instance, and then start instance. You need a "fixer" machine in the same AZ.

janalsncm• 2 years ago

This outage may be more expensive and cause more damage than any cyberattack in history.

rs999gti• 2 years ago

So can crowdstrike be classified as malware now?

Currently waiting in line for 2 hours + waiting for Delta to tell me when my connecting leg can be booked. My current flight is delayed 5 hours.

Zaskoda5• 2 years ago

I want to add something to the discussion but it's difficult for me to accurately summarize and cite things. In a nutshell, there appears to be a lot of tomfoolery with CrowdStrike and the stuff that happened with the DNC during the 2016 election. Here's some of what I'm talking about:

There's a strong link between the DNC, Hillary, and CrowdStrike. Here's once piece that links a cofounder of CrowdStrike with Hillary pretty far back: https://www.technologyreview.com/innovator/dmitri-alperovitc...

This 2017 piece talks about doubt behind CrowdStrike's analysis of the DNC hack being the result of Russian actors. One of the groups disputing CrowdStrike's analysis was Ukraine's military. https://www.voanews.com/a/crowdstrike-comey-russia-hack-dnc-...

This detailed analysis of CrowdStrike's explanation of the DNC hack goes so far as to say "this sounded made up" https://threatconnect.com/resource/webinar-guccifer-2-0-the-...

The Threat Connect analysis is also discussed here: https://thehill.com/business-a-lobbying/295670-prewritten-gu...

"For one, the vulnerability he claims to have used to hack the NGP VAN ... was not introduced into the code until an update more than three months after Guccifer claims to have entered the DNC system."

Noted at the end of this story they mention that CrowdStrike installed it's software on all of the DNC's systems: https://www.ft.com/content/5eeff6fc-3253-11e6-bda0-04585c31b...

Finally, there's this famous but largely forgotten story of the time Bernie's campaign was accused to accessing Hillary's data: https://www.npr.org/2015/12/18/460273748/bernie-sanders-camp...

"This was a very egregious breach and our data was stolen," Mook said. "We need to be sure that the Sanders campaign no longer has access to our data."

"This bug was a brief, isolated issue, and we are not aware of any previous reports of such data being inappropriately available," the company said in a blog post on its website.

(edited for spelling)

Kye9• 2 years ago

There's a workaround: reboot 10-15 times. I've seen two people say it independently, so maybe it's for real.

tonymet6• 2 years ago

No rolling updates? How could a 100% repro BSOD pass QC? I'm more concerned about the deployment process than the crash itself. Everyone experiences a bad build from time to time. How did this possibly go live?

energy123• 2 years ago

On the plus side this will help us develop an immune system against cyber attacks in any future war. Businesses will start thinking of contingencies.

Key89• 2 years ago

is there an ELI5 on how can this happen? Like i get its a boot loop, but what did crowdstrike do that cause it? How can non malicious code trigger boot loop?

monocasa6• 2 years ago

Kernel driver bug that essentially defaults, then on reboot loads the same driver early on segfaults and reboots again, ad nauseum.

baryphonic• 2 years ago

s/defaults/segfaults/ # stupid autocorrect

sieste• 2 years ago

Best ELI5 ever!

__MatrixMan__21• 2 years ago

Crowdstrike needs to be the first person in the room so that they can act like the boss. If other people show up before crowdstrike, there's a possibility that they'll somehow prevent crowdstrike from being the boss. For this reason, crowdstrike integrates with the boot process in ways that most software doesn't.

Their ability to monitor and intervene against all software on the system also puts them in a position to break all software on the system.

more accurately: s/boss/most informed spy/g

fsloth11• 2 years ago

I would not call Crowdstrike ”non-malicious”. It’s incredibly incompetently implemented kit that’s sold to organizations as snakeoil that ”protects them from cybercrime”. It’s purpose is to give incompetent IT managers to ”implement something plausible” against cyberincidents, and when an incident happens, it gives them the excuse that ”they followed best practices”.

It craps the users PC while at it too.

I hope the company burns to the ground and large organizations realize it’s not a really great idea to run a rootkit at every PC ”just because everyone else does it”.

mulletbum• 2 years ago

I have to say, it saved our ass a few months ago. Some hacker got access to one of multiple brands server infrastructure, started running PowerShell to weed through the rest and CrowdStrike notified us (the owning brand) that something was off about the PowerShell being ran. Turns out this small brand was running a remote in tool that had an exploit. Had Crowdstrike not been on that server we wouldn't have known until someone manually got in there to look at it.

kspacewalk22• 2 years ago

What specifically makes it "incredibly incompetently implemented", and would you simply derisively describe any system that can push updates requiring admin access a "rootkit", or is there some way you envision a "competently implemented rootkit" operating? Your opinion seems incredibly strong so I'm just curious how you arrived at it? I'm not in IT, but the idea of both rolling out updates remotely and outsourcing the timely delivery of these updates to my door* is a no brainer.

* if not directly to all my thousands of PCs without testing, which is 100% a "me" task and not a "that cloud provider over there" task

pas7• 2 years ago

What they did is that they forgot to write a graceful failure mode for their driver loader. (And what they did on top of it is to ship it without testing.)

jethro_tell• 2 years ago

My assumption is that when you have graceful failure for something like this, you risk a situation where someone figures out how to make it gracefully fail, so no it's disabled on this huge fleet.

It's likely that there have been multiple discussions about graceful failure at the load stage and decided against for 'security' reasons.

skocznymroczny• 2 years ago

Crashes in kernel mode usually result in BSODs.

piuantiderp• 2 years ago

how come does anyone still use crowdstrike?

1024core2• 2 years ago

Read on Mastodon: https://infosec.exchange/@littlealex/112813425122476301

The CEO of Crowdstrike, George Kurtz, was the CTO of McAfee back in 2010 when it sent out a bad update and caused similar issues worldwide.

If at first you don't succeed, .... ;-) j/k

localfirst3• 2 years ago

Kurtz response is ridiculous blaming the customer on X. He will probably find another company to hire him as CEO tho. Just an upside down world in the C-suite world.

BaldricksGhost2• 2 years ago

Don't forget the golden parachute. These guys always seem to fail upward.

jeffrallen4• 2 years ago

That guy is gonna fail all the way right up to the top. Sheesh.

bloopernova4• 2 years ago

Worth $3.7B, paid $148M in 2022.

Edited to add: I wonder what the economic fallout from this will be? 10x his monetary worth? 100x? (not trying to put a price on the people who will die because of the outage; for that he and everyone involved needs to go to jail)

mk894• 2 years ago

Nothing at all.

He will be the guy that convinced the investors and stakeholders to pour more money into the company despite some world-wide incident.

He deserves at least 3x the pay.

PS: look at the stocks! They sank, and now they are gaining again value. People can't work, people die, flights get delayed/canceled because of their software.

matrix874• 2 years ago

who is hiring these fucking idiots? they need to be blacklisted

hbn• 2 years ago

Reminds me of Phil Harrison who always seems to find himself in an of executive position, botching launches of new video game platforms - PlayStation 3, Xbox One, Google Stadia

Zanneth• 2 years ago

Crowdstrike is run by humans just like you and me. One mistake doesn’t mean they are completely incompetent.

markus_zhang8• 2 years ago

CXOs usually have deep connection and great contracts (golden parachutes, etc.) that make them extremely difficult to fire and amiable to hire :)

duped• 2 years ago

He founded the company

SkyPuncher4• 2 years ago

Eh. To be fair, the higher profile your job is, the more likely you'll be the face of one of these in your career.

OsrsNeedsf2P• 2 years ago

Ok but he faced two

anonymous88884• 2 years ago

fool me once...

mvkel• 2 years ago

If anything, this just shows how short-term our memory is. I imagine crowdstrike stock will be back to where it was by the end of next week.

I bet they don't even lose a meaningful amount of customers. Switching costs are too high.

A real shame, and a good reminder that we don't own the things we think we own.

akira25016• 2 years ago

this just shows how short-term our memory is.

I've been out of IT proper for a while, so to me, I had to ask "the Russiagate guys are selling AV software now?"

_heimdall6• 2 years ago

I don't partake in the stock market these days, but this is the kind of event that you can make good money betting the price will come back up.

When a company makes major headlines for bad news like this investors almost always over react and drive the price too far down.

omeid2• 2 years ago

With a P/E of over 573? Doubt it will recover that fast.

pseudopersonal• 2 years ago

Yet another successful loser who somehow continues to ascend corporate ranks despite poor company performance. Just shows how disconnected job performance is from C-suite peer reviews, a glorified popularity contest. Should add the unity and better.com folk here

BobbyTables2• 2 years ago

I didn’t understand why in 2010, it didn’t seem to make most news…

Took out the entire company where I worked.

People thought it was a worm/virus — few minutes after plugging in laptop, McAfee got the DAT update, quarantined the file; which caused Windows to start countdown+reboot (leading to endless BSODs).

belter4• 2 years ago

It's not the first time they pull something similar...1 month ago: "CrowdStrike bug maxes out 100% of CPU, requires Windows reboots" - https://www.thestack.technology/crowdstrike-bug-maxes-out-10...

75 Billion dollars valuation, CNBC Analysts praising the company this morning on how well the company is run!...When in reality they can't master the most basic of the phased deployment methodologies known for 20 years...

Hundreds of handsomely paid CTO's, at companies with billions of dollars in valuations, critical healthcare, airlines, who can't master the most basic of the concepts of "Everything fails all the time"...

This whole industry is depressing....

monkmartinez5• 2 years ago

This borked our dispatch/911 call center then as well. However, it wasn't as bad as this one. This outage put our entire public safety system into the stone age and with that we were at stone age efficiency.

BaldricksGhost2• 2 years ago

I work IT at a regional 911 center. We're fine but I sympathize with those who are back to pen and paper dispatching. Hard for most current dispatchers to realize the way we did it back in the day.

chronid3• 2 years ago

What I find definitely depressing is the fact we used to roll out progressively even OS upgrades (I guess now that is done through intune?) and was one point in favor of windows (on Linux you had to do things yourself at the time AFAIK, I don't think the situation has improved much).

Nowadays we get mandated random software upgrading at once on the entire company fleet and no one bats an eye - I counted more than a dozen agents installed for "security" and "monitoring" purposes in my previous company servers, many of those with hooks in the kernel obviously, and many of those installed with random policies to tick yet another compliance box...

consp• 2 years ago

(on Linux you had to do things yourself at the time AFAIK, I don't think the situation has improved much)

You can schedule the updates any time you want, want to do it staggered then configure that, want to do it all at the same time then do that, want it with a random interval also possible. I don't see the "you need to do everything yourself" option as much as any managed environment.

segasaturn• 2 years ago

Those "CNBC analysts" truly know nothing, especially when it comes to tech. They're just cheerleaders who repeat talking points all days.

nothercastle5• 2 years ago

The worst part is that nobody will be held accountable. A F up like this should wipe out the entire company but instead everyone will just shrug it off as an opposie a few low level employees will get punished and nothing will change.

alexose• 2 years ago

This whole industry is depressing....

I'll take it a step further and say that every industry is depressing when it comes to computers at scale.

Rather than build efficient, robust, fault-tolerant, deterministic systems built for correctness, we somehow manage to do the exact opposite. We have zettabytes and exaflops at our fingertips, and yet, we somehow keep making things slower. Our user interfaces are noisier than ever, and our helpdesks are less helpful than they used to be.

mihaaly5• 2 years ago

Exactly! :(

I am drifting towards hating to turn on my computer in the morning. The whole day is like pissing into the wind, trying to find workaround of annoyances or even malfunctions, getting rid of obstructive noise from all direction, my productivity using modern computer systems is diminishing compared to where it was just mere 10-15 years ago (still better than 25 years ago not only becuase of experience but also the access of information on demand). Very depressing. I should have became a farmer perhaps.

nu11ptr• 2 years ago

This whole thing likely would have been averted had microkernel architectures caught on during the early days (with all drivers in user mode). Performance would have likely been a non-issue, not only due to the state of the art L4 designs that came later, but mostly because had it been adopted everything in the industry would have evolved with it (async I/O more prevalent, batched syscalls, etc.).

I will admit we've done pretty well with kernel drivers (and better than I would have ever expected tbh), but given our new security focused environment it seems like now is the time to start pivoting again. The trade offs are worth it IMO.

lizknope8• 2 years ago

Not disagreeing with you but we need operating systems with snapshots before updates and a trivial way to rollback the update.

Linux has some immutable OS versions and also btrfs snapshots and booting a specific snapshot from the GRUB bootloader

crote• 2 years ago

I wonder if for critical applications we'll ever go back to just PXE booting images from a central server: just load a barebones kernel and the app you want to run into a dedicated memory segment, mark everything else as NX, and you don't even have to worry about things like viruses and hacks anymore. Run into an issue? Just reboot!

chinathrow• 2 years ago

Yet their stock tanked only a couple of dollars. They (and their customers) should face some rather unpleasant lawsuits. If you let others own your systems, you should not be allowed to provide critical infrastructure.

rabidonrails2• 2 years ago

I don't think you're going to see as many lawsuits are you think. Most of these contracts probably state that they had to follow reasonable precautions for business continuity and data recovery. Having Crowdstrike in the path seems to have been a reasonable and potentially best practice before today's outage.

I don't think that companies are going to be held liable at all.

akdev1l2• 2 years ago

Crowdstrike should be held liable and sued out of existence.

fsloth11• 2 years ago

I do hope some organizations realize though it’s not a great idea to have a half-baked rootkit as your lucky charm against cybercrime.

choilive• 2 years ago

Once they are taken to court and all their crap gets subpoena'd I think we might find that reasonable precautions were not taken.

Its possible that this update was never properly QA'd and was just rushed out the door. If thats the case then it could be found to be negligence, and no amount of legal jargon protects you from negligence. It could be the end of CrowdStrike. /end fud.

layer87• 2 years ago

They didn’t force anyone to use their software in critical infrastructure. The customers deploying the software as part of critical infrastructure should take the necessary precautions or insist on a contractual agreement that makes the vendor liable for any causally related failures of the critical infrastructure. The mistake is that so much software is being put into use without any substantial liability. Doing so would also make software much, much more expensive.

paulmd2• 2 years ago

unqualified people on the internet shouldn't give legal advice, but since we're doing it anyway: no, this is definitely not true and if you make assurances about the fitness of the good you are on the hook when it fails, even if it's some absurdly improbable "I had no way of knowing that our pencils intended for schoolchildren would be used on a spacecraft" situation.

There is a reason you will see a ton of warranties and terms of service/EULA specifically forbid or disclaim the use in life-critical situations, in which case you are safe, because you said don't do it. But if you don't, you generally are going to be liable.

Sadly there is a reason the chainsaws say "do not stop chain with genitals". Not only did someone probably do that, but the damages stuck.

example, I was talking about the CUDA license yesterday and of course one of the clauses is:

You acknowledge that the SDK as delivered is not tested or certified by NVIDIA for use in connection with the design, construction, maintenance, and/or operation of any system where the use or failure of such system could result in a situation that threatens the safety of human life or results in catastrophic damages (each, a “Critical Application”). Examples of Critical Applications include use in avionics, navigation, autonomous vehicle applications, ai solutions for automotive products, military, medical, life support or other life critical applications. NVIDIA shall not be liable to you or any third party, in whole or in part, for any claims or damages arising from such uses. You are solely responsible for ensuring that any product or service developed with the SDK as a whole includes sufficient features to comply with all applicable legal and regulatory standards and requirements.

Why is this here? because they'd be liable otherwise, and more generally they want to be on the record as saying "hey idiot don't use this in a life-critical system".

There might well be a clause like that in crowdstrike's license too, of course. But the problem is it's generally different when what you are providing is a mission-critical safety/security system... hard to duck responsibility for being in critical places when you are actively trying to position yourself in critical places.

umanwizard8• 2 years ago

Their stock is down 13% right now... pretty huge for an intra-day drop.

Bluestein7• 2 years ago

I mean, seriously: You can cause a worldwide outage of gargantuan proportions, affecting actual human lives and untold points off GDP in several countries ...

... and the market gives you the equivalent of a wrist slap? No lawsuits?

What is it down to? Anonymous hit markets? Where is justice going to be served here?

PS. Most outlets are reporting "a fix has been issued" - as in "Whoopsy. No biggie" ...

... I mean, who's going to make affected (still alive!) people whole?

Ringz• 2 years ago

By chance, I watched a few episodes of 911 and kept thinking that it was all completely unrealistic nonsense. Then there's an episode where the entire emergency call system for LA goes down, and even though there were different reasons in the episode (a transformer fire), I couldn't have imagined that it was actually possible to completely disable the emergency call system (and what else) of a city.

qwerty456127• 2 years ago

WTF is CrowdStrike and why is it affecting so many people and companies? I've never heard of it before. And apparently it isn't anything relevant to all Windows users as it didn't affect any computer of any person I personally know.

tonymet6• 2 years ago

Very popular corporate endpoint protection (malware detection and spyware) that runs telemetry & monitoring agents installed as kernel-mode drivers on windows. Thus if there is a crash, it crashes the entire kernel (BSOD) . And their drivers load at boot.

nothercastle5• 2 years ago

I love how their company name foreshadows this exact event. It’s malware pretending to be a security suite.

satisfice2• 2 years ago

I want to say the problem is that the industry has systematically devalued software testing in favor of continuous delivery and the strategy of hoping that any problems are easy to roll back.

But it's deeper than that: the industry realizes that, once you get to a certain size, no one can hurt you much. Crowdstrike will not pay a lasting penalty for what has just happen, which means executives will shrug and treat this as a random bolt of lightning.

etc-hosts2• 2 years ago

Mission critical systems should be running something like ChromeOS.

Too bad ChromeOS seems be on the way out at Google.

01100011• 2 years ago

This might be a good time for folks to go back and watch the first episode of James Burke's Connections: The Trigger Effect

https://www.youtube.com/watch?v=NcOb3Dilzjc

Interconnected systems can fail spectacularly in unforeseen ways. Strange that something so obvious is so often dismissed or overlooked.

rincebrain3• 2 years ago

If you realize something horrific, your options are to decide it's not your problem (and feel guilty when it blows up), carefully forget you learned it, or try to do something to get it changed.

Since the last of these involves overcoming everyone else's shared distress in admitting the emperor has no clothes, and the first of these involves a lot of distress for you personally, a lot of people opt for option B.

HPsquared• 2 years ago

Option C is to quietly distance yourself from it.

__MatrixMan__21• 2 years ago

overcoming everyone else's shared distress in admitting the emperor has no clothes

I don't disagree, but why do we do we react this way? Doesn't knowing the emperor has no clothes instill a bit of hope that things can change? I feel for the people who were impacted by this, but I'm also a little bit excited. Like... NOW can we fix it? Please?

yawnxyz• 2 years ago

there are way too many horrific things in the world to learn about... and then realizing you can't do something about every of those things. But at least you can tackle one of them! (In my case, antibiotic resistance)

fsloth11• 2 years ago

Crowdstrike though is not part of a system of engineered design.

It’s a half-baked rootkit sold as a figleaf for incompetent it managers so they can implement ”best practices” in their companys PC:s.

The people purchasing it don’t actually know what it does, they just know it’s something they can invest their cybersecurity budget into and have an easy way to fullfill their ”implement cybersecurity” kpi:s without needing to do anything themselves.

lawlessone3• 2 years ago

they just know it’s something they can invest their cybersecurity budget into and have an easy way to fullfill their ”implement cybersecurity” kpi:s

To be fair. This is what companies like Crowdstrike are selling to these managers. Emphasis on the world "SELLING"

bryant2• 2 years ago

This is an interesting response. I'm curious why you specifically believe "it’s a half-baked rootkit sold as a figleaf for incompetent it managers."

jacurtis2• 2 years ago

That is how all of these tools are. I have always told people that third-party virus scanners are just viruses that we are ok with having. They slow down our computers, reduce our security, many of them have keyloggers in them (to detect other keyloggers). We just trust them more than we trust unknown ones so we give it over to them.

CloudStrike is a little broader of course. But yeah, its a rootkit that we trust to protect us from other rootkits. Its like fighting fire with fire.

ilya_m• 2 years ago

have an easy way to fullfill their ”implement cybersecurity”

There's a typo in there. "Do cyber" is how the said managers would phrase it.

thelittleone• 2 years ago

Perhaps true in some cases but in regulated insustries (example fed regulated banks) a tool like crowdstrike addresses several controls that if uncontrolled result in regulatory fines. Regulated companies rarely employ home grown tools due to maintainance risk. But now as we see these rootkit or even agent based security tools bring their own risks.

rozap2• 2 years ago

I think a lot about software that is fundamentally flawed but gets propelled up in value due to great sales and marketing. It makes me question the industry.

It's interesting that this is being referred to as a black swan event in the markets. If you look at the SolarWinds fiasco from a few years ago, there are some differences, but it boils down to problems with shitty software having too many privileges being deployed all over the place. It's a weak mono culture and eventually a plague will cause devastation. I think a screw up for these sorts of software models shouldn't really be thought of as a black swan event, but instead an inevitability.

isaacremuant2• 2 years ago

Very well put. Compliance over actual ops or security.

Bluestein7• 2 years ago

Again, as I feel I must every time Connections is mentioned: That series is an unmitigated masterpiece. It really is ...

cchance• 2 years ago

My issue is WTF do sooooooo many companies trust this 1 fucking company lol, like its always some obscure company that every major corporation is trusting lol. All because crowdstrike apparently throws good parties for C-Level execs lol

keeganpoppen• 2 years ago

Nassim Taleb is having a good day

eviks3• 2 years ago

Except this is not that, good and known practices exist against these kind of fails, and are used by others.

coderinsan• 2 years ago

CrowdStrike today has shown why it's absolutely crucial to test code before deployment, say no to YOLO deployments with LLM powered software testing https://github.com/codeintegrity-ai/mutahunter

LrnByTeach3• 2 years ago

what many people of not taking is that why we are here:

one simple reason: all eggs in one Microsoft PC basket

why in one Microsoft PC basket?

- most corporate desktop apps are developed for Windows ONLY

Why most corporate desktop apps are developed for Windows ONLY?

- it is cheaper to develop and distribute since, 90% of corporations use Windows PCs ( Chicken and Egg problem)

- alternate Mac Laptops are 3x more expensive, so corporations can't afford

- there are no robust industrial grade Linux laptops from PC vendors (lack of support, fear of Microsoft may penalize for promoting Linux laptops etc.)

1/ Most large corporations (Airlines, Hospitals etc..) can AFFORD & DEMAND their Software vendors to provide their ' business desktop applications' both in Windows and Linux versions and install mix of both Operating systems.

2/ majority of corporate desktop applications can be Web applications (Browser based) removing the single vendor Microsoft Windows PC/Laptops

ahachete• 2 years ago

Lenovo and Dell have some laptops with Linux, and they are very good ones.

(not sure if you meant rugged ones, that may not be the case, but I guess this is a tiny percentage of the market)

linuxlizard• 2 years ago

Crowdstrike also has an Ubuntu Linux version. We're required to install it at work.

trhway• 2 years ago

SolarWinds story was quickly forgotten, and this one will be too, and we'll continue to build such special single points of global catastrophic failure into our craftly architected decentralized highly robust horizontally scaled multi-datacenter-region systems

rtev• 2 years ago

Solarwinds is still dealing with the reputation damage and fallout today from that breach. People don’t forget about this stuff. the lawsuits will likely be hitting crowdstrike for years to come

TeaBrain4• 2 years ago

The SolarWinds story wasn't forgotten. Late last year the SEC launched a complaint against SolarWinds and its CISO. It was only yesterday that many of the SEC's claims against the CISO were dismissed.

ttyprintk2• 2 years ago

No less than three baskets, or you cannot apply for bailouts. If you want to argue your industry is a load-bearing element in the economy: no less than three baskets.

watermelon0• 2 years ago

Windows is not the issue here. If all of the businesses used Linux, a similar software product, deployed as widely as Crowdstrike, with auto-update, could result in the same issue.

Same goes for the OS; if let's say majority of businesses used RHEL with auto updates, RedHat could in theory push an update, that would result bring down all machines.

antihero2• 2 years ago

I would posit that RedHat have a slightly longer and more proven track record than Crowdstrike, and more transparent process with how they release updates.

No entity is infallible but letting one closed source opaque corporation have the keys to break everything isn’t resilient.

phkahler• 2 years ago

> Windows is not the issue here.

Yes it is. Windows was created for the "Personal Computer" with zero thought initially put in to security. It has been fighting that heritage for 30 years. The reason Crowdstrike exists at all is due to shortcomings (real or perceived) in Windows security.

Unix (and hence Linux and MacOS) was designed as a multi-user system from the start, so access controls and permissions were there from the start. It may have been a flawed security model and has been updated over time, but at least it started some notion of security. These ideas had already expanded to networks before Microsoft ever heard the word Netscape.

LrnByTeach3• 2 years ago

yes, staggered software update is the way to go. there was reply in this thread why Crowdstrike did not do it -- don't want extra cost of Engineering for that

having 1/3 of Airlines computers Windows, RHEL, Ubuntu .. all unlikely to hit same problems at same time.

doubled1122• 2 years ago

https://access.redhat.com/solutions/7068083

Just last month there were issues between RHEL's kernel update and Crowdstrike.

fulafel2• 2 years ago

That's assuming in this alternate universe we'd also be using kernel antivirus software to counter malware. It's far from obvious.

aflag• 2 years ago

But then it'd be putting all eggs in the Linux pc basket, wouldn't it? I think they point was that more heterogeneity would make this not be a problem. If all your potatoes are the same potato it only takes one bad blight epidemic to kill off all farmed potatoes in a country. If there's more heterogeneity things like that doesn't happen.

kjellsbells• 2 years ago

Agree. The monoculture simply accelerates the infection because there are no sizable natural barriers to stop it.

Windows and even Intel must take some blame, because in this day and age of vPro on the board and rollbacks built into the OS it's incredible that there is no "last known good" procedure to boot into the most recent successfully booted environment (didnt NT have this 30 years ago?), or remotely recover the system. I pity the IT staff that are going to have to talk Bob in Accounting through bitlocker and some sys file, times 1000s.

IT get some blame, because this notion that an update from a third party can reach past the logical gatekeeping function that IT provides, directly into their estate, and change things, is unconscionable. Why dont the PCs update from a local mirror that IT has that has been through canary testing? Do we trust vendors that much now?

Poor Crowdstrike. This might be the end for them.

Natsu3• 2 years ago

More secure OSes would consider an application being able to take down the entire OS as a security issue and would make that impossible.

lanstin4• 2 years ago

The difference being that RHEL has a QA process which crowd strike apparently does not. The quality practices for open source involved companies is apparently much higher than for large closed source "security" firms.

I guess getting whined at because obscure things break in beta or rc releases has a good effect for the people using LTS.

Ajedi32• 2 years ago

Maybe this is pie-in-the-sky thinking, but if all the businesses used some sort of desktop variant of Android, the Crowdstrike app (to the extent that such a thing would even be necessary in the first place) would be sandboxed and wouldn't have the necessary permissions to bring down the whole operating system.

__MatrixMan__21• 2 years ago

Making everything browser based doesn't help (unless you can walk across the room and touch the server). The web is all about creating fast-acting local dependency on the actions of far-away people who are not known or necessarily trusted by the user. Like crowdstrike, it's about remote control, and it's exactly that kind of dependency that caused this problem.

I love piling on Microsoft as much as the next guy, but this is bigger than that. It's a structural problem with how we (fail to) manage trust.

rewgs5• 2 years ago

The most concerning thing about this is the realization of just how many incredibly critical systems run on Windows.

apantel• 2 years ago

This just in ‘CrowdStrike Strikes Crowd’

insane_dreamer• 2 years ago

How is it that these major companies aren't rolling out vendor updates to a small number of computers first to make sure that nothing broke, and then rolling out to the entire fleet? That's deployment 101.

rustcleaner3• 2 years ago

Thank Chronos I switched to Qubes OS almost two years ago!

siliconc0w• 2 years ago

The postmortem will should interesting, can't imagine how even just basic integration testing didn't catch this. Much less basic best practice like canarying.

low_tech_punk• 2 years ago

Crowdstruck

m3kw9• 2 years ago

It’s that easy. A hacker that controls the update terminal at crowdstrike controls the world?

cjbgkagh13• 2 years ago

Edit; it appears my comment has been moved to a top level comment, i.e. peer with the parent without any way of telling what happened - so now there is the whole other pointless branch polluting the relevance of the tree.

Previously;

It appears that someone was able to take my previous comment in this thread completely off hacker news, it's not even listed as flagged. It was at 40pts before disappearing, perhaps there is some reputation management going on here. If it was against the site rules it would be helpful to know which ones.

Edit; the link is https://news.ycombinator.com/item?id=41007985 it was a high up comment that no longer appears even though flagged comments do appear. I checked if it has been moved but the parent comment is still the same. This feels like hellbanned in that there isn't an easy way for me to see if I've been shadowbanned. But I really don't know. I was commenting in good faith.

nine_k• 2 years ago

Oh, nevermind, it's just a third-party cybersecurity tool running on the server detected a potential threat and quarantined the offending database record, just in case!

cjbgkagh13• 2 years ago

I was looking forward to spending the day talking to people about cyber security but if my comments are going to disappear like that then maybe hacker news is not the site for me. A shame really.

Edit; I don't know for sure but this is possibly the last straw for me on hacker news. It really has gone downhill. If good faith discussions from experts are being secretly deleted for what I can only now assume are for nefarious reasons then I can't trust what I find here is in anyway representative. It's unfortunate in that there really isn't anywhere else to go. Now my best discussions are in small WhatsApp groups / Discords with friends. It's ok for me where I have had a career to get to know people personally and have such groups but if public forums are tainted in this way then younger people in this field will end up only talking to each other.

MyFirstSass• 2 years ago

what was the comment about?

cjbgkagh13• 2 years ago

I edited my post to include the link, I guess if people see these posts I'm not completely banned and maybe there isn't anything nefarious going on, they have a situation where the comments to a single comment are already off the first page so there could be a software bug. But it does appear that there is some sort of reputation management going on.

secondcoming2• 2 years ago

I can see you lamenting the loss of your internet points. Chin up

cjbgkagh13• 2 years ago

I still keep the points, so perhaps a loss of possible future points. The points don't bother me, I restart antonymous accounts on occasion. I kind of use points to get an idea of where other peoples opinions which is half the reason I use this site. A negative signal to something I think is good is actually more interesting than a positive signal. I'm more worried about the damage such actions do to the 'market place of ideas' and if it hasn't pushed me away then perhaps it has pushed others that I'm interested to hear from. And if so where have they gone. Once I become disinterested in the opinions of others on this site then it's unlikely I'll have any further use for hacker news - and I'm getting pretty close to that point.

dang3• 2 years ago

(I've detached this offtopic subthread from https://news.ycombinator.com/item?id=41002977.)

Your original comment was https://news.ycombinator.com/item?id=41007985. I detached it from its original parent (https://news.ycombinator.com/item?id=41002977) because it was more of a generic tangent than a specific reply.

It's a vital moderation function to do this, particularly when the parent is the top comment of the entire thread. Those tend to attract non-reply-replies, and that has bad effects on the thread as a whole. It causes the top part of the page to fill up with generic rather than specific content, and it makes the top subthread too top-heavy.

I'm not saying that you did anything wrong or that your post was bad or that it was unrelated to the original parent. The problem is that the effects I'm describing pile up unintentionally and end up being a systemic problem. It isn't anybody's fault, but there does need to be someone whose job it is to watch out for the system as a whole, and that's basically what moderators do.

Sometimes we comment that we detached a post from its original parent (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...) and sometimes not. (Perhaps the software should display this information automatically.) I'm less likely to do it when a comment stands on its own just fine, which was the case with your post https://news.ycombinator.com/item?id=41007985 and which is usually the case with the more generic sort of reply—in fact it's one test for deciding that question.

so now there is the whole other pointless branch polluting the relevance of the tree

Yes, please don't do that—especially in the top subthread. I understand the frustration of "WTF where did my comment go", but you can always reach us at hn@ycombinator.com and get an answer.

darkhorn• 2 years ago

Why they don't use Windows' own anti-virus?

Nemo_bis4• 2 years ago

That's called [Microsoft Defender for Endpoint](https://learn.microsoft.com/en-us/defender-endpoint/), which is used even on Linux servers in big corporations. (Largely because it's the easiest way to complete box ticking exercises with Windows servers: once you have it, it's easy to decide to extend it to non-Windows machines as well.)

The binary self-upgrades and runs in highly privileged mode, so it might not be immune from the kind of failure CrowdStrike had here. Though apparently there's at least a way to use a local mirror so you have some control on the updates: https://learn.microsoft.com/en-us/defender-endpoint/linux-su...

pfortuny• 2 years ago

This is the Irish potato famine (essentially due to the farming of a single species of potato) equivalent in IT infrastructure: a single vendor.

mindcrash• 2 years ago

This has all the hallmarks of a SSCA (Software Supply Chain Attack).

Either that or Crowdstrike is testing critical software meddling in ring zero so poorly, causing crashes and bootloops out in the wild on 100% of the deployments, that they need to get sued out of existence.

I hope for their sake its the former.

felipesabino• 2 years ago

Does anyone know how to proceed if I do not have administrator level access to the computer?

I do not have access to c:\windows\system32\drivers\crowdstrike folder to delete the corrupted .sys file

I was able to boot on recovery mode with network, after waiting 30 min, I rebooted and BSOD persisted.

Are there other alternatives on how to recover?

victor9000• 2 years ago

Create a bootable USB stick from Ubuntu, then see if you can mount the windows drive and delete the file that way.

einpoklum3• 2 years ago

If I weren't an atheist I would say this is god's punishment for installing malware on your employees' machines, on one hand, and for being a spineless patsy for management by letting them install that crap on your work machine.

jonplackett2• 2 years ago

How do so many super critical things rely on… windows? I wouldn’t trust windows to run a laptop reliably but here it is running prettty ucy everything. I guess that’s why they need crowdstrike.

wly_cdgr• 2 years ago

It's because Windows is a mature, professional-grade operating system developed and maintained by extremely competent people with a lot to lose.

oblio4• 2 years ago

Crowdstrike can run on Linux and some companies mandate it, with similar issues...

jonplackett2• 2 years ago

All the crazy people banging the drum for war with Russia and/or China.

Imagine what our IT systems would look like with someone _intentionally_ messing with them.

awahab92• 2 years ago

closed source will always fuck you in the ass

zx10rse• 2 years ago

There is hardly a better time if you write software to watch - "The Mess We're In" by Joe Armstrong - https://www.youtube.com/watch?v=lKXe3HUG2l4

I am not sure in which one of his talks he briefly mentioned that one of his concerns is that we are basically building a digital Alexandria library, and if it burns, well ...

Even more devastating events like this will happen in the future.

We stand on the shoulders of giants and yet we learned nothing.

outside1234• 2 years ago

Texas, where software goes to die. Or maybe that is where killer software is developed?

dboreham8• 2 years ago

Now things are serious: I can't place a mobile order at Starbucks.

kelembu• 2 years ago

Is there a way to estimate number of affected devices? 10 million? 100 million?

photonbeam2• 2 years ago

It should be obvious to everyone now that kernel extensions for ‘security’ is not worth it

wufufufu• 2 years ago

Anyone have a technical writeup of the actual bug? I'm trying to explain how this could happen to people who think this is related to AI or cyber attacks.

What happened to the QA testing, staggered rollouts, feature flags, etc.? It's really this easy to cause a boot loop?

To me, BSOD indicates kernel level errors, which I assume Crowdstrike would be able to cause because it has root access due to being a security application. And because it's boot-looping, there's not a way to automatically push out updates?

nrr8• 2 years ago

I don't have a technical writeup to offer, but your assessment around the BSOD seems correct enough. Without having an affected machine but knowing how NT loads drivers like this, I'd hazard a guess that the OS likely isn't even getting to the point where smss.exe starts before the kernel bugchecks. This means no userspace, which almost certainly means no hope of remotely remediating the problem.

_hcuq2• 2 years ago

Ok... Would a Linux based infrastructure be more resilient.

Does Linux require Crowdstrike style AV software?

tryauuum• 2 years ago

technically no os requires such software, it's people who make a decision to run it

Someone in comments described crowd strike bringing down their fleet of Linux servers in April

rtkwe• 2 years ago

I'd hate to be on Microsoft's teams today. They're catching a lot of stray blame for this in the public eye where it's entirely not their fault.

sytelus3• 2 years ago

CloudStrike had managed to invade into StarBucks IT. All of the online order taking systems are down.

exabrial• 2 years ago

Why are people still using Windows?

sagebird• 2 years ago

It is interesting that operating systems exist for server applications at all.

What is the problem they are solving?

What is the difference between what an operating system contains and can do and what you need it to do?

Why would I want to rent a server to run a program that performs a task, and also have the same system performing extra tasks - like intrusion detection, intrusion detection software updates, etc.

I just don't understand why compiled program that has enough disk and memory would ever be asked to restart for a random fucking reason having nothing to do with the task at hand. It seems like the architecture of server software is not created intelligently.

banku_brougham• 2 years ago

The IT security chief at my co (paraphrasing):

talked to pres of Crowdstrike. His forthrightnes was refreshing. He said “We got it wrong.”
They are working with Microsoft to understand why this happened.

Pretty much the message minus even more boilerplate talk.

wojo1206• 2 years ago

Why Crowdstrike doesn't follow standard deployment strategies such as canary or rolling? Gradual update would uncover this bug before reaching critical mass. Doing all-at-once update is unacceptable to critical systems.

downrightmike4• 2 years ago

This is what happens when you treat IT as a cost center.

nineteen999• 2 years ago

Speaking as somebody who manages a large piece of a 911 style system for first responders and has done so for 10 years (and is not affected by this outage) - this is why we do not allow third parties to push live updates to our systems.

It's unfortunate, the ambulances are still running in our area of responsibility, but it's highly likely that the hospitals they are delivering patients to are in absolute chaos.

ijidak• 2 years ago

How do you patch software causing a BSOD?

It seems like a chicken and egg problem.

I ran a team that developed a remote agent, and this was my nightmare scenario.

grumpyprole3• 2 years ago

Hopefully now people might wake up to the idea that these tech monopolies are not leading to safe, secure and reliable systems. They will wonder how a third party component could cause such breakage. I expect many will be calling for regulation.

badgersnake• 2 years ago

I don’t really understand why AV updates aren’t tested before being pushed out to critical systems and I don’t understand why every system would run the same AV.

But also I don’t understand why this corporate garbageware is still a thing in 2024 when it adds so little value.

xyst4• 2 years ago

why the fuck is our critical infrastructure running on WINDOWS. Fuck the sad state of IT. CIOs and CTOs across the board need to be fired and held accountable for their shitty decisions in these industries.

yes CRWD is a shitty company but seems they are a "necessity" by some stupid audit/regulatory board that oversees these industries. But at the end of the day, these CIOs/CTOs are completely fucking clueless as to the exact functions this software does on a regular basis. A few minions might raise an issue but they stupidly ignore them because "rEgUlAtOrY aUdIt rEqUiReS iT!1!"

lnrd• 2 years ago

The OS doesn't matter, the question should be why is critical infrastructure online and allowed to receive OTA updates from third parties.

teej• 2 years ago

The OS absolutely matters

aylons• 2 years ago

It matters as in it makes it easy for this kind of issue to cause this much damage with little to no recourse for a fast correction.

Not that Linux or whatever are all immune, but it definitely matters.

bg24• 2 years ago

With so much of tooling and products, it should come down to

- What am I running and their current security state

- Supply chain of any change that's happening

- Test/Stage/Rollout any change - do not trust the vendor, as they do not know your infrastructure

By allowing OTA update, they assumed that the vendor has tested all permutations.

cpill3• 2 years ago

Of course the OS matters! Windows is a nasty ball of patches in order to maintain backward compatibility with the 80s. Linux and OSX don't have to maintain all the nasty hacks to keep this backward compatibility.

Also, Crowdstrike is a security (patch) company because Windows security sucks to the point they have, by default, real-time virus protection running constantly (runs my CPU white hot for half the day, can you imagine the global impact on the environment?!).

It's so bad on security that its given birth to a whole industry to fix it i.e. Crowdstrike. Every time I pass a bluescreen in a train station or advertisement I'd like "hA! you deserve that for choosing Windows".

tivert3• 2 years ago

The OS doesn't matter, the question should be why is critical infrastructure online and allowed to receive OTA updates from third parties.

Not exactly. I think the question is why is critical infrastructure getting OTA updates from third parties automatically deployed directly to PROD without any testing.

These updates need to go to a staging environment first, get vetted, and only then go to PROD. Another upside of that it won't go to PROD everywhere all at once, resulting in such a worldwide shitshow.

lambda• 2 years ago

While Linux isn't a panacea, the OS does matter as Linux provides tools for security scanners like Crowdstrike to operate entirely in userspace, with just a sandboxed eBPF program performing the filtering and blocking within the kernel. And yes, CrowdStrike supports this mode of operation, which I'll be advocating we switch over to on Monday. So yeah, for this specific issue, Linux provides a specific feature that would have prevented this issue.

killerstorm2• 2 years ago

OS absolutely does matter. Windows has an enormous attack surface because Microsoft doesn't care.

There's a number of minimal operating systems without all bells and whistles. The reason they aren't as popular choice is the "OS doesn't matter".

If OS is minimal it doesn't need OTA updates, let alone from a third party...

mvdtnz• 2 years ago

Windows is no more or less vulnerable to this class of issues than any other OS.

evilmwnci• 2 years ago

Debatable. macOS did away with third-party kernel extensions. On Windows, CS runs in the kernel, and the kernel can't load properly because of CS.

fortran775• 2 years ago

Windows isn’t Crowdstrike.

cpill3• 2 years ago

No, its just soooooo bad at security/stability that it gave birth to Crowdstrike. They very fact that Crowdstrike is so big and prevalent means is proof of the gapping hole in Windows security. Its given birth to a multibillion dollar industry!

afavour7• 2 years ago

why the fuck is our critical infrastructure running on WINDOWS

Because it’s cheaper.

I feel like many in this thread are obsessing over the choice of OS when the actual core question is why, given the insane money we spend on healthcare, are all healthcare systems shitty and underinvested?

A sensible, well constructed system would have fallbacks, no matter if the OS of choice is Windows or Linux.

rbanffy16• 2 years ago

The difference is that lots of different companies can share the burden of implementing all that in Linux (or BSD, or anything else) while only Microsoft can implement that functionality in Windows and even their resources are limited.

freeopinion• 2 years ago

Cheaper? Well, perhaps when you require your OS to have some sort of support contract. And your support vendor charges you unhealthy sums.

And then you get to see the value of the millions of dollars you've paid for support contracts that don't protect your systems at all. But those contracts do protect specific employees. When the sky falls down, the big money execs don't have a solution. But it's not their fault because the support experts they pay huge sums don't have solutions either. Somehow paying millions of dollars to support contractors that can't save you is not seen as a fireable offense. Instead it is a career-saving scapegoat.

Within companies that have been bitten this time, the team that wasn't affected because they made better process decisions will not be promoted as smarter. Their voice will continue to be marginalized by the people whose decisions led to this disaster. Because, hey, look, everyone got bit right? Nobody looks around to notice the people who were not bitten and recognize their better choices. And "I told you so" is a pretty bad look right now.

VHRanger• 2 years ago

I feel like many in this thread are obsessing over the choice of OS when the actual core question is why, given the insane money we spend on healthcare, are all healthcare systems shitty and underinvested?

Because it's basically impossible to compete in the space.

Epic is a pile or horseshit, but you try convincing a hospital to sign up to your better version.

ar_lan• 2 years ago

Odd that you choose Windows to swipe at when this was largely CRWD's problem + a mix of awful due diligence by IT departments.

marcosdumay3• 2 years ago

yes CRWD is a shitty company but seems they are a "necessity" by some stupid audit/regulatory board that oversees these industries.

Yep, this is the problem. The part about Windows is a distraction here.

That bullshit regulation is a much larger security issue than Windows. Incomparably so. If you run it over Linux, you'll get basically the same lack of security.

Saris2• 2 years ago

Didn't CRWD cause a similar issue with Debian/RHEL a little while ago?

It sounds to me that the problem lies with CRWD and not with whatever OS it's installed on.

rbanffy16• 2 years ago

A kernel driver can, definitely, take down a Linux machine.

The question is whether someone should implement something like this as a kernel module when there are better ways.

worik2• 2 years ago

why the fuck is our critical infrastructure running on WINDOWS.

That hits the nail on the head.

But it is a rhetorical question. We know why, generally, software sacks, and specifically why Windows is the worst and is the most popular

Good software is developed by pointy headed needs (like us) and successful software is marketed to business executives are have serious pathologies

There are exceptions (I am struggling to think of one) where a serious piece of good software has survived being mass marketed, but the constraints (basically business and science) conflict

luxuryballs• 2 years ago

More specifically why is critical stuff not equipped properly to revert itself and keep working and/or fail over? This should be built-in stuff at this point, have the last working OS snapshot on its own storage chip and automatically flash it back, even if it takes a physical switch… things like this just shouldn’t happen.

aenis9• 2 years ago

Nah, nope.

1/ linux is as vulnerable to kernel panics induced by such software. In fact, CS had a similar snafu mid April, affecting linux kernels. Luckily, there are far fewer moronic companies running CS on linux boxes at scale.

2/ it does offer protection - if you are running total shit architecture and you need to trust your endpoints not to be compromised, something like this is sadly a must.

Incidentally, google, which prides itself at running a zero-trust architecture, sent a lot of people home on Friday. Not so zero-trust after all, it seems.

Lots of armchair CIOs/CTOs in the comments today.

pertymcpert3• 2 years ago

Source on google sending home people?

heurist• 2 years ago

The primary answer to your question is because it's expensive to switch.

zjaffee• 2 years ago

Tons of critical infrastructure in the US is run on IBM zOS. It doesn't matter what operating system you use, what matters is updates aren't automatic and everything is as air gapped as possible.

wolfspaw• 2 years ago

(FORCE) Pusheedd to Prod on FRIDAYY -- Burneeeddd by its Sins

2OEH8eoCRo02• 2 years ago

They should be sued into bankruptcy

fargle• 2 years ago

such stupidity. our $$$ corporate geniuses mandate multiple so-called security software which is:

- unaccountable black boxes

- of questionable, and un-auditable, quality

- requires kernel modules, drivers, LocalSystem, root access, etc.

- updates at random times with no testing

- download these updates from where? and immediately trust and run that code at high privilege. using unaccountable-black-box crypto to secure it.

- all have known patterns of bad performance, bugs, and generally poor quality

all in the name of security. let's buy multiple "solutions" and widely deploy them to protect us from one boogeyman, or at least the shiny advertisements say. while punching all sorts of serious other holes in security. why even look for a Windows ZeroDay when we can look for a McAfee or Crowdstrike zero day?

mensetmanusman• 2 years ago

Stopped by a gas station in rural Wisconsin leaving from MSP. Thank God we were on a full tank when we left, nothing was operational except the bathrooms (which is why we stopped).

I left thinking about how anti-anti-fragile our systems have become. Maybe we should force cash operations…

accra4rx• 2 years ago

Never heard mainframe going down

mikewarot• 2 years ago

Why are we still running ANY operating systems based on Ambient Authority, as part of our infrastructure?

DoD shouldn't have given up on MULTICS. That premature optimization is going to sink the US and the Free World.

Personally, I'm still waiting for Genode to be my daily driver.

GabeIsko• 2 years ago

So - what is the lesson learned? The only clear message for me is that critical programs that also demand kernel level access maybe shouldn't update themselves.

jpgvm• 2 years ago

If you run RATs like these on your machines then I'm sorry, this is just a case of fucking around and finding out.

Just don't do it. Windows Defender is a thing, it does just fine. For everything else there is least-privilege and group policy.

watersb2• 2 years ago

I wonder how unknown (yet) malware will be wiped out or enabled before this is over.

ckemere• 2 years ago

It seems that an unexplored weirdness here is the prevalence of virtual Windows in the medical world. It seems that this has approach has become commonplace for HIPAA reasons (though it's unclear that it makes the world better versus using secure applications to handle HIPAA data). In the case of this Crowdstrike outage, one would think that virtual machines would simplify getting things up and running again, but instead there seems to be just the opposite going on, where lack of hardware access is limiting restoring them.

Any insight from those affected?

Group_B• 2 years ago

fun times...

smcleod• 2 years ago

I mean... installing what is essentially a 3rd party enterprise rootkit that not only has root access to all files and network activity but also a self-update mechanism ... who could have seen this coming?

gpderetta• 2 years ago

The machine stops.

tigerlily• 2 years ago

Meanwhile the linux desktop just keeps on truckin'.

Bluestein7• 2 years ago

I think we have reached and inflection point. I mean we have to make an inflection point out of this.-

This outage represents more than just a temporary disruption in service; it's a black swan célèbre of the perilous state of our current technological landscape. This incident must be seen as an inflection point, a moment where we collectively decide to no longer tolerate the erosion of craftsmanship, excellence, and accountability that I feel we've been seeing all over the place. All over critical places.-

Who are we to make this demand? Most likely technologists, managers, specialists, and concerned citizens with the expertise and insight to recognize the dangers inherent in our increasingly careless approach to ... many things, but, particularly technology. Who is to uphold the standards that ensure the safety, reliability, and integrity of the systems that underpin modern life? Government?

Historically, the call for accountability and excellence is not new. From Socrates to the industrial revolutions, humanity has periodically grappled with the balance between progress and prudence. People have seen - and complained about - life going to hell, downhill, fast, in a hand basket without brakes since at least Socrates.-

Yet, today’s technological failures have unprecedented potential for harm. The Crowdsource outage killed, halted businesses, and posed serious risks to safety—consequences that were almost unthinkable in previous eras. This isn't merely a technical failure; it’s a societal one, revealing a disregard for foundational principles of quality and responsibility. Craftsmanship. Care and pride in one's work.-

Part of the problem lies in the systemic undervaluation of excellence. In pursuit of speed and profit uber alles. Many companies have forsaken rigorous testing, comprehensive risk assessments, and robust security measures. The very basics of engineering discipline—redundancy, fault tolerance, and continuous improvement—are being sacrificed. This negligence is not just unprofessional; it’s dangerous. As this outage has shown, the repercussions are not confined to the digital realm but spill over into the physical world, affecting real lives. As it always has. But never before have the actions of so few "perennial interns" affected so many.-

This is a clarion call for all of us with the knowledge and passion to stand up and insist on change. Holding companies accountable, beginning with those directly responsible for the most recent failures.-

Yet, it must go beyond punitive measures. We need a cultural shift that re-emphasizes the value of craftsmanship in technology. Educational institutions, professional organizations, and regulatory bodies must collaborate to instill and enforce higher standards. Otherwise, lacking that, we must enforce them ourselves. Even if we only reach ourselves in that commitment.-

Perhaps we need more interdisciplinary dialogue. Technological excellence does not exist in a vacuum. It requires input from ethical philosophers, sociologists, legal experts. Anybody willing and able to think these things through.-

The ramifications of neglecting these responsibilities are clear and severe. The fallout from technological failures can be catastrophic, extending well beyond financial losses to endanger lives and societal stability. We must therefore approach our work with the gravity it deserves, understanding that excellence is not an optional extra but an essential quality sine qua non in certain fields.-

We really need to make this be an actual tuning point, and not just another Wikipedia page.-

ok123456• 2 years ago

Make a live CD Linux image that mounts the NTFS drives, locates the Windows directories from the bootloader, and deletes the file.

Also, you can mount BitLocker partitions from Linux iirc. If it encounters a BitLocker partition, have it read a text file of possible keys off the USB drive.

clarity20• 2 years ago

Has any thread had 3000 comments before?

Cyphase3• 2 years ago

This story is about to break into the top 15 of upvoted stories on HN, but it already seems safely within the top 10 by number of comments.

type04• 2 years ago

video summary from fireship: https://www.youtube.com/watch?v=4yDm6xNeYas

grigy• 2 years ago

I could not imagine so many critical systems run on Windows.

hughw2• 2 years ago

So, if CrowdStrike licenses didn't say "We're responsible for nothing" and if all affected users sued them, they'd be worth negative 90 trillion dollars or so right now. iow out of business.

I can understand the frustration their customers feel. But how could a software company ever bear liability for all the possible damage they can cause with their software? If they built CrowdStrike to space mission standards nobody could afford it.

JSDevOps• 2 years ago

Remember, there's someone out there right now, without irony, suggesting that AI can fix this. There's someone else scratching their head, wondering why AI hasn't fixed this yet. And there's someone doing a three-week bootcamp in AI, convinced that AI will fix this. I’m not sure which is worse

beepbooptheory• 2 years ago

The saddest one is definitely the bootcamp one.

1oooqooq13• 2 years ago

when even jsDevOpsv can see the king is naked....

sys32768• 2 years ago

SMB here. Just spent a nine hour day fixing this. We had two machines that after a couple of reboots just came back up fine.

We were trialing CrowdStrike and about to purchase next week. If their rep doesn't offer us at least half off, we are going with Sentinel One which was half the price of CS already.

The incompetence that allowed this is baffling to me. I assumed with their billions of dollars they'd have tiers of virtual systems to test updates with.

I remember this happening once with Sophos where it gobbled up Windows system files. If you had set to Delete instead of Quarantine, you were toast.

troad• 2 years ago

We were trialing CrowdStrike and about to purchase next week. If their rep doesn't offer us at least half off, we are going with Sentinel One which was half the price of CS already.

You’re still considering them after all this?

integricho3• 2 years ago

It is baffling to me that you are still considering them.

utkarsh858• 2 years ago

Adding a comment to make this the most commented piece on hackernews and hence highlight the bad impact a bug can make on lives founded on IT.

tammer2• 2 years ago

When I saw 'Global IT Outage' trending I assumed it was another major cloud service failure. Obviously this has far wider impact because of the need for intervention on individual endpoints.

The irony is dawning on me that for much of the recent computing era we've developed defenses against massive endpoint outages (worms, etc.) and one of them is now inadvertently reproducing the exact problem we had mostly eradicated.

unixhero3• 2 years ago

4 hour delay at the airport in Los Cabos. At least they have tacos!

SoftMachine• 2 years ago

Am I supposed to use some AI bot to summarize all this shit? Ain't no one got time to read 3000+ comments. Any good links?

hansvm• 2 years ago

Random strangers running unknown, untrusted code on your computers is the worst. It's a good thing we patched that security flaw by letting the _right_ random strangers run unknown, untrusted code on our computers.

As something of a friendly reminder, it was Microsoft this time, but it's a matter of "when" not "if" till every other OS with that flavor of security theatre is similarly afflicted (and it happens much more frequently when you consider the normal consequences of a company owning the device you paid for -- kicked out of email forever, ads intruding into basic system functions, paid-in-full device eventually requires a subscription, ...). Be cautious with automatic updates.

type04• 2 years ago

clownStrike does make cool commercials though

https://www.youtube.com/watch?v=8gatiYW5kWs

https://www.youtube.com/watch?v=p4OGQQPMiXQ

TexanFeller• 2 years ago

A heuristic that has served me well for years is that anyone who uses the word “cybersecurity” is likely incompetent and should be treated with suspicion.

My first encounter with CrowdStrike was overwhelmingly negative. I was wondering why for the last couple weeks my laptop slowed to a crawl for 1-4 hours on most days. In the process list I eventually found CrowdStrike using massive amounts of disk i/o, enough to double my compile times even with a nice SSD. Then they started installing it on servers in prod, I guess because our cloud bill wasn’t high enough.

Laforet• 2 years ago

Cyber- is pretty much a code prefix for anything targeted at the public sector. I too see it as a kind of dirty word TBH.

gukov2• 2 years ago

"Cyber," used on its own, is the worst of them all.

photochemsyn• 2 years ago

It rather looks like Crowdstrike marketed heavily to corporate executives using a horror story about the bad IT tech guy who would exfiltrate all their data if they didn't give Crowdstrike universal access at the kernel level to all their machines...? It seems more aimed at monitoring the employees of a corporation for insider threats than for defense against APT actors.

neverrroot• 2 years ago

The employees is a very important attack vector, we had multiple incidents, after they downloaded the wrong kind of stuff.

nbtm• 2 years ago

I heard all Windows PCs at the University of New South Wales were also boot-looping.

elchief• 2 years ago

maybe this was just an enormous distraction while Spetssvyaz did a bunch of fun stuff

zoom6628• 2 years ago

By way of a data point for everyone else I live in HongKong and haven't seen any of this level of disruption yet. I also was in Shenzhen China yesterday, probably the words highest density of Win95 machines, and everything was fine. At home we have only one old laptop on win10 that only gets opened when the 8yo gets windows homework - otherwise it's MacOs and Linux on all laptops, desktops and SBCs.

If I see some news I will update this comment.

godelmachine• 2 years ago

I wonder what happens to the engineer who deployed this patch.

jiggawatts2• 2 years ago

While initially everyone blamed Microsoft and then quickly pointed the finger at CrowdStrike, I'd like to call out Microsoft especially their Azure division for making the recovery process unnecessarily difficult.

1) A key recovery step requires a snapshot to be take of the disk. The Portal GUI is basically locking up, so scripting is the only way to do this for thousands of VMs. This command is undocumented and has random combinations of strings as inputs that should be enums. Tab-complete doesn't work! See: https://learn.microsoft.com/en-us/powershell/module/az.compu...

E.g.: What are the accepted values for the -CreateOption parameter? Who knows! Good luck using this in a hurry. No stress, just apply it to a production database server at 1 am in the morning.

2) There has been a long-standing bug where VMs can't have their OS disk swapped out unless the replacement disk matches its properties exactly. For comparison, VMware vSphere has no such restrictions.

3) It's basically impossible to get to the recovery consoles of VMs, especially VMs stuck in reboot loops. The serial console output is buggy, often filled with gibberish, and doesn't scroll back far enough to be useful. Boot diagnostics is an optional feature for "reasons". Etc..

4) It's absurdly difficult to get a flat list of all "down" VMs across many subscriptions or resource groups. Again, compare with VMware vSphere where this is trivial. Instead of a simple portal dashboard / view, you have to write this monstrous Resource Graph query:

    Resources
    | where type =~ 'microsoft.compute/virtualmachines'
    | project subscriptionId, resourceGroup, Id = tolower(id), PowerState = tostring( properties.extended.instanceView.powerState.code)
    | join kind=leftouter (
      HealthResources
      | where type =~ 'microsoft.resourcehealth/availabilitystatuses'
      | where tostring(properties.targetResourceType) =~ 'microsoft.compute/virtualmachines'
      | project targetResourceId = tolower(tostring(properties.targetResourceId)), AvailabilityState = tostring(properties.availabilityState))
      on $left.Id == $right.targetResourceId
    | project-away targetResourceId
    | where PowerState != 'PowerState/deallocated'
    | where AvailabilityState != 'Available'

vpshastry• 2 years ago

Don’t they have canary deployments? Such huge updates happen al at once?

cookiengineer• 2 years ago

I'm a little late to the party, but I've uploaded my source codes to GitHub in case anyone needs a more convenient tool to deploy/execute on running machines and/or needs something fast on USB flash drives to run around the office:

https://github.com/cookiengineer/fix-crowdstrike-bsod

Releases section contains prebuilt binaries, but of course, I always recommend to check the source and then build it yourself.

miffy900• 2 years ago

I'm sure you mean well, but it's not going to be most programmers or devs who will need to apply a fix for this, it'll be sysadmin/network/SREs who'll be doing this and they're not going to download Go to build this random github code repo. Because it affects only Windows systems, it'll be way better writing a bat or powershell script that can non-programmers can read and comprehend before they execute anything in production/live systems.

1oooqooq13• 2 years ago

Why aren't we upvoting a list of alternatives to CrowdStrike here?

dang3• 2 years ago

All: there are over 3000 comments in this thread. If you want to read them all, click More at the bottom of each page, or like this:

https://news.ycombinator.com/item?id=41002195&p=2

https://news.ycombinator.com/item?id=41002195&p=3

https://news.ycombinator.com/item?id=41002195&p=4 (...etc.)

uptownfunk• 2 years ago

The correct solution is to have IT force push updates only when they deem fit (after they have tested internally on some ghost machines).

rkagerer7• 2 years ago

This is why I don't like fully automatic updates. I prefer having control over the "deploy" button for the ability to time it when I can tolerate downtime. In mission-critical production systems all updates should go through test staging pipelines that my team controls, not a vendor.

Broken updates have cause far more havoc than being a few hours or even days late on a so-called critical patch.

TeeWEE• 2 years ago

Even if you deploy manually all at once: you have the same problem.

A solution is slow rollout. Not a manual deploy button

rkagerer7• 2 years ago

It's not the same problem at all.

Troubleshooting an issue like this when I have the time and am prepared for a potential outage (with human resources hot and standing by for immediate action) is VASTLY different than encountering it the evening before some critical deadline for a multi-million dollar project (as Murphy's Law will be sure to have it).

When I have control over the deployment of updates I can push them through my own QA environment first, to uncover many of these kinds of issues before they hit production. Vendors pushing them out on their whim leaves me subject to whatever fast and loose practice they use and prevents me from being able to properly manage my own infrastructure.

A slow rollout certainly helps but doesn't satisfy the kind of 9's I demand in the environments I care for.

TowerTall• 2 years ago

Microsoft to give Vista kernel access to security firms (2006)

https://arstechnica.com/information-technology/2006/10/7998/

256_• 2 years ago

I used to laugh at Dijkstra's idea that all code should be mathematically proven correct. I thought of it as a laughable idea from yet another out-of-touch mathematician.

I suppose true genius is seldom understood within someone's lifetime.

belter4• 2 years ago

I used to laugh at Dijkstra's idea that all code should be mathematically proven correct.

That would not help for this outage.

beardyw• 2 years ago

Microsoft to give Vista kernel access to security firms (2006) | Hacker News

https://news.ycombinator.com/item?id=41014426

cranberryturkey• 2 years ago

Someone accidentally shut down the planet with a code push -- rofl

kwhitefoot• 2 years ago

Can someone explain to me why such systems need anti-virus in the first place?

Windows has pretty good facilities for locking down the system so that ordinary users, even those with local admin rights, cannot run or install unauthorised code so if nothing can get in why would the system need checking for viruses?

So why do most companies not lock down their machines?

1oooqooq13• 2 years ago

easier to show a paid bill than to show true due diligence to your insurance when you're hit with ransomware.

that's the whole CS business model.

tomthumb• 2 years ago

Someone on X has shared the kernel stack trace of the crash

The faulting driver in the stack trace was csagent.sys.

Now, Crowdstrike has got two mini filter drivers registered with Microsoft (for signing and allocation of altitude).

1) csagent.sys - Altitude (321410) This altitude falls within the range for Anti-Virus filters. 2) im.sys - Altitude (80680) This altitude falls within the range for access control drivers.

So, it is clear that the driver causing the crash is their AV driver, csagent.sys.

The workaround that CrowdStrike has given is to delete C-00000291*.sys files from the directory: C:\Windows\System32\Drivers\CrowdStrike\

These files being suggested to be deleted are not driver files (.sys files) but probably some kind of virus definition database files.

The reason they name these files with the .sys extension is possibly to leverage Windows System File Checker tool's ability to restore back deleted system files.

This seems to be a workaround and the actual fix might be done in their driver, csagent.sys and the fix will be rolled out later.

Anyone having access a Falcon endpoint might see a change in the timestamp of the driver csagent.sys when the actual fix rolls out.

rldjbpin• 2 years ago

it is humbling (and lowkey reassuring?) to know that not all large players use the absolute cutting edge approaches in their workflow.

it seems and i hope that after all is said and done there is no major life-threatening consequence of this debacle. at the same time, heart goes out to the dev who pushed the troubling code. very easy to point at them or the team's processes, but we need to introspect at our own setup and also recognize that not all of us work in crucial systems like this.

attentive11• 2 years ago

the funny thing it's often labeled "Microsoft IT outage" - theguardian.com as example

vlod2• 2 years ago

I haven't heard ask this, but would this have happened on linux. Obviously not many people run virus s/w, but would something similar like this have caused this?

Are there any protections to prevent repeating reboots?

peanut-walrus2• 2 years ago

I feel we are at a point in the evolution of our digital society where relying on general purpose OS-s is just not an option when moving forward.

omnee• 2 years ago

This event raises the question: What is the liability of Crowdstrike given its erroneous update caused the meltdown, and the impact certainly had negative personal or business outcomes globally.

See for example 6000 flights cancelled or the many statements posted here regarding it negatively impacting healthcare and other businesses.

1oooqooq13• 2 years ago

we are bound to see the YouTube ads equivalent of late night spot ads for lawyers with accelerated audio "have you lost someone to the 2024, 2025 or 2029 crowdstrike global hospital outages? if so you may be entitled to compensation. DM law5237 on X to find more"

hahamaster• 2 years ago

Someone at CrowdStrike got fired for this. I'm curious to know who this person is.

ngneer9• 2 years ago

The whole thing needs to be redesigned, so that antivirus and EDR solutions do not require such high privilege. We need a high-performance way for a possibly privileged service to export all the data that is needed for a decision, and then let the AV/EDR do its thing. If the AV/EDR is broken by an update, fine. At least the system won't go down.

tuxone• 2 years ago

And in critical production systems AV/EDR upgrades should be first tested on lower environments.

ngneer9• 2 years ago

Absolutely. Discipline can make all the difference.

ngneer9• 2 years ago

Have we failed as an industry?

ngneer9• 2 years ago

Allow me to give a different, information-theoretic, perspective. How much damage can flipping a single bit cause? How much damage can altering two bits cause?

The fanout is a robustness measure on systems. If we can control the fanout we increase reliability. If all it takes is a handful of bits in a 3rd party update to kill IT infrastructure, we are doing it wrong.

__MatrixMan__21• 2 years ago

Are you suggesting that a 3kb update be tested 3k times to assess the impact of each possible bit flip, and 9M times for the impact of each possible pair bit flips?

Because I think that's effort better spent in other ways.

ngneer9• 2 years ago

Not remotely. I am aware of the state space explosion and the difficulty with brute forcing the testing. I am suggesting that the damage a broken antivirus update can do should be restricted.

tonymet6• 2 years ago

Question : was this update delivered by Crowdstrike’s update agent or Windows Update ?

j-bos2• 2 years ago

My read is that Crowdstrike's update agent downloaded new security threat definitions and those definitions exposed a bug in the existing Crowdstrike drivers, causing the disaster.

tartavull• 2 years ago

The front page of https://www.accel.com/

"Fail Fast. Evolve Faster"

mbrumlow2• 2 years ago

We have collaborated with Intel to remediate affected hosts remotely using Intel vPro and with Active Management Technology.

This worries me. Does this mean intel has access to remotely access my machine?!?!

anvuong2• 2 years ago

What is with this paranoia on HN?

5s Google says that vPro is the Remote management platform offered by Intel for business, basically a fleet management tool, which is typical for corp

mbrumlow2• 2 years ago

Yah. But the way it was worded is that a 3rd party company who goofed was able to go to another company and ask for access to it.

I assume that some sort of configuration and opt in step would need to be had.

If this had been done prior to the event it seems like this would have been the first thing to do, not a Hail Mary after being down.

ai4ever• 2 years ago

what would be funny is if crowdstrike demanded ransom from their castomars.

security is a great business - you play on people's fears, your product does not have to deliver the goods.

like the lock maker, you sell a lock, the thief breaks it, but it is not your problem, and you sell a bigger badder lock the next year which promptly gets broken.

as a business, you dont have any consequences for how your product works or doesnt work, what a great business to be in !!

meta-level• 2 years ago

Wow, CrowdStrike made it to #6 of all time HN threads by now.. https://hn.algolia.com/?q=

meetpateltech• 2 years ago

CrowdStrike’s faulty update crashed 8.5 million Windows devices, says Microsoft

https://www.theverge.com/2024/7/20/24202527/crowdstrike-micr...

Yawrehto• 2 years ago

Oh wow, this is #5 for all time already, beating out Steve Jobs.

retrocryptid• 2 years ago

My Commodore 64 never gave me a blue screen of death and my Atari ST never lapsed into a hardcore boot loop.

just sayin'

peter_d_sherman• 2 years ago

Greenspun's tenth rule:

"Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp."

(https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule)

Arthur C. Clarke's third law:

"Any sufficiently advanced technology is indistinguishable from magic."

(https://en.wikipedia.org/wiki/Clarke%27s_three_laws#:~:text=....)

Apparently we now have the following, as well:

"Any sufficiently bad software update is indistinguishable from a cyberattack…"

(https://x.com/leighhoneywell/status/1814278230704111792)

minhoryang• 2 years ago

Can we find an uptime(availability) graph for the CrowdStrike agent? Don't you think this graph should be included in the postmortem?

nurettin• 2 years ago

If I were a cloud vendor, I would provide a "CrowdStrike recovery" button which queues the recovery image and restores the system for the entire project. Why didn't hetzner, linode, DO, gcp, aws do something like this? Why leave people to their devices? Isn't this a basic application of centralization? It feels to me like this should be easier than managing your data center.

CrowdStrike Update: Windows Bluescreen and Boot Loops

Related Stories