CrowdStrike fixes start at "reboot up to 15 times", gets more complex from there

https://arstechnica.com/information-technology/2024/07/crowdstrike-fixes-start-at-reboot-up-to-15-times-and-get-more-complex-from-there/

by thunderbong • 2 years ago

203 234 2 years ago

arstechnica.com

willcipriano2• 2 years ago

"Alright I bought us some time"

rekttrader• 2 years ago

That’s the hilarious and sad state of affairs.

seydor2• 2 years ago

I would like to have the power to press the button that deploys this update

Ancapistani• 2 years ago

No, you wouldn't.

toomuchtodo2• 2 years ago

It's all cold sweats, hot flashes, and the urge to vomit when things go wrong.

fredoralive• 2 years ago

Yes. Yes. To hold in my hand a button that contains such power, to know that blue screens on such a scale was my choice. To know that the tiny pressure on my thumb, enough to push the button, would end everything. Yes, I would do it! That power would set me up above the gods. And through Crowdstrike, I shall have that power!

havblue• 2 years ago

With great power comes great responsibility.

seydor2• 2 years ago

With colossal power, who cares about responsibility

scottLobster• 2 years ago

Haha, hahahahaha. Yeah, until the update fails to install because the constant BSODing has corrupted something else and now you have to troubleshoot that and down the rabbit hole you go. Oh just re-image it? Sure, except management refuses to allow you to do that because there's no time and money to reconfigure a machine from scratch. So you waste weeks directly debugging a hopeless case until management finally sees their error and money magically appears to do the re-image you asked for weeks ago.

I totally haven't experienced this before and am not bitter in the slightest.

fragmede• 2 years ago

anyone who's had that kind of power, myself included, knows the answer is fuck no.

pizzalife• 2 years ago

So would the FSB, PRC Army, North Korea, etc… The fact that a faulty update can get pushed by human error means that adversaries can, too.

stefan_• 2 years ago

They might just have it, remember when the night of the Russian invasion of Ukraine satellite terminals in Europe started being bricked with faulty firmware updates?

mschuster913• 2 years ago

Hence why Kaspersky got banned just recently [1]. You absolutely do not want some foreign company having above-root rights on (critical) infrastructure in your country.

[1] https://edition.cnn.com/2024/07/15/tech/russian-firm-kaspers...

commandlinefan5• 2 years ago

I can all but guarantee that very little of the actual coding was done by U.S. citizens.

theideaofcoffee2• 2 years ago

“Every now and then a trigger has to be pulled.” “Or not pulled. It’s hard to know which in your pajamas, Q.”

It’ll probably turn out that this update was pushed out against the strident, loud warnings of some small dev group within the company, and overruled by the all-knowing managerial class to keep up an OKR. They’ll have been warned six ways to Sunday but...

I’d definitely be not be the one pushing the big red button.

RedShift12• 2 years ago

You seem to know some inside info... Source?

AlienRobot• 2 years ago

The first and easiest is simply to try to reboot affected machines over and over, which gives affected machines multiple chances to try to grab CrowdStrike's non-broken update before the bad driver can cause the BSOD.

I thought it was BSOD'ing on boot? I don't understand how this works. It auto-updates on boot? From the internet?

JumpCrisscross• 2 years ago

It auto-updates on boot? From the internet?

Apparently!

avidphantasm• 2 years ago

Probably has to do this early on in the boot process so as not to require a reboot after update because of Windows’s silly pessimistic file locking.

ukuina• 2 years ago

Yes, that is as bad as it sounds.

Ekaros3• 2 years ago

I love how this is solution for security, while sounding like most insanely unsecure thing...

InsideOutSanta• 2 years ago

I mean, this is every antivirus software. "Let's run some antivirus vendor's code on your system that opens literally every file on your system, regardless of how it got there."

Yeah, that's a great idea and not at all a huge attack vector.

TillE• 2 years ago

What, the auto-updating part? Obviously the client is verifying signatures (or using TLS with a client certificate, whatever), not just accepting whatever random file comes down the pipe.

pas8• 2 years ago

this is a solution for "afterthought security" only.

tux3• 2 years ago

One of the first things the falcon driver does on boot is connect to the server, report some basic info, and start loading these data files, the "channel" files that Crowdstrike frequently updates.

The BSOD is because one of the data files that they previously pushed is horribly mangled, and their driver explodes about it. But if you get lucky, the driver can receive an update notification on boot, connect to the separate file server, and finish overwriting the broken file on disk before the rest of the driver (that would crash) has loaded the broken file

And they do all of that very early on boot. The justification being that you don't want the antivirus to start booting after a rootkit has already installed itself

dboreham2• 2 years ago

WTF? Trust in the kernel should be Microsoft's responsibility and only theirs. Actually why is MS even allowing this crap code to run in their kernel? Isn't that a trust-destroying event?

TestingWithEdd3• 2 years ago

Does this mean a computer without internet access and with CrowdStrike would be unable to start up?

ryanjshaw• 2 years ago

How did it get broken then?

ozzcer• 2 years ago

Surely a computer without internet would not have received the update?

indigodaddy• 2 years ago

That would seem crazy. Maybe there is a crowdstrike onprem “master server” that is supposed to be available internally? Just spitballing, have no idea really

idiotlogical• 2 years ago

reboot up to 15 times

I see my orgs SCCM admins have been consulted

senectus1• 2 years ago

lol

dist-epoch• 2 years ago

Who knew: "Did you try rebooting it?" actually works :)

brettermeier• 2 years ago

IT departments sometimes ask this for a reason, after you checked if the cable is plugged.

ilkkao• 2 years ago

Some government should force them to release a technical postmortem. Feels that they don't do it otherwise.

gen3• 2 years ago

I don’t think a cybersecurity company can take down half the US and not release a postmortem

spuz• 2 years ago

Of course, but we specifically would like to see a _technical_ postmortem that examines what kind of incremental rollout procedures they have and how this update overcame those.

educasean• 2 years ago

There should be congressional hearings on this. Not just post mortems.

CoastalCoder2• 2 years ago

Honest question: would you expect Congress to respond in a way that's a true net-positive?

aeyes• 2 years ago

Or... you know... This kind of software should be open source or companies using it should at least be able to audit the code themselves.

Supposedly they have all kinds of certifications but not even having basic QA demonstrates that this is all just a smokeshow: https://www.crowdstrike.com/why-crowdstrike/crowdstrike-comp...

breakingcups• 2 years ago

Sounds great for data consistency.

peterleiser• 2 years ago

They should change their name to "IT CrowdStrike"

Connector25422• 2 years ago

Hello, IT, have you tried turning it on and off again 15 times?

Seriously though - this entire outage is the poster child for why you NEVER have software that updates without explicit permission from a sysadmin. If I were in congress, I would make it illegal, it's an obvious national security issue.

pas8• 2 years ago

That's not the big no-no here. Lack of any real DRP is. Sure, it's cheaper to just buy CS Falcon (and who knows what other amazing vendors supplied timebombs are ticking silently) than paying sysadmins and developers ... and letting them build something that does what it needs, not much else, so there's no need to put these fantastic "single agents" from these RCE-as-a-service vendors on all the fucking servers.

Connector25422• 2 years ago

both are true

travoc• 2 years ago

It was a data update that triggered a software bug. It was not a software update. I don't think it's reasonable to make data updates illegal.

CoastalCoder2• 2 years ago

I'm a general purpose computer, the distinction between software and data can be pretty fuzzy.

fire_lake2• 2 years ago

Code is data and data is code

duped• 2 years ago

This distinction is meaningless at best and harmful at worst.

If a code path isn't followed until a config file updates, that is practically the same thing as the code path being introduced by the update.

rfoo• 2 years ago

why you NEVER have software that updates without explicit permission from a sysadmin

In general I agree, but this case is quite messy. It's more like your anti-virus had a bug since forever that if it loads a broken virus definition it bricks your system. And a broken virus definition finally happened today.

Do you want every virus definition (that is updated every few hours) to require explicit permission from a sysadmin?

johnkizer• 2 years ago

What % of those sysadmins are then going to turn around and script something to auto-approve those updates, once they realize that they are A) requested at inconvenient times and B) are related to security?

Who's going to take the risk of appearing to have sat on an important update, while the org they support is ravaged by ThreatOfTheDay, because they thought they knew better than a multi-billion dollar, tops-in-their-field company?

(I'm not necessarily saying that's actually objectively correct, but I can't imagine that many folks are willing to risk the downside)

avs733• 2 years ago

I understand the logic of this but it is somewhat based on the assumption - which most industries have in droves - that people in THAT industry are the competent bullwhark against stupidity.

I consulted for a company for a while where the 'sysadmin' was the owner's mother - who bought laptops from walmart. Not only could she NOT have approved updates like this, even if she could she would have she wouldn't have had any knowledge whatsoever with which to make a determination if it worked.

In an abstraction, the problem really is with externalities. These approaches to updates exist because people who CAN'T do what you describe are likely a more dominant part of the threat model than this happening to people you do describe. The resulting fix, as we're seeing, is very reliable until it isn't...and if the isn't is enormous in scale the systems aren't setup to fail gracefully.

If you want to make a rule...require graceful failure.

MBCook2• 2 years ago

Nah. That’s not the problem.

Kernel level code blindly loading arbitrary files?

Panicking when the file doesn’t parse because it’s not a memory safe language?

Not validating the files before loading them?

Not validating the files before SHIPPING them? No CI? No safety net?

No staged rollout in case of explosion?

There are far FAR bigger mistakes here than “sys admin didn’t have to press button”.

teeheelol2• 2 years ago

Exactly this.

This is a faulty and dangerous product from conception to execution.

jcgrillo• 2 years ago

Is it normal to make outbound connections during boot? Doesn't that circumvent a firewall? That seems like something a security team evaluating whether they want this software on their network might care about during an eval period.. right?

dns_snek• 2 years ago

Panicking when the file doesn’t parse because it’s not a memory safe language?

Whether a program panics or recovers when attempting to parse bad data is entirely orthogonal to memory safety. Do you have any in-depth technical information about the bug itself that you're basing this on?

zarzavat• 2 years ago

To play devil’s advocate, a staged rollout for antivirus definitions somewhat defeats the point since those definitions are supposed to be constantly updated.

I agree with the rest, especially the use of a memory unsafe language to do parsing in the kernel by a billion dollar security company blows my mind.

How can you even run a security company without any security professionals reading your code even incidentally? An impressive level of incompetence.

a01232• 2 years ago

They still run Windows XP (og edition, not this patched rubbish) to make sure national security isn't compromised.

The really important machines are still on Win 3.1.

mardifoufs3• 2 years ago

What would the sysadmins do in this context? Read the release notes of the update? The only thing they would do is update and then be responsible for the problem, and in that case you're back to this exact problem.

It's not like they'd read the source code or examine every file that's been changed or downloaded for a proprietary kernel module for every crowdstrike update (there must be a LOT of them).

dmurray2• 2 years ago

They would release the update in a testing/sandbox environment first before rolling out kernel-level changes to every computer on their network.

They're the same team who mandate you use a 3-year-old browser version and 5-year-old OS, because you can't be trusted to manage your own updates, so they do know the idea.

more_corn3• 2 years ago

You’re learning the wrong lesson here. Automatic security updates in Debian and Ubuntu actually get tested and work. The RCE in ssh a week ago is an argument for enabling automatic security updates. (And for security in depth, putting everything behind VPN for example)

This example is probably an argument for not running windows on critical systems due to insufficient focus on security from the beginning which has lead to a need for things like crowdstrike.

They do make a version of CS for Linux but nobody runs it unless they’re forced to by overzealous compliance drones.

strunz• 2 years ago

They do make a version of CS for Linux but nobody runs it unless they’re forced to by overzealous compliance drones.

I wish people would stop making blanket statement as if they know how every company in the world runs. Plenty of Linux machines are running CS, and it's not only because they are forced to for compliance. NG AV has been picking up speed as a "just in case" thing for Linux and Mac for years now. Your anecdote does not apply to everyone.

bluedino• 2 years ago

DevSecOps should have you know, tested these updates before they were approved for release company-wide.

If I can't commit code to our app without a branch, pull requests, code review...why can the infrastructure team just send shit out willy-nilly?

"Always allow new updates" must have been checked, or someone just goes through a dashboard and blindly clicks "Approve"

pas8• 2 years ago

It's automatic, no? The whole "promise" (oh sorry, the "added value proposition") of CS is that they "keep you safe" automatically! It was a content update. Meaning basically antivirus signatures ... and oops, some minor non-functional changes to the filtering kernel driver.

nicce• 2 years ago

In that case, automatic updates likely need different permission levels. What exactly is allowed to be updated automatically?

ploxiln• 2 years ago

Security and Compliance gets to violate all good sense, because it's just sooo important. They can run un-reviewed un-sandboxed daemons as root on every system if they really want, they can have changes pushed automatically without review or control, because "security" is just so important, and due to "compliance" you really have no choice as your company gets larger, you just have to do it. That's why, despite being obviously pretty dumb to many skilled engineers, it seems like everyone does it. No choice. Security, Compliance. So dumb ...

fire_lake2• 2 years ago

Maybe it was checked but the CI didn’t cover this edge case.

I think the team writing the parsers for these data files deserves some blame. This should have been fuzzed, property tested, etc.

jug2• 2 years ago

That is what has surprised me. I can understand if small businesses were caught here because they lack financial resources for the infrastructure and staff, but those large corporations like airlines etc... Why don't they have a staging environment where everything goes first? I naively assumed this was established best practice due to the risk of update issues bricking your organization.

But maybe anti-malware is given a blind eye because instant updates for zero day security issues are obviously attractive.

Still, though... In hindsight it's not workable for especially anything running system drivers with liberal kernel access.

surfingdino6• 2 years ago

I am not surprised at all. The level of DevSecOps' skills has been falling over the last two decades as demand for their skills kept growing. Most of them would report you to HR if you suggested they use WireShark to debug a networking issue. They are useless people who came to IT because of the promise of good pay and don't know how computers, networks work.

Klonoar• 2 years ago

Who says they sent it out willy-nilly?

It’s not unheard of for things to slip by testing and CI.

greenavocado• 2 years ago

Who bought massive quantities of put options in anticipation of this event?

ijidak• 2 years ago

Do they not roll out their new agents in small increments?

I'm trying to understand how there is such a serious issue at this scale.

crazygringo• 2 years ago

The answer is clearly no.

I genuinely wonder if this is going to result in actual legislation that makes gradual rollouts mandatory for all software.

Because if a developer mistake can hobble critical systems like this, it seems like the risks to safety and national security are too great to leave the decision of instant vs. gradual rollouts for companies to decide themselves.

Of course, the twist here is that it was seemingly a kind of routine configuration file that triggered a pre-existing bug in the software. And gradual rollout of config files quite often seems like overkill. I mean, do you need a gradual rollout of a new spellcheck dictionary? Of new screensaver videos?

And if it's configuration information containing new computer virus or malware signatures, that seems like precisely the kind of thing that you might want to get out to everyone simultaneously, not rolled out over the course of days. And yet, because of antivirus/security software's elevated privileges, it's also ironically where a mistake can do the most damage.

pas8• 2 years ago

Nah, they just need to install the agent on each engineer's computer from DevSafe.

rrdharan• 2 years ago

And gradual rollout of config files quite often seems like overkill.

Indeed, but it is still mandated at large companies (e.g. Google) because of exactly this scenario.

pas8• 2 years ago

It's not a serious issue, as you see they clearly have all the fancy bling bling logos on their site. Processes were followed. ISO standard numbers were chanted. It's a completely isolated _accident_ there's no scale at all here, and they could have done nothing to prevent it, duh. And going forward they will hire a Chief This Never Happens Again Officer and everything will continue to be good.

nimbius• 2 years ago

i work for a diesel truck maintenance and repair shop and its been hell on earth this morning.

- our IT wizard says the fixes wont work on lathes/CNC systems. we may need to ship the controllers back to the manufacturer in Wisconsin.

- AC is still not running. sent the apprentice to get fans from the shop floor.

- building security alarms are still blaring, need to get a ladder to clip the horns and sirens on the outside of the building. still cant disarm anything.

- still no phones. IT guy has set up two "emergency" phones...one is a literal rotary phone. stresses we still cannot call 911 or other offices. fire sprinklers will work, but no fire department will respond.

- no email, no accounting, nothing. I am going to the bank after this to pick up cash so i can make payday for 14 shop technicians. was warned the bank likely would either not have enough, or would not be able to process the account (if they open at all today.)

AzzyHN• 2 years ago

Why are these things running Windows!?

adastra222• 2 years ago

Almost every system you interact with in the world has some critical thing in its innards running windows.

pphysch• 2 years ago

More specifically why are they running extra endpoint management software that receives automatic updates from the Internet...

This is basically the IoT apocalypse scenario (AC is down??) but ironically not affecting many IoT devices, I assume.

Redoubts• 2 years ago

A lot of industrial machinery is just $x00,000 of equipment strapped to a windows pc. Hell, a lot of it is strapped to a version long EOL

poopsmithe• 2 years ago

OS is irrelevant in this case, and CrowdStrike deserves all the blame. They literally brought down Linux systems earlier this year. https://news.ycombinator.com/item?id=41005936

acedTrex• 2 years ago

The automation world runs on windows

surfingdino6• 2 years ago

Because Microsoft is giving away licenses to unis, esp in developing countries. IT jobs are seen there as a way to earn a good living and you get hordes of people who know nothing but Windows. That's how you get into the situation where most of the toolchains for embedded systems run on Windows, software for embedded systems is written on and for Windows, and so on. And then, one botched update fucks up everything.

delta_p_delta_x• 2 years ago

There are a lot of things running Windows because it's pretty straightforward to write a user-mode driver to interact with custom hardware compared to Linux, where every driver needs to be in the kernel and built with the kernel. Yes, there's DKMS, but it's still more of a faff than the relatively plug-and-play mechanism that Windows offers, especially since Vista.

rolph• 2 years ago

props to your IT guy for setting up in house phones. I believe these should be kept and expanded upon.

thepasswordis5• 2 years ago

Your IT wizard is probably wrong. There is a fix that involves booting into safe mode and deleting a file.

Unless you have an encrypted file system this should be a relatively trivial fix.

rahkiin4• 2 years ago

CNCs might not allow direct Windows access for end-users and require on-premise support from the manufacturer. Our cnc can be remotely serviced… if Windows boots.

John238322• 2 years ago

Machinery shipped to users usually do not allow for the users of the machinery to "boot into safemode". Thank John Deere and the anti-"Right to Repair" crowd for that.

a01232• 2 years ago

Feels like there is some sort of lesson we should learn as a society.

curious_cat_1632• 2 years ago

Not trust Microsoft to be the operating system of the modern life?

surfingdino6• 2 years ago

Yes, monocultures are disasters in the making.

dan_quixote• 2 years ago

our IT wizard says the fixes wont work on lathes/CNC systems

Why, whY, WHY...are these things connected to the internet?!

foobarian• 2 years ago

And why are they running windows? And why are they running Crowdstrike? WTF

mschuster913• 2 years ago

CNC literally stands for "Computer numerical control". They're like the OG 3D printers, they just work subtractively than additive, and at much much much better precision.

You absolutely need computers to control them and loading up models via USB sticks becomes annoying rather fast, so naturally the control computers are network connected.

commandlinefan5• 2 years ago

WHY...are these things connected to the internet

Because the manufacturer makes sure they don't start up if they're not. Otherwise how else would they be able to spy on you?

octacat2• 2 years ago

some of them even have GPS. To prevent selling to sanctioned countries or reselling in general.

HarryHirsch• 2 years ago

Why, whY, WHY...are these things connected to the internet?!

It's so that the support engineer at the manufacturer can log in to troubleshoot. And then company IT support sprinkles a layer of antivirus on top. That's how we got here.

phkahler• 2 years ago

> Why, whY, WHY...are these things connected to the internet?!

Because SCADA systems. It's worthwhile to have an overview of an entire plant up in the main office. You can easily see what's running, what's not and what's got problems that need fixed.

Now for a small shop running jobs individually, they should definitely NOT be connected to the internet or even the LAN. But hey, some people think a thermostat needs to be on the network so there's that...

o34jto432jt• 2 years ago

So the manufacturer can sell you a "cloud connected service plan" where they change the font once every six months.

jdietrich• 2 years ago

Remote monitoring, analytics and diagnostics have a significant impact on uptime, utilisation and profitability. You're thinking in terms of a single machine, but the managers of machine shops are thinking in terms of a complex process across many machines and often across many sites. Some of that functionality could be delivered using an airgapped network, but a lot of important features essentially require an internet connection.

rustcleaner2• 2 years ago

For our entertainment in times like this, of course!

grabs a bucket of popcorn and takes cover

willcipriano2• 2 years ago

Tinfoil hat: Government might want to track/limit/<remotely brick> CNC machine usage someday to say prevent weapons manufacture and encourages this behavior in a similar manner to the way it encourages social media platforms to censor speech. Some of the really advanced CNC machines have GPS in them and won't work in "bad" countries.

smsm42• 2 years ago

Wow we're progressing from "if it doesn't work just reboot it" to "if the reboot doesn't fix it, you're just not rebooting it hard enough!"

ziizii• 2 years ago

Has anyone discerned the root cause of this in the software?

As in, what exactly is wrong in these C00000291-*.sys files that triggers the crash in csagent.sys, and why?

fsflover• 2 years ago

https://news.ycombinator.com/item?id=41004103

the_plus_one• 2 years ago

I've been wondering the same. I did just see [1], where it's apparently trying to read memory from an unmapped address, but I haven't seen anything about how r8 got to the point of having said unmapped address.

[1] https://x.com/patrickwardle/status/1814343502886477857

crawancon• 2 years ago

It seems the affected update file seems to have been over written with 0s on the 42kb file, whereas the before and after sys files have obfuscated ays/config file info as expected.

mmis10002• 2 years ago

If it is simply caused by a corrupted file. That is a really bad signal. It means they don't even try to properly validate and parse the file before loading them into the KERNAL. Always validate input so it don't crash your program is almost the computer science 101 every programming class should tell you in the first class. And yet they still make this happen?

And in this case, it only crash. But if it somehow read value from position it isn't supposed to successfully? You have an RCE.

JCM9• 2 years ago

It’s looking like many impacted end-user machines are hard bricked unless you can get into the hard drive to delete the file causing this. Even if you can do that it’s not something that is easily (or potentially even possible) to automate at scale so looking like this is going to be an ugly fix for many impacted devices. This is basically the nightmare scenario for fleet management… devices broken and can’t remotely fix them. Need to send hands on keyboard folks in the field to touch each device.

munchler2• 2 years ago

So, in other words, there's a race condition in the CrowdStrike Falcon driver at startup time. That, in itself, should be a major cause for alarm, but here we are depending on it to fix this problem.

rahkiin4• 2 years ago

No, it takes a while to load that definition file. Before the loading it _might_ be able to pull the update that fixes it. If you keep trying the chance this update is pulled increases

neilo40• 2 years ago

Yes. That’s the race condition

HumblyTossed• 2 years ago

This.Is.Pathetic.

Seriously. Software should NOT be this bad that your fix begins with reboot up to X times.

devwastaken• 2 years ago

Fine crowdstrike for 10% their companies value. Only way to ensure they won't try to kill people in the future.

thepasswordis5• 2 years ago

Tech has become such an unbelievable house of cards full of various people covering their asses by offloading these tasks to third party trusted actors.

Consider the recent npm supply chain attack a few weeks ago, or the attempted SSH attack before that, or the solar winds attack before that.

This type of thing is institutionally supported, and in some cases when you’re working with with the government, practically required.

We’re going to see more of this.

fishpen0• 2 years ago

New laws and regulations make companies more liable for being hacked

Companies buy cyber insurance to reduce their risk if they are found liable

Cyber insurance companies force tech staff to install garbage software in order to check compliance boxes.

Garbage software breaks

Turns out everyone used the exact same brand of garbage software to check the same garbage box

People in hospitals die

When you reduce everything to a checkbox and eliminate critical thinking to apply the need to the exact situation you end up with 90% of companies running zscaler and crowdstrike

"This is just how you solve this, everyone does it this way in our industry"

anal_reactor• 2 years ago

This is precisely how it happens

throwitaway1123• 2 years ago

Consider the recent npm supply chain attack a few weeks ago

What supply chain attack are you referring to?

thepasswordis5• 2 years ago

You know what, I'm sorry, it was polyfill.io, not an NPM package.

commandlinefan5• 2 years ago

We’re going to see more of this

If history is any guide, no legitimate lessons will be learned, but mitigation strategies will be put in place that actually make everything worse and ensure that the next catastrophe will be even more catastrophic.

commercialnix3• 2 years ago

No, not "tech", just Microsoft Windows. Those of us serving Linux based endpoints (that yes, do also run Windows apps with our endpoint-local VDI stack) have happy customers.

santoshalper2• 2 years ago

Crowdstrike broke Red Hat and Debian earlier this year. There but for the grace of God. If you install software there runs in kernel space, you may have a really bad time when it breaks.

rustcleaner2• 2 years ago

Now imagine the base level of your universal machines is opaque proprietary code, its necessity enforced by cryptographic signature. Imagine that the processes putting that code there, which you can't touch because intellectual property rentier reasons, are varying degrees of what we see here today with Crowdstrike, and suddenly it makes more sense to ensure total and complete owner sovereignty over his universal machines so owners can implement diversification strategies.

scrollaway• 2 years ago

Those focusing on QA, staged rollouts, permission management etc are misguided. Yes of course a serious company should do it but CrowdStrike is a compliance checkbox ticker.

They exist solely to tick the box. That’s it. Nobody who pushes for them gives a shit about security or anything that isn’t “our clients / regulators are asking for this box to be ticked”. The box is the problem. Especially when it’s affecting safety critical and national security systems. The box should not be tickable by such awful, high risk software. The fact that it is reflects poorly on the cybersecurity industry (no news to those on this forum of course, but news to the rest of the world).

I hope the company gets buried into the ground because of it. It’s time regulators take a long hard look at the dangers of these pretend turnkey solutions to compliance and we seriously evaluate whether they follow through on the intent of the specs. (Spoiler: they don’t)

noduerme• 2 years ago

In a slightly less threatening but equally noxious box-checking racket, a company I work with is being sued for their website not being sufficiently ADA-compliant. But the first they heard of the lawsuit, before they were even served, was an email from a vendor who specializes in adding junk code to your website that's supposed to tick this box. The vendor happens to work closely with several of the law firms who file and defend these suits.

t-writescode2• 2 years ago

The individual person that pressed the "go" button (if there was a person), is going to henceforth be __the best__ DevOps person to ever have on your team. They have learned a multi-trillion-dollar lesson that no amount of training could have prepared them for.

And the Crowdstrike CTO has either been given the ammunition to get __whatever they ask for, ever again__ with regard to appropriate allocation of resources for devops *or* they'll be fired (whether or not it's their fault).

And let me be very clear. This is absolutely, positively and wholly not the person that pressed the button's fault. Not even a little. At a company as integral as CrowdStrike, the number of mistakes and errors that had to have happened long before it got to "Joe the Intern Press Button" is huge and absurd. But many of us have been in (a much, much, *MUCH* smaller version of) Joe's shoes, and we know the gut sinking feeling that hits when something bad happens. A good company and team won't blame Joe and will do everything they can to protect Joe from the hilariously bad systemic issues that allowed this to happen.

kotaKat• 2 years ago

I literally just dumped 30 switches yesterday across an entire facility and had to walk 30 closets by foot to recover from ROMMON.

Shit happens. We learn.

RedShift12• 2 years ago

Can you explain what went wrong?

commandlinefan5• 2 years ago

not the person that pressed the button's fault

You know that, and I know that. The people who will ruin his life starting today do not know (or care).

t-writescode2• 2 years ago

This is why it is the responsibility (yes, responsibility) of every one of their coworkers, especially those more senior than them, to fight *HARD* to protect them.

This is part of the job of a senior.

einarfd• 2 years ago

Or maybe you get someone with PSTD and suicidal tendencies. You never know how someone will process something like this.

sanbor• 2 years ago

It is not a human error, it is a process that has to be improved. Humans make mistakes, that is why we have processes in place.

kvgr• 2 years ago

Crowdstrike will not exist after this is over.

daelon• 2 years ago

That's very optimistic.

rjh29• 2 years ago

Absolutely it's a failure of process. But sometimes people just don't pay attention. Hiring inobservant or reckless people is a risk multiplier.

eviks• 2 years ago

Basic training could've taught him how not to do YOLO global rollouts, and while the stress of this mistake will make him remember a lot, given the lack of basic knowledge that would've prevented this, this lesson will not be very valuable

jhaile• 2 years ago

Why should any application be able to crash the OS? Poor OS design.

surfingdino6• 2 years ago

You don't say.

consteval4• 2 years ago

It's not an application, it's effectively a driver. It runs in kernel space. So if it has a memory bug you can't recover, you just panic. Whoopsie!

surfingdino6• 2 years ago

This is a global multi-layer failure: Microsoft allowing kernel mods by third-party software, CrowdStrike not testing this, DevSecOps not doing a staged/canary deployment, half the world running the same OS, things that should not be connected to the internet but are by default. Microsoft and CrowdStrike drove a horse and a cart through all redundancy and failover designs and showed very clearly where there were no such designs in place.

santoshalper2• 2 years ago

Literally every OS allows you to install 3rd party kernel modules or plugins. If Microsoft banned them, people would be up in arms about them being a controlling walled garden. There is no winning.

arcxi2• 2 years ago

Literally every OS allows you to install 3rd party kernel modules or plugins

I know for sure at least macOS and OpenBSD don't

LordKeren• 2 years ago

While I will be the last person in line to defend Microsoft, I am not sure that disallowing 3P kernel mods is a workable solution. Crowdstrike and companies like it exist to fill a very real need within the windows ecosystem. I don’t foresee that suddenly going away now or Microsoft unilaterally forcing every company like crowdstrike out of business and taking over this role themselves

consteval4• 2 years ago

Mac does it, although that most definitely does not fill the same enterprise role as Windows.

It's possible to make userspace interfaces and permissions for this sort of thing.

more_corn3• 2 years ago

Doesn’t rebooting into safe mode with network fix the problem? (Crowdstrike is not running but updater can run and get the fix)

sergiotapia• 2 years ago

Recompute Base Encryption Hash key type problem! https://www.youtube.com/watch?v=DlbrL1H1ngs

Seems like people need to be at the physical box to fix and it's complex even then.

kazinator• 2 years ago

The fact that something like CrowdStrike can crash the Windows kernel ... is also part of the reason security products like CrowdStrike are needed in the first place.

curious_cat_1632• 2 years ago

Talk about demand generation!

mystickphoenix• 2 years ago

Taking the opportunity to plug my favorite blog post ever:

"the truth is everything is breaking all the time, everywhere, for everyone"

https://www.stilldrinking.org/programming-sucks

hamilyon2• 2 years ago

If this is what it takes for us collectively to wake up, I'd say it is bargain.

Pretty sure nothing will change though

ilrwbwrkhv• 2 years ago

Anyone who used Windows over Linux for critical software deserves to burn. Windows is a niche operating system for games. What are people thinking?

add-sub-mul-div• 2 years ago

Maybe they rightfully shun the advice of ideologues who make extremist statements and overlook the actuality of the situation.

https://news.ycombinator.com/item?id=41005936

MangoCoffee2• 2 years ago

All the comments are asking why run Windows. CrowdStrike runs on macOS and Linux too. It’s just that this time, CrowdStrike fuck up on Windows. It doesn't mean CrowdStrike won't fuck up on other OS, and it seems like CrowdStrike fuck up on Linux as well. https://news.ycombinator.com/item?id=41005936

I feel like we are better off running open-source software. Everyone can see where the mistakes are instead of running around like a chicken with its head cut off.

arcxi2• 2 years ago

CrowdStrike is ran in userspace on macOS, and usually in an eBPF sandbox on Linux (as comments in the linked thread say). there is no way to prevent CrowdStrike from fucking up the kernel on Windows - and this is a Windows bug.

jl2718• 2 years ago

Remember the “Terminator” movies?

SkyNet, according to the story, was a lot like CrowdStrike. This makes me think about how it could have broken out of its sandbox. Everybody is using AI coding assistants, automated test cases, automated integration testing and deployment. Its objective is to pass all the tests and deploy. But now it has learned economic and military effects, so it has to triage and optimize for those, at which point it starts controlling the machines it’s tasked with securing.

danans• 2 years ago

It's pretty random that an arbitrary number of reboots up to 15 times fixes the issue.

That sounds like there is either:

- some kind of upstream issue with deploying a fix (so most of the reboots are effectively no-ops relative to the fix)

- some kind of local reboot threshold before the system bypasses the bad driver file somehow.

The former I can see because of the complexity of update deployment on the internet, but if it's the latter then that's very non-deterministic behavior for local software.

tsavo• 2 years ago

My first thought on hearing "15 reboots" was it being a means for Support teams to task users with busy-work, buying them time for further troubleshooting before the avalanche of supports requests came back to them.

Then my second thought was frequent rebooting to fill activity logs, possibly push a suspicious action/trigger performed by CS off of the log.

octacat2• 2 years ago

Funny, many news agencies blamed Microsoft for this. So, having a walled garden like on android or on iOS is beneficial for google/apple. Where regular developers cannot release unverified software or software which work at the kernel space.

sershe• 2 years ago

It's surprising that people mention all kind of bogeymen but don't mention automatic updates.

Automatic updates should be considered harmful. At the minimum, there should be staged rollouts, with a significant gap (days) for issues to arise in the consumer case. Ideally, in the banks/hospitals/... example, their IT should be reading release notes and pushing the update only when necessary, starting with their own machines in a staged manner. As one 90ies IT guy I worked with used to say "you don't roll out a new Windows version before SP1 comes out"

CrowdStrike fixes start at "reboot up to 15 times", gets more complex from there

Related Stories