The CrowdStrike file that broke everything was full of null characters?

https://twitter.com/jeremyphoward/status/1814364640127922499

by behnamoh • 2 years ago

307 351 2 years ago

TillE• 2 years ago

That probably explains how it got past internal testing. Something went wrong after that, during deployment.

behnamoh• 2 years ago

It could be as simple as cosmic radiation that flipped a bit (it has happened before: https://www.independent.co.uk/news/science/subatomic-particl...), or as sophisticated as an adversarial hacking.

rvnx• 2 years ago

The same cosmic radiation that flips the bits to make some specific political party win.

zimpenfish• 2 years ago

Someone on the Fediverse conjectured that it might have been down to the Azure glitch earlier in the day. An empty file would fit that if they weren't doing proper error checking on their downloads, etc.

homero• 2 years ago

It's crazy if they weren't signing and verifying downloads

janice1999• 2 years ago

It's still crazy that a security tool does not validate content files it loads from disk that get regularly updated. Clearly fuzzing was not a priority either.

Zigurd• 2 years ago

How many years has this Crowdstrike code been running without issues? You have put your finger on it: Fuzzing should have been part of a test plan. Even TDD isn't a bastard test engineer writing tests that probe edge cases. Even observing that your unit tests have good code coverage isn't a substitute for fuzzing. There is even a counter-argument that something that been reliable in the field should not be fixed for reasons like failing a test case never seen in real deployments, so why go making trouble.

password4321• 2 years ago

https://news.ycombinator.com/item?id=41006104#41006555

the flawed data was added in a post-processing step of the configuration update, which is after it's been tested internally but before it's copied to their update servers

gbin• 2 years ago

I don't understand, how the signature even worked? Please please tell me those drivers are signed... Right? ...

Retr0id5• 2 years ago

While I won't discount it entirely, I think the people acting like this (alone) implies malice are being very silly.

JumpCrisscross• 2 years ago

It demonstrates terrible QC.

loloquwowndueo• 2 years ago

Don’t mess with Quebec :D

Retr0id5• 2 years ago

Clearly something failed catastrophically, but it could well be post-QC

midtake2• 2 years ago

I disagree. In the current day the stakes are too highly to naively attribute software flaws to incompetence. We should assume malice until it is ruled out, otherwise it will become a vector for software implants. These are matters of national security at this point.

Retr0id5• 2 years ago

You can (and should) want to identify the root cause, without assuming malice.

gerdesj2• 2 years ago

"These are matters of national security at this point."

Which nation exactly? Who on earth "wins" by crashing vast numbers of PCs worldwide?

Many of the potential foes you might be thinking of are unlikely to actually run CS locally but its bad for business if your prey can't even boot their PCs and infra so you can scam them.

I might allow for a bunch of old school nihilists getting off on this sort of rubbish but it won't last and now an entire class of security software, standards and procedures will be fixed up. This is no deliberate "killer blow".

Who knew that well meaning security software running in Ring 0 could fuck up big style if QA takes a long walk off a short plank? Oh, anyone who worked in IT during the '90s and '00s! I remember Sophos and McAfee (now Trellix) and probably others managing to do something similar, back in the day.

Mono-cultures are subject to pretty catastrophic failures, by definition. If you go all in with the same thing as everyone else then if they sneeze, you will catch the 'flu too.

fourteenfour• 2 years ago

At least it compressed well, which must have saved network resources during the update. :)

sitkack• 2 years ago

Resources like wall clock time.

bloopernova2• 2 years ago

Note to self: on Monday, add a null character check to pre-commit hooks, and add the same check to pipelines.

Retr0id5• 2 years ago

It's perfectly normal for binary artifacts to contain null bytes, even long runs of them.

bloopernova2• 2 years ago

Yeah, I'd need to figure it out properly, but for unicode text files it should be OK. Good point about the binaries though, thank you!

MilStdJunkie4• 2 years ago

I mentioned it in a separate parent, but null purge is - for the stuff I work with - completely non-negotiable. Nulls seem to break virtually everything, just by existing. Furthermore, old-timey PDFs are chock full of the things, for God knows what reason, and a huge amount of data I work with are old-timey PDF.

toast0• 2 years ago

Furthermore, old-timey PDFs are chock full of the things, for God knows what reason, and a huge amount of data I work with are old-timey PDF.

Probably UCS-2/UTF-16 encoding with ascii data.

mrguyorama2• 2 years ago

The problem ISN'T the null character though. The problem is that they tested the system, THEN changed stuff, then uploaded the changed stuff.

Their standard methodology was to deploy untested stuff.

thrill• 2 years ago

So ... the checksum of all the Seinfeld episodes?

geor9e• 2 years ago

Explain joke please

themagicteeth• 2 years ago

Seinfeld is a show about nothing

http://seinfeldscripts.com/ThePitch.htm

WhyCause• 2 years ago

Seinfeld was "a show about nothing."

kristjansson3• 2 years ago

Something like

   zero_output_file(fh, len(file))

   flush()

   fill_output_file(fh, data)

with an oops in line 3?

AndrewKemendo3• 2 years ago

This should not have passed a competent C/I pipeline for a system in the critical path.

I’m not even particularly stringent when it comes to automated test across-the-board but for this level of criticality of system, you need exceptionally good state management

To the point where you should not roll to production without an integration test on every environment that you claim to support

Like it’s insane to me that this size and criticality of a company doesn’t have a staging or even a development test server that tests all of the possible target images that they claim to support.

Who is running stuff over there - total incompetence

martinky245• 2 years ago

A lot of assumptions here that probably aren't worth making without more info -- For example it could certainly be the case that there was a "real" file that worked and the bug was in the "upload verified artifact to CDN code" or something, at which point it passes a lot of things before the failure.

We don't have the answers, but I'm not in a rush to assume that they don't test anything they put out at all on Windows.

EvanAnderson3• 2 years ago

I haven't seen the file, but surely each build artifact should be signed and verified when it's loaded by the client. The failure mode of bit rot / malice in the CDN should be handled.

chrisjj5• 2 years ago

it could certainly be the case that there was a "real" file that worked and the bug was in the "upload verified artifact to CDN code" or something

I.e. only one link in the chain wasn't tested.

Sorry, but that will not do.

We don't have the answers, but I'm not in a rush to assume that they don't test anything they put out at all on Windows.

The parent post did not suggest they don't test anything. It suggested they did not test the whole chain.

0xcafecafe• 2 years ago

They could even have done slow rollouts. Roll it out to a geographical region and wait an hour or so before deploying elsewhere.

xyst2• 2 years ago

Or test in local environments first. Slow rollouts like this tend to make deployments very very painful.

daseiner1• 2 years ago

You say even (emphasis mine). Is this not industry standard?

saati• 2 years ago

In theory CrowdStrike protects you from threats, leaving regions unprotected for an hour would be an issue.

notabee• 2 years ago

Without delving into any kind of specific conspiratorial thinking, I think people should also include the possibility that this was malicious. It's much more likely to be incompetence and hubris, but ever since I found out that this is basically an authorized rootkit, I've been concerned about what happens if another Solarwinds incident occurs with Crowdstrike or another such tool. And either way, we have the answer to that question now: it has extreme consequences. We really need to end this blind checkbox compliance culture and start doing real security.

sonotathrowaway• 2 years ago

That’s not even getting into the fuckups that must have happened to allow a bad patch to get rolled out everywhere all at once.

carterschonwald• 2 years ago

The strange thing is that when I interviewed there years ago with the team that owns the language that runs in the kernel, they said their ci has 20k or 40k machine os combinations/configurations. Surely some of them were vanilla windows!

dboreham• 2 years ago

They used synthetic test data in CI that doesn't consist of zeros.

russdill2• 2 years ago

You can have all the CI, staging, test, etc. If some bug after that process nulls the file, the rest doesn't matter

Jtsummers• 2 years ago

If a garbage file is pushed out, the program could have handled it by ignoring it. In this case, it did not and now we're (the collective IT industry) dealing with the consequences of one company that can't be bothered to validate its input (they aren't the only ones, but this is a particularly catastrophic demonstration of the importance of input validation).

fabian2k• 2 years ago

Those signature files should have a checksum, or even a digital signature. I mean even if it doesn't crash the entire computer, a flipped bit in there could still turn the entire thing against a harmless component of the system and lead to the same result.

LorenPechtel• 2 years ago

Yup. I had quite a battle with some sort of system bug (never fully traced) where I wrote valid data but what ended up on disk was all zero. It appeared to involve corrupted packets being accepted as valid.

It doesn't matter how much you test if something down the line zeroes out your stuff.

Cerium• 2 years ago

What sort of sane system modifies the build output after testing?

Our release process is more like: build and package, sign package, run CI tests on signed package, run manual tests on signed package, release signed package. The deployment process should check those signatures. A test process should by design be able to detect any copy errors between test and release in a safe way.

jononor• 2 years ago

The issue is not that a file with nulls was produced. It is that an invalid file (or any kind) can trigger a blue screen of death.

hnlmorg• 2 years ago

It seems unlikely that a file entirely full of null characters was the output of any automated build pipeline. So I’d wager something got built, passed the CI tests, then the system broke at some point after that when the file was copied ready for deployment.

But at this stage, all we are doing is speculating.

dagaci2• 2 years ago

/* Acceptance criteria #1: do not allow machine to boot if invalid data signatures are present, this could indicate a compromised system. Booting could cause presidents diary to transmit to rival 'Country' of the week */

if(dataFileIsNotValid) { throw FatalKernelException("All your base are compromised"); }

EDIT+ Explanation:

With hindsight not booting may be exactly the right thing to do since a bad datafile would indicate a compromised distribution/ network.

The machines should not fully boot until file with valid signature is downloaded.*

arp2426• 2 years ago

Like it’s insane to me that this size and criticality of a company doesn’t have a staging or even a development test server that tests all of the possible target images that they claim to support.

Who is saying they don't have that? Who is saying it didn't pass all of that?

You're making tons of assumptions here.

martinky245• 2 years ago

Yeah... the comment above reads like someone who has read a lot of books on CI deployment, but has zero experience in a real world environment actually doing it. Quick to throw stones with absolutely no understanding of any of the nuances involved.

JKCalhoun2• 2 years ago

To be sure. But the fact is the release broke.

I'm not sure: is having test servers that it passed any better than none at all?

ikiris• 2 years ago

Dude, the fact that it breaks directly.

You sound like the guy that a few years ago tried to argue (the company in question) tested os code that didn't include any drivers for their gear's local storage. Its obvious it wasn't to anyone competent.

dheera• 2 years ago

I don't know if people on Microsoft ecosystems even know what CI pipelines are.

Linux and Unix ecosystems in general work by people thoroughly testing and taking responsibility for their work.

Windows ecosystems work by blame passing. Blame Ron, the IT guy. Blame Windows Update. Blame Microsoft. That's how stuff works.

It has always worked this way.

But also, all the good devs got offered 3X the salary at Google, Meta, and Apple. Have you ever applied for a job at CrowdStrike? No? That's why they suck.

* A disproportionately large number of Windows IT guys are named Ron, in my experience.

kabdib2• 2 years ago

That's a pretty broad brush.

miki123211• 2 years ago

Keep in mind that this was probably a data file, not necessarily a code file.

It's possible that they run tests on new commits, but not when some other, external, non-git system pushes out new data.

Team A thinks that "obviously the driver developers are going to write it defensively and protect it against malformed data", team B thinks "obviously all this data comes from us, so we never have to worry about it being malformed"

I don't have any non-public info about what actually happened, but something along these lines seems to be the most likely hypothesis to me.

Edit: Now what would have helped here is a "staged rollout" process with some telemetry. Push the update to 0.01% of your users and solicit acknowledgments after 15 minutes. If the vast majority of systems are still alive and haven't been restarted, keep increasing the threshold. If, at any point, too many of the updated systems stop responding or indicate a failure, immediately stop the rollout, page your on-call engineers and give them a one-click process to completely roll the update back, even for already-updated clients.

This is exactly the kind of issue that non-invasive, completely anonymous, opt-out telemetry would have solved.

adzm• 2 years ago

This was a .dll in all but name fwiw.

ar_lan• 2 years ago

tests all of the possible target images that they claim to support.

Or even at the very least the most popular OS that they support. I'm genuinely imagining right now that for this component, the entirety of the company does not have a single Windows machine they run tests on.

tinytime• 2 years ago

It's wild that I'm out here boosting existing unit testing practices with mutation testing https://github.com/codeintegrity-ai/mutahunter and there are folks out there that don't even do the basic testing.

bryanlarsen4• 2 years ago

Segue: What the heck is about Windows files and null characters? I've been almost exclusively dealing with POSIX file systems for the last 30 years, but I'm currently shipping a cross-platform app and a lot of my Windows users are encountering corrupted files which exhibit a bunch of NULs in random places in the file. I've added ugly hacks to deal with them but it'd be nice to get down to root causes. Is there a handy list of things that are safe on POSIX but not on Windows so I can figure out what I'm doing wrong?

I'm at the stage where I'm thinking "%$#@ this, I'm never going to write to the Windows file system again, I'm just going to create an SQLite DB and write to that instead". Will that fix my problems?

nradov• 2 years ago

The Windows NTFS is safe and reliable. It doesn't corrupt files. You have probably misunderstood the problem.

bryanlarsen4• 2 years ago

However I'm using it it certainly isn't. It's certainly quite likely the problem is me, not Windows.

Most of the files that are getting corrupted are being written to in an append-only fashion, which is generally one of the mechanisms for writing to files to avoid corruption, at least on POSIX.

omoikane• 2 years ago

I have observed a file filled NULs that was caused by a power loss in the middle of a write -- my UPS alerted me that utility power is gone, I tried to shutdown cleanly, but the battery died before a clean shutdown completed. This was NTFS on a HDD and not a SSD.

I am not saying it happens often, but it does happen once in a while.

tatersolid• 2 years ago

Returning all zeros randomly is one of the failure modes of crappy consumer SSDs with buggy controllers. Especially those found in cheap laptops and on Amazon. If it’s a fully counterfeit drive it might even be maliciously reporting a size larger than it has flash chips to support. It will accept writes without error but return zeros for sectors that are “fake”. This can appear random due to wear-leveling in the controller.

alexisread• 2 years ago

I'd guess yes, and you get the SQL goodness to boot :)

Sounds like you have an encoding issue somewhere, windows has it's own charset - Windows-1252, so I'd vet all your libs that touch the file (including eg. .Net libs etc). If one of them defaults to that encoding you may get it either mislabelling the file encoding, or adding in null after each append etc.

SQLite is tested cross-platform so 100% the file will be cross-platform compatible.

dist-epoch4• 2 years ago

A lot of your users encountering file corruption on Windows either means that somehow your users are much more likely to have broken hardware, or more realistically that you have a bug in your code/libraries.

bryanlarsen4• 2 years ago

I think omoikane pinpointed the primary cause, hard power loss. I definitely do some writing when receiving the shutdown signal, and I bet some people initiate a normal shutdown but remove power before it finishes. I think trying to avoid writes upon receipt of a shutdown signal will help.

o11c• 2 years ago

Normally, there are 3 notable things that are unique about Windows files:

* It's standard for "security" software to be installed and hook into the filesystem. Normally this only means giving up on several orders of magnitude of performance, but as this incident reminds us: security software is indistinguishable from malware.

* The system has a very long history of relying on exclusive access to files. This is one of the major reasons for Windows needing to restart so many times, but also leads to a lot of programs being badly written, among other things making false assumptions regarding atomicity in the presence of system failure.

* There's a difference between text files and binary files, and thus a whole pain of adding/removing \r before \n (which among other things means byte counts are wrong and checksums mismatching).

neffy• 2 years ago

There's a claim over on Mastodon from Kevin Beaumont that the file is different on every customer he´s received the file from.

https://cyberplace.social/@GossiTheDog/112812454405913406

(scroll down a little)

drewg123• 2 years ago

I thought windows required all kernel modules to be signed..? If there are multiple corrupt copies, rather than just some test escape, how could they have passed the signature verification and been loaded by the kernel?

dist-epoch4• 2 years ago

This is not even a valid executable.

Most likely is not loaded as a driver binary, but instead is some data file used by the CrowdStrike driver.

millero• 2 years ago

Yes, this fits in with what I heard on the grapevine about this bug from a friend who knows someone working for Crowdstrike. The bug had been sitting there in the kernel driver for years before being triggered by this flawed data, which actually was added in a post-processing step of the configuration update - after it had been tested but before being copied to their update servers for clients to obtain.

Apparently, Crowdstrike's test setup was fine for this configuration data itself, but they didn't catch it before it was sent out in production, as they were testing the wrong thing. Hopefully they own up to this, and explain what they're going to do to prevent another global-impact process failure, in whatever post-mortem writeup they may release.

finaard2• 2 years ago

[flagged]

dgfitz2• 2 years ago

Hmm, I post-process autonomous vehicle logs probably daily.

Why is this stupid? It’s pretty useful to see a graph of coolant temp vs ambient temp vs motor speed vs roll/pitch.

I must be especially stupid I suppose. Nuts.

dist-epoch4• 2 years ago

"We need to ship this by Friday. Just add a quick post-processing step, and we'll fix it next week properly" - how these things tend to happen.

arp2426• 2 years ago

"I heard on the grapevine from a friend who knows someone working for Crowdstrike" is perhaps not the most reliable source of information, due to the game of telephone effect if nothing else.

And post-processing can mean many things. Could be something relatively simple such as "testing passed, so lets mark the file with a version number and release it".

swat535• 2 years ago

I think Crowdstrike is due for more than just "owning up to it". I sincerely hope for serious investigation, fines and legal pursuits despite having "limited liabilities" in their agreements.

Seriously, I don't even know how to do the math on the amount of damage this caused (not including the TIME wasted for businesses as well as ordinary people, for instance those taking flights)

There has to be consequences for this kind of negligence

machine_coffee2• 2 years ago

At last an explanation that makes a bit of sense to me.

Hopefully they own up to this, and explain what they're going to do to prevent another global-impact process failure

They probably needn't bother, every competent sysadmin from Greenland to New Zealand is probably disabling the autoupdate feature right now, firewalling it off and hatching a plan to get the product off their server estate ASAP.

Marketing budgets for competing product are going to get a bump this quarter probably.

MilStdJunkie4• 2 years ago

Holy smokes. I'm no programmer, but I've built out bazillions of publishing/conversion/analysis systems, and null purge is pretty much the first thing that happens, every time. x00 breaks virtually everything just by existing - like, seriously, markup with one of these damn things will make the rest of the system choke and die as soon as it looks at it. Numpy? Pytorch? XSL? Doesn't matter. cough cough cough GACK

And my systems are all halfass, and I don't really know what I'm doing. I can't imagine actual real professionals letting that moulder its way downstream. Maybe their stuff is just way more complex and amazing than I can possibly imagine.

wormlord• 2 years ago

Not a C programmer, why is 0x00 so bad? It's the string terminator character right?

bagful• 2 years ago

Indeed, '\0' is the sentinel character for uncounted strings in C, and even if your own counted string implementation is “null-byte clean”, aspects of the underlying system may not be (Windows and Unix filesystems forbid embedded null characters, for example).

tedunangst• 2 years ago

It's a byte like any other. You're more likely to see big files full of 0x0 than 0x1, but it's really not so different.

hawski2• 2 years ago

Binary files are full of null bytes it is one of the main criteria of binary file recognition. Also large swaths of null bytes are also common, common enough we have sparse files - files with holes in them. Those holes are all zeroes, but are not allocated in the file system. For an easy example think about a disk image.

MilStdJunkie4• 2 years ago

Ah, I'm a dummy. Of course these would be binaries getting chucked around.

j-wags• 2 years ago

It's possible that these aren't the original file contents, but rather the result of a manual attempt to stop the bleeding.

Someone may have hoped that overwriting the bad file with an all-0 file of the correct size would make the update benign.

Or following the "QA was bypassed because there was a critical vulnerability" hypothesis, stopping distribution of the real patch may be an attempt to reduce access to the real data and slow reverse-engineering of the vulnerability.

0cf8612b2e1e2• 2 years ago

On the plus side of this disaster, I am holding out some pico-sized hope that maybe organizations will rethink kernel level access. No, random gaming company, you are not good enough to write kernel level anti cheat software.

majormajor• 2 years ago

I can't imagine gaming software being affected at all, unless MS does a ton of cracking down (and would still probably give hooks for gaming since they have gaming companies in their umbrella).

No corporate org is gonna bat an eye at Riot's anti-cheat practices, because they aren't installing LoL on their line of business machines anyway.

InitialLastName2• 2 years ago

Right, MS just paid $75e9 for a company whose main products are competitive multiplayer games. They are never going to be incentivized to compromise that sector by limiting what anti-cheat they can do.

minetest2048• 2 years ago

Until the malware bring their own compromised signed anti cheat driver on their own, like what happened with Genshin Impact anti cheat mhyprot2

tgsovlerkhgsel• 2 years ago

because they aren't installing LoL on their line of business machines anyway

But if their business is incompatible with strict software whitelisting, their employees might...

pvillano• 2 years ago

imo anti-cheat should mostly be server-side behavior based

gruez• 2 years ago

How are you going to catch wallhackers that aren't blatantly obvious?

frizlab• 2 years ago

Unless I’m mistaken on macOS at least kernel access is just not possible, so at least there’s that.

heraldgeezer3• 2 years ago

And Valorant, CS2, CoD dont run on macOS due to anti cheat :)

pityJuke• 2 years ago

The problem you’re fighting is cheat customers who go “random kernel-level driver? no problem!”

hn_throwaway_992• 2 years ago

On a related note, I don't think that it's a coincidence that 2 of the largest tech meltdowns in history (this one and the SolarWinds hack from a few years ago) were both the result of "security software" run amok. (Also sad that both of these companies are based in Austin, which certainly gives Austin's tech scene a black eye).

IMO, I think a root cause issue is that the "hacker types" who are most likely to want to start security software companies are also the least likely to want to implement the "boring" pieces of a process-oriented culture. For example, I can't speak so much for CrowdStrike, but it came out the SolarWinds had an egregiously bad security culture at their company. When the root cause comes out about this issue dollars-to-donuts it was just a fast and loose deployment process.

koliber2• 2 years ago

Don't forget heartbleed, a vulnerability in OpenSSL, the software that secures pretty much everything.

Lennie2• 2 years ago

I don't see that as a business problem directly.

But I see it as the XKCD 2347 problem.

NegativeK• 2 years ago

Alternate hypothesis that's not necessarily mutually exclusive: security software tends to need significant and widespread access. That means that fuckups and compromises tend to be more impactful.

hn_throwaway_992• 2 years ago

100% agree with that. The thing that baffles me a bit, then, is that if you are writing software that is so critical and can have such a catastrophic impact when things go wrong, that you double and triple check everything you do - what you DON'T do is use the same level of care you may use with some social media CRUD app (move fast and break things and all that...)

To emphasize, I'm really just thinking about the bad practices that were reported after the SolarWinds hack (password of "solarwinds123" and a bunch of other insider reports), so I can't say that totally applies to CrowdStrike, but in general I don't feel like these companies that can have such a catastrophic impact take appropriate care of their responsibilities.

compacct27• 2 years ago

The Austin tech culture is…interesting. I stopped trying to find a job here and went remote Bay Area, and talking to tech workers in the area gave me the impression it’s a mix of slacker culture and hype chasing. After moving back here, tech talent seems like a game of telephone, and we’re several jumps past the original.

When I heard CrowdStrike was here, it just kinda made sense

moandcompany2• 2 years ago

Crowdstrike was originally founded and headquartered in Irvine, CA (Southern California). In those days, most of its engineering organization was either remote/WFH or in Irvine, CA.

As they got larger, they added a Sunnyvale office, and later moved the official headquarters to Austin, TX.

They've also been expanding their engineering operations overseas which likely includes offshoring in the last few years.

nullify88• 2 years ago

They bought out Humio in Aarhus, Denmark. Now Falcon Logscale.

ajsnigrutin• 2 years ago

Security software needs kernel level access.. if something breaks, you get boot loops and crashes.

Most other software doesn't need that low level of access, and even if it crashes, it doesn't take the whole system with it, and a quick, automated upgrade process is possible.

rahkiin2• 2 years ago

Security software needs kernel level access.. *on Windows. macOS has an Endpoint Security userland extension api

cedws• 2 years ago

the "hacker types" who are most likely to want to start security software companies are also the least likely to want to implement the "boring" pieces of a process-oriented culture

I disagree, security companies suffer from "too big to fail" syndrome where the money comes easy because they have customers who want to check a box. Security is NOT a product you pay for, it's a culture that takes active effort and hard work to embed from day one. There's no product on the market that can provide security, only products to point a finger at when things go wrong.

Andrex• 2 years ago

The market is crying for some kind of '10s "agile hype" equivalent for security evangelism and processes.

worstspotgain2• 2 years ago

The industry's track record is indeed quite peculiar. Just off the top of my head:

- CrowdStrike

- SolarWinds

- Kaspersky

- https://en.wikipedia.org/wiki/John_McAfee

I'm sure I'm forgetting a few. Maybe it's the same self-selection scheme that afflicts the legal cannabis industry?

heraldgeezer3• 2 years ago

I would add Kaseya VSA Ransomware attack in 2021 to that list.

OutOfHere2• 2 years ago

This looks like a test file that got deployed. Perhaps a QA test was newly added which ran and overwrote the build. This is all I can think of.

OutOfHere2• 2 years ago

This could happen if there was a test for the above QA test. Such a meta test would necessarily have to write a test file to the same path as the build path. Unfortunately, the meta tests could have got incorporated into the same suite as the regular tests. Someone then could have decided to activate a test order randomizer for fun. The random order finally hit the anti-jackpot.

markus_zhang• 2 years ago

I'm starting to think that the timing (Friday) and the scale as well as other things (like this finding) might -- just might point to a bad actor.

We will probably have to wait for CS' own report.

breadwinner4• 2 years ago

I blame Microsoft. Why? Because they rely on third parties to fill in their gaps. When I buy a Mac it already has drivers for my printers, but not if I buy a Windows PC. Some of these printers drivers are 250 MB, which is a crazy size for a driver. If it is more than a few 100 KB it means the manufacturer does not know how to make a driver software. Microsoft should make it unnecessary to rely on crappy third party software so much.

luuurker3• 2 years ago

CrowdStrike's mess up is CrowdStrike's fault, not Microsoft's. We might not like the way Windows works, but it usually works fine and more restrictive systems also have downsides. In any case, it was CrowdStrike who dropped the ball and created this mess.

I don't like what Microsoft is doing with Windows and only use it for gaming (I'm glad Linux is becoming a good option for that), so I'm far from being a "Microsoft fan", but Windows is very good at installing the software needed. Plug a GPU, mouse, etc, from any well known brand and it should work without you doing much.

I didn't have to install anything on my Windows PC (or my MBP) last time I bought a new printer (Epson). The option to let Windows install the drivers needed is enabled though... some people disable that.

breadwinner4• 2 years ago

CrowdStrike's mess up is CrowdStrike's fault, not Microsoft's.

Disagree. It is everyone's fault. It is CrowdStrike's fault for not testing their product. It is Microsoft's fault for allowing CrowdStrike to mess with kernel and not vetting such critical third parties. It is the end customers' fault for installing crapware and not vetting the vendor.

taftster2• 2 years ago

Windows should NEVER blue screen for third-party software. This is definitely as much Microsoft's fault as anyone's.

Maybe Microsoft doesn't offer an ABI or whatever other access is needed for the CS module. So there's some work that needs to be done on the kernel to enable the functionality to run in user space. Third-party libraries should not have privileged access to the kernel internals.

In this case, the CloudStrike kernel module attempted to read a bogus data file and crashed. The Windows kernel should detect the module crash, inform you in some meaningful way, but continue to boot. Maybe even in some limited execution model, but not a blue screen.

CloudStrike should have tested better. A LOT better. Multiple layers of tests better. They should most definitely take serious their obligations to the millions of customers that are affected by this. But Microsoft needs to own some of the responsibility too, by allowing a 3rd party kernel module to even be able to bring down its system. Just because the program was signed, doesn't mean when it crashes, Windows should crash with it.

Alghranokk• 2 years ago

I think this is unfair; m$ does provide perfectly usable generic printer drivers, as long as you only use basic universal features. The problem is that the printer producers each want to provide a host of features on top of that, each in their own proprietary way, with post print hole-punching, 5 different paper trays, user boxes, 4 different options for duplex printing.

Also, label printers, why the heck does zebra only do EPL or ZPL? Why not pcl6 or PS like the rest of the universe?

The point is that printers are bullshit. Nobody knows how they work, and assuming that microsoft should just figure it out on its own is at least in my opinion, unreasonable.

breadwinner4• 2 years ago

What Windows was known for in the 1990s, is good quality 1st party drivers. Then after Windows achieved monopoly status they shifted driver responsibility to device manufacturers. I have never had to install a third party driver on a Mac, but on Windows I do. If Apple can do it Microsoft can too.

mardifoufs• 2 years ago

Which printer did you try it with? I've never had issues with printing out of the box with windows or mac. At least not for the past 5 years.

Also, I'm glad Microsoft doesn't provide an easy way to get what is essentially complete control over a machine, and every single event/connection/process that it has.

cmrdporcupine• 2 years ago

Is there not responsibility at some level as well to Microsoft for having a kernel which even loaded this? Not just because of the apparent corruption, but also ... it was, I heard.. signed and given a bit of an MS blessing.

This crap shouldn't be run in kernel space. But putting that aside, we need kernels that will be resilient to and reject this stuff.

ale422• 2 years ago

The thing is that, despite the file has a confusing .sys extension, it's not the driver, but rather a data file loaded by the Crowdstrike driver.

ThinkBeat• 2 years ago

Maybe Crowdstrike has adopted the modern ethos Move fast and break things With continuous integration we ship a thousand times a day. fuck QA.

motohagiography2• 2 years ago

conspiracy prediction: I don't think CS will give a complete public RCA on it, but I do think the impact and crisis will be a pretext for granting new internet governance powers, maybe via EO, or a co-ordinated international response via the UN/ITU and the EU.

cherryteastain• 2 years ago

EU recently passed a law in this domain: https://www.eiopa.europa.eu/digital-operational-resilience-a...

motohagiography2• 2 years ago

DORA is coincidental, but worth watching as a potential source of authority under some partisan disinformation initiatives. security in general is governance, where previously it had a high bar for competence to practice, and you can see it getting stacked in institutions of people without practitioner competence who can be relied on to defer to narrative.

as soon as security can come fully uncoupled from physical competence and the factual constraints of engineering, it will get subsumed into the managerialist hivemind as a rhetorical lever on narrative actuated cogs, and become as meaningless as everything else they do in service to administrative power for its own sake. I have a real problem with these people. :)

kragen7• 2 years ago

this seems like the second or third test file any qa person would have tried, after an empty file and maybe a minimal valid file. the level of pervasive incompetence implied here is staggering

in a market where companies compete by impressing nontechnical upper management with presentations, it should be no surprise that technically competent companies have no advantage over incompetent ones

i recently read through the craig wright decision https://www.judiciary.uk/judgments/copa-v-wright/ (the guy who fraudulently claimed to be satoshi nakamoto) and he lacked even the most basic technical competence in the fields where he was supposedly a world-class specialist (decompiling malware to c); he didn't know what 'unsigned' meant when questioned on the witness stand. he'd been doing infosec work for big companies going back to the 90s. he'd apparently been tricking people with technobabble and rigged demos and forged documents for his entire career

george kurtz, ceo and founder of crowdstrike, was the cto of mcafee when they did the exact same thing 14 years ago: https://old.reddit.com/r/sysadmin/comments/1e78l0g/can_crowd... https://en.wikipedia.org/wiki/George_Kurtz

crowdstrike itself caused the same problem on debian stable three months ago: https://old.reddit.com/r/debian/comments/1c8db7l/linuximage6...

it's horrifying that pci compliance regulations have injected crowdstrike (and antivirus) into virtually every aspect of today's it infrastructure

GardenLetter27• 2 years ago

Also ironic that the compliance ended up introducing the biggest vulnerability as a massive single point of failure.

But that's government regulation for you.

kragen7• 2 years ago

pci-dss is not a government agency but it might as well be; it's a collusion between visa, mastercard, american express, discover, and jcb to prevent them from having different data security standards (and therefore being able to compete on security)

acdha• 2 years ago

But that's government regulation for you.

You misspelled “private sector”. Use of endpoint monitoring software is coming out of private auditing companies driven by things like PCI or insurers’ requirements – almost nobody wants to pay for highly-skilled security people so they’re outsourcing it to the big auditing companies and checklists so that if they get sued they can say they were following industry practices and the audit firms okayed it.

babypuncher• 2 years ago

this had nothing to do with government regulation, thank private sector insurance companies.

moandcompany2• 2 years ago

It's definitely ironic, and compatible with the security engineering world joke that the most secure system is one that cannot be accessed or used at all.

I suppose one way to "stop breaches" is to shut down every host entirely.

In the military world, there is a concept of an "Alpha Strike" which generally relates to a fast-enough and strong-enough first-strike that is sufficient to disable the adversary's ability to respond or fight back (e.g. taking down an entire fleet at once). Perhaps people that have been burned by this event will start calling it a Crowdstrike.

phatfish• 2 years ago

It seems government IT systems in general faired pretty well the last 12 hrs, but loads of large private companies were effectively taken offline, so there's that.

bandyaboot• 2 years ago

What’s frustrating is that after falsely attributing pci to government regulation and being corrected, you’re probably not going to re-examine the thought process that led you to that false belief.

worstspotgain2• 2 years ago

I don't mean to sound conspiratorial, but it's a little early to rule out malfeasance just because of Hanlon's Razor just yet. Most fuckups are not on a ridonkulous global scale. This is looking like the biggest one to date, the Y2K that wasn't.

martin-t2• 2 years ago

We as a society need to start punishing incompetence the same way we punish malice.

Of course, we also need to first start punishing individuals for intentionally causing harm through their decisions even if the harm was caused indirectly through other people. Power allows people to distance themselves from the act. Distance should not affect the punishment.

worik4• 2 years ago

We as a society need to start punishing incompetence the same way we punish malice.

Yes

But competence is marketed

The trade names like "Crowdstrike" and "Microsoft "

yourapostasy• 2 years ago

Of course, we also need to first start punishing individuals for intentionally causing harm through their decisions even if the harm was caused indirectly through other people.

Legal teams are way ahead of you here, by design. They’ve learned to factor out responsibility in such a diffuse manner the “indirectly” term loses nearly all its power the closer one gets to executive ranks and is a fig leaf when you reach C-levels and BoD levels. We couldn’t feasibly operate behemoth corporations otherwise in litigious environments.

I personally suspect we cannot enshrine earnestness and character as metrics and KPI’s, even indirectly. From personal experience selling into many client organizations and being able to watch them make some sausage, this is where leadership over managing becomes a key distinction, and organizational culture over a long period measured in generations of decisions and actions implemented by a group with mean workforce tenure measured in decades becomes one of the few reliable signals. Would love to hear others’ experiences and observations.

chuckadams3• 2 years ago

So which individual engineer are you locking up and telling their kids that they don't get to see their mommy or daddy again?

intentionally causing harm

Maybe we need to criminalize groundless accusations too.

worik4• 2 years ago

george kurtz, ceo and founder of crowdstrike, was the cto of mcafee when they did the exact same thing 14 years ago: https://old.reddit.com/r/sysadmin/comments/1e78l0g/can_crowd...

I find it amusing that the people commenting on that link are offended this called a "Microsoft " outage, when it is "Crowdstrike's fault".

This is just as much a Microsoft failure.

This is even more, another industry failure

How many times does this have to happen before we get some industry reform that lets us do our jobs and build the secure reliable systems we have spent seven decades researching?

1988 all over again again again

TeMPOraL4• 2 years ago

It's simple: the failure is not specific to the OS.

Crowdstrike runs on MacOS and Linux workstations too. And it's just as dangerous there; the big thread has stories of Crowdstrike breaking Linux systems in the past months.

Crowdstrike isn't needed by/for Windows, it's mandated by corporate and government bureaucracies, where it serves as a tool of employee control and a compliance checkbox to check.

That's why it makes no sense to blame Microsoft. If the world run on Linux, ceteris paribus, Crowdstrike would be on those machines too, and would fuck them up just as bad globally.

kragen7• 2 years ago

the reforms you're hoping for are not going to happen until the countries with bad infosec get conquered by the countries with good infosec. that is going to be much worse than you can possibly imagine

it's not microsoft's fault at all; crowdstrike caused the same problem on debian systems three months ago. the only way it could be microsoft's fault is if letting users install kernel drivers is microsoft's fault

baxtr• 2 years ago

I think the worst part of the incident is that state actors now have a clear blueprint for a large scale infrastructure attack.

IAmNotACellist• 2 years ago

I can think of a lot better things to put in a kernel-level driver installed on every critical computer ever than a bunch of 0s.

Guthur• 2 years ago

You're assuming it wasn't an attack.

Just the same week Kaspersky gets kicked from the us market...

emporas2• 2 years ago

he lacked even the most basic technical competence in the fields where he was supposedly a world-class specialist

Pretty ironic Craig also suggests to put digital certificates on the blockchain instead of them issued by a computer or a cluster of computers, in one place, which makes them a target. A peer to peer network is much more resilient to attacks.

Was the problem, or a part of the problem, that the root certificate of all computers was replaced by CrowdStrike? I didn't follow that part closely. If it was, then certificates registered on the blockchain might be a solution.

I'm totally not an expert in cryptography, but that seems plausible.

As a side note, it is only when i heard Craig's Wright explanation of the blockchain that the bitcoin whitepaper started making a lot of sense. He may not be the greatest of coders, but are mathematicians useless in security?

kragen7• 2 years ago

none of your comment makes any sense. possibly it was generated by a large language model with no understanding of what is being talked about

olliej• 2 years ago

We can argue all we want about CI infrastructure, manual testing, test nets/deployment, staged deployment.

All of that is secondary: they wrote and shipped code that blindly loaded and tried to parse content from the network, and crashed when that failed. In kernel mode.

Honestly it’s probably good that this happened, because presumably someone malicious could use this level of broken logic to compromise kernel space.

Certainly the trust they put in the safety of parsing content downloaded from the internet makes me wonder about the correctness of their code for reading data from userspace.

bb88• 2 years ago

We've had security software in the past break software compilation in this method by replacing entire files with zeros. I'm not saying this is the case, but it wouldn't surprise me if it were.

Basically the linker couldn't open the file on windows (because it was locked by another process scanning it), and didn't error. Just replaced the object code to be linked with zeros.

People couldn't figure out what was wrong until they opened a debugger and saw large chunks of object code replaced with zeros.

heywire• 2 years ago

The number of times I encounter issues with Visual Studio not being able to write to the file I just compiled, ugh... We've even got an mt.exe wrapper that adds retries because AV likes to grab hold of newly created binaries right around the time mt.exe needs to write to it. Even our CI build process ends up failing randomly due to this.

slashdave• 2 years ago

I don't get it. Shouldn't the file have a standard format, with a required header and checksum (among other things), that the driver checks before executing?

nikau2• 2 years ago

you would think that...

fhub• 2 years ago

Anytime critical infrastructure goes down I always have a fleeting thought back to “Spy Game” movie where the CIA cut power to part of a Chinese city to help with a prison escape.

Thaxll2• 2 years ago

I'm not versed enough into windows loading dll / driver, but isn't the caller able to handle that situation? Or windows itself? Does loading an empty file driver can be handled in a way that it does not make the OS crash?

yamumsahoe• 2 years ago

thats a lot of prod to test in.

jmspring• 2 years ago

Poor testing. But we also need to stop CISOs, etc doing "checkbox compliance" and installing every 3rd party thing on employee laptops. My prior employer, there were literally 13 things installed for "laptop security" - 1/2 of them overlapped. Developers had the same policy as an AE and as a Sales Engineer as well as an HR person. Crowdstrike was one of the worst. Updating third party packages in go was 30-40% faster in an emulated arm64 VM (qemu) - virtualized disk / disk just a large file - on an Intel MBP compared to doing the same operation on the native system in OSX.

snowwrestler2• 2 years ago

I have flagged this because it is wrong, and there is no other way to kick it off the front page.

Finding a file full of zeroes on a broken computer does not mean it was shipped as all zeroes!

https://x.com/craiu/status/1814339965347610863

https://x.com/cyb3rops/status/1814329155833516492

execveat• 2 years ago

CrowdStrike does this trick where it replaces the file (being transferred over a network socket) with zeroes if it matches the malware signature. Assuming that these are the malware signature files themselves, a match wouldn't be surprising.

bombcar• 2 years ago

CrowStrike foot gunning itself would be amusing, if expected.

tsavo• 2 years ago

This actually makes the most sense, and would help explain how the error didn't occur during testing (in good faith, I assume it was tested).

In testing, the dev may have worked from their primary to deploy the update to a series of secondary drives, then sequential performed a test boot from each secondary drive configured for each supported OS version. A shortcut/quick way to test that would've bypassed how their product updates in customer environments, also bypassing checks their software may have performed (in this case, overwriting their own file's contents).

dang• 2 years ago

Ah thanks. I've made the title questionable now.

snowwrestler2• 2 years ago

Finally confirmed by CrowdStrike themselves:

https://www.crowdstrike.com/blog/tech-analysis-channel-file-...

keketi• 2 years ago

Found this post on 4chan's technology board:

What actually happened was two-folded:

Step 1: CSAgent.sys is a certified durgasoft kernel driver that parses clownstrike virus definition files

Step 2: Clownstrike never fixed cases for malformed virus definition files that could trigger an invalid memory access in CSAgent.sys

Step 3: Clownstrike ships the buggy CSAgent.sys and it works for months without problems on customer computers with good virus definition files

Step 4: For some reason the webserver serving virus definition files to all endpoints started serving malformed virus definition files, sometimes they were all blank or filled with random bytes

Step 5: All clownstrike updaters now download malformed LE NEXT GEN APT PROTECTION CLOUD AI LIVE UPDATES into C:\Windows\System32\drivers\clownstrike

Step 6: CSAgent.sys reloads virus definition files

Step 7: CSAgent.sys parses them and crashes with PAGE_FAULT_IN_NONPAGED_AREA (in kernel that means memory at an oopsie address was accessed)

Step 8: Computer BSOD and reboots

Step 9: CSAgent.sys loads virus definition files

Step 10: Goto Step 7

The kernel driver was a ticking timebomb just waiting for a bug in the CDN. I think it was some funny caching bug with le cloud http loadbalancer. Users reported that their C-00000291-00000000-00000032.sys contained random shit and actual real files that were a completely different part of the software, like US localization files.

You can see it in the diff between CSAgent.sys_18511.sys and CSAgent.sys_18513.sys, they changed size checks and increased sizes of buffers so that future malformed virus definition files wouldnt crash.

denisdamico2• 2 years ago

Step 4. If this has occurred so many times, it is a hacker attack, targeting the poorly written Kernel driver. (But hey, right now at this election moment, with Trump's beloved company. It could be a show of strength)

justinclift2• 2 years ago

For some reason the webserver serving virus definition files to all endpoints started serving malformed virus definition files, sometimes they were all blank or filled with random bytes

That sounds like some virus protection on the server was stopping the reads from disk, and instead of throwing an error it was providing 0's for the output data instead.

It'll be funny if there's actually some antivirus package that caused it. ;)

denisdamico2• 2 years ago

I assume the Kernel driver itself has been signed, tested and verified by MS. The channel update files would be the responsibility of the CS and contain most of the information, including urgent (untested) information. This CS kernel driver is there to inspect, collect everything, receive CS settings and commands, and act in privileged mode. It's not a so complex system, the intelligence is in CS, that's why MS partners with them. But the code in the driver could at least do a perfect input check. Shits happens. Blaming MS: I want to believe they do extensive sandbox testing, but in this case they could do better canary deployments, staged updates and not affect worldwide.

commandersaki• 2 years ago

I wonder if this is linked to the Azure outage that happened just prior to the Crowdstrike incident.

havkom• 2 years ago

Lots of very strong opinions here what they should have done in terms of code/pipelines/etx from people who have no idea how CrownStrikes stack or build systems works and people who have no clue how to write that type of software themselves (including Windows kerneln modules).

ExoticPearTree• 2 years ago

So basically they don't do input validation. Any file signed (maybe) by CS is treated as valid and the kernel driver loads it. Wow.

I'm curious how this scenario, malformed definitions file signed by CS itself is not verified in any way on the endpoint. Like a try {} catch something...

I mean, the failure is spectacular in its simplicity.

kmxr3• 2 years ago

I am going to take a guess on how the CrowdStrike event might have happened. This is based on two (2) observations: (1) CS said they have always followed the same process in the past without issues (2) (personal observation) When certain NAS drives get full and you modify a file, it ends up with all NULL bytes. Here's my guess on how their process was: Typically, when their content file is ready, they push it to their farm of QA test machines for testing and also to this NAS drive that's ready for deployment. And when all of their test deployments are successful, they press this BIG RED button that picks up this content file from the NAS drive for global deployment. During their prior deployments, this NAS drive had plenty of free space and deployments went well. Now, when the NAS was full, upon copy, it produces a file with all NULL bytes. In summary, the content file they tested with was not the file that got globally deployed.

The CrowdStrike file that broke everything was full of null characters?

Related Stories