CrowdStrike accepting the PwnieAwards for "most epic fail" at defcon

Did you click through to the video? Because the acceptance speech seemed to show the opposite of hubris to me. Specifically in owning up to the mistake, and using the award as a reminder to do better in the future.

tgsovlerkhgsel4• 2 years ago

Refusing to show up generates the same, if not more, negative PR without the opportunity to show humility and promise to do better.

nonrandomstring• 2 years ago

Comes a point when some people just have an "ethical breakdown (breakthrough)". It's a positive thing. It's where recovery starts. He's owning it. There's no absolution until you throw yourself in front of the lions. At this point who cares what the PR and legal teams have to say. They'll be lucky to have a job in a few months.

I really hope he makes the most of a great opportunity to tell some truth, so that we can break the cycle of bullshit solutions causing further pain and loss in the future: Something like;

   "Thanks for the award. Well, we all knew this managed endpoint
   cybersecurity shit was never gonna fly. And on Windows? Seriously?!
   You all knew it too, but you pays yer money and takes yer chance
   for a lucky charm to keep the auditors and insurance ghouls
   away. So here we are. We all got caught with our pants round our
   ankles. It was a good racket while it lasted. Oh well... Anyone
   hiring?"

motohagiography• 2 years ago

something along the lines of, how do you reach your most influential customers all at once with a sincere message. this was the right thing to do.

anyone who makes serious decisions will see acknowledging this in front of peers was correct. it's funny how the hacker ethic of celebrating failures as lessons becomes impossible when you have a chorus angling for leverage all the time. the failure mode of most tech is catastrophic, where all the convenience you get from it disappears suddenly and randomly. I'd be mad about the lost time during the recovery and over missed flights or even health services, but managing that risk is the job.

to anyone else, next time something fails and messes up your plans or puts you in a spot, try to remember a time when you had a chance to do something well but didn't because you were thinking, "not my problem."

xyst7• 2 years ago

Probably firm they outsourced their public relations to thought this would be a good idea. It’s backfired

chongli• 2 years ago

I appreciate that we’re finding the humour in this catastrophe but what about the question of liability? I have seen a few stories on HN of the billions lost by this event but so far not much in the way of lawsuits.

What is the situation? Are the licenses so ironclad that customers have no recourse? I could understand this in the case of consumers who might suffer minor inconvenience as their home PC is out of service for a few hours/days but it seems totally unacceptable for industries to accept this level of risk exposure.

This is one of the big reasons civil engineering is considered such a serious discipline. If a bridge collapses, there’s not only financial liability but the potential for criminal liability as well. Civil engineering students have it drilled into their heads that if they behave unethically or otherwise take unacceptable risks as an engineer they face jail time for it. Is there any path for software engineers to reach this level of accountability and norms of good practice?

gjsman-10007• 2 years ago

Is there any path for software engineers to reach this level of accountability and norms of good practice?

Heck, no.

Civil engineering doesn’t change. Gravity is a constant. Physics are constants. If Rome wrote an engineering manual, it would still be quite valid today.

Imagine if we had standardized software engineering in 2003. Do you think the mandatory training about how to make safe ActiveX controls is going to save you? Do you think the mandatory training on how to safely embed a Java applet will protect your bank?

Software is too diverse, too inconsistent, and too rapidly changing to have any chance at standardization. Maybe in several decades when WHATWG hasn’t passed a single new spec into the browser.

(Edit: Also, it’s a fool’s errand, as there are literally hundreds of billions of lines of code running in production at this very moment. If you wrote an onerous engineering spec; there would not be enough programmers or lawyers on earth to rewrite and verify it all, even if given decades. This would result in Google, Apple, etc. basically getting grandfathered in while startups get the burden of following the rules - rules that China, India, and other countries happily won’t be enforcing.)

2OEH8eoCRo05• 2 years ago

I'd imagine we wouldn't have ActiveX controls in the first place.

theideaofcoffee• 2 years ago

There is nothing saying that allowing for some standardization means that we have to be stuck at 2003-levels of state of the art. And actually, yes many engineering disciplines do change, Civil engineering brings in new construction techniques, methods for non-destructive testing, improvements to materials and on and on, but it doesn't do so like the coked-up industry of software does it in such a free-for-all manner. It's a proper engineering discipline because there's the control, testing the best way to do things and rolling that out.

If we (meaning software 'engineers' and I tepidly include myself in that group) had half the self control in introducing insanity like the 10000th new javascript framework to read and write to a database like the 'proper' disciplines do, maybe it would be better because there's less churn. Why does it have to move so fast? Software is diverse and inconsistent and rapidly changing because 'the industry' (coked-out developers chasing the next big hit to their resume to level up) says it should. I just don't agree that we need that amount of change to do things that amount to mutating some data. If the techniques didn't grow beyond what was cool in 2007, or they were held there until the next thing could be evaluated and trained, but the knowledge and process around them did, perhaps we'd be in a better position. I know I certainly wouldn't mind maintaining something that was created in the last decade of the previous millennium knowing it was built with some sort of self-control and discipline in mind, and that the people working on it with me had the same mindset as well.

figassis2• 2 years ago

Physics will still be the same when your faulty software tells an airplane to dive.

BartjeD3• 2 years ago

This is so wrong

Most suspension bridges were built without a theoretical model, because didnt have one yet. Theory caught up much later.

Innovation often happens in absence of Theory.

Anechoic• 2 years ago

Civil engineering doesn’t change. Gravity is a constant. Physics are constants.

Physics may be a constant, but materials and methods are not. There is a reason why ISO/IEC/ICC/ASTM/ANSI/ASME/ASHRAE/DIN/IEEE/etc standards have specific dates associated with them.

If Rome wrote an engineering manual, it would still be quite valid today.*

Considering many engineering standards from a few years ago are no longer valid, this is almost certainly not true.

jjmarr• 2 years ago

In Canada, we have software and computer engineering programs accredited by the same entity (CEAB) that does civil engineering.

My program is more out of date (Java Server Pages, VHDL) but the school can't lower the quality of their programs. Generally, the standard learning requirements aren't on technology but principles, like learning OOP or whatever else. The CEAB audits student work from all schools in Canada to make sure it meets their requirements.

The culture itself is probably the most important part of the engineering major. They don't round up. If you fail, you fail. And I had a course in 3rd year with a 30% fail rate. Everything's mandatory, so you just have to try again over the summer.

A lot of people drop out because they can't handle the pressure. But the people that stay understand that they can't skip out on stuff they aren't good at.

8note• 2 years ago

Gravity is not constant, instead it varies by location and by height.

Bubble sort however, is always bubble sort. A similarly large portion of what engineers do with in software is constant

chestertonsgate9• 2 years ago

I hope you realize that "sowwy, there's too much code :3" will not fly with whatever government decides to regulate software after the next major cock-up. We can either grow up and set our own terms, or we can have them forced on us by bureaucrats whose last computer was an Apple II. Choose.

otterley3• 2 years ago

If Rome wrote an engineering manual, it would still be quite valid today.

“How to conduct water efficiently: first, collect a whole bunch of lead. Then construct pipes from said lead…”

2OEH8eoCRo05• 2 years ago

Is there any path for software engineers to reach this level of accountability and norms of good practice?

There is no reason that software couldn't be treated with the same care and respect. The only reason we don't is because the industry resists that sort of change. They want to move fast and break things while still calling themselves "engineers." Almost none of this resembles engineering.

wizzwizz43• 2 years ago

Software engineering, of course, presents itself as another worthy cause, but that is eyewash: if you carefully read its literature and analyse what its devotees actually do, you will discover that software engineering has accepted as its charter "How to program if you cannot.".

— Edsger Wybe Dijkstra, 1988. (EWD1036)

figassis2• 2 years ago

I’m a software engineer, with a degree, and SWE does have the same ethical principles and the same engineering process, from problem definition and requirements a the way to development lifecycle, testing, deployment indigent management, etc. none of it includes sprints and story points.

Suffice it to say most SWEs are not being hired to do actual engineering, bc the industry can’t get over the fact that just because you can update and release SW instantly doesn’t mean you should.

throwaheyy• 2 years ago

To be fair, software and technology is so magically transformative that even with warranty disclaimers like “this software comes with no warranty, and we limit our liability to the purchase price”, every company in the world still lines up to buy it. Because for them it’s effectively magic, that they cannot replicate themselves.

No individual software developer, nor corporation, is foolish enough to claim their software is free of bugs, that’s why they put the risk on the customer and the customer still signs on the dotted line and accepts the risks anyway. After all, it’s still way more profitable to have the potentially-faulty software than needing an army of clerks with pen and paper instead.

Most software has to be this way or it would be exorbitantly expensive. That’s the bargain between software developers and the companies that buy the software. Customer accepts the risks and gets to pocket the large profits that the software brings (because of the software’s low cost), because it’s better than the software developer balking at the liability, no software being written at all, and having an army of staff every airport writing out boarding passes by hand. There are only a few softwares that aren’t this way - example the software in aircraft or nuclear power plants. That software is correspondingly extremely expensive. Most customers that can, choose to accept the risks so they can have the larger profits.

belter• 2 years ago

https://zlk.com/pslra-1/crowdstrike-lawsuit-submission-form?...

https://www.sauderschelkopf.com/wp-content/uploads/2024/07/A...

MattGaiser2• 2 years ago

Are the licenses so ironclad that customers have no recourse?

Even on Hacker News, there was agreement that CrowdStrike screwed up, but then people also blamed IT staff, Microsoft (even after realizing it was a CrowdStrike issue), and the EU/regulators.

I imagine responsibility of each entity would need far more clarification than it does now.

If you want to define liability, there needs to be a clear line saying who is responsible for what. That doesn’t currently exist in software.

There are also considering how people respond to risk.

Consider how sesame regulation led to most bread having sesame deliberately put into it. Industry responded by guaranteeing contamination.

Crowdstrike and endpoint security firms might respond by saying that only Windows and Mac devices can be secured. Or Microsoft may say that only their solution can provide the requisite security.

miki123211• 2 years ago

Civil engineering mostly requires you to have a government-verified certificate and to work in the country your infrastructure will be deployed in.

Software engineering doesn't, and that makes criminal prosecutions that much harder. There's no path to making it happen.

Financial liability for the company in question? Sure, that's probably doable. "Piercing the corporate veil" and punishing the executives who signed off on it? Harder but not impossible. Punishing the engineer who wrote that code, and who lives in a country with no such laws? Won't happen.

WarOnPrivacy4• 2 years ago

Civil engineering mostly requires you to have a government-verified certificate and to work in the country your infrastructure will be deployed in.

It's a relatively small (and sharply defined) pool of people who can be called a civil engineer.

Are we saying we want to segment software engineering (from coding) - the same way civil engineering is segmented from construction?

Otherwise we're talking about placing specialist liability upon a non-specialist group. This seems unethical.

JumpCrisscross20• 2 years ago

If a bridge collapses, there’s not only financial liability but the potential for criminal liability as well

If a bridge collapses people die. To my knowledge, nobody died or was put in mortal peril as a result of the Crowdstrike debacle.

rightbyte2• 2 years ago

The deaths if any where probably indirect. E.g. ambulances not turning up in time etc. due to paper and pen fallbacks.

With all the hospitals victim of the attack, I would be surprised if the amount of patients that died are zero.

SkiFire13• 2 years ago

Hospitals were affected too, I don't think it's that far fetched to think some people died, or at least some could not have been saved due to this incident.

bt1a3• 2 years ago

I am positive that people in hospitals died as a direct result of this incident.

trentnix2• 2 years ago

Hospitals and doctor’s offices were paralyzed by the outage. Transplant organs are often delivered by couriers on commercial flights. Many pharmacies were unable to fulfill prescriptions.

It wasn’t just vacation travelers that were affected by Crowdstrike’s incompetence.

chestertonsgate9• 2 years ago

Small reminder that the law already has a way of deciding liability for damages, and you don't have to directly drop a bridge on someone to get in trouble.

LordKeren3• 2 years ago

There is recourse, just not for normal people, as you eluded to. Companies are and will be continuing to sue crowdstrike, and based on the papers that crowdstrike has posted, the impacted companies are extremely likely to be successful. It seems overwhelmingly likely that the companies are going to be able to convince a judge/jury/arbiter that crowdstrike acted grossly negligent and very plainly caused both direct losses and indirect reputational harm to the companies.

I’m not sure crowdstrike will even fight it, to be honest. I would assume most of this is going to be settled out of court and we will see crowdstrike crumble in the coming years.

JumpCrisscross20• 2 years ago

not sure crowdstrike will even fight it

To my knowledge only Delta is suing and CrowdStrike is kicking and screaming about it [1].

[1] https://www.cnn.com/2024/08/05/business/crowdstrike-fires-ba...

dredmorbius6• 2 years ago

NB: alluded, not eluded.

<https://dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=Al...>

<https://dict.org/bin/Dict?Form=Dict2&Database=gcide&Query=el...>

jedberg3• 2 years ago

Delta threatened to sue them for their $500M loss. Crowdstrike replied (publicaly) pointing out that their contract limits Crowdstrike's liability to single digit millions.

Then then gave them a list of things they would seek in discovery, such as their backup plans, failover plans, testing schedules and results, when their last backup recover exercise was, etc.

Basically, they said, "if you sue us, we will dig so deep into your IT practices that it will be more embarrassing for you than us and show that you were at fault".

JumpCrisscross20• 2 years ago

they said, "if you sue us, we will dig so deep into your IT practices that it will be more embarrassing for you than us and show that you were at fault"

But CrowdStrike said this publicly. If they’d privately relayed it to Delta, it would have been genuine. By performatively relaying it, however, it seems they’re pre-managing optics around the expected suit.

reaperducer• 2 years ago

Delta threatened to sue them for their $500M loss. Crowdstrike replied (publicaly) pointing out that their contract limits Crowdstrike's liability to single digit millions.

Delta's move seems like an attempt to assuage shareholders and help the C.E.O. save face.

Crowdstrike shouldn't be afraid of Delta. Crowdstrike should be afraid of the insurance companies that have to pay out to those businesses that have coverage that includes events like this.

Even if the payout to a company is $10,000, a big insurance company may have hundreds or thousands of similar payouts to make. The insurance companies won't just let that go; and they know exactly what to look for, how to find it, and have the people, lawyers, and time to make it happen.

Crowdstrike will get its day of reckoning. It won't be today. And it probably won't be public. But the insurance companies will make sure it comes, and it's going to hurt.

trentnix2• 2 years ago

They should be providing all that information regularly to auditors anyway. If they don’t have it handy, then their IT leadership should be replaced.

mc322• 2 years ago

That’s odd. One is an internal process which has no obligation to an external party, and the other one who is specifically responsible for being liable for any repercussions due to deviating from their own SDLC process[1]they totally skipped themselves?

If I were Delta, I’d get other affected parties and together sue CrowdStrike and get all their dirty laundry out in the open.

[1] I haven’t checked but they used to list all their ISO certs, etc. Wonder if those get revoked for such glaring violations…

VirusNewbie• 2 years ago

They might find out delta does embarrassing things like not testing out of bounds array access or does global deployments without canarying.

richardw• 2 years ago

I’m not sure anything else was material given that the machines were bricked and client roll-out approaches were evaded by Crowdstrike. What client actions would have helped?

Surely someone is looking at a class action? People died. The contract can’t make that everyone else’s problem, can it?

mikeocool• 2 years ago

It really seems funny that Crowdstrike’s defense is basically “you should have been better prepared for us to knock all of your systems offline.”

It’s probably true, but seems like an odd stance to take from a PR perspective or a “selling other clients in the future” perspective.

icehawk• 2 years ago

I'd LOVE to see Crowdstrike do this. The last time I dealt with the specifics of this sort of validation testing for security software was a decade and from what I saw in the RCA Delta can just keep pointing out that whatever they had worked until Crowdstrike failed to understand that the number 20 and the number 21 are not the same:

The new IPC Template Type defined 21 input parameter fields, but the integration code that invoked the Content Interpreter with Channel File 291’s Template Instances supplied only 20 input values to match against. This parameter count mismatch evaded multiple layers of build validation and testing, as it was not discovered during the sensor release testing process, the Template Type (using a test Template Instance) stress testing or the first several successful deployments of IPC Template Instances in the field.

This combined with the lack of partitioning updates, makes me draw the conclusions they're missing table stakes WRT to validation.

otterley3• 2 years ago

That’s not the way legal process works. CrowdStrike might be permitted to conduct discovery, but that won’t entitle them to share what they might find with the public, embarrassing or otherwise. Business records and other sensitive information relating to parties in civil matters are frequently sealed.

minkles• 2 years ago

Engineering safety culture is built on piles of bodies and suffering unfortunately. I suspect in software the price of failure is mostly low enough that this motivation will never develop.

xyst7• 2 years ago

Probably in a decade or so after the AI crash. I have yet to see anything that comes close to “liability” for the digital realm.

US governments and businesses get hacked/infiltrated all the time by foreign adversaries yet we do not declare war. Maybe something happens in the dark or back channels. But we never know.

dylan6044• 2 years ago

A lot of companies have insurance on events causing them to lose sources of income. Whether that's farmers having crop insurance, big box retailers having insurance for catastrophic damage to their big box, I would assume there's something for infrastructure collapse to bring sales to $0 for the duration.

Even if everyone that was affected sued ClownStrike for 100% of their losses, it's not like ClownStrike has the revenue to cover those losses. So even if you're a fan of shutting them down, nobody recovers anything close to actual losses.

So what would you actually propose? Bug free code is pretty much impossible. Some risk is accepted by the user. Do you seriously think that software should be absolutely 100% bug free before being able to be used? How do you prove that? Of course, the follow up would be how clean is your code that you feel that's even achievable?

Rinzler892• 2 years ago

Bug free code is pretty much impossible. Some risk is accepted by the user.

This wasn't your average SW bug, it was gross negligence on behalf of Crowdstreike, who seems to not have heard of SW testing on actual systems and canary deployment. Big difference.

Yeah SW bugs happen all the time but you have to show you took some steps to prevent them, while some dev at Crowdstrike just said "whatever, it works on my machine" and directly pushed to all customer production systems on a Friday. That's the definition of gross negligence that they didn't have any processes in place to prevent something like this.

That's like a surgeon not bothering to sterilize his hands and then saying "oh well, hospital infections happen all the time".

chestertonsgate9• 2 years ago

Bug-free code is impossible. Stupid, negligent bug-free code, however, is very much doable. You just can't hire anyone who happens to be able to fog a mirror to write it.

ttymck3• 2 years ago

but so far not much in the way of lawsuits

It hasn't been that long? The situation might be that there hasn't been sufficient time to yet gather evidence to commence lawsuits.

cesarb• 2 years ago

Is there any path for software engineers to reach this level of accountability and norms of good practice?

Yes, time. Civil engineering has thousands of years of history. Software engineering is much newer, the foundations of our craft are still in flux. There have been, at least in my country, legislative proposals for licensure of system analysts, electronic computer programmers, data processing machine operators, and typists(!) since the late 1970s; these laws, if approved, would have set back the progress of software development in my country for several decades (for instance, one proposal would make "manipulation and operation of electronic processing devices or machines, including terminals (digital or visual)" exclusive to those licensed as "data processing machine operator").

Cpoll• 2 years ago

set back the progress
exclusive to those licensed

Sounds to me like it just would've made a lot of money for whatever entities give out the licenses.

On the other hand, I've read speculation on here that some countries are short on entrepreneurs entirely due to the difficulty of incorporating a small business, so maybe.

tsujamin• 2 years ago

I’m interested in what those who suffered outages as a result of crowdstrike told their insurers with respect to “QA’ing production changes”

It’d be interesting to see if anyone tries to claim the outage as some sort of insurance event only to lose out because they let Crowdstrike roll updates into a highly regulated environment without testing

crote• 2 years ago

Civil engineering students have it drilled into their heads that if they behave unethically or otherwise take unacceptable risks as an engineer they face jail time for it. Is there any path for software engineers to reach this level of accountability and norms of good practice?

The problem is that with civil engineering you're designing a physical product. Nothing is ever designed to its absolute limit, and everything is built with a healthy safety margin. You calculate a bridge to carry bumper-to-bumper freight traffic, during a hurricane, when an earthquake hits - and then add 20%. Not entirely sure about whether a beam can handle it? Just size it up! Suddenly it's a lot less critical for your calculations to be exactly accurate - if you're off by 0.5% it just doesn't matter. You made a typo on the design documents? The builder will ask for clarification if you're trying to fit a 150ft beam into a 15.0ft gap. This means a bridge collapse is pretty much guaranteed to be the result of gross negligence.

Contrast that to programming. A single "<" instead of "<=" could be the difference between totally fine and billions of dollars of damages. There isn't a single programmer on Earth who could write a 100% bug-free application of nontrivial complexity. Even the seL4 microkernel - whose whole unique selling point is the fact that it has a formal correctness proof - contains bugs! Compilers and proof checkers aren't going to complain if you ask them to do something which is obviously the wrong thing but technically possible. No sane person would accept essentially unlimited liability over even the smallest mistakes.

If we want software engineers to have accountability, we first have to find a way to separate innocent run-of-the-mill mistakes from gross negligence - and that's going to be extremely hard to formalize.

Analemma_• 2 years ago

This entire comment boils down to "we can't be held accountable because it's soooo hard you guys", which isn't even convincing to me as someone in the industry and certainly won't be to someone outside it.

wannacboatmovie2• 2 years ago

Contrast that to programming. A single "<" instead of "<=" could be the difference between totally fine and billions of dollars of damages.

I fail to see the difference between a misplaced operator and a misplaced bolt (think Hyatt walkway collapse), both of which could have catastrophic consequences. Do you think the CAD software they use to perform the calculations is allowed have bugs simply because it's software?

Maybe go back to entering code on punch cards if you're so fixated on the physical domain being the problem.

cyberax• 2 years ago

This is all true. But we _do_ have known best practices that reduce the impact of bugs.

A most trivial staged rollout would have caught this issue. And we're not talking about multi-week testing, even a few hours of testing would have been fine. Failure to do that rises to the level of gross negligence.

ang_cire3• 2 years ago

To add onto this, the Pwnie Awards also go to people who get attacked, which is something that e.g. civil engineers certainly don't get blamed for (i.e. if a terrorist blows up their bridge).

We would need a way to draw a liability line between an incident that involves a 3rd party attack, and one that doesn't, but things like SolarWinds even blur that line where there was blame on both sides. When does something become negligence, versus just the normal patching backlog that absolutely exists in every company?

And why are people aiming the gun already at software engineers, rather than management or Product Architects? SE's are the construction workers at the bridge site. Architects and Management are responsible for making, reviewing, and approving design choices. If they're trying to shift that responsibility to SEs by not doing e.g. SCA or code reviews, that's them trying to avoid liability.

Honestly, this reaction by the CEO is great for taking responsibility. Even if there's not legal liability, a lot of companies are still going to ditch CrowdStrike.

btilly2• 2 years ago

The other side of it is this. By law, a licensed civil engineer must sign off on a civil engineering project. When doing so, the engineer takes personal legal liability. But the fact that the company needs an engineer to take responsibility means that if management tries to cut too many corners, the engineer can tell them to take a hike until they are willing to do it properly.

Both sides have to go together. You have to put authority and responsibility together. In the end, we won't get better software unless programmers are given both authority AND responsibility. Right now programmers are given neither. If one programmer says no, they are just fired for another one who will say yes. Management finds one-sided disclaimers of liability to be cheaper than security. And this is not likely to change any time soon.

Unfortunately the way that these things get changed is that politicians get involved. And let me tell you, whatever solution they come up with is going to be worse for everyone than what we have now. It won't be until several rounds of disaster that there's a chance of getting an actually workable solution.

exe34• 2 years ago

did they take basic precautions like staged releases, code reviews, integration tests?

if not, then it's literally the engineer equivalent of gross negligence and they do deserve to be sued to oblivion.

NegativeK• 2 years ago

Doctors, engineers, and lawyers aren't infinitely accountable to their equivalent of bugs. Structures still fail, patients die, and lawyers lose cases despite the reality of the crime.

But they're liable when they fuck up beyond what their industry decides is acceptable. If Crowdstrike really wasn't testing the final build of their configuration files at all, then yeah -- that's obviously negligent given the potential impact and lack of customer ability to do staged rollouts. But if a software company has a bug that wasn't caught because they can't solve the halting problem, then no professional review board should fault the license holder.

we first have to find a way to separate innocent run-of-the-mill mistakes from gross negligence - and that's going to be extremely hard to formalize.

I think we just (oh god -- no sentence with a just is actually that easy) need to actually look at other professional licenses to learn how their processes work. Because they've managed to incorporate humans analyzing situations where you can't have perfect information into a real process.

But I don't think any of this will happen while software is still making absolute shit loads of money.

spott• 2 years ago

This is less complicated than you think.

Civil engineering rules, safety margins and procedures have been established through the years as people died from their absence. The practice of civil engineering is arguably millennia old.

Software is too new to have the same lessons learned and enacted into law.

The problem isn’t that software doesn’t have the kind of practices and procedures that would prevent these kinds of errors, (see the space shuttle code for example), it is that we haven’t formalized their application into law, and the “terms of service” that protects software makers has so far prevented legal case law from ensuring liability if you don’t use them.

Software engineering, compared to other engineering disciplines, has had a massive effect on the world in an incredibly short amount of time.

TeMPOraL3• 2 years ago

Nothing is ever designed to its absolute limit, and everything is built with a healthy safety margin. You calculate a bridge to carry bumper-to-bumper freight traffic, during a hurricane, when an earthquake hits - and then add 20%. Not entirely sure about whether a beam can handle it? Just size it up! Suddenly it's a lot less critical for your calculations to be exactly accurate

That may have been true a couple hundred years ago. It's not been true for a couple decades now, because budget became a constraint even more important than physics, and believe it or not, you will have to justify every dollar that goes into your safety margin. That's where the accuracy of modern techniques matter: the more accurate your calculations (and the more consistent inputs and processes builders employ), the less material you can use to get even closer to the designed safety margin. Accidentally making a bridge too safe means setting money on fire, and we can't have that.

That's the curse of progress. Better tools and techniques should allow to get more value - efficiency, safety, utility - for the same effort. Unfortunately, economic pressure makes companies opt for getting same or less[0] value for less effort. Civil engineering suffers from this just as much as software engineering does.

[0] - Eventually asymptotically approaching the minimum legal quality standard.

bsaul• 2 years ago

I like the analogy. What would the equivalent of « adding safety margins » to a piece of critical code ? Building three of them with different technologies and making sure all return the same results ?

darksim905• 2 years ago

Do people actually believe when a company says something caused billions of dollars of damage? unless you can quantify that, much like law enforcement and articulate suspicion, it's pretty useless as a metric. If you can pull something out of your ass, what does it matter?

jjav2• 2 years ago

Contrast that to programming. A single "<" instead of "<=" could be the difference between totally fine and billions of dollars of damages.

Disagree. This is true purely at the coding level, yes. Anyone could make a typo.

If you're running a company that releases software with the risk exposure of crowdstrike, you better not have a release model where that typo goes straight to production. There need to be many layers of different kinds of testing. If carefully built, now there are many layers all of which have to fail for the bug to go live. You can bring down the failure probability down to negligible levels with enough layers of validation.

find a way to separate innocent run-of-the-mill mistakes from gross negligence - and that's going to be extremely hard to formalize.

I don't think it's that hard. Not saying it is trivial, but it is well within the capability of the industry if we just focused a little bit on quality instead of 100% in profit.

Standardize models and layers of testing coverage. If you implement them all then you're not being negligent and thus should not be liable. If you decide to skip them, liable.

wannacboatmovie2• 2 years ago

Is there any path for software engineers to reach this level of accountability and norms

Potentially controversial stance here, but most software engineers are not engineers. They study computer science, which doesn't include coursework on engineering ethics among other things. I would say that by design they are less prepared to make ethical decisions and take conservative approaches.

Imagine if civil engineers had EULAs for their products. "This bridge has no warranty, implied or otherwise. Cross this bridge AT YOUR OWN RISK. This bridge shall not be used for anything safety critical etc."

lupire3• 2 years ago

How many bridges, would you say, does the average civil engineering firm deliver each year, each on only 1 day notice, in response to a surprise change in requirements due to a newly developed adversarial attack?

Crowdstrike does this constantly.

You could demand the same level of assurance from software, but in exchange, you don't get to fly, because the capacity won't be there

dakiol• 2 years ago

I appreciate that we’re finding the humour in this catastrophe but what about the question of liability?

One of the biggest and most used piece of software (the Linux kernel) comes with zero warranties. It can fail, and no one would be liable. Are we fine with that? Is the CS case different because it costs money? From an user perspective we don’t want software failing in the middle of an airplane landing, so whether the software comes from CS or github, it’s of lesser importance.

guax2• 2 years ago

I would find it more useful if liability here we're attributed to the need to purchase such draconian tools. Certifications that require it and C levels who approve it. We would be better by it.

cqqxo4zV46cp5• 2 years ago

Oh Christ. Just drop it. A by all accounts legitimate security function of a product targeted at company-owned endpoints.

Please don’t devolve this conversation into you being upset about not getting admin rights on your work computer or whatever this is about.

Any (esp. larger) org would be criminally negligent to eschew using something like CrowdStrike in order to capitulate to some nerd that thinks that they have ownership over their work equipment.

maccard3• 2 years ago

I completely agree. When I've negotiated contracts for my workplace, and we explicitly write in the contract that the vendor is responsible for XYZ, it is my understanding (and confirmed by legal, multiple times) that this means in case of XYZ going wrong, they are liable for up to the amount in the SLA, however that isn't a cap on liability in extenuating circumstances.

If this all gets brushed away, it significantly devalues the "well we pay $VENDOR to manage our user data, it's on them if they store it incorrectly" proposition, which would absolutely cause us to renegotiate.

cqqxo4zV46cp5• 2 years ago

You aren’t showing us the specific language that you’re referring to, nor do we know what a typical CrowdStrike contract looks like. You could be talking about apples and oranges here. I’ve seen both.

wcunning• 2 years ago

Software (controls) engineers at VW during the emissions scandal went to jail, engineers at GM were held liable for the ignition switch issue (not mostly in software, but still). I expect we'll eventually see some engineers/low level managers thrown under the bus with Boeing. It definitely happens, but not as frequently as it could. That said, I definitely prefer Amazon's response to the AWS East 1 outage back in 2016 -- the engineer wasn't blamed, despite the relatively simple screw up, but the processes/procedures were fixed so that it didn't happen again in the last 8 years. Crowdstrike is a little bit gray on that regard -- people should have known how bad the practice of zero testing on config updates was, but then again, I've seen some investigating saying that the initial description wasn't fully accurate, so I'm waiting for the final community after action report before I really pass judgement.

ezoe• 2 years ago

They totally deserved it.

Those who think running third-party closed-source Windows kernel driver(which parse files distributed from Internet in realtime) are good for the security, they must also accept the consequence.

I'm sick of these so-called security consultants who always insist check lists like installing proprietary close-source binary blob Linux kernel module to the system consists of otherwise mostly free softwares except for hardware drivers and think they did their job, or executives who pays a lot of money to these idiotic so-called security consultants.

oglop• 2 years ago

On this website you are asking a population that would be responsible for this, so you will likely only get answers about how hard this is to solve and how it’s not software engineers fault and how we need to understand software engineering is not civil engineering and we need to be careful with this analogy and how it’s not our fault! Don’t blame us when things go wrong, but also, give us all the money when things go right.

This is not the place for this question is what I’m saying.

barelysapient• 2 years ago

Your barber has more licensing requirements than a senior software engineer.

Regulations have not caught up with developers (yet).

omoikane• 2 years ago

Previous Pwnie Award winners for comparison:

https://en.wikipedia.org/wiki/Pwnie_Awards

rldjbpin• 2 years ago

most of the previous "most epic fail" awards are naturally dominated by microsoft.

blame shame can continue but if you believe if "production failures" such as this one is due to bad processes, then m$ definitely played a big role here.

xyst7• 2 years ago

This comes across as incredibly tone deaf. People suffered degraded medical care, billions lost in the airline industry, billions more lost in productivity, and ultimately its time that people cannot get back. Yet these clowns are accepting joke awards as if this is something to hang on your trophy wall.

This is actually a c-level executive at ClownStrike, by the way.

Michael Sentonas serves as President and is responsible for CrowdStrike’s product and go-to-market functions, including its sales, marketing, product & engineering, threat intelligence, privacy & policy, corporate development, corporate strategy and CTO teams

https://www.crowdstrike.com/about-crowdstrike/executive-team...

The whole C-level executive suite at ClownStrike needs to go. This company needs a real CTO like Jeremy Rowley. Although I suspect a good person like him would never join the ranks of ClownStrike

sigmoid10• 2 years ago

Well, this is the second time their CEO caused a major outage by pushing a flawed update for a security product. This whole thing is probably a joke to him.

hn_throwaway_992• 2 years ago

Did people actually watch the video??? I just don't understand how they think Michael Sentonas was making a joke of all this. If anything, he was acknowledging the horrible outcome of what happened.

I don't think this absolves CrowdStrike of responsibility at all, but what would you like him to do, commit harikari on stage?

xyst7• 2 years ago

Clearly, step down. Including CEO (George Kurtz)

The shit rolls down hill, starting from the c-suite. These clowns clearly cannot change the org and are blind to the issues. Keeping the same leadership means nothing will change. The fact that they even poke their head up for what is clearly a marketing/PR stunt without showing any substance shows how clueless they are.

Guy has “20 years” of experience which clearly doesn’t amount to shit. Maybe 20 years of junior experience and falling upwards.

lupire3• 2 years ago

People who write "ClownStrike" aren't contributing to discourse. Downvote and move on

mvdtnz2• 2 years ago

I watched the video. I saw this asshole executive with a huge shit eating grin on his face the entire time he gave his PR-managed speech and lapped up the applause of perhaps the stupidest audience in tech history.

Xen9• 2 years ago

I would like to have had him wear a red clown nose & walk to stage on soundtrack of The Empire Strikes Back. Nothing more, nothing less.

textlapse2• 2 years ago

This may end up in one of those court evidence videos or lawsuits - this isn’t a funny thing.

This would have been a closed moment (just a bunch of security nerds discussing something) but instead this is now freely available for the wider general public who had major grievances to lampoon them.

hn_throwaway_992• 2 years ago

This may end up in one of those court evidence videos or lawsuits - this isn’t a funny thing.

I didn't take the CrowdStrike's executive as making light of the situation, at all. If anything, I thought his speech took it seriously, acknowledged it was a major, major fuckup, and basically said he was accepting the trophy as a mark of shame and as a cautionary tale for future CrowdStrike employees.

I thought the exec accepting this was a true class act (to emphasize, saying that in no way should imply that I think it absolves CrowdStrike of responsibility, or liability, for what happened).

textlapse2• 2 years ago

Context is everything. They had every chance to own up from the day of until now. A ‘lulz haha we goofed up’ in a nerdy security conference doesn’t seem like the right place or time.

ncr1003• 2 years ago

Can make it funny, via a T-shirt:

When I use REGEXP I use it in my KERNEL CODE

Tragedy and comedy, same coin...

Brett_Riverboat3• 2 years ago

I work in IT and I happened to be the poor bastard on call when Clown Strike took out the majority of infrastructure. If it wasn't for my own personal refusal to use cloud based bullshit we would have probably been down for days instead of hours. The fact that people like my IT director saw nothing wrong with this and is taking 0 steps to negate such bullshit makes me quite worried that I will soon have to deal with some other catastrophic cloud based failure in the near future.

I keep repeating ad-nauseam "only idiots rely on other peoples computers" And I stand behind that statement 100%

xyst7• 2 years ago

Can only do so much when idiot CTOs take their advice from CTO summits, consultants with their own perverse incentives, and of course random conferences

system2• 2 years ago

I 100% agree. Additionally, I am amazed to see that people can pay outrageous fees for cloud services such as Azure VD. For the fraction of the yearly cloud budget, companies can create crazy stable, offline-capable infrastructure themselves.

cqqxo4zV46cp5• 2 years ago

You sound like a very aggressive person. I don’t think I’d like to work with you regardless of whether or not you were right. Maybe you’d get your point across better if you weren’t so aggressive about it.

Brett_Riverboat3• 2 years ago

citrin_ru2• 2 years ago

What strange that there are mangers/executives who understand why it’s bad to have a SPOF (single point of failure) in the internal infrastructure but who are OK with having a product/service from an external vendor as an SPOF. Like if you have a contract and are paying money for it means that it is created and maintained by infallible superhuman (as opposite to internal engineers they don’t trust). Such misplaced trust puzzles me.

Brett_Riverboat3• 2 years ago

My director mistakenly thinks the more it costs the better it is, he refuses to even consider anything FOSS. Before he came along we ran everything in house and we still had issues of course but down time was nearly nonexistent because we could take action immediately instead of waiting for whatever cloud service messed up today to feel like getting around to it.

And of course we now get to pay monthly for the privilege of being at someone else's mercy as opposed to before when we paid once and went on our merry way.

aa_is_op• 2 years ago

They weren't even nominated, so this is such an epic failure they won it as a late entry.

Also, that pony isn't typically super-glued to that structure, so this was also a special trophy! :)))))))))

fredgrott• 2 years ago

No, the most epic fail is still held by Boeing...they launched an alpha capsule to ISS and still cannot get the astronauts back to Earth...

swarnie• 2 years ago

Glad you find it funny Chaps, forwarding this to my legal dept.

addicted• 2 years ago

The only reason this could be funny is because the software industry has found a way to excuse itself from any liability.

There is no other industry where someone could cause so much damage and laugh about it. Least of all because the liability itself would have led to its collapse.

Can you imagine a company hired to reinforce a bridge to protect it from damage from a ship instead causes its collapse.

How long is that company gonna last? Even if no one dies or is injured it will be run out of business.

Only in Tech can such a company not only survive but laugh about it.

And that’s even before we get to how amateurish the mistake these guys made was.

xyst7• 2 years ago

100% agree. As mentioned in another post, the acceptance of this joke award is completely tone deaf.

Hospitals, banks, airlines, governments, and of course various IT operations at companies that are forced to use this endpoint security crap and Windows were impacted. Many people suffered degraded quality of care at medical facilities. Surgeons losing access to critical imaging/labs during surgery. Probably many canceled and rescheduled surgeries as well.

People’s flights were delayed or canceled. Imagine having to take a last minute flight to visit a loved one on their death bed only to get canceled because ClownStrike shit and incompetent IT departments/CTOs fucked them over.

Many people/businesses unable to access critical banking services.

Then the amount of lost productivity for office workers. Many hours lost for IT folks, often even working into the weekends. Time lost to dealing with ClownStrike bullshit when that time could have been spent with their families and friends.

Fuck ClownStrike, George Kurtz, and this latest clown, Michael Sentonas

vasco• 2 years ago

The time to take yourself seriously is before the stuff happens. Being stuffy now doesn't add anything. Accepting the award means they have something to show every single new hire and everyone in that office will have a physical reminder to do better in the future.

I get the other points about consequences, but I don't think accepting this award is in anyway problematic. It's one of those things that I expect only people that care about "appearances" would complain about.

cnasc• 2 years ago

I agree. They show up, cop to it, and collect a memento mori that will hopefully help motivate improvement in the future. They have a lot of work to do to repair their reputation, and I don’t think they’re foolish enough to think that this is anything more than a small step on a long path.

bb88• 2 years ago

Ever had someone make a mistake that cost you time or money and then tried to laugh it off as no big deal? That's what this feels like.

The security industry needs to grow up.

The best thing they should have done is fire the CEO, apologized profusely, and then use their army of sales people to help make things right on a one on one basis with their customers.

bootlooped• 2 years ago

A less charitable way to look at it is that they weren't taking things seriously before the incident, and they're still not taking things seriously now.

What the most appropriate way to view it is, I don't know. I think I'd need to know way more than I do about Crowdstrike leadership.

chestertonsgate9• 2 years ago

I'm sorry, how has CrowdStrike at all demonstrated that they're going to do better?

Aardwolf• 2 years ago

On the other hand (maybe I'm just playing devil's advocate here): nobody died (I hope at least! It's possible if some hospital equipment, 911 calls, ... failed...) from this incident despite being such huge scale that almost everyone knows about it. It's almost as if humanity can be... fine... when all this computing equipment fails.

evilduck3• 2 years ago

Then what value do they provide?

joe_the_user• 2 years ago

Many hospitals, including emergency rooms, were shutdown. Maybe no single death can be directly tied to the event but I'm pretty this effectively resulted in greater death.

Maybe some of those inconvenience had to time stop and contemplate the world but there are parts of the world where computers stopping don't leave people just fine.

allendoerfer2• 2 years ago

Stopping air travel has probably canceled a lot of unnecessary meetings and slowed down global warming.

Maybe events like this provide value in that they indentify which systems are actually mission critical.

mrinfinitiesx• 2 years ago

Fuck up the world's computers, piss off all of its IT teams, and then send people $10 uber eats gift cards as if that'll get you anything, maybe a coke at best.. but its further admitting fault. That's like, a tip. Like here, have a sandwich while you fix our fuckup.

They don't care. They'll all get 6 figure bonuses too in management for 'weathering the storm' after the mishap and probably get more money because look what they can withstand, literally technology-murder and get away with it.

It's almost movie-grade evil villainy tier stuff lol

trhway2• 2 years ago

Because people on the receiving end are the same - they accepted and rolled out the update without even as much as “canarying” it. SolarWind was the same - the customers weren’t bothered even by mismatched integrity hashes. It is a tacit pact in our industry - we all screw up and cut the slack to each other. Who will cast the first stone?

bitexploder2• 2 years ago

Honest question, what should they do? Uninstall the company and give up? Ignoring their actual response, what is a good response to this?

mywacaday2• 2 years ago

Crowdstrike should be held accountable but so should any reasonably sized enterprise that allows code to be pushed to their whole enterprise without testing. All the large enterprises I've worked for required Windows patches to be tested before being pushed to production why are crowdstrike updates treated differently?

delusional• 2 years ago

I don't think this is all that accurate. In the engineering space, Boeing has so far accepted responsibility for two fatal crashes and the fucking door falling out of an airplane and is still in business.

spacechild1• 2 years ago

To add insult to the injury, these people call themselves "software engineers".

codetrotter• 2 years ago

There is sparsely little real “engineering” that goes on in the field of “software engineering”, industry wide, in terms of ensuring that our software is reliable and secure. Performance and development velocity seem to take the driver seat always, in a whole lot of software.

See also this talk by Bryan Cantrill, “Scale by the Bay 2018: Rust and Other Interesting Things” https://youtu.be/2wZ1pCpJUIM where he talks about software platform values, in the sense of what different programming languages and other things focus on. He also touches on the fact that while higher level layers in the stack might value security it kind of falls apart when the very microcode in our CPUs sacrifices security for performance.

This was in a period of time where Spectre https://en.m.wikipedia.org/wiki/Spectre_(security_vulnerabil... and Meltdown https://en.m.wikipedia.org/wiki/Meltdown_(security_vulnerabi... had reared their ugly heads.

wayeq• 2 years ago

Statistically speaking it seems likely people really did die from this mistake, if only indirectly due to for example delays in medical care caused by the outage.

gleenn3• 2 years ago

It would be hard to imagine it didn't: https://www.washingtonpost.com/business/2024/07/19/windows-o...

thayne• 2 years ago

So, you would prefer that they don't accept this "award", and in so doing admit that they messed up?

And honestly, crowdstrike is more likely to go under than a company that failed to re-inforce a bridge. Their mistake caused measurable harm to many well funded companies that have the resources to sue crowdstrike in court.

If crowdstrike survives, it will be because there isn't a lot of competition in their market, not because they can excuse themselves of liability.

ospray• 2 years ago

I would have liked to see Crowdstrikes legal councils face as they accepted this award. There is no way they ran it by them.

lispisok• 2 years ago

We've professionalized industries like engineering and medicine because incompetent practitioners are a threat to public health and safety. Software is now in everything and incompetent practitioners have been a threat to public health and safety for a long time now yet we do nothing about it.

Onavo3• 2 years ago

Found the gatekeeper

scott_w2• 2 years ago

In construction, Grenfell happened and witnesses demanded immunity from prosecution to testify, because they knew they’d broken numerous laws in its construction and certification. Residents at similar buildings are the ones paying to make them safe for habitation, not the crooks that built them. Professionalisation is not a magic bullet.

robocat2• 2 years ago

Blaming individuals when systems fail: does it work reliably?

And how can certification work across borders between jurisdictions?

I've seen a fair share of engineering disasters in my own developed country where a few signatures by engineers didn't prevent the causes.

Regulations and jail doesn't seem to be enough of a disincentive? How do you force someone to do a good job?

Open source is not very compatible with certification.

Certification either (1) makes all software proprietary or (2) requires people to sign that particular open source software is safe or (3) maybe we should disallow the liability evasion clauses of open source software licenses?

zvmaz• 2 years ago

As if "professionalized industries" like engineering and medicine don't ever commit mistakes.

Onavo3• 2 years ago

Wait till you learn about finance, or oil and gas, or mining, or countless other legacy industries that quietly run America since the Rockefeller days. Many parts of Texas lost power for weeks due to Beryl and all the power companies got was a slap on the wrist (despite being explicitly warned about this scenario many times).

scott_w2• 2 years ago

Hell look at Grenfell in London! 72 people died and the people that designed it wanted immunity from prosecution to testify at the inquiry! Similar buildings are charging the cost of upgrading the cladding to residents instead of doing the right thing and eating the cost themselves.

robocat2• 2 years ago

Companies are too easy a target...

What were the regulatory incentives? Was the electricity market designed to encourage resiliency?

There was a systematic fault - it needed a collective solution not one that relies on individual companies doing the right thing???

chestertonsgate9• 2 years ago

You would not want for a sympathetic ear if you also criticized these companies. The point is not that CrowdStrike is uniquely incompetent, not at all. Every critical organization needs to be held to a higher standard, not just incompetent security firms.

lapphi• 2 years ago

I can think of at least one industry where the price of failure is almost always borne by the users and not the companies. Very closely integrated with tech as well.

booleandilemma• 2 years ago

It just illustrates how low the stakes are in tech really.

boxed• 2 years ago

This attitude is why we have a culture of fear and do-nothings.

maxlin• 2 years ago

This shows great grit and ownership. Props

hartator• 2 years ago

So 911 not working in several cities and hospitals flows being slow down to an halt, and they have time to go to defcon for a joke?

mminer237• 2 years ago

Are there hospitals that haven't fixed their computers yet? Obviously CS messed up, but aside from paying for damages and making process changes, I'm not sure what else they can do at this time. Warning people to not repeat their mistakes doesn't seem like a bad use of time.

citrin_ru2• 2 years ago

CS deserves the blame here but putting CS into a critical system like 911 IMHO is a huge mistake (but whoever done this likely knew they will dodge the blame so why they should care).

thedougd• 2 years ago

They understand who’s buying their product. It’s not the information security teams who cleaned up this mess, but rather the operations and end user compute teams.

mikeweiss• 2 years ago

I'm wondering how on earth has the CEO and CTO managed to keep their jobs after this fiasco?

skirge• 2 years ago

Serious companies should have better QA. This includes Crowdstrike customers.

badgersnake• 2 years ago

It took our IT department until Monday afternoon after to put out a message reaffirming their confidence in crowdstrike. Before any postmortem on their side or ours. I’m guessing they got offered a big discount to renew.

My IT team have a lot more confidence in crowdstrike than I have in them.

Aerbil313• 2 years ago

Never in a hundred years I’d expect to see the CEO of Crowdstrike at Defcon. The two are the two extremes on a spectrum of corporate.

bravetraveler• 2 years ago

This is how counter-culture is systematically uprooted

ein0p• 2 years ago

That seems unwise and tone deaf - to laugh it up on stage after causing easily tens of billions of dollars in losses worldwide.

hypeatei3• 2 years ago

This issue goes beyond CrowdStrike and points to the general approach to security that is buying products off the shelf to satisfy regulators and insurers while not actually caring what it does or how it works.

I'm not saying tech shouldn't be regulated, but our current model of "buy this thing to shed liability" doesn't work. The worst part is, the people who saw this coming (i.e. your IT department) probably can't do a damn thing about it because it's mandated at high levels in the company either for "cyber insurance" requirements or some other regulation. Madness.

maccard3• 2 years ago

The worst part is, the people who saw this coming (i.e. your IT department) probably can't do a damn thing about it because it's mandated at high levels in the company either for "cyber insurance" requirements or some other regulation.

I've worked with many excellent IT people who feel this way, but the vast majority of my experience with IT departments has been that as long as the contract covers what it needs to, they don't actually care if it solves the problem or not. At a previous job, software similar to crowdstrike was installed on my workstation over a weekend, and I came back to 20% slower compile times (I was working on them at the time so I had dozens of measurements). I had ETL traces showing the problem was the software, but IT refused to acknowledge it because the vendor contract said there was no performance impact expected for our workload.

hypeatei3• 2 years ago

That is my experience, too. I attribute it to IT / sysadmin jobs having a lower bar to entry and becoming more of a "watered down" business unit that just follows orders without much say or care for anything.

cqqxo4zV46cp5• 2 years ago

Most IT departments wouldn’t have seen this coming, and certainly would’ve been right to not base their entire security strategy around it. I’m not sure where this narrative is coming from. Falcon delivered and still delivers real, genuine security benefit to its customers. That does not mean that it eliminates all risk, and does not mean that it doesn’t introduce risk of its own.

It’s literally a game of trafeoffs like all engineering problems. This shouldn’t be that foreign to anyone here. Suddenly HN is full of security experts that are fuelled with 20/20 hindsight and recency bias, explaining how companies could’ve dodged this bullet without considering what very real bullets were being dodged by using Falcon in the first place.

hypeatei3• 2 years ago

Suddenly HN is full of security experts that are fuelled with 20/20 hindsight and recency bias

That is incorrect, many in tech saw these blanket IT policies being implemented and didn't like the prospects but couldn't change anything. At my workplace, policies like password rotations every 90 days (NIST recommends against), resource heavy machine scans, and nonsensical firewall rules are all a result of the company buying "cyber insurance".

It’s literally a game of trafeoffs like all engineering problems

Adding a single point of failure to all of your systems is a pretty big tradeoff to make for questionable gains.

Falcon delivered and still delivers real, genuine security benefit

Rhetorical question but I'll ask why some of the machines affected in the CrowdStrike outage even needed EDR software installed in the first place? Examples are flight status displays, critical 911 and healthcare machines, warehouse cranes, etc. things that don't immediately pass the smell test for having an internet connection.

solarkraft• 2 years ago

It would’ve been the best place to say sorry, but as far as I understand this would have legal consequences.

mikewarot• 2 years ago

Computer security issues cropped up during Viet Nam, and the US did the work, and found actually effective computer security models. We're in a society that has effectively memory holed them.

Why is it necessary to have scanners running 24x7 on everything a computer is about to run?

Why is it necessary to have Operating Systems that rely on ambient authority?

Blaming Crowdstrike does nothing but distract from the fundamental design failure we ignore every day in our operating systems, Linux, MacOS, Windows, et al.

Rick762• 2 years ago

I think they’re owning up to their mistakes instead of dodging the issue. I still feel that if they did the right testing, they shouldn’t be blamed for everything. It’s pretty standard for IT teams to avoid auto-updates and instead manually review them—especially in critical sectors like healthcare, aviation, and government. For instance, at my workplace, we’re not allowed to auto-update VsCode.

They mentioned they ran tests which unfortunately returned false positives. While it’s true they could’ve been more thorough, the affected companies also dropped the ball by not doing their own checks

wolrah• 2 years ago

I still feel that if they did the right testing, they shouldn’t be blamed for everything.

This update crashed 100% of the Windows systems it got installed on, which means either their testing did not involve actually loading it on real world computers at all or that blue screening and boot looping did not cause the test to fail. It is objectively clear that they did not do the right testing. There is no excuse for this update having ever left the earliest stages of a proper test process.

It's not like this is a case of an unexpected interaction with a configuration not found in the test lab.

It’s pretty standard for IT teams to avoid auto-updates and instead manually review them—especially in critical sectors like healthcare, aviation, and government.

This component was not able to be controlled in this way. Systems that were configured to be delayed on other CrowdStrike updates still got this particular update immediately with no ability for IT departments to control them.

They mentioned they ran tests which unfortunately returned false positives.

Again, whatever tests they actually ran clearly didn't involve actually loading the update in to the actual driver. Their explanation sounds like they may have validated the formatting of their update or something like that but then just sent it.

While it’s true they could’ve been more thorough, the affected companies also dropped the ball by not doing their own checks

No they did not because they could not. They may have dropped the ball when installing Crowdstrike in the first place, but the whole reason this was such a widespread thing affecting so many high priority systems is that it wasn't able to be controlled in the ways IT departments would want.

Rick762• 2 years ago

This component was not able to be controlled in this way. Systems that were configured to be delayed on other CrowdStrike updates still got this particular update immediately with no ability for IT departments to control them.

I had to look this up because I had not heard about this. I didn't understand that this bypassed companies' protections. I take back what I said, I guess I'm used to companies like those to having poor IT standards but once something goes wrong, they pretend that they had no part in it.

Thompson88• 2 years ago

While they're still blaming Microsoft as if it isn't possible to run the code they update outside of the kernel and only use the kernel mode code for observation and action, but not logic.

CrowdStrike accepting the PwnieAwards for "most epic fail" at defcon

Related Stories