Multiple airlines disrupted due to Microsoft Azure outage

https://www.nytimes.com/2024/07/18/us/frontier-flights-grounded-microsoft.html

by panarky • 2 years ago

371 121 2 years ago

crazytony• 2 years ago

This rollercoaster is not over yet. There's a crowdstrike issue causing windows machines/servers to brick globally and this industry is heavily windows dependent. It may or may not be related to the Azure issue but it's suspicious to me.

https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e...

Taniwha• 2 years ago

HN discussion here - needs to be on the front page - here in NZ ATMs, supermarkets, satellite TV channels are all down

https://news.ycombinator.com/item?id=41002195

caf• 2 years ago

Yeah, it's far from just airlines affected.

https://www.abc.net.au/news/2024-07-19/global-it-outage-crow...

mirekrusin• 2 years ago

Europe will wake up to flood of problems as well. This needs to be at the top of HN. We are experiencing multiple issues here in EU.

This issue feels extremely widely spread.

Maybe we don't hear ppl complaining because we're offline? :)

javaunsafe20192• 2 years ago

Berlin airport is down

vesinisa• 2 years ago

Maybe they have rolled back the update and Windows boxes in Europe are no longer pulling it?

surfingdino2• 2 years ago

Ryanair is unable to check in passengers online. You can check in at the airport.

geek_at• 2 years ago

it's already noon in central europe and yes everythings fucked. except for the linux powered companies

dagaci2• 2 years ago

On the wireless they are reporting a bad Crowdstrike update and a major Azure failover in central USA as separate events, are they they the same or different?

SSLy• 2 years ago

They seem either unrelated, or the Azure one was caused by CrowdStrike.

dagaci2• 2 years ago

A whole lot of people are running Crowdstrike in the cloud and on local PC An crowdstrike update last night caused a windows kernel panic Azure/Crowdstrike personel have spend rolled back the update in the cloud Local IT people will have to revert it from local machines manually

bun_terminator2• 2 years ago

I have never heard of crowdstrike. Is that some kind of antivirus? How is that related to PCs not booting? And why does it affect so many PCs if I've never heard of it? I'm so confused

shadow28• 2 years ago

They make security software that is really popular in various industries.

shrikant• 2 years ago

It's enterprise anti-malware that [in addition to other bits] has a client component installed on all PCs in the corporate network. An update to that client component (called an "endpoint") is causing those Windows machines to BSOD.

It's unlikely you'd have heard of it unless you've worked at a large enterprise that runs primarily Microsoft IT.

Crowdstrike does have Mac/Linux "endpoints" also (IIRC) but I'm unsure if they're affected as well.

maeil• 2 years ago

Almost certainly Azure using Crowdstrike on Windows in one way or another.

Not surprising that AWS and GCP don't seem to be hit as they wouldn't run anything on Windows, unlike Azure, who I'm sure are forced to do so under MS' infamous interdepartmental structure.

raverbashing• 2 years ago

Ugh.

Though I can image there's a Azure market for "Citrix server" kinda thing in the cloud

(or maybe it's SaaS - Solitaire as a Service)

classified• 2 years ago

Companies using Windoze for anything touching customer business should get sued by their customers.

craxdevil7• 2 years ago

true, got some insights why this happened https://medium.com/@confusedcyberwarrior/when-security-becom..., but how they didn't had an update process like testing or QA?

protosam• 2 years ago

The article is pay walled. Seems like this would be the fault of the airlines though. There is a reason to be distributed between different geographic areas.

tanelpoder2• 2 years ago

But if the Azure outage is due to Windows machines crashing because of the currently ongoing CrowdStrike crash/reboot loop issue, then such servers might end up being down in all regions. Looks like there might be some advanced lessons to be learned about blast radius here...

switch0075• 2 years ago

Azure VM host machines are running CrowdStrike...??

politelemon2• 2 years ago

It's more coincidental than likely, since these are managed services that were down, while the crowdstrike issue is closer to company deployments.

But never say impossible, it's best to wait and see what the actual problem is rather than throwing shade so early.

panarky• 2 years ago

https://archive.ph/FByL6

FearNotDaniel2• 2 years ago

A side issue, but since we’re on the subject of global tech being generally fscked: I’m currently on holiday in Italy and just discovered the entire archive.ph domain is blocked by the government, apparently due to kiddie porn. Shrug emoji…

amarcheschi• 2 years ago

I'm in italy and I can access archive.ph

Perhaps just the ISP you're roaming with blocks the site?

temporarely• 2 years ago

what does archive.ph have to do with that? Isn't it just a way to get around paywall?

kachinga123• 2 years ago

No flights at Germanys Berlin BER airport as well

lifestyleguru3• 2 years ago

I bet BER staff is screaming at the passengers right now "you wanted this, it's your own fault".

kolme• 2 years ago

Oh, you know them as well.

LeoPanthera• 2 years ago

BBC live coverage, no paywall. https://www.bbc.com/news/live/cnk4jdwp49et

switch0075• 2 years ago

Better described as a worldwide IT outage https://www.bbc.com/news/live/cnk4jdwp49et

Sky News UK is off the air. Some UK train companies are having an IT outage. Berlin, Melbourne airports flights disrupted...

vergessenmir• 2 years ago

Waitrose tills and card machines aren't working

switch0075• 2 years ago

OK now the outage is getting real. How will Tarquin get his cinnamon and gooseberry yoghurt?

openopenopen• 2 years ago

Does Microsoft use canary deployments? or was it deployed on everything?

pplante• 2 years ago

Sounds like it's more of a problem caused by Crowdstrike. I'm sure azure fell over when millions of servers all freaked out simultaneously.

jaggederest2• 2 years ago

Good lesson in rollout and heterogenous software stacks.

eecc• 2 years ago

I guess a canary rollout of the Crowdstrike patch would have contained the impact of this.

BSDobelix2• 2 years ago

I love it! Just keep putting all your eggs in one basket, because at the end of the day, it's not your fault, but Azure's.

switch0075• 2 years ago

I've had large customers seriously see that as a big appeal, when we were selling them some IT. They loved being able to blame someone else!

tormeh• 2 years ago

The corporate training always says you are responsible for your chosen cloud vendor's problems, but in reality everyone assumes they can successfully blame the vendor.

SteveSmith16384• 2 years ago

"No-one ever got fired for choosing Microsoft".

undebuggable• 2 years ago

Although many got locked out for choosing them.

BSDobelix2• 2 years ago

Well, the older version is about IBM, and it would probably even be true (today) if we were talking about mainframes, because they are one hell of a stable basket ;)

tkubacki• 2 years ago

That’s why it’s important to demonopolize desktop OS market. Microsoft should be divided long time ago

arcxi2• 2 years ago

nobody is forcing these companies to depend on MS products and services

logicchains• 2 years ago

Many in regulated industries actually are forced to use crapware like Crowdstrike, to tick a box on the security checkbox.

YeahThisIsMe• 2 years ago

How would that change anything about Windows market share?

switch0075• 2 years ago

You don't think the bundling of windows with pcs has anything to do with their market share?

commercialnix• 2 years ago

See my comment here https://news.ycombinator.com/item?id=41003860

cm2187• 2 years ago

To those who remember the Y2K bug scare, Microsoft delivered it: https://www.youtube.com/watch?v=WhF7dQl4Ico

dools• 2 years ago

I'm so old that I was like "Do that many people really play Crowdstrike?" then I realised that's Counter Strike, then I looked up Counter Strike and it came out 24 years ago.

yells at cloud

robertlagrant2• 2 years ago

And yes, they do! Counterstrike 2 is out now, to a mixed reception.

ClassyJacket• 2 years ago

Which is actually Counter Strike 4

_the_inflator• 2 years ago

The dawn of the hybrid model: on premise will be back soon.

The impact is vast: imagine being blocked by an hostile administration in this way. Disaster recovery of this magnitude is like a global pandemic.

AI won’t save us. Network topology and admins will have their comeback.

seydor• 2 years ago

Unless attacks take down infrastructure regularly, we won't go back to decentralized model. The internet itself was created decentralized to withstand a a war .

KineticLensman• 2 years ago

No, the ‘internet’ was created to allow US defence researchers to access shared computers more easily. It adopted the packet switched model that had been developed theoretically to support command and control applications, but which was never actually implemented by the extant C2 providers (in a dentist’s waiting room now so don’t have links)

DanielHB• 2 years ago

I dunno, I have been around and I never seen an on-prem infra being more reliable than your average cloud.

The only difference is that when on-prem goes down you can shout to your infra engineers, when cloud goes down you shout at your enterprise cloud representative. The first is more effective, but even with that it still doesn't achieve the same reliability and disaster-recovery of your average cloud provider.

rcxdude• 2 years ago

It also has the property of being less correlated with other failures, which may or may not be an advantage.

tankenmate• 2 years ago

With small events cloud engineers scale better, with large events local engineers scale better.

ocdtrekkie• 2 years ago

I have never seen an on-premise environment down as much as Azure is.

medion3• 2 years ago

It’s a general political trend too: de-globalization. Everyone sold the idea of globalisation and off-premise ethereal globalized cloud services. Both good when they work, but a total disaster when they don’t.

Decentralization is the way.

zmgsabst2• 2 years ago

There’s a reason that I admire the federal system of the US:

For all the US’s problems, devolving critical functions to layers of differing granularity has proven surprisingly robust to many faults.

I suspect we’ll see economic equivalents, where critical functions are spread around at various scales. We dont need to be either totally globalized or totally domestic.

Yizahi2• 2 years ago

Except that anything local will be more expensive than anything globally scaled, and the same people complaining about globalization "suddenly" don't want to pay more for the same.

It's not like alternatives for MS services doesn't exist for decades. There smaller and more people friendly hostings, email services, file shares, office software etc. The problem is that people complaining about MS services don't wan't to use them or pay for them.

totallywrong5• 2 years ago

I wish. Only (mostly small) tech savvy companies might maybe make that move at some point. Herd mentality and short term convenience have already won that battle. AI will only add to that since for 99.9% of people LLMs and cloud are synonyms.

kaptainscarlet• 2 years ago

This is why I sometimes think on prem hosting is better.

belter• 2 years ago

It's 2024. Using Azure should be a firing offense....

https://www.lastweekinaws.com/blog/azures_vulnerabilities_ar...

lifestyleguru3• 2 years ago

Obviously you've never been working at enterprise purchasing department and participating in business meetings with Microsoft sales people.

benterix• 2 years ago

I see no relationship between understanding that Azure has a poor security posture and participating in business meetings with salespeople.

totallywrong5• 2 years ago

Yeah, you're completely out of touch of the industry. As horrendous as it is Windows on Azure (and the entire ecosystem it comes with) is an easy sell for the dinosaur IT leaders that haven't seen a Linux server in their life. You'd be surprised how common that is.

javaunsafe20192• 2 years ago

Berlin airport is down. I guess it’s related

davidmurdoch2• 2 years ago

Waiting in a queue at an airport in Palma, Mallorca, Spain right now and the check in staff are currently flipping through printed sheets of paper to check us all in. It's going to be a very long wait.

robertlagrant2• 2 years ago

Don't buy tills that run Windows. You must be crazy.

steve1977• 2 years ago

I wonder if these airlines were really affected by that Azure problem or if they were affected by the CrowdStrike issue and were just mixing things up.

adamsiem• 2 years ago

ALB airport down. Check in systems are in BSOD ‘Recovery Mode’.

surfingdino2• 2 years ago

So, anti-malware software turns out to be malware? How odd. SecOps out on anti-open source training, I presume?

totallywrong5• 2 years ago

Antivirus company causing exponentially more harm in hours than it's prevented during its entire existence.

totallywrong5• 2 years ago

Interesting that I happened to transit 3 different Asian airports today and had zero issues. I haven't seen anything at all related to the outage over here.

angra_mainyu• 2 years ago

Hmm, maybe they've migrated their systems? I know SK had a plan to migrate govt machines to Linux.

totallywrong5• 2 years ago

How is this update released at all? Something that affects literally every machine does not get caught in testing? From a 80B company no less.

adultSwim• 2 years ago

Real life operations of hospitals were affected (eg scheduled surgeries not taking place).

Multiple airlines disrupted due to Microsoft Azure outage

Related Stories