Major Microsoft 365 outage caused by Azure configuration change

https://www.bleepingcomputer.com/news/microsoft/major-microsoft-365-outage-caused-by-azure-configuration-change/

by doener • 2 years ago

102 62 2 years ago

bleepingcomputer.com

vb-84486• 2 years ago

I guess we can call it a "black Thursday" for microsoft.

bluedino• 2 years ago

Why is MS getting any blame for the Crowdstrike thing?

yelnatz• 2 years ago

They allow a 3rd party to change Kernel stuff with an update. Apple banned this a while ago.

AmVess• 2 years ago

Letting an app take down your OS, perhaps.

vb-84486• 2 years ago

Well, if you look on all news sites, they put highlight that is a windows issue.

I personally think that IT managers should be blamed for the disaster, but in the collective imagination this will e a microsoft/windows problem.

bguebert• 2 years ago

Seems like microsoft is getting put into the headlines because the Crowdstrike versions for mac and linux aren't affected. Plus Crowdstrike has a history of pushing people onto microsoft tech support when their software causes problems.

skywhopper• 2 years ago

Because their OS allowed a third-party tool to crash it, worldwide, with no automated way of recovery?

jameskilton• 2 years ago

And CloudStrike hitting them at the same time must have been just unbelievable.

patmorgan23• 2 years ago

Also Microsoft using Crowd strike instead of their own defender for endpoint?

Guess it could be part of a risk mitigation strategy to not but all their eggs in that product.

joelvalleroy2• 2 years ago

The Crowdstrike issue is causing the Azure issue as I understand, lots of machines using that software that all updated to the blue screen feature around 6 PM apparently.

rcarmo• 2 years ago

Nope. Azure does not run CrowdStrike. It doesn't even run Windows as you know it.

hypeatei• 2 years ago

The Central US outage really exposed a lot of shortcomings both at my company and at MSFT apparently.

Even having regional fail over available wouldn't help because the control plane was unreliable meaning it couldn't be triggered by anyone.

macote• 2 years ago

That's an interesting recommendation...

We've received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage.

https://azure.status.microsoft/en-ca/status

underdeserver2• 2 years ago

Why would multiple reboots make sense? I can accept three reboots triggering some condition that tries three times before it stops trying, but fifteen?

krembo• 2 years ago

With my experience dating back to 286 machines, sometimes even kicking the box (literally) solves the problem

easton• 2 years ago

It sounds like a race condition; you want the CrowdStrike updater to start and pull down the fix before the affected virus definition file is loaded and kills the box.

If you keep rebooting, you eventually may get those to load in the right order.

zdw• 2 years ago

Have you tried turning it off and on and off and on again?

terom2• 2 years ago

https://azure.status.microsoft/en-us/status/history/ doesn't seem to have links to the individual incidents. Some reports claim the Azure / Microsoft 365 outages were related to crowdstrike, but this sounds like an entirely separate incident.

AFAIK the broken crowdstrike channel update happened at 2024-07-19 06:05 UTC and was "fixed" (rolled back) at 06:47 UTC, but I don't have a proper source for that timeline?

EDIT: https://azure.status.microsoft/en-gb/status claims 2024-07-18 19:00 UTC as the approximate start of impact for the crowdstrike update. It would be nice to find a proper source for the start and mitigation timelines...

EDIT: reddit threads reporting symptoms start at approx 2024-07-19 05:00 UTC. That would mean the crowdstrike impact started soon after the azure recovery.

---

What happened?

Between 21:56 UTC on 18 July 2024 and 12:15 UTC on 19 July 2024, customers may have experienced issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services. A storage incident impacted the availability of Virtual Machines which may have also restarted unexpectedly. Services with dependencies on the impacted virtual machines and storage resources would have experienced impact.

What do we know so far?

We determined that a backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks hosted on impacted storage resources.

How did we respond?

21:56 UTC on 18 July 2024 – Customer impact began

22:13 UTC on 18 July 2024 – Storage team started investigating

22:41 UTC on 18 July 2024 – Additional Teams engaged to assist investigations

23:27 UTC on 18 July 2024 – All deployments in Central US stopped

23:35 UTC on 18 July 2024 – All deployments paused for all regions

00:45 UTC on 18 July 2024 – A configuration change as the underlying cause was confirmed

01:10 UTC on 19 July 2024 – Mitigation started

01:30 UTC on 19 July 2024 – Customers started seeing signs of recovery

02:51 UTC on 19 July 2024 – 99% of all impacted compute resources recovered

03:23 UTC on 19 July 2024 – All Azure Storage clusters confirmed recovery

03:41 UTC on 19 July 2024 – Mitigation confirmed for compute resources

Between 03:41 and 12:15 UTC on 19 July 2024 – Services which were impacted by this outage recovered progressively and engineers from the respective teams intervened where further manual recovery was needed. Following an extended monitoring period, we determined that impacted services had returned to their expected availability levels.

ar0• 2 years ago

The German BSI (Federal Office for IT Security) quotes the advisory from CrowdStrike (which is behind their customer login portal) as saying you need to roll-back to snapshots prior to 04:09 UTC:

https://www.bsi.bund.de/SharedDocs/Cybersicherheitswarnungen...

So your Redditor saying 05:00 UTC seems to be close.

terom2• 2 years ago

https://old.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e... quotes the CrowdStrike advisory verbatim, which has timestamps of 04:09 UTC for the problematic version and 05:27 for the reverted (good) version.

gjsman-1000• 2 years ago

I'm not sure if this has anything to do with it; but I got an email this morning at 2 AM saying that my Microsoft account password was changed. The email was authentic - it came from Microsoft's servers, and had no buttons for me to click. It said that the IP Address of the password change was my own ISP in my area... and that the reset came from a phone number I didn't recognize on my security info, that even now does not show on my account dashboard.

I ran in a panic to reset my password... only to discover, the password was "changed" to the exact same password? And how would they even get in the account without 2FA? My "sign in history" also showed no trace of anything unusual. At this point, and reading these headlines, I feel more confident something's broke.

isaacremuant• 2 years ago

This entire thing is not surprising but it's fascinating to see unfold. In many dimensions, even listening to the wild speculation or weird reporting around it.

CWuestefeld• 2 years ago

OK, it's a pedantic nit, but doesn't anybody have editors anymore?

This massive outage started around 6:00 PM EST

We're currently in daylight savings time, so the time should read "EDT".

jftuga• 2 years ago

Furthermore, the article states that it was "affecting customers across the Central US region". Doesn't it seem like they should have used "5:00 PM CDT" in the article?

boringg• 2 years ago

Honestly is anyone surprised that Microsoft is having these issues? They still run the same sloppy software as they always have.

ilrwbwrkhv• 2 years ago

With some of the worst engineers. I had the misfortunate of working with a couple of Microsoft principal engineers and I still shudder at how bad they were and how many layer of abstractions they could write.

prerok2• 2 years ago

Not at MS, but I had the misfortune of working with such "engineers" as well.

I belive the term is "architecture astronauts".

Major Microsoft 365 outage caused by Azure configuration change

Related Stories