Azure Active Directory down

https://twitter.com/AzureSupport/status/1371544292502564866

by Decabytes • 5 years ago

91 44 5 years ago

twitter.com

bombcar2• 5 years ago

I was seeing:

DX10501:+Signature+validation+failed.+Unable+to+match+keys:+ kid:+'[PII+is+hidden]',+ token:+'[PII+is+hidden]'.

and sure enough, it seems it was some authentication thing going sideways

EMM_386• 5 years ago

It seems to be coming back online now.

Strange that HN was down at exactly the same time, although this is reported to be unrelated.

CURRENT STATUS: Engineering teams are currently rolling out mitigation worldwide. Customers should begin seeing recovery at this time, with full mitigation expected within 60 minutes. This message was last updated at 21:12 UTC on 15 March 2021

deathanatos2• 5 years ago

It is a huge pet peeve of mine that,

a. the status page update took as long as it did.

b. the status page still only claims that AAD is down. You're not "up" if a required dependency of yours is down, IMO, and most of the Azure services right now are severely degraded to the point of not being useful due to the AAD outage. Now, that would make your status page look bad, yes. (And I would clearly message on it that the other outages are fallout of AAD on that.)

c. I don't have any real expectation that we'll get a public PM about this. (And a PM must include sufficient information for me to understand what went wrong, and what's been done to prevent it going forward.)

I had been starting to wonder what sort of incident it would take for them to actually update the status page. I guess this is it. We've witnessed a number of outages in various services over the past few months, none of which made it to the status page. I pushed a support rep on the issue, and was told that they don't want to cause a panic.

jrockway• 5 years ago

I'm also here because my work is blocked by the inability to do anything on Azure right now.

I looked at their past status histories, and they have provided pretty detailed postmortems for much more minor incidents. I don't have any reason to believe that they won't do that this time. I'm new to Azure, so this kind of outage really looks bad in my mind, but it's par for the course with Cloud stuff. They'll fix it and there will be one less thing to break in the future.

spondyl2• 5 years ago

As someone who has been using Azure for 6 months, it's fairly bad but most of the Azure pain is more death by 1000 cuts than large explosions.

Documentation that contains triple negatives, UI elements that have no consistency (clicking off a modal on one page does nothing while doing so on another closes it and clears your data), widgets and modals forgetting state when you navigate to different submenus.

The sort of unexciting stuff that by itself is ok but it slowly makes you descend into madness.

asabla• 5 years ago

Been an active user of Azure for almost five or six years now. Things were a lot more rough in then (especially if you'd seen the old portal), unlike what it is now.

People always talk about cloud as an entity which can/shouldn't have problems at all. When it's more about convenience then anything else. So I wouldn't worry too much about this if I were you.

breatheoften• 5 years ago

Bad status pages are a very bad sign in my opinion.

I had to use the IBM cloud for a project once upon a time and the status monitoring was almost always wrong ...

I really hope that as customers we can succeed in making "excellent status reporting" an absolute minimum requirement for cloud services. Observing the quality of the status page reporting history is one of the first things I do now when evaluating a new service -- and no I'm not just looking for a page that shows all greens -- I'm looking for a page that connects to some conceivably meaningful metric as well as detailed reports about major and minor outages ... something to prove that they actually even know if their service is working at a given point in time ...

The IBM cloud page was a page that tended to update in response to my own issue reports (usually if I also pointed out that nothing about the outage appeared on the status page ...). I don't think that's at all an acceptable standard ...

jmartens2• 5 years ago

Are status pages the right answer though? They seem to be a failure all around, so let's find a better solution.

jmartens2• 5 years ago

Honest question for the sake of discussion/learning: What impact would it have on you if (1) the status page was updated instantly and (2) All impacted capabilities were marked as down/degraded?

NDizzle2• 5 years ago

Affecting Teams, Azdo, and likely more services.

It turned the whole afternoon into learning time at our company. Thankfully our Okta integration goes through our on prem AD servers and not purely AAD. Otherwise I wouldn't be able to get to learning resources which authenticate through AD!

spondyl2• 5 years ago

As someone working at a company that pays for Azure support, it's pretty sad that Microsoft 365's Status Twitter account has been providing more insight than Azure's own status page

https://twitter.com/MSFT365Status/status/1371554704518352896

They actually acknowledged it was due to a "recent change to an authentication system" where Azure Support just vaguely mentions an outage

llama0522• 5 years ago

I have zero expectations from Azure support as a whole, in fact 80% of the time I'll rebuild production services in Azure to avoid having them tell me to do just that in most tickets. It is abysmal levels of actual support.

heyflyguy2• 5 years ago

I am naive, but it is crazy to me that we experience outages of any magnitude like this from any major supplier (GOOG/MS/AMZN). I get alot of crap for being crotchety about making my backups in and out of the cloud, and it's stuff like this that proves my point!

ocdtrekkie3• 5 years ago

"The more they overthink the plumbing, the easier it is to stop up the drain." - Scotty

Trying to provide globally available, replicated service that meets every and all needs 24/7/365 is basically making Active Directory about 600x more complicated than it would be otherwise. It's basically impossible for a service that complicated to meet the uptime of... installing Windows Server 2019 on a VM and patching it monthly. (Bear in mind, if I know when my business is not affected by an outage, I can do it without being disruptive. Microsoft, by definition, cannot. Every moment is critical since their customers are everywhere.)

There are types of businesses to which the former might be a necessary solution, but most would be better off with the latter. It'll be really interesting when people start realizing the cloud is mostly just a scam to get people on subscription revenue streams, and not actually providing any greater reliability or less management overhead than what they had before.

heyflyguy2• 5 years ago

I agree!

Spooky23• 5 years ago

Plus, with a massive customer base and eating their own dog food makes their schedule be a priority over other customers. Your priority is lower than theirs.

Microsoft is always a shitshow this time of year. Infrastructure changes usually land ahead of spring releases.

Severian• 5 years ago

Teams completely borked because of this. My employer hugely depends on this as we all WFH, and basically has stopped all communication outside of email (thank god we are on-prem).

king_magic3• 5 years ago

This should be a huge shitshow for Microsoft. This never should have happened.

Azure AD taking down pretty much the entirety of Azure / O365 / Teams is frankly inexcusable. Astounding incompetence, and an astonishing single point of failure that needs to be re-architected.

macintux3• 5 years ago

In fairness to Microsoft, there's no way to make authentication not a single point of failure.

You have to make the service as resilient as possible, which clearly they failed at, but you can't very well fail over from your AD service to something else.

king_magic3• 5 years ago

Microsoft is rolling back some update they pushed that borked Azure AD.

Early signs are not pointing to this as a service resilience issue - this appears to be sheer incompetence - pushing an update that was likely not tested well enough that broke pretty much everything. More than that, why aren't updates to Azure AD being rolled out regionally? Why is Azure AD architected in such a way that the entire thing going down can do so much damage?

It's pretty hard to be understanding with a preventable screwup with this kind of global scale.

hsbauauvhabzb• 5 years ago

They could quite easily have progressive migrations, blue/green deployments, or a failover environment..

wallacoloo• 5 years ago

In fairness to Microsoft, there's no way to make authentication not a single point of failure.

Huh? x509-type authentication works without a single point of failure. I can authenticate myself to any client via public-key cryptography without any external dependencies, after we've agreed on a root of trust.

Active Directory chose a solution that requires centralized availability. They didn't have to do it this way (though it does make certain administrative tasks, like revocation, simpler).

Note that even AD itself does not require a single point of failure though! You can define secondary (i.e. failover) DCs (usually via DNS). Then DNS becomes your point of failure -- and again most DNS implementations support failover, such that any single point of failure gets pushed lower to the stack (e.g. NICs), which also support failover or multipathing, etc, if you really need high availability.

Anyway, it's totally possible to make authentication not be a single point of failure. Of course, you have to make some tradeoff to do it (see: the CAP theorem).

ocdtrekkie3• 5 years ago

Every time someone tells me we should move to the cloud, it seems like there's an outage the same day. Microsoft isn't actually any good at managing your IT infrastructure for you.

Between Exchange 365 outages and Azure AD outages, I'm not sure how anyone thinks moving Windows infrastructure to the cloud is not an existential risk to business operations.

hateful• 5 years ago

cloud=a single point of failure for the whole world!

dnndev• 5 years ago

I run a saas company and we use azure ad. B2B sales. Good thing is we are down but so are they. This is strangely not that bad. Left the office early today

_Understated_• 5 years ago

You know, in some ways I'm actually glad that Azure AD is down: I ran a SQL update and right after it I started seeing errors and thought I'd done something wrong... I've been shitting myself for an hour :D

chevman• 5 years ago

Was interesting watching this happen today at a large Fortune xx that is a year or so in to wide scale Azure rollout.

Took out Outlook, Teams and all meetings over the course of 5 minutes, then moved on to internal service alarms going off across multiple (10+) major customer facing digital products that got vaporized due (presumably) to various dependencies on AD. I think some of this was service accounts on backend that lost connection to various dbs, file systems, etc.

What a total shit show.

Company basically down from 2pm CST and barely coming back up 5 hours later.

IT Buddy who is 2 people away from CIO pinged me and said “DNS issue, data center power outage or Azure just crapped its pants.”

JCM9• 5 years ago

Azure AD instability has been a big pain point for us for a while. We’ve had more of these outages than I care to remember. One the one hand the ability to have one central auth between on-prem and other services is great but every time this happens so many services just stop working. Microsoft really needs to fix their stability issues.

jiggawatts• 5 years ago

What's crazy is that apparently there's a "fallback" Azure AD authentication endpoint that is a delayed read-only replica of the production endpoint with a subset of the features. During the last global AAD outage, they claimed that they learned their lesson and improved the failover to the fallback endpoint. That didn't seem to happen...

What is more crazy is that low level service-to-service authentication in Azure is primarily based on Azure AD. This includes Key Vault and Storage Accounts. Both were mentioned in the Microsoft notification page as affected by this outage.

How can anyone build a reliable service on top of Azure if what is essentially the failure of the "Office 365 login" also breaks your VM's access to its SSL certificates (or whatever)?

Azure Active Directory down

Related Stories