Duo Outage

https://status.duo.com

by davewritescode • 3 years ago

76 75 3 years ago

status.duo.com

f1refly• 3 years ago

I bet both of duos users will be bummed to hear about this outage.

Rebelgecko• 3 years ago

It's mandatory 2FA at the college I went to (where, coincidentally, classes are starting today)

harrego5• 3 years ago

College-induced software!

MattSteelblade• 3 years ago

This is Cisco's Duo used for MFA and SSO and not Google's discontinued video app.

davewritescode2• 3 years ago

Yep, and my company uses it as part of access to all of our internal applications including our AWS console.

Not a super productive morning

Gabrys1• 3 years ago

I didn't realize this Duo was affiliated with Cisco. Now it makes sense it's so popular

commandlinefan• 3 years ago

It's impacting me, but I was a little surprised to find out that Duo was "popular" enough for this outage to make it the front page of HN.

davewritescode2• 3 years ago

I posted it and honestly I am too

afavour2• 3 years ago

Never underestimate the number of college students hiding in the online shadows

spike021• 3 years ago

I've used it while working at two major companies everyone's heard of and used. Definitely seems to be popular.

pavel_lishin• 3 years ago

I thought this was one of Google's messaging apps.

jolux2• 3 years ago

It was a video call service: https://en.wikipedia.org/wiki/Google_Duo

miked85• 3 years ago

Wow, never heard of this one - how many messaging/chat applications has Google created now?

gsa• 3 years ago

Is it really discontinued yet? If anything it's an ongoing naming blunder with no end in sight.

There are two products called Google Meet now. The web interface for the former Google Duo is duo.google.com, just rebranded to Google Meet. The former Google Meet still exists on meet.google.com. Both also have their own Android apps - one called Meet and other Meet (original). Both products have a different set of features and neither completely replaces the other.

drcongo• 3 years ago

Same here.

mattmaroon• 3 years ago

Same name, different app.

toomim• 3 years ago

I thought this was Duolingo, a language learning app.

And before that, I thought Duolingo was a crowdsourced translation app, that doubled as a language learning app.

ChrisArchitect• 3 years ago

Same. For anyone wondering:

Duo is a multi-factor authentication/single sign-on platform owned by Cisco

stefanpie• 3 years ago

This is also an unfortunate coincidence, as today is Georgia Tech's first day of classes, and the institution's Single Sign-On (SSO) for almost all its services, such as the Canvas LMS, registration, and the Bursar, is based on Duo's service. Right now it seems to be having some issues and is timing out / not logging in.

However, I'm not sure if GT's Duo service is self-hosted or is hooked into Duo's service "in the cloud".

MattPerry• 3 years ago

I am reading this, slacking off, because I couldn't get into Georgia Tech's Canvas. Judging by the status.gatech.edu message, it seems like changing the instance of Duo that responses is a quick fix.

afavour2• 3 years ago

I wonder if it’s more causation than coincidence.

steve1977• 3 years ago

Single Sign-On, single point of failure… maybe having lots of different logins wasn’t such a bad idea after all.

kube-system• 3 years ago

There are a lot more other problems that happen when each service manages their own authentication. The move to SSO has been in response to problems that existed then.

adbachman• 3 years ago

Same for my kid picking up info for the beginning of their first year at Univ of Maryland.

Millions of students coming online, textbooks are digital now, SSO all the things. Perfect storm.

PAPPPmAc• 3 years ago

University of Kentucky too, first day of classes, no one can get into anything they aren't already logged in to with a valid cookie.

I spent 20 minutes trying to figure out what new cookie I needed to grey-list for the half dozen redirections in the M365 auth flow to not bork before I thought to check if it was generally broken.

Great success.

nonameiguess• 3 years ago

I'm actually taking a course there right now, as part of my pointless masochistic love of continuing education in stuff I like knowing to be well-rounded but no one will ever give a crap about professionally, and I logged in fine. Duo's push notifications seemed to stop working, but it lets you fall back on the TOTP passcode, which continued working.

repiret2• 3 years ago

Duo seems to have had lots of success, selling into higher education. To add to the other schools mentioned in this thread, Oregon State, and Stanford both use it too. And lots of schools are starting class today, which might be what contributed to their downtime.

To be honest, their useR experience is pretty slick compared to most two factor solutions. I am happy for them, (and their users) that they have been successful, selling into large organizations – good user experience usually isn’t found in large enterprise software due in part to the principal agent problem.

l33t7332273• 3 years ago

Ohio state is another ~60k duo users this week.

godshatter• 3 years ago

University of Idaho checking in, first day of classes as well.

And, yes, Duo is normally a great solution for us.

alsodumb• 3 years ago

University of Illinois Urbana-Champaign starts today too, another 60,000 or so users.

janejeon2• 3 years ago

selling into higher education

Just want to add another data point: Dartmouth was a relatively early adopter, and while a bit annoying, it chugged along just fine.

JCharante• 3 years ago

My duo workflow was figuring out I can request an SMS OTP and read the OTP code as a notification on my laptop instead of unlocking my phone and confirming via the app. Hopefully they get volume pricing on texts.

candleknight• 3 years ago

university of waterloo as well, although thankfully classes start in two weeks for us

elashri3• 3 years ago

UC (Cincinnati, not California) first day of classes is today too, I always hate that out IT doesn't allow 2FA except Due. I always wanted to request an exception, but my previous experience with them refusing to allow IMAP access to my email because it is not secure!.

cridenour• 3 years ago

Not often I see UC mentioned on HN. What program are you in?

yodigital• 3 years ago

Did you notice that 2FA went away around 10 am? I worked on the team that made that happen. Its back now that the outage is over.

We do have an app that lets faculty exempt students from 2FA, but it's mostly for students who need the exception when taking tests etc.

You are going to see a big change in the tech at UC very soon. The old guard is getting the boot.

ajaimk• 3 years ago

I miss Tech. Still using Oscar and T-square?

jdechko• 3 years ago

Ugh. Those memories.

I was among the last to have to use dial-up from off-campus (which required a specialty ISP @ 26.6k or some awful speed). Fortunately, they upgraded their system while I was there, so I could use the “much” faster DSL. Still not as nice as on-campus internet.

runnerup• 3 years ago

This doesn’t mean GT is offline. There are a couple dozen Duo server #’s and this only affects DUO1. If GT is on DUO5 or DUO19 or DUO25 they’ll be fine today. It does raise the question of how to provide continuity of services if your server is offline though.

harrego5• 3 years ago

I was always curious why Duo cloud 2FA was chosen over traditional offline 2FA codes in education.

ihattendorf• 3 years ago

User experience is nicer: click confirm prompt from lockscreen vs. having to find and open app, select correct site, and enter code.

harrego5• 3 years ago

From the student side I end up carrying an extra 2FA app that only works with my college vs. Google Auth (and similar) that carry my other 15 sites.

lolinder• 3 years ago

click confirm prompt from lockscreen

This is also a huge vulnerability that has been exploited.

https://www.theregister.com/2022/11/03/mfa_fatigue_enterpris...

acedTrex• 3 years ago

Offline 2fa codes seems like a nightmare to roll out to an academic campus

reaperman2• 3 years ago

Indeed. They are generally understaffed and salaries are very low so they're very lucky to get any "1x-5x" developers who stick around long enough to understand the infrastructure. Outsourcing as much as possible makes a lot of sense in that environment, it does create major single points of failure but "roll-your-own" would likely fail more often anyways.

SkyPuncher• 3 years ago

It's simply so much easier.

Okta offers a similar feature. So much easier to click a confirmation on my phone than to scroll through dozens of 2FA codes (some of which might be orphaned).

cute_boi• 3 years ago

well apps like 1password makes it easy to fill those 2fa codes from computer and phones easily.

harrego5• 3 years ago

This implementation sounds better. Though for me I still have to manually input a code from the Duo app (that doesn't auto refresh after code entry since it's not time based).

thesuitonym• 3 years ago

People really don't understand 2FA codes. Imagine trying to tell thousands of students to get the code from their 2FA app (Which app?). What happens when a student goes home over the summer and gets a new phone, but doesn't transfer the app info? Duo offers a level of management that other apps don't. If a student is struggling, you can send them a text with a direct link to the app they need to download. You can temporarily bypass 2FA from the Duo console. For the longest time, it was the only 2FA app that offered any kind of management. Okta has it now, too, but most higher ed already has a different SSO provider, so switching to Okta just to get 2FA management (And I'm not sure it's as good as Duo's) is probably an impossible task to get off the ground.

Spooky232• 3 years ago

It’s two-step auth, not MFA.

dmattia• 3 years ago

I love that the status page is being updated regularly.

But I have no idea what the difference is between DUO1, DUO2, etc. through DUO73. I feel like they should have a better way to clarify which users are affected.

electroly• 3 years ago

They're all identical deployments of their whole stack. They shard customers onto the deployments to reduce blast radius. In your Duo admin dashboard, look for "Deployment ID" in the left pane.

sriske• 3 years ago

FWIW I just logged into my university portal and Duo worked just fine. Perhaps it's not widespread?

btbuildem• 3 years ago

We use Duo at work for 2fa. I am laughing my ass off, I literally can't do anything -- can't get on the VPN, can't get into my emails, can't access company services. They locked everything down so hard, they've literally chained themselves to a radiator and tossed the key out of reach. We didn't receive internal communications about the outage yet -- and my bet is it's because whoever's in charge of that, is locked out due to the outage.

merb• 3 years ago

I would never do duo 2fa only. It sucks if one of your factors does not work. Most of the time I always try to use two 2fa apps.

Spooky232• 3 years ago

TOTP tokens don’t meet alot of compliance requirements. You usually need a PIN or crypto device with a dedicated solution.

Like the popular password vault says in the name… 1Password for everything. If everything can stand alone in 1Password, it ain’t MFA.

gchallen• 3 years ago

And the proud Illinois tradition of some mission-critical service crashing on the first day of class continues.

In this case, it is an external service. However, I also suspect that the Duo outage is probably shielding other on-campus services from load surges that would probably be causing them to get crashy.

I guess I don't know how we could ever prevent such incidents. Given that the first day of classes is a well-kept secret /s.

senectus1• 3 years ago

no issue here in australia

billbrown• 3 years ago

They published the postmortem, and the penetration into colleges is hinted at as being causal.

Increased load on DUO1 due to significantly increased adoption and simultaneous peak usage across multiple larger customers led to authentication failures.

https://status.duo.com/incidents/rw7g0q7ztj8f

Duo Outage

Related Stories