Meta outage

https://metastatus.com/

by geocrasher • 2 years ago

698 818 2 years ago

metastatus.com

jameswatling• 2 years ago

We just started having issues with google oauth too....

ivanmontillam• 2 years ago

Yes, I noticed it too[0]. This is going to send a truckload of traffic onto Twitter/X and Telegram.

[0] https://twitter.com/IvanMontillaM/status/1765036872290681089

joltis• 2 years ago

Can't see the post, and can't login on twitter either, so probably :D

quartz• 2 years ago

Seems like it. Pretty messy too because it sometimes pushes you to reset your password which then doesn't work so there are going to be a LOT of reset email codes floating out there.

mfrommil4• 2 years ago

Agreed. Probably could be a much better UX for handling a mass outage like this. Graceful, clear error messaging that FB login is down would be better than the current UI.

Triggering millions of people to unnecessarily reset their password yet still be unable to login is not a great UX. This seems like one of those cases that's high impact when it does happen, never likely to occur on any given day, but likely to happen at some point; probably just wasn't much focus put on handling a case like this.

hnfong2• 2 years ago

From a process/QA perspective I doubt this can ever be properly tested.

Sure you can set up a UX to show that the auth server is somehow down and discourage users from trying to login/reset passwords, but when shit hits the fan, you actually never know the precise error that gets thrown to the client because it could be any layer between the backend and the client that failed...

jonnycomputer2• 2 years ago

It logged me out and told me that my credentials were incorrect; I thought my credentials had been stolen, so I'm kinda personally glad that it seems to be happening to a lot of other people too. I know that's a bit selfish, but :shrug:

MyFirstSass4• 2 years ago

Same, thought all of my friends were getting spammed and i'll look like a "boomer" who got phished.

Then i remembered i have a very long and secure password, then immediately panicked about someone having access to my Gmail.

The sense of security is more brittle than i thought.

sandspar4• 2 years ago

On a psychological note, I think the threat detection part of our brain doesn't always notify our conscious thought that it's actively monitoring for threats. I've often noticed that when I'm carefully handling a hot frying pan then my ringing phone is more likely to startle me than usual.

1010082• 2 years ago

I panicked the same. Even when I tried to recover my password, it said my email address wasn't associated to any account. I thought I lost it forever

marcosdumay5• 2 years ago

That's the natural endgame of the "user-facing services must not stop, if something they depend upon stops, they must only degrade" philosophy.

kortex3• 2 years ago

Same same. I went through the password reset flow (I was overdue anyways), it never sent anything to my SMS, so I did it again with email, reset the password and went to log in with the new password, "Incorrect password" error. Old password, also incorrect.

Didn't help that I had just posted a lukewarm spicy take on how linguistic prescriptivism is BS.

All the while the website felt like it was unstable, hard to describe, but it felt like it was bouncing around between URLs too much and reloading a lot.

Definitely feels like a botched update on their end.

E: Instagram is misbehaving as well, banner loads but big "Something is wrong" error on the feed.

E: now youtube has "Something went wrong" - WTF. I can't believe I'm saying this, but thank goodness for reddit and X[itter]???

E: interesting, seeing a big spike across multiple platforms on downdetector, including AWS: https://downdetector.com/status/aws-amazon-web-services/ I'm not able to log in right now, but that could be PEBCAK, I have too many saved IDs and I don't want to fail2ban myself

pratnala2• 2 years ago

The password reset flow was broken too. And I got logged out on every device. My friend who works there said they can't login either.

cratermoon2• 2 years ago

Discord is also having issues.

samaritano• 2 years ago

downdetector reports has gone down but to me is still bugged out, been catching a livestream on youtube all along though, meta stocks are back up from the dip so I take it some regions are restored to normality

NikolaNovak• 2 years ago

Yes. My spidey sense went off and I told my work I'll be off for an hour while I redo all my passwords... might still do that but glad to know it's not necessarily me getting hacked.

pratnala2• 2 years ago

don't bother. fb's forgot password flow is broken too.

smcl2• 2 years ago

I called out some comment for being racist a little earlier (yeah I know, just report and move on...) and figured they'd managed to pwn my account somehow. Good to know it's not just me.

fishnchips4• 2 years ago

Strictly speaking, just because there's an outage does not mean you're not pwn'ed.

suyash7• 2 years ago

me too, also Instagram, could be that Facebook got hacked ?

suyash7• 2 years ago

Pro tip: in chase meta actually got hacked, it would be good idea to not use that password on any other websites, change them immediately.

mfrommil4• 2 years ago

A much better UX would be clear error messaging informing users that the service is down and there is no problem with their individual account.

This would prevent people from panicking they've been hacked and/or unnecessarily resetting their password.

eurekin• 2 years ago

There was for a brief moment. I got that once

Kalium4• 2 years ago

You are absolutely correct. That would be a much better experience.

That said, getting there strikes me as pretty challenging. Automatically detecting a down state is difficult and any detection is inevitably both error-prone and only works for things people have thought of to check for. The more complex the systems in question, the greater the odds of things going haywire. At Meta's scale, that is likely to be nearly a daily event.

The obvious way to avoid those issues is a manual process. Problem there tends to be that the same service disruptions also tend to disrupt manual processes.

So you're right, but also I strongly suspect it's a much more difficult problem than it sounds like on the surface.

boring_twenties2• 2 years ago

Well you can't expect to hire engineers with half a brain for the pitiful compensation Meta offers, can you?

barbazoo• 2 years ago

Not the worst thing that a bunch of Facebook users are resetting their passwords.

mns• 2 years ago

There are quite some harsh comments here below. You can't plan for every possible failure point, who knows what part of a system/infra out of everything that they have went down and triggered this behaviour. Some things you just can't catch/predict. Especially in huge systems like theirs. I would expect people here to understand things like these and not just call people names for something like this, we all know things seem simple/clear from the outside, but the job of debugging and fixing something like this take quite some effort.

jonnycomputer2• 2 years ago

It would be a better UX, but, depending on the outage, that might be a really hard behavior to guarantee.

Twirrim• 2 years ago

Likewise, started password reset process that won't complete, asked my wife to double check my account wasn't compromised and posting cryptocurency crap or somesuch.

KMag2• 2 years ago

Yea, the wife came to me in a bit of a panic that her Facebook account got hacked. I tried logging in to FB to check if I had been unfriended, and I also got errors indicating my password was incorrect. My FB password is 96 bits from /dev/urandom in a GPG-based password manager I wrote for myself a couple decades ago. So, no my password wasn't wrong, and I'm not a big enough target for someone to put enough effort into figuring out how to snarf up my password data and crack my GPG passphrase.

Anyway, when FB thought my password was wrong I calmed way down. I thought maybe FB corrupted their password DB or something, so I just tried to reset my password, got into an odd workflow loop, and then quacked "downdetector facebook".

sigil• 2 years ago

My FB password is 96 bits from /dev/urandom in a GPG-based password manager I wrote for myself a couple decades ago.

We have the same approach to password management!

alkonaut• 2 years ago

Yeah your product should NEVER confuse an auth service being down with a failed auth. This is really terrible by FB.

simlevesque• 2 years ago

It really is.

jph• 2 years ago

Facebook is throwing up the sign in page, as if your sign in has failed or someone's logged you out. Instagram and Messenger also seem to be affected.

TheOtherHobbes• 2 years ago

And Threads.

coffedean2• 2 years ago

And the Quest headsets.

jzw88332• 2 years ago

Yes, just checked Instagram and it says “Couldn’t refresh feed” and is generally not loading. Messenger has also logged me out.

subtra3t2• 2 years ago

For some reason I couldn't browse the home page but I could still access my messages (website).

corobo2• 2 years ago

Oh good, I thought iOS Firefox was forgetting to load its cookie jar again

xenospn• 2 years ago

Yup. Kicked me out and won’t let me back in.

greatquux• 2 years ago

This will be great for productivity!

cm2187• 2 years ago

I thought the average facebook user was of retirement age.

epcoa• 2 years ago

Instagram is also affected.

graemep2• 2 years ago

Not sure about globally, but middle aged, 45 - 65, slightly more women.

I would bet those who are not working in full time jobs are a lot more active though.

subtra3t2• 2 years ago

In a lot of developing countries, Facebook is often the only app that a significant part of the population uses. And that includes young adults as well.

aranw• 2 years ago

Instagram has stopped working for me and I've also noticed other sites having issues like YouTube?

MyFirstSass4• 2 years ago

Yes both Youtube and Instagram seems to be down too for me.

jzw88332• 2 years ago

Instagram is down but Youtube seems to be fine for me

zalyh• 2 years ago

YouTube seems fine

acquacow2• 2 years ago

Yup, got logged out of everything about 25min ago.

jmiller1138• 2 years ago

yeah, randomly logged me out and now my pw doesn't work. The reset password options don't seem to be connecting.

rvz2• 2 years ago

Oh dear.

Facebook, WhatsApp, Instagram, Threads are all down. and its time to contact the CEO of Meta to bring them all back up.

At least Twitter / X is still up, so time to complain about it there.

lebean• 2 years ago

I'm on it! I WhatsApped Zuckerberg directly. He's aware of the situation.

lsh123• 2 years ago

WhatsApp works for me

maerF0x02• 2 years ago

+1 i was freaking out i got hacked cause I tend to overreact like that xD

afturner• 2 years ago

[deleted]

ink_13• 2 years ago

Meta doesn't do that. For one thing, "Patch Tuesday" is a Windows thing, and 0% of production traffic is served from Windows there. For another, they are constantly and always redeploying.

More likely that someone bungled a deploy of user auth. No doubt they are rolling back as we speak.

chongli• 2 years ago

More likely that someone bungled a deploy of user auth

How would that affect YouTube?

EarthLaunch• 2 years ago

I tried to do a SMS verify when I couldn't login, and the text never came. That service must be overloaded. That's when I realized it was an outage. Hundreds of thousands (millions?) must be doing that.

Also, going to /r/facebook doesn't load, heh, there must be per-subreddit load issues?

mikeytown2• 2 years ago

Don't panic, you didn't get hacked

neogodless6• 2 years ago

https://metastatus.com/

They are not down. (They just don't work for lots of people!)

acquacow2• 2 years ago

That hasn't even been updated since Mar 1... Edit, oh, now it has today's date, but still doesn't reflect reality.

neogodless6• 2 years ago

For me, the home page and Messenger Platform have date/time stamps like

Updated Mar 5 2024 10:33 AM EST

and

Updated Mar 5 2024 10:37 AM EST

maerF0x02• 2 years ago

25% down, but 100% Down for me is the same as 100% down.

neogodless6• 2 years ago

I'd say "down for lots of people" is the same as "having widespread issues." If it was just a single person, then it would be annoying but I wouldn't assume it's a system problem.

My point was just that, of course, the status page is not reflecting reality.

darkwater2• 2 years ago

There are lies, damned lies and... status pages.

According to them Whatsapp Business on premises solutions has issues since end of February.. But also looks like that WAB is the only product with API issues according to the status page.

shrikant• 2 years ago

They are not down. (They just don't work for lots of people!)

Seems about par for the course for big tech these days. There's currently an issue affecting the Google Ads API causing timeouts when sending data to it, but the Google Ads Status Summary page shows nothing [0]. However, there's an incident detail page showing some vague hand-wavy information about incidents [1], which appears to be unreachable from anywhere on the Summary page. Gah.

[0] https://ads.google.com/status/ [1] https://ads.google.com/status/publisher/incidents/gNG1ppoY3y...

p.s.: The actual incident details URLs are available in the "RSS feed" link very transiently and tend to disappear -- the feed, which incidentally for fun reasons, is actually an Atom feed.

zhan_eg• 2 years ago

It looks like Meta acknowledged a problem on their status page [0]

[0] https://metastatus.com/whatsapp-business-api at "5:32 PM GMT+2"

max_hammer• 2 years ago

Facebook login is also not working but status page is not updated.

https://metastatus.com/facebook-login

Platform Status : No known issues Mar 5 2024 4:38 PM GMT+1

The service is up and running with no known issues.

Crosseye_Jack• 2 years ago

They can't log in to update the status page about facebook login because they use facebook login for auth on the status page!

Joking aside, I wouldn't be surprised if this was the case, because during their last major outage (something related to DNS iirc) they were having issues pushing fixes because they couldn't login because their DNS was down.

EDIT: seems like the status page was recently updated.

basch5• 2 years ago

What would take Google and Facebook auth down at the same time? Coordinated 0day patch?

toomuchtodo• 2 years ago

[Edit: Misread the comment, mea culpa]

basch5• 2 years ago

For both companies? Or is the report of Google auth being down only tied to meta logins, and a misdiagnosis?

nathanaldensr4• 2 years ago

I had immediately started the account hijack process which, funnily enough, is also down. I was kicked off Messenger and Facebook and then my password was rejected by the login page. This whole flow made it seem like an account takeover and not an authentication outage.

Bad UX, Meta. Bad!

1010082• 2 years ago

What's the account hijack process btw?

saturn5k• 2 years ago

Session timed out. I'm in Macedonia. In the quick login menu there was another person with name and profile photo beside my name and profile photo. It seems the girl was from my country, don't know her. I thought I was hacked. This is messed up.

IG is not working as well. Feeds, profiles, messages are all blank.

dewmal2• 2 years ago

I'm from Sri Lanka. It's not working here; I was suddenly logged out from all devices.

throwaway3563• 2 years ago

In USA, same story here. Logged out and cannot log in.

OskarS• 2 years ago

So lovely that the status page indicates that everything is fine except the WhatsApp Business API!

Ragnarokkk• 2 years ago

I'm in France and same for me and my friends

miyuru2• 2 years ago

Sri Lankans cannot live without FB and its the afternoon here.

mdrzn• 2 years ago

FB logs you out and claims "Wrong password" if you try to login. IG just doesn't load new posts or comments, but doesn't log me out.

MyFirstSass4• 2 years ago

Anyone have a good guesstimate of how much money is "lost" when Youtube/Facebook/Instagram/Whatsapp (more?) are down each minute?

thehappypm5• 2 years ago

Naively, divide ad revenue by time to get a dollars-per-time.

But thats naive because ad serving isn’t totally sold out so they can make up for it by increasing the density of ads in the next time window. If the outage is short, then the impact is small.

But some markets are totally sold out and there’s no making up for lost impressions.

MarcellusDrum2• 2 years ago

This is something I've thought about a while back. Like Facebook probably has a "maximum number of ads shown to users per post" value. So theoretically, they have a ceiling for how many ads can be bought in a specific time frame before having to increase the ceiling/find new users.

Do they share data about this?

bmau52• 2 years ago

They also do not refund spend during this time typically so wouldn't be a 1:1

bananapub• 2 years ago

they can make up for it by increasing the density of ads in the next time window

no need to do that, for google search, people will come back later to make the search

kevincox• 2 years ago

they can make up for it by increasing the density of ads in the next time window

Not only that but the bigger spenders will have more budget so the bidding after a large outage should return higher bids on average leading to increased profit per ad slot.

mfrommil4• 2 years ago

This is rough napkin math, no need to downvote if anyone knows the real number and this is way off :)

Meta 2023 ad revenue was $131 billion. To make it easy, let's assume an even spread for # of users and ad revenue generation per hour/minute of the day and day of the year (which I'm sure is not the case).

This would be:

$358 million per day

$15 million per hour

$249k per minute

This also assume a minute down won't be somewhat or totally offset by a spike in users when it comes back online.

fullstop• 2 years ago

I didn't check your math, but your last number is probably per minute, not per hour.

seydor• 2 years ago

I assume they will run all those campaign views after yhe outage, so very little will be actually lost

zeven7• 2 years ago

I can’t answer your question, but when I was at Google I made a mistake that caused ads serving on Google results to become unclickable. For the postmortem they had me calculate (I don’t think a dollar amount but) the number of ad clicks that would have happened during the time it was down. Of course I looked up average cost per click rates. Not sure if I could share even if I remembered, but it really put things in perspective.

Overall it was a good learning experience. I didn’t get reprimanded; several months later I got a promotion.

strunz• 2 years ago

I can't believe you typed all that and didn't include a number, what a tease

ZiiS• 2 years ago

A hell of a lot less than when they are up.

FinNerd• 2 years ago

Instagram is also down

Rudraksh_15143• 2 years ago

Yes. The app does start (unlike Facebook) but then it's "Couldn't refresh feed" and nothing visible on the explore page.

darnfish• 2 years ago

Instagram wasn't working for me in New England, USA.

dadver• 2 years ago

Sweden, logged out from all devices, Google authentication to reset password results in error. Authorization codes sent via SMS to reset passwords seems non-responsive. Authorization codes to reset password sent by e-mail works, but setting passwords results in set password page two times in a row and after second try "An unexpected error occurred. Please try logging in again."

simne• 2 years ago

Greetings from Ukraine. And I confirm. First I sought, it is because some war issue in Ukraine (we are at war if somebody don't know), or somebody hacked me, but decided to check HN and seen this thread.

God bless HN and America!

kerirenee217• 2 years ago

Prayers for Ukraine!

johncalvinyoung• 2 years ago

Note that this status page is... less than helpful in the face of an obvious outage.

igorguerrero• 2 years ago

Yet another status board that's a lie... great.

RomanPushkin• 2 years ago

San Francisco, Instagram, Facebook both not working

mh-• 2 years ago

I don't think SF not working is related to this outage, though.

fddrdplktrew3• 2 years ago

Why do you say that? It could be exactly why Facebook is not working.

magixx• 2 years ago

Same issue on the East Coast in the US. All devices were logged out afaict.

ironmagma• 2 years ago

Neither is the only Facebook status page I found, interestingly. https://metastatus.com/ At least, it shows everything green (except for WhatsApp Business API).

dimillian• 2 years ago

Being logged out of every app is so insane. No automatic recovery from that. Big fuckup.

rightbyte3• 2 years ago

I guess there are alot of unsuspecting victims being logged in as dorment users and tracked.

A lot of people would probably also don't bother log back in, if there will be a password prompt?

egorfine• 2 years ago

Poland and Ukraine: all down.

jordemort• 2 years ago

Also got logged out an all devices, and can't get logged back in, or successfully complete a password reset. In the US. Glad it's not just me...

aeolis_mons• 2 years ago

Yep, I’m logged out and can neither log in nor reset. Mountain west, USA.

twoquestions• 2 years ago

Thank you for posting, I'm glad you can generally find service status better here than status pages that never show downtime.

fforflo• 2 years ago

Google and Meta auth, both down at the same time? Oooh. Conspiracy theories (or not) incoming...

lmlsna• 2 years ago

google oauth probably be getting slammed by people trying to use for facebook login

_zoltan_3• 2 years ago

you think Google can't scale it? I doubt.

coobird• 2 years ago

Same situation here in Japan -- all devices logged out of Facebook and cannot log back in.

rvz2• 2 years ago

Not just Facebook, all of Meta (Instagram, Threads, WhatsApp) is taking a breaks.

The Metaverse is temporarily closed for now.

Lets see where the majority of the discussion migrates to. (Twitter / X)

Rudraksh_15143• 2 years ago

WhatsApp seems to be working here in India.

intelkishan2• 2 years ago

True it's also working for me

graemep2• 2 years ago

and in the UK

dingnuts• 2 years ago

real discussion happens in private spaces. everything on Twitter/Facebook/etc is performative. including my comment.

palad1n• 2 years ago

Seoul Korea and can confirm. “Logged out.”

ykonstant2• 2 years ago

And rather than an indication that it is not working, I was just told to login again; since the attempts were unsuccessful, I was led to reset my password. Now I have no clue if the password has been reset properly or an old one is used, or login with google is used, or if I will continue to be logged out after they fix stuff.

If your service is down, please say "my service is down".

jeltz7• 2 years ago

Yeah, same thing happened to me so will likely need to reset my password again once Facebook is up again.

CivBase• 2 years ago

Logging everyone out across all platforms and telling them to just try logging back in sounds like an excellent way to DDOS yourself.

This should be a fun postmortem.

bbarn3• 2 years ago

Yeah, and they've dug themselves a hole with having near permanent login persistence. I can't remember the last time on a computer I use frequently that I've had to log in.

the42thdoctor• 2 years ago

That was my cue to check hacker news

dimask• 2 years ago

Me too, after the first couple of "is it down" websites I tried failed to load.

HN never fails us when a big website is down.

I was also almost ready to reset my password.

mrzaggy4• 2 years ago

Yup - same here...

I fear my initial attempts to 'reset my password' and getting that same security(at)facebookmail.com email with the SAME reset code, have not helped me - if only I'd been asleep, I probably could have slept through it.

At least Insty came back up - though to be honest I cant remember that password either and now terrified to try and recover or change that now too!

geerlingguy• 2 years ago

I lament the fact error messages are rarely, if ever, displayed anymore. Just a generic message in its place, usually not even indicating whether it's a 'you' problem or a 'them' problem :(

UniverseHacker2• 2 years ago

It amazes me how most smartphone apps- which operate in a wireless environment of likely flaky connections, don't seem to be tested or designed at all to handle loss of connections. They tend to just become non-responsive or do erroneous things like play the first second of an audio file and then auto-pause, without reporting any errors.

mewpmewp24• 2 years ago

There are so many microservices everywhere with different teams responsible, that if just one team doesn't forward errors properly it is messed up.

diggan3• 2 years ago

Yeah, but at least it's a bit understandable that its changed in that direction, and usually you can get a developer-friendly error message if you look at the HTTP responses in the devtools.

I remember one time a company I worked at received a support message where the title was (paraphrased) "DONT HURT MY CHILDREN" from some person who saw an error message saying something about "couldn't dispose of child" or something similar, when the frontend broke. "Child/children" being kind of common in programming, I'm sure others faced similar scenarios.

I'm not sure if they were genuinely scared or just decided to have some fun with us while reporting the issue, but after that we made the error messages even more generic, so nothing could be misunderstood.

I guess it is this quest to not alienate the lowest common denominator that is the reason behind the stupidification of error messages. On one hand, people won't get scared, on the other hand, people get less context about why the error happened in the first place.

Wonder how many of the average users actually care?

MIlBianco2• 2 years ago

I thought I got hacked and tried to retrieve my password.

Got a weird email with the same code every time from facebookmail.com

ceejayoz11• 2 years ago

I really hate that companies do this. PayPal does it sometimes, too.

sircastor2• 2 years ago

My first thought was that I'd been hacked as well...

I have gotten 4-6 "here's your facebook reset" type emails over the last month. All unrequested. I've always assumed they're casual attacks (mostly interested in seeing if my password can be stuffed into another, more valuable account)

teeray• 2 years ago

I'm just waiting for all my relatives to breathlessly call me saying "I THINK MY FACEBOOK WAS HACKED".

basch5• 2 years ago

Already got a call from someone who regularly gets caught in unprompted xfinity password reset loops saying “it’s happening again.”

Inexcusable to catch people in a reset loop during a login outage, and not confirm which password is the current one.

getlawgdon4• 2 years ago

I have exactly this issue. Been frantically jumping around this morning dealing with perceived security issues and have no idea where my many FB resets led to.

ffsm82• 2 years ago

Same... This was literally the first time I wanted to use my Facebook account in years, so I changed my password twice before I realized that there must be an issue on their end.

I just wanted to use their oauth login to buy a jonsbo n3 case from AliExpress. Sadge

mrweasel• 2 years ago

One weekend I was on-call, we had a system we ran for the government, but authentication was handled by another government owned service, hosted and managed by another company.

A user called in early Saturday evening, saying that the system was down. After a bit of debugging, wondering where our alerting had failed, I concluded that the system was perfectly fine, but the authentication service had been returning 500 errors for around an hour. When I called the user back he made a comment that rather changed how I think about monitoring and systems. He says: Well, from my point of view it really doesn't matter which part isn't working, it's just down.

muttled2• 2 years ago

I've learned the same lesson, and why things like ping checks aren't enough. Even just a WGET of the front page isn't enough. You need the monitor to login and replicate at least part of the user experience if you have a complex setup. It's very embarrassing when someone tells a user "shows it's up from my side." and they're relying on a naïve monitor.

bertan14• 2 years ago

MY SERVICE IS DOWN.... I did exactly as you.. I have no clue now

mrzaggy4• 2 years ago

Password works on THREE other machines, fails completely on ONE.

I was logged in when the outage happened on machine A and phone B - immediately tried to reset password, which took me into the hellish abyss many of us are experiencing now...

This evening - about 18hrs about the event, I fire up machine C - it logs STRAIGHT INTO Messenger. OK... This is something, too scared to open a browser in case that triggers the session disconnect...

So I fire up laptop D - STRAIGHT INTO MESSENGER... OK, open browser, STRAIGHT INTO FB. Log into Google password manager, VISUALLY CHECK password, and it is my last known good password (in use at time of outage).

Fire up iPad E - Straight into Messenger!!! All these machines are on the same network!

Back to Machine A - clear cookies and try to log in, no joy; different browser and try to log in, no joy; try to reset password on this different browser (I might add, I did get a new Change password Token number off this attempt), but no joy; clear cookies and restart, then attempt to log in, no joy!

WHAT THE F?!?!?!?

I initially thought it may have been a 24hr block due to password change attempts? But now not so sure... I've also tried logging in on via Machine A in a VM from a different O/S to see if it may have something to do with it - but again no joy - this environment had NOT been logged in to FB before...

Thoughts????

mrzaggy4• 2 years ago

MAC Address being flagged?

But on the flip side, I was able to create a back up profile in a different VM on this machine.....

nonfamous• 2 years ago

Threads isn’t working either.

ergonaught• 2 years ago

Thankfully there are no known issues.

SSLy2• 2 years ago

threads.net seem down too.

world2vec• 2 years ago

UK: Instagram iOS app not working.

alephnan• 2 years ago

If there was an AI armageddon, this would be how it starts

alex_young• 2 years ago

  > Facebook is not working

That’s very meta.

aaomidi2• 2 years ago

This might be the end of Facebook if it lasts long enough.

Invalidating people’s sessions on apps? That’s a HUGE cost. There’s a huge % of users that won’t be able to get back in.

AlfredBarnes• 2 years ago

I hope so.

dddddaviddddd• 2 years ago

I've known several people who, when they got signed out, couldn't remember their password and access their email, and so made a new account. I'm sure Facebook will see a spike of this.

jiminymcmoogley• 2 years ago

yeah i'm hoping it was just an auth server outage and browser sessions werent invalidated because this basically described my situation, but i wont have access to the only device i own that still (hopefully) has a an active login session until the weekend to find out

00deadbeef• 2 years ago

It might not have actually invalidated the session

aaomidi2• 2 years ago

It does on the app. Logs you out.

ur-whale2• 2 years ago

Looks like their entire auth service is completely borked.

[EDIT] : and if it's a hack, by now half of the world has typed in their password to re-identify ... scary.

qaq• 2 years ago

I thought someone hacked my account ...

basch5• 2 years ago

What would take Google and Facebook auth down at the same time? Coordinated 0day patch?

Or is the report of Google auth being down actually tied only to meta logins?

jawns2• 2 years ago

That's my best guess. And for them to log out every user in the world makes me incredibly curious about what would have happened if they'd chosen a different course.

basch5• 2 years ago

I think I’ve experienced 2 or 3 global session resets on fb in my life. Usually followed by some kind of reason they had to protect everyone. This probably isn’t great, hopefully a precaution not an active exploit.

drewg123• 2 years ago

I was wondering if Google was just melting down under a tidal wave of password reset emails from FB/IG..

My gmail seems to be working fine, both personal and work.

neogodless6• 2 years ago

See https://news.ycombinator.com/item?id=39604590 (5 minutes earlier)

Facebook, Instagram, WhatsApp outage (downdetector.com)

MandieD3• 2 years ago

Special bonus: today is Super Tuesday, which is when a large number of US states have their primary elections.

bbarn3• 2 years ago

I've seen this posted everywhere like it's some grand conspiracy.

What is the impact of it being Super Tuesday? Are people worried they can't vote without social media?

ak_1114• 2 years ago

Actually I can see this morphing into a Russia interference conspiracy.

If the main social media used by liberals goes down, while the one used by Trump (forgot what it's called) stays up, surely that is an advantage for him?

onlyrealcuzzo• 2 years ago

Social Media is very successful in Get Out the Vote campaigns for younger people - who typically vote for one party disproportionately.

mynameishere• 2 years ago

The primaries are pretty well decided, so I suppose the conspiracy would be "trial run".

lxgr• 2 years ago

Facebook and other social media platforms have played somewhat of a role in the 2016 election, for example. Meta has been under a lot of pressure for that, compare e.g. this recent Instagram change: https://about.instagram.com/blog/announcements/continuing-ou...

kortex3• 2 years ago

Messenger is often used to coordinate logistics among friends. I would not be surprised if a Meta disruption lead to at least momentary confusion if someone was planning on carpooling. Not to mention it messes with "get out the vote" posting.

I'm not very conspiracy-minded but this does smell a little weird.

At the very least, "there's a big event in the country of one of our biggest userbases, maybe hold off on risky deploys until tomorrow"

InitialLastName• 2 years ago

I can't get into the US-local conspiracy-mongering, but I'm not sure how you've missed social media becoming a hot-spot for election-day information/misinformation?

Shutting down social media has gone to the top of the list for regimes either "attempting to fabricate positive election results" or "attempting to combat the spread of misinformation about elections".

More sympathetically, for better or for worse (definitely the latter) there will be people trying to look up election information ("what are my local polling hours", "who is on my ballot") on social media websites, who will now not be able to be guided to the correct information.

nso• 2 years ago

I'm sure people will be able to vote even if their vacation pictures are unavailable

jeltz7• 2 years ago

I agree with the first part but the second is taking it too far. Plenty of people use Facebook messenger for communication about semi-important things. Not seeing how that would affect the primaries but it is not jsut for vacation pictures.

Ancapistani2• 2 years ago

Not relevant to the primary election in any way, but this has been more than annoying to me already.

We have a small livestock operation, and won an online auction late last night for a pig about four hours away. Facebook was the only listed means of contacting the person, and we were planning on driving to pick it up this morning.

Now I get to re-arrange my day today to deal with that, and will probably have to take a PTO day from work to drive there later in the week.

Real businesses are in fact impacted by Facebook being down - including those not based around Facebook and that you might never expect.

Brian_K_White• 2 years ago

Did they imply anything as ridiculous as needing facebook to vote? The implication is merely that everyone would otherwise have been talking about voting and the results, ie, simply higher than normal traffic. (I'm not sure it would be that much but that was the reasonable interpretation of the comment.)

apetresc• 2 years ago

Neither party's primary is in the slightest way contested, so what do you think the "bonus" is here?

ak_1114• 2 years ago

If we get huge surge of "Uncommitted" votes to Biden as we did in Michigan, you can argue this can cause significant pressure on him and might even lead to his ousting...

issafram2• 2 years ago

Seems like somehow YouTube is also affected? I'm starting to think that this is an attack of sorts and not Meta specific

issafram2• 2 years ago

Google Play Store also affected

beejiu• 2 years ago

Given Facebook is like 10% of total Internet bandwidth, I'm curious what happens when all that traffic suddenly gets "redirected" elsewhere by people going to other sites and platforms. Could YouTube cope with a sudden 10% uptick in streams for example?

Edit: YouTube seems now to be defaulting to a lower bitrate, leading me to guess it's a demand issue.

ykonstant2• 2 years ago

That seems to be the actual case. Most of the "usual web" is struggling. Even HN is struggling for me.

bigwheeler2• 2 years ago

I was logged in on mac, and the page refreshed and logged me out without any interaction from me, while I was reading a post. Then I went to check on my phone and I got the "We had a problem with the page you tried to reach." dialog box.

julianeon• 2 years ago

Facebook is not working, but you wouldn't know it from this page, which says everything is working.

intelkishan2• 2 years ago

It's the same for me in India

yakshaving_jgt• 2 years ago

I guess we all better do some work today then.

usrusr• 2 years ago

For me it happened when my long-dormant messenger lighted up (for a perfectly valid reason, no surprise there) and made me go through those unbundling options presumably mandated by EU. Certainly a coincidence, but it does irrationally strengthen that satisfying feeling I get whenever I see the ad giant stumble.

indigodaddy• 2 years ago

Does FB still have that corporate chat/slack competitor offering? Personally I’m glad for the FB “forced” break, but may not be great for those users..

DyingAdonis• 2 years ago

Sounds like they messed up rotating their encryption key.

ivraatiems• 2 years ago

I was logged out of all devices and told my password was incorrect. It looks identical to a hijacked account attack and was quite scary until I started seeing similar stories from others.

amingilani• 2 years ago

Google’s oauth logins are timing out too.

nyolfen• 2 years ago

many independent major services are having significant spikes https://downdetector.com/

MIlBianco2• 2 years ago

I thought I got hacked and tried to retrieve my password.

Got a weird email with the same code every time from facebook ail.com

neogodless6• 2 years ago

Status has just updated (as of 10:49 AM ET) to show that there are outages.

TrackerFF• 2 years ago

The first thing I did was to reset me password(s), and wonder if my devices had been compromised.

daemonologist2• 2 years ago

It seems to not just be Meta - sites like Downdetector[0] are showing a spike in reported issues from AWS, Google services, and X/Twitter as well. I noticed issues with Google myself.

[0] https://downdetector.com/

jcwayne• 2 years ago

The sparklines on Downdetector's homepage can't be compared to each other. Spikes that look similar can actually have a difference of several orders of magnitude. Only meta's services have truly large spikes.

daemonologist2• 2 years ago

That's true and an excellent point. I commented about the reported issues elsewhere mainly because I experienced them myself (google.com and drive.google.com not loading or being extremely slow to load content). That could be entirely sympathetic though - people having issues with Meta flooding other services and briefly overwhelming them.

uticus2• 2 years ago

what are stats for Facebook uptime? i can't remember last time large parts of it went down like this.

jeltz7• 2 years ago

Probably not very good. They had a major outage in 2021 too. And from people I know who work at Meta I am not surprised at all.

https://www.theverge.com/2021/10/4/22709575/facebook-outage-...

MiddleEndian2• 2 years ago

Discord also seems to be down for me

kortex3• 2 years ago

Discord loaded from scratch for me, but it took a much longer time than I usually remember for the feed to come up.

MiddleEndian2• 2 years ago

I've given it a lot of time, no luck. It displays:

ISSUE STARTING SESSIONS

We are investigating the issue

trashtestcrash• 2 years ago

and I thought I was having a bad day

giancarlostoro2• 2 years ago

This means IG is also down. Which means probably a network tier screw up yet again. Idk why they design a monolith network that basically can put out all their infrastructure.

jeltz7• 2 years ago

Does not look like a network issue, my guess is that they fucked up something major with their auth server when deploying a change related to the EU Digital Markets Act.

giancarlostoro2• 2 years ago

Theres other apps that went down including Discord how is that not a network issue?

Euphorbium• 2 years ago

They recently started demanding that I log in on a phone app, to make some choice to use messenger on ipad. That is pretty much the end of messenger for me.

uticus2• 2 years ago

me: wonder if there's a problem? i'll find a status site

me 2 seconds later: nah, i'll check HN

me 5 seconds later: ahh, there it is

dewmal2• 2 years ago

Its work all the time

sircastor2• 2 years ago

I assumed it was a "me" issue. They really ought to put something on the login page. Then I found out my wife had been logged out too.

I started to go through the password reset process and that failed as well. Then I got here.

_zoltan_3• 2 years ago

not only Facebook, Google loggeed me out as well, and had trouble logging back in. downdetector shows nicely that almost everything had a hickup. what are the chances of this?

captainbland• 2 years ago

The domain reads a lot like "metastasis" which fits quite well with the social media landscape and oodles of terrible "suggested" content on Facebook in general

wewu2• 2 years ago

logged out of fb, i sta, and msgr in NY

HK4ur• 2 years ago

I had just added Google Authenticator as a two factor ligon for both Instagram and FB not ten min before it happened so I thought it was something got messed up when setting that up. I got a msg on FB saying session expired and tried to log back on with Unexpected error. Then thought I was hacked until my husband confirmed same on his account.

wewu2• 2 years ago

logged out of fb, insta, and msgr in NY

kossTKR2• 2 years ago

What are the chances this is GPT-5 that has escaped and gone rogue?

Jest aside, i wonder when/if stuff like that will actually happen.

fishnchips4• 2 years ago

Negligible. It would have gone after Twitter first.

prepend• 2 years ago

Comically twitter seems to be working fine and is where I was able to find confirmation that it wasn’t just me.

Woshiwuja3• 2 years ago

You mean X right?

causal• 2 years ago

Clearly the work of Llama 3

meindnoch5• 2 years ago

About 95%.

corytheboyd• 2 years ago

My shadow is extremely good at mimicking my every move, but I don’t live in fear that it will make the jump to the third dimension, kill me, and assume my identity. Should I?!

TeMPOraL• 2 years ago

Well, maybe - if the source of the shadow keeps getting more and more powerful, it will eventually turn you into a permanent shadow.

1970-01-01• 2 years ago

Which one this time - BGP or DNS?

Either way, global GDP should be up :)

ototot2• 2 years ago

Given the range. Doesn't look like DNS

marcosdumay5• 2 years ago

So... No way it could be DNS?

kordlessagain• 2 years ago

My sense is that it's BGP.

gene313372• 2 years ago

I'm betting memory.

Althuns• 2 years ago

Every outage there is a discussion about how these status pages are failing to adequately notify and describe the problem. Is there anyone out there doing it right?

HK4urCAN• 2 years ago

I had just added Google Authenticator as a two factor logon for both Instagram and FB not ten min before it happened so I thought it was something got messed up when setting that up. I got a msg on FB saying session expired and tried to log back on with Unexpected error. Then thought I was hacked until my husband confirmed same on his account.

davisr• 2 years ago

There isn't a single part of my life that is affected by this. Good riddance; Facebook is a psychological cancer.

TedHartDavis• 2 years ago

The lines on DownDetector are fun.

askonomm• 2 years ago

Estonia: all down

wenbin• 2 years ago

Google login seems to be slow just now for 2 of our websites

synergy20• 2 years ago

yes it's all broken now, so I can take a short break here :)

notsahil• 2 years ago

Discord is having issues too! https://discordstatus.com/

montekristooGDB2• 2 years ago

wddkcs• 2 years ago

Could this be related to Google's log in page change? Seemed cosmetic only, but funny timing that Google's page update happens the same time all these log in issues pop up.

chimeracoder3• 2 years ago

Could this be related to Google's log in page change? Seemed cosmetic only, but funny timing that Google's page update happens the same time all these log in issues pop up.

That rollout is staggered over time, so not all users receive it at the same time. It's unlikely to be related.

shombaboor• 2 years ago

They've been previewing that change for weeks, and I wondered: why do they need to change it at all? Is it some product manager justifying his need to exist?

Shrezzing• 2 years ago

I think the downtime associated with other services could just be people choosing alternative sites for their social media time.

Facebook + Insta makes up a huge share of the social media market, and when they go offline, it'd be natural for their competitors to receive large sudden upticks in trafic they're not immediately prepared for on a Tuesday morning.

happosai• 2 years ago

Scary idea but could indeed be. Who let the zombies out!

BaldricksGhost2• 2 years ago

Threads is down too. Given its integration with Instagram that's not exactly surprising though.

suyash7• 2 years ago

Official page to check status: https://metastatus.com

mips_r4300i2• 2 years ago

Status page is down for me now too

monsieurgaufre2• 2 years ago

Also can't logged into my openAI account.

Not saying is related, but timing is weird.

r4nd0m_jump• 2 years ago

if you are logging in with facebook account that might be the reason

monsieurgaufre2• 2 years ago

Indeed if it was the case. But i always create an account with my email for this kind of situation.

wolfpanke2• 2 years ago

if you try to reset your password you will receive each time the same CODE number for auth....

mips_r4300i2• 2 years ago

USA, Instagram app broke about 5 minutes ago.

vincnetas2• 2 years ago

every one panicked that they got hacked. i got a slim hope that i also got hacked and that i will not bother to recover my account and just roll without FB :) im too week to quit my self.

revscat2• 2 years ago

Maybe post your password to Twitter or something. Let it escape into the wild, where it can be observed in its natural state.

Liquidor• 2 years ago

I know your intention is to help, but please don't share your FB password (if that wasn't obvious already lol). Letting randoms log into your FB account will just have massive consequences with your friends and family thinking it's you talking to them etc.

noman-land• 2 years ago

Send me your password and I'll log in and change it to nonsense. Get out of that heck hole.

avgDev• 2 years ago

I tried logging in, I couldn't and was like "nice, I guess I want to have to look at garbage today".

I really would like a social network just for friends without all the garbage bloat.

baby6• 2 years ago

That's what Facebook used to be. I think they really lost their ways when they went from "useful tool" to "let's try to get users to spend as much time as possible".

I dream of a "facebook-like" app where you can only add someone as friend via a bluetooth protocol, forcing you to only add people that you've met in real life. Then text only, or with very limited image options.

dv_dt2• 2 years ago

Well, I've been spitballing 1-3 years for effects of layoffs to affect company performance, so this timeframe is about right

lazide4• 2 years ago

Yup. That it’s related to the elections is also predictable, due to stress.

Made worse in big corp due to affirmative action + lack of enough qualified candidates meeting diversity criteria.

Which is inevitable when you have coarse criteria applied to such a large industry this way so quickly, as it takes decades for anyone to be qualified for the senior roles, and many years for junior/mid level, even if there were no pipeline issues, which there are.

And unqualified folks in leadership, and mid level == stupid mistakes.

And, with the DOL rules, the company can’t even pay people differently, so no bueno even giving the high performers keeping things afloat better bonuses - unless they happen to meet the diversity criteria and it makes the stats look good.

Which it’s already hard enough to do properly when there is only one dimension, and impossible when there are 2-3.

so the bigger the company, the faster it has to cut its own throat.

binaryblitz3• 2 years ago

Blaming an outage on DEI... Man that's a new one for me.

Could you let us know where you work? I want to make sure I never apply there.

dv_dt2• 2 years ago

Your spitball is flying in a completely different universe than my spitball. Imo anytime you have layoffs that cut so deep, you cut through informal capability, knowledge and relationships that take a long time to form. If anything DEI helps create internal resilience because the personal networks end up a little different, giving you wider and more angles of coverage.

HN talks about people in open source holding up major functionality with little to no recognition. That happens within corporations too. Indiscriminate layoffs may directly fire those people, or signal to them that it's better to move elsewhere leaving gaps that only get discovered over time.

tapland• 2 years ago

Shortly before I received password recovery codes to my email, from all facebook accounts I have.

My email requires 2FA, as does my facebook account, but when I went there and clicked "forgot password" there was an unknown email address added to my account. That shouldn't be possible.

yreg3• 2 years ago

Your mouse click broke it.

montekristooGDB2• 2 years ago

Discord too ?

neogodless6• 2 years ago

https://metastatus.com/ seems to be suffering a hug of death now...

perfectritone• 2 years ago

It really should be a static page.

slig• 2 years ago

Built on React, though.

stefandesu• 2 years ago

Such irony...

Galacta7• 2 years ago

It could save others from clicking on downed sites, but not itself.

hiccuphippo• 2 years ago

Tangent to this but I saw a link to metastatus last week and thought it would be a status page for other services' status pages. This makes it sound like a useful thing now, too bad the name is taken.

jcgrillo• 2 years ago

Could call it metastasis, apropos when one outage cascades into others.

thomasjudge• 2 years ago

During the outage it was cheerily reporting no problems

Tarball10• 2 years ago

I see Corporate Memphis has even infected status pages now.

suyash7• 2 years ago

Official page to check status: https://metastatus.com

suyash7• 2 years ago

It would have been smarter if they hosted this on a separate reliable server altogether.

Ambroos2• 2 years ago

It's on AWS, but likely just has issues with provisioned capacity now that it's actually being hammered.

Meta has a small collection of tools on AWS to deal with large SEV0 events like these. Another one of them is a basic communication tool that does not use Meta's own servers for anything (including auth), a super basic version of the internal SEV tool.

pwenzel• 2 years ago

Might as well rename it from Meta Status to Metastasis.

Woshiwuja3• 2 years ago

Thats down too. lmao

wolfpanke2• 2 years ago

IF you try to recover your password you will receive the exact auth code each time that you request it.

meindnoch5• 2 years ago

This should be considered an act of war.

kerirenee217• 2 years ago

What do you mean by this? I find all of this very odd. Everything that’s been happening.

meindnoch5• 2 years ago

You're witnessing the greatest attack on Internet infrastructure in history.

Amphibious assault on Taiwan imminent.

coldcode• 2 years ago

It appears to be more than Meta, lots of things are showing a spike in reports, including AWS.

rightbyte3• 2 years ago

That might be the FB traffic instead going to the content producers? I mean half the internet users need to go somewhere else if they want procrastination.

MPSimmons• 2 years ago

My password didn't work, and I was like, "oh man, it's finally happened - I've clicked on the wrong link somewhere and my account has been owned".

Whew.

meindnoch5• 2 years ago

Meta stock in freefall. Going short at 20x leverage.

robjan• 2 years ago

Risky move. The slight drop (and subsequent recovery) today is unlikely related to this outage in any way.

getlawgdon4• 2 years ago

Can't tell if serious, but I've seen worse trades. Wouldn't call this a freefall yet: https://imgur.com/a/0UN3HZa

greenie_beans• 2 years ago

OH MY GOD WHAT IS THE WORLD GONNA DO!!!!!?????

nathanaldensr4• 2 years ago

Touch grass. xD

albatros13132• 2 years ago

I love how no one here is talking about the "coincidental" timing that this is happening on Super Tuesday. Carry on, nothing to see here.

cjohnson318• 2 years ago

We've seen continuous layoffs for the past year or so, and the stability of everything is wonky now. Yesterday I had trouble with LinkedIn, last week it was Coinbase, a month or so ago Gmail was hanging. I don't think it's because of Super Tuesday.

nathanaldensr4• 2 years ago

I have yet to see one plausible reason for why this would affect primary voting in any way. Voting is done at the local level on local hardware; we don't vote on Facebook.

dawnofdusk• 2 years ago

Messenger is also down. For some people this is an important means of communication (many young people I know in the US). People may either be in a panic or otherwise cannot coordinate sufficiently the plans they may have had to get to the polls.

That being said this year's primaries have got to be historically uncontested that it could not matter less.

angst_ridden2• 2 years ago

It’s not direct disruption.

Grassroots organizers use FB and messenger and Insta for “get out the vote” communications. People use these services to goad their friends to go and vote. People use FB as a search engine to identify their polling place.

Waterluvian3• 2 years ago

Same reason we're also not talking about the ten million non-coincidences that didn't seem to happen today. It's a kind of survivorship bias that coincidences seem like they mean something.

Or you could imagine it as this: your animal brain has evolved to notice patterns. Seeing coincidences like this is akin to seeing faces in the stars. The challenge of evolved being is to override those impulses when they're not logically sound.

getlawgdon4• 2 years ago

Sure. But if millions of evolved brains are "seeing the same pattern", they will behave as such, and so the meaning of that otherwise random noise in fact becomes something material.

binaryblitz3• 2 years ago

If this was 2016, I'd think there could be a possibility. Given that the candidates for the election are almost 100% a foregone conclusion at this point, what purpose would it serve?

rantee• 2 years ago

Fueling clickbait conspiracy theories is one probable outcome that comes to mind. For example, even a winning candidate may claim that a "woke" employee sabotaged the site in an effort to subvert the will of the people to have free and open discourse, they won despite the media deck stacked against them, etc.

granger07022• 2 years ago

Same here in Vietnam, nothin working at all

Codegres• 2 years ago

Yup. Google OAuth and Facebook, Instagram all are down.

I think its a case of bad deployment by Google.

They recently introduced new OAuth flow.

Might be related.

No idea how long it will take to fix it.

granger07022• 2 years ago

same here in Vietnam, I can't contact my colleagues now

angst_ridden2• 2 years ago

I’m seeing a lot of non-FB services down too. Mostly AWS-based, but not all. My original conspiracy thoughts (this being Super Tuesday in the US) are giving away to thinking it’s some low level routing issue.

albatros13132• 2 years ago

It's definitely related to Super Tuesday. Why kill 2 birds with one stone when you can kill 10?

Waterluvian3• 2 years ago

I'm having one heck of a time getting into Discord or getting YouTube to show videos. Anyone else experiencing issues outside of the Meta realm?

NKosmatos• 2 years ago

Obligatory DNS joke :-)

It's not the DNS, It can't be the DNS, It might be the DNS, It's the DNS

Thaxll• 2 years ago

SRE now have real work to do.

wly_cdgr2• 2 years ago

I've never seen an outage this bad and I've been on Facebook since the oughts.

Ambroos2• 2 years ago

The 2021 outage was worse, it completely killed the network access to FB data centers for hours, and even led to issues with FB employees accessing offices (since the badge readers were also offline). It was so bad it got it's own Wikipedia page: https://en.wikipedia.org/wiki/2021_Facebook_outage

wly_cdgr2• 2 years ago

I don't remember that one - must have been on a hiking trip. This one might turn out to be worse yet! Hopefully not.

unavida• 2 years ago

my heart rate was pounding when I couldn't login and couldn't reset my password just now. Seeing this HN post made me come back down to normal. I'm now going to move away from FB, Google, and all other major tech companies I rely on.

jeromegv• 2 years ago

Mastodon is up!

marcosdumay5• 2 years ago

Just checked again to be sure, even lemmy.world is up.

kerirenee217• 2 years ago

Anyone else find this shit weird? First Nation wide outages with all phone carriers, mainly AT&T and Verizon and T-Mobile… massive outage… and then now Facebook and instagram. Then banks were having outages and pharmacies.. and when the phone outages accrued mine was working fine and my fiancés was not. I just find all this weird… feel like there’s more to it!

joemanaco• 2 years ago

Has a meta developer used Chat GPT again?

DeusExMachina• 2 years ago

Looking at the Downdetector home page [1], it looks like many more services are having outages, not just the ones owned by Meta, including:

- Google

- YouTube

- Google Play

- T-Mobile

- X (Twitter)

- Discord

- TikTok

- Pokemon Go

- Snapchat

It looks like they all have the same failure point.

[1] https://downdetector.com

andtheboat• 2 years ago

iirc these all use GCP which would make sense for them all to be disrupted at the same time. I wouldn't have thought Meta was GCP reliant though?

Yiin• 2 years ago

all of them are using oauth, likely auth provider issue?

bburnett44• 2 years ago

There’s no way fb uses gcp

ajross2• 2 years ago

Most of those seem OK for me now, and DD agrees. This seems to have been a temporary blip for all of them, possibly some kind of service switchover/fallback "not entirely unrelated" to the Meta outage?

Edit: actually a more attractive theory, given the very short timelines and near simultaneity of all those failures, is that downdetector itself had a failure, possibly a Meta-dependence, that they noticed and corrected quickly.

peterleiser3• 2 years ago

GCP seems fine, and no issues logging into Google Cloud Console.

peterleiser3• 2 years ago

S**'s gettin' real: https://downdetector.com/status/pokemon-go/

AndyJames• 2 years ago

Some piece of core infrastructure went down because everyone got spike at the same time. Surprisingly DoorDash and Steam was up

agilob• 2 years ago

Everything is going down, except Steam. You know where to find me.

dwighttk• 2 years ago

Scrolling down their list, why is DoorDash the only one that didn’t have a spike this morning?

esnard• 2 years ago

Pokémon GO player here, the app was absolutely fine unless you tried to use Facebook as your login provider.

tuan• 2 years ago

Fb is down but YT is still up for me.

jon-wood• 2 years ago

Or people are using Facebook Auth for them. I don't really trust Down Detector, which despite the claims is really People Winging on Twitter Detector.

redserk• 2 years ago

Yeah, it's wild that it's now treated as an authoritative source, especially by some news organizations.

It's as good as asking a neighbor what happened with a loud noise down the street. Sometimes you'll get something good, sometimes it'll be completely wrong.

rimunroe7• 2 years ago

I don't really trust Down Detector, which despite the claims is really People Winging on Twitter Detector.

I'm confused. Isn't listening for spikes in complaints about outages a great way to detect them? I know for a fact some service companies monitor social media channels for this purpose (among others). I'd be surprised if that wasn't more or less standard practice.

I've checked Down Detector for ISP outages in my area many times now. It's always confirmed them before my ISP did.

johnfn• 2 years ago

People are using Facebook Auth for YouTube?

KineticLensman• 2 years ago

Seems likely. TikTok and YouTube are currently working for me, while Meta platforms aren't.

tflol• 2 years ago

i trust Down Detector more than the (majority of) companies who are silent during outages

hell, i'm surprised Down Detector hasnt been outright sued due to the graphs being an actual honest representation of availability that shitty companies cannot hide

consumer451• 2 years ago

Or people are using Facebook Auth for them.

Gmail is also on the list. You can't use FB auth to login to Gmail, can you?

tzs• 2 years ago

YouTube was definitely doing something weird that doesn't seem likely connected to Facebook.

A couple hours ago after watching a video I went to my home page, which usually shows recommendations based on what I've recently watched plus a few videos labeled as sponsored that have nothing to do with any of my interests.

Instead everything on the home page was either a sponsored video, or a movie that was free to view with ads, or something from one of their music products.

I tried from an incognito window to see if it had something to do with being logged in. Normally going incognito loses the history-based recommendations but at least recommends user uploaded content. But now it has just like my logged in home page. No user content. Just ads and videos from Google's movie and music services.

Refreshing gave an error that said something went wrong. I then logged in on that page and again got something went wrong. Another refresh got a page with some user content. Another refresh was the ads and Google stuff page.

A little later it seemed to clear up and now my home page is back to normal.

EscargotCult• 2 years ago

Their outage heatmap is also basically a population density map too. https://xkcd.com/1138/

baby6• 2 years ago

It's not that, getting some issues using X

harryquach• 2 years ago

Thanks for sharing this, from what I have read it looks like an issue beyond just Facebook.

sbrother2• 2 years ago

HN seems to be struggling too, but that could just be everyone here to talk about the outages.

passwordoops2• 2 years ago

Happens whenever Sama sneezes too

GaggiX• 2 years ago

That's the standard HN experience, this site runs on a single core I believe.

paulddraper2• 2 years ago

HN is always struggling.

Couldn't use it two nights ago, IDK why.

govg• 2 years ago

There is no consistent scale on that graph, so any local maxima of reports received would look similar to any other.

mankyd• 2 years ago

Exactly this. FB topped out around 520,000 reports. Google topped out around 1,400. That's a massive difference in scale.

Both are above their baselines, but I bet some is just mis-reports, or increases in awareness due to more people checking in.

Meta seems to be the only one really affected from what I can tell.

NelsonMinar• 2 years ago

I made that same mistake after seeing someone post an unlabeled set of graphs to a Slack. The Google peak reported outages is about 0.25% of the Facebook peak. It seems reasonable some people just made a mistake.

btbuildem2• 2 years ago

What's your fave conspiracy theory? Massive cyberattack for Super Tuesday? Powers-that-be mandated takedown? Mossad sleeper agents activated? Covid-brain struck that one engineer attending to that one wire that kept everything going?

N19PEDL2• 2 years ago

Houthis destroying underwater cables in the Red Sea.

dylan6046• 2 years ago

I guess the swamp got drained, so there's no more flow through the tubes.

eddof13• 2 years ago

Google was acting up for me as well, so that could be

jedberg• 2 years ago

Man the postmortem on this is gonna be fun.

“Yeah so it turns out when Facebook and Instagram goes down so does Google”

I do not envy the SREs at either company. I'm pretty sure all those other ones use Facebook or Google as their OAuth provider which is why they are all being reported as down.

MandieD3• 2 years ago

Additional fun factor: today is Super Tuesday - primary elections in a lot of US states.

This outage will result in absolutely no ridiculous conspiracy theories.

kurthr2• 2 years ago

If your election integrity relies on Facebook, YouTube, or even DNS to be up... there are bigger issues.

Actually, if all of them including Xitter went down, maybe things would get better? All the sunlight photons might get sucked in by too many eyeballs though, and there could be grass trampling.

chakintosh• 2 years ago

Cloudflare or AWS ?

nickpsecurity• 2 years ago

A lot of sites have features that say “log in with your X or Y account.” They connect to each other somehow. I never studied that protocol. I wonder if authentication failures across services could be tied to it.

For process of elimination, do all of these services do multi-platform logins? Or do some not connect to anyone else?

dylan6046• 2 years ago

that actually infuriates me more than cookie banners. the one from Googs is the worst offender.

tdudhhu• 2 years ago

Services are coming back to live.

Interesting to see that all static content was still working during the outage (at least for Instagram). It was still possible to swipe through all reels (I assume the list was cached).

yosito• 2 years ago

I think the content was probably cached on your device.

partiallypro• 2 years ago

Down Detector is so unreliable. People that can't call an AT&T phone via Verizon will think (and report) that Verizon is down, when it's really AT&T. People can try logging in using Facebook's on click login and not be able to get in, so they think Tiktok is down. It's not all that useful. I hate when journalists cite it.

dylan6046• 2 years ago

It's only slightly better than "my mom claims". My mom would ask if I had the internet at my house. Yup. all of it. in a rack in my bedroom closet. She'd also report the "internet is down" when a single website was having issues. To me, that is down detector carrying on the legacy of moms everywhere.

jordanthoms5• 2 years ago

It has false positives and noise for sure, but it's also very sensitive and shows issues very quickly.

I wouldn't trust it as a single source, but in a case like this where our internal monitoring shows a spike of issues with the Google APIs and we can see a huge spike in reported issues for Google on Downdetector starting at the same time, it's useful to confirm that the issues have an external source.

pixl972• 2 years ago

Eh, mostly it's people misunderstanding what it represents.

If I can't login to tiktok because FB is down, then tiktok is effectively down for me. When it comes to technology most people don't care about the trip, they care about the destination.

So yea, tiktok isn't "down" but for a lot of people it might as well be, hence coupling your infrastructure/auth on other providers has side effects like this you must take into account.

tonymet• 2 years ago

Facebook audience is in the billions, so you will see 100k false positives when a big site like that goes down.

callalex4• 2 years ago

Downdetector has successfully detected 150 of the last 20 outages.

Its mention should honestly be banned from this site.

dzlobin• 2 years ago

This is just a knock-on affect from 1B+ users moving their timespent elsewhere during the outage

blastadon• 2 years ago

Yeah, I had issues accessing GCP's documentation site for AlloyDB around 8:00-9:00ish Eastern this morning. The page just said 'Service Unavailable'. Had to use Google Cache.

Vicinity96352• 2 years ago

X (Twitter) wasn't affected:

https://twitter.com/elonmusk/status/1765048551023734801

Downdetector is user reports, not automated monitoring. It's... semi-trustworthy.

chimeracoder3• 2 years ago

In general, linking to an Elon Musk tweet is hardly proof that a Twitter service degradation didn't happen.

Even more so when the tweet in question isn't even a direct claim about Twitter, but just a meme making fun of a competitor.

TuringNYC• 2 years ago

All the electronic candy/soda vending machines in our office are also not working. Shudder to think of the chain of dependencies inside these machines.

adawg4• 2 years ago

Is downdetector known for accuracy in these types of situations? Seems like a large amount of services out.

johntiger1• 2 years ago

Ironically, Meta employees are also affected by this

jeltz7• 2 years ago

Just like last time they had an outage.

lenerdenator3• 2 years ago

Gonna have my Angle Grinder controller ready for the new Sysadmin Simulator release

ototot2• 2 years ago

They layoff too many ppl.

C4RN12• 2 years ago

Way to go meta team...

fmsf2• 2 years ago

I was considering buying the Quest 3 this morning, this outage is timely. The fact that I can't use my perfectly working headset to play an offline game because facebook is down, makes me wonder if I should go for a different provider. Any recommendations? Excluding apple vision pro since it is too expensive.

practice92• 2 years ago

Unless it's a recent change, it works perfectly fine offline (wifi turned off).

As for alternatives, there is Pico, but Quest 3 may be superior in games selection. Or go wired which is of course less portable

fmsf2• 2 years ago

In my case I was doing iRacing with a cable, it requires the Oculus app to run for the link to work. Which in turn needs the login to be active.

dartos• 2 years ago

I think you need to be online for the first setup of the headset, otherwise it won’t work at all.

extraduder_ire• 2 years ago

This is different from it being offline, it's like the device is kicked off the associated account. A "something went wrong" window pops up with a "generate device code" button, with instructions on how to remove and re-pair the headset.

There is no way to even access wifi settings or anything else to disconnect the device from the internet. If it's still a problem much later in the day, I'll try turning off my router to see what that does.

bloaf• 2 years ago

I think the current buzz in the VR space is the "Bigscreen Beyond" which eschews all of the nice-to-haves in order to make the headset as light as possible, and the result is surprisingly compelling.

TulliusCicero13• 2 years ago

It looks compelling for high-end PC gaming VR enthusiasts, but if GP is more of a mainstream consumer it probably won't make sense for them.

At least from what I've read, there's a bunch of downsides for regular consumers: very expensive ($1000) -- SUPER expensive if you bundle in controllers and tracking points (~$1600), needs external tracking, wired instead of wireless, no built-in audio, can effectively only be used by one person (because each one is built custom to your face), and of course it's not a standalone headset, it has to be hooked up to a gaming PC.

basilgohar• 2 years ago

As a principle I think it is a good idea to avoid any technically unneccessary coupling of hardware or software to something else that is not strictly user-servicing, else the priorities are inverted in favor of the vendor's priorities and not your own.

CobrastanJorji• 2 years ago

I'm pretty sure the priorities are almost always inverted these days, unnecessary coupling or not.

bdangubic• 2 years ago

I have been in the industry for the last 25 years and have done more “technically unnecessary” things than there are grains of sand on Earth :)

lewsid• 2 years ago

I'm typing this from Immersed in the Quest 3, no issues.

TulliusCicero13• 2 years ago

The Quest 3 is kind of the obvious mainstream consumer choice for a VR headset.

Standalone wireless headset, reasonably powerful chipset, can optionally stream from a PC either wired or wireless, good optics/resolution, decent controllers/tracking, large game library, large suite of features (including hand tracking and color passthrough), all for a reasonable price. Not sure any other headset really competes on all those things at once.

citrusybread• 2 years ago

can we really call it standalone if you need it to phone home to Meta? and doesn't it require an account one of their services too?

they did a fantastic job with hardware. I just wish they didn't couple the software so tightly.

Analemma_• 2 years ago

The parent's comment is about how it quite literally is not a standalone headset, as is currently being demonstrated.

DonHopkins2• 2 years ago

Lucky you didn't already have a Quest 3, and you were in CyberSpace when Facebook crashed, because then you would be trapped in CyberSpace and die. That's how it works, you know.

idle_zealot2• 2 years ago

I can confirm that I was unable to use my Quest 3 this morning. I left it connected to the internet, it tried to phone home I guess, and then locked itself into a "please connect this headset to your Meta account" state.

I am so sick of companies "selling" computers that they continue to control. In what universe does Meta have the right to remotely lock my headset and prevent me from using it to run the software I installed on it? If I were to sell my current desktop computer, or phone, or whatever, on any marketplace, and leave a remote login account on it that I then used to continue to operate the computer as though it were mine remotely, installing software, playing games, and occasionally peeking at what the current owner was doing, that would be obviously criminal. How is this any different? Because I signed away my rights when I "agreed" to their Terms and Conditions box (which I was compelled to do to use the hardware I purchased)?

Something is so fundamentally broken in the current ownership/property landscape. We somehow ended up in a world where people don't own the most critical tools in their lives, companies have managed to recreate feudal fiefdoms within the bounds of the market.

rixthefox• 2 years ago

Ubisoft Director of Subscriptions really opened the floodgates of bad behavior when they came out saying "Gamers need to get comfortable with not owning their games".

I think these companies need to be reminded they do not own our PCs either.

I'm really starting to like that mantra of "If buying isn't owning then piracy isn't stealing".

bonton893• 2 years ago

I can understand your frustration but were you not aware of the software lock in when you bought it? I'm not defending the ownership erosion, but I avoided these things specifically because of who was selling it and how it was locked to them.

d00nut• 2 years ago

Steam Index is great if you don't care about cords / needing a PC. Easy to do VRchat, IRacing, or even blade & sorcery game sessions for 3-4+ hours without any eye strain, headaches, motion sickness, or discomfort from the headset.

It also fits over / around glasses

Biggest reason not to IMO is of the rumors around an "index2".

CoastalCoder• 2 years ago

Seconded, I think. I've been very happy with my Valve Index, but I don't know how it compares to newer headsets on the market.

More broadly, I find that Valve represents my interests far more than Facebook/Meta does. So I'd much rather send my money to Valve.

cesarb3• 2 years ago

Biggest reason not to IMO is of the rumors around an "index2".

Another big reason not to buy a Steam Index is "not available in your country". The only VR headsets I've seen in stores here are the Quest 2/3 and the PSVR2; and the Steam store page for the Valve Index (and the Steam Deck) says "not available in your country".

darrenf2• 2 years ago

I used my Quest 3 for a couple of gaming breaks during the outage with no issues.

darrenf2• 2 years ago

I am wondering why I had no issues, since Meta themselves are saying yep, Quests booted people out. I was definitely online (albeit not playing multiplayer). Odd. https://www.uploadvr.com/meta-explains-why-quest-headsets-st...

YeBanKo4• 2 years ago

I could not play a steam game on a PC in offline mode when my internet was down. This issue is not exclusive to fb.

samtheprogram• 2 years ago

Have never had this problem. There must have been something specific to your scenario, like the game wasn't downloaded, or it's a specific title?

IggleSniggle4• 2 years ago

That's not a Steam thing though, but rather the specific software. Steam explicitly has an exit path for the user if Valve disappeared overnight that allows their downloaded games to continue working offline.

The difference being that we are discussing platforms, not the things that run on those platforms.

paulryanrogers• 2 years ago

Doesn't Steam still offer an offline mode that works with most games?

cesarb3• 2 years ago

Were you still connected to your local network? Next time it happens, try completely disconnecting from your local network before starting Steam, it seems to use the presence of a local network connection to decide whether to enable offline mode or not.

ZealousFerret• 2 years ago

Quest is the best option right now

makestuff• 2 years ago

Is there any reason to get the quest pro over the quest 3? Or should I wait for a quest pro refresh?

callalex4• 2 years ago

Whatever you do, get one of the all-in-ones that doesn’t require setting up tracking beacons. They are universally a pain and will cause you to never bother playing.

m3kw9• 2 years ago

You can’t have everything. If you want a VR headset you described, there may not be any good ecosystem of software yet and you’d have to wait. This means you are gonna use your headset even less, think about how often fb goes down, it goes down for 2-3 hours once a year and it also has to coincide with the time you are using it? It doesn’t make sense to be this risk adverse

squidsoup• 2 years ago

The Sony PSVR2 is getting PC support by end of year apparently.

zer0zzz• 2 years ago

This is pretty cool. I might get one if I’m convinced it’s better than my quest3s display link.

ahupp• 2 years ago

There may be a bug or change since I left, but I built the app library and authorization logic, and it was explicitly designed to work offline. Of course, using it day-to-day and initial setup are different and I'd imagine if Apple is down it's hard to setup an AVP as well.

EMCymatics• 2 years ago

Offline gaming should never have been destroyed.

Wonder what data they collect from gamers.

Jemm• 2 years ago

I was able to use my Quest 3 throughout the outage for offline games.

xp842• 2 years ago

imho this is a stance you certainly have every right to take, but good luck. If you want to be part of the world of things like VR, smartphones, etc. then refusing (on principle) to participate in things like "accounts" and "cloud" is going to cost you far more time than the number of hours a massive company like Meta may have downtime. Likewise, yes, at some undetermined future date a lot of this hardware will become a complete doorstop due to their supporting servers being taken offline, however again, if you are doing this advanced gadget thing, it'll be long after you would have decided to upgrade to new hardware anyway because it can't do any of the latest stuff.

(and yes, there are ways if you're devoted enough, to roll your own everything and run Linux on a Framework laptop, and use some kind of custom ROM on your phone without Google anything, 3d print yourself a VR headset, etc. But all of this will cost you several orders of magnitude more time than Meta outages ever would.

wojciem• 2 years ago

Llama3 become sentient

rany_2• 2 years ago

The status page seems down for me as well. I just get a never-ending loading screen.

dudeinhawaii• 2 years ago

If you have an adblocker, try disabling that. The page fails to load with ublock Origin running.

XCSme• 2 years ago

What could affect so many different services? Some DNS server? Clould provider like AWS?

jgehrcke3• 2 years ago

Underwater cables seemingly got attacked; the Red Sea situation is likely related: https://apnews.com/article/red-sea-undersea-cables-yemen-hou...

epgui• 2 years ago

Timeline doesn't seem to line up

jeltz7• 2 years ago

Probably the authentication serer being down.

nonrandomstring• 2 years ago

Lovely bit of cloaked dramatic irony in "Leave The World Behind" t=1:11:20 [0]

Ruth (Myha'la): This is for the better.

G.H (Mahershala Ali): For who?

[0] https://en.wikipedia.org/wiki/Leave_the_World_Behind_(film)

dmk• 2 years ago

Google login also seems to be having issues, multiple people reported to me that the login isn’t working and they’ve been logged out of their Google accounts.

ExoticPearTree• 2 years ago

Yes, I tried logging in today in two distinct Google accounts on separate Chrome profiles and it would sign me out in about ~ 5 seconds after logging in. And the login process was very sluggish.

slowhadoken2• 2 years ago

YouTube is down too. Only thing working is Twitter.

bookofjoe3• 2 years ago

YouTube works fine for me in Charlottesville, Virginia, U.S.A.

slowhadoken2• 2 years ago

It’s working again. It’s been experiencing outages since 6am I guess.

coffedean2• 2 years ago

YouTube seems okay here. East coast US.

jgehrcke3• 2 years ago

Might be related to this event?

The cut lines include Asia-Africa-Europe 1, the Europe India Gateway, Seacom and TGN-Gulf, Hong Kong-based HGC Global Communications said. It described the cuts as affecting 25% of the traffic flowing through the Red Sea.

https://apnews.com/article/red-sea-undersea-cables-yemen-hou...

https://www.bbc.com/news/world-middle-east-68478828.amp

suyash7• 2 years ago

time to move away from undersea cables to satellites.

BearOso• 2 years ago

LEO satellites would be too inconsistent, and further orbits have way too much latency.

pksebben• 2 years ago

We have satellites. We use cables b/c they lack the speed and bandwidth necessary to support the total requirements of the modern internet. Satellite-only is only feasible if you're fine with going back to waiting minutes for your saucy jpegs to load (elder millennials, you know what I'm talkin' about).

SSLy2• 2 years ago

I was thinking is deploy of DMA "compliant" unbundling the day before it takes force.

Could be both.

bhawks• 2 years ago

BY JON GAMBRELL Updated 8:25 PM PST, March 4, 2024

Timelines don't match nicely

danbala• 2 years ago

maybe a long tail consequence of further shifting traffic?

hsuduebc25• 2 years ago

This shouldn't affect europe. It just stopped working

wongarsu• 2 years ago

Somebody might have fat-fingered a BGP configuration while trying to improve traffic routing that was impacted by the cut cables.

elorant• 2 years ago

It's not unreachable. I can easily see the FB page on my browser. It's just that even after resetting my password it doesn't accept it. Probably something's fucked up in the credentials database.

ajross2• 2 years ago

Those lines were cut yesterday, so it seems like a poor candidate for explaining the current outages. Likewise the geography doesn't match up with the outages.

toast0• 2 years ago

I'm pretty sure that report is just mainstream media reporting this week old cut https://www.datacenterdynamics.com/en/news/at-least-one-subs...

Pretty neat if a week after a cable is cut, FB falls over.

Especially when most of the source of truth databases are in the US and Europe, and that sort of data flow doesn't cross the Red Sea. FB has datacenters and points of presence all over, but outside the US/EU it's almost all caching.

jgehrcke3• 2 years ago

Thanks for the reference. Yeah, may also be entirely unrelated.

dylan6046• 2 years ago

Pretty neat if a week after a cable is cut, FB falls over.

that'd be one helluva cache!!

passwordoops2• 2 years ago

I'm 95% sure it'll turn out to be a mundane config error somewhere

barkingcat• 2 years ago

It will be DNS related.

newsclues2• 2 years ago

Nothing of value was lost.

deadbabe• 2 years ago

Entire businesses and livelihoods would crumble without Meta’s platforms.

newsclues2• 2 years ago

They can migrate to other platforms.

tjpnz• 2 years ago

If your business only exists at the sole discretion of Meta then it's not a viable business, at least not in the longterm.

d3zzyd• 2 years ago

Down here in South Africa too.. it's weird that it's not affecting everyone, yet it's worldwide.. It's almost as if it's taking down a percentage or section of addresses/uid's on their database... I wouldn't be surprised to hear that everyone affected, joined the app at a similar date, or the sequential uid's. To me this sounds like more than a bug.. I wouldn't be surprised if it's a hack.

kerirenee217• 2 years ago

I agree with this 100%! I do not believe this is just a little bug. This is more. There’s been a lot of things happening that people are trying to ignore. But this just adds to it..

bluetidepro• 2 years ago

On Super Tuesday in the US today, too. Very interesting timing for how much political nonsense is on social media.

stevefan1999• 2 years ago

This is so funny because the status page itself actually shows 500 Internal Server Error to me in the API calls. So the status page clearly isn't isolated from the FB network itself. I highly suspect it is either a BGP convergence issue, or their OIDC service hits the dirt.

hsuduebc25• 2 years ago

Funny. This page won't work too and is clearly down. So there is possiblity of pretty big amateurism from facebook or failure at dns servers.

bigwheeler2• 2 years ago

Tip for Meta engineers: when your service is failing, don't just log people out and prevent logins. Display a cute image that shows that the service is drastically failing (like a whale or something), and then people will know to stop trying to repeatedly log in. The public might even come up with a catchy name for the whale.

brk• 2 years ago

This is a great idea! I know that I was flailing about trying to figure out why I couldn't login, so I'd suggest calling it a "Flail Whale".

bandwidth2• 2 years ago

Surely the repeated logins can't be helping the situation. I suppose it is entirely auth related across all Meta products. The repeated strain could pose a cold start problem for example.

zabzonk• 2 years ago

catchy name

Petunia?

boring_twenties2• 2 years ago

Beyond unbelievable that going on an hour later, they're still showing "incorrect password" errors. How many hundreds of millions of people have wasted time frantically trying (in vain) to reset their passwords and pointlessly freaking out that their account might be compromised? What a bunch of careless, incompetent excuses for engineers.

alwyn• 2 years ago

Blame the managers and product owners that don't think this is an issue, not the developers that likely raised it a million times already.

ConorSheehan1• 2 years ago

What a poor bunch of overworked human beings, with almost no control over the product they work on. Frantically following the whims of managers, reduced to labour units in this late stage capitalist hellscape.

nuz2• 2 years ago

Imagine how many hundreds of millions of users waste their time using instagram and facebook on a daily basis. Safe to say they don't mind wasting their customers time

matsur• 2 years ago

Major outages are periods of intense stress and extremely difficult to operate in. The folks troubleshooting may be many things, but careless and incompetent are unlikely to be among them.

Solvency2• 2 years ago

I can almost guarantee you're getting mercilessly downvoted because half of the people here are sympathetic Meta worshippers who desperately (1) wish they worked there and (2) know they'd probably contribute similarly to this same horribly engineered system.

newman8r• 2 years ago

How many other people here just assumed they had been banned for some arbitrary, uncontestable reason? I just use instagram to post hiking photos...

TonyTrapp• 2 years ago

Having had a look at desperate Twitter posts during a major outage of a big German email provider with similar failure mode (login failed silently), it seemed like many people assumed that their email account was hacked. Close enough.

sumitgt• 2 years ago

Could be an outage in the auth service itself

Pxtl2• 2 years ago

Unless it's an outage in their ability to log into their own servers, they should be able to swap out the login page with a static HTML page explaining the outage. Maybe a 503 status code.

remram• 2 years ago

Yes of course it was. The point is, an hour later, they could have hit a circuit-breaker to get people to stop trying and going crazy over an error that is completely inaccurate.

Pxtl2• 2 years ago

Right? I know I was like "oh, I haven't typed in my FB password in eons... maybe I changed it at some point and forgot? But if I change it what happens to all related services, is it going to log out my kids' Messenger Kids devices? Those are such a pain to log in. Should I change my password or not? What do I do?"

Then I saw the news that it was an outage.

zerr• 2 years ago

For anything outside like-ing and post-ing, facebooks UI/UX is horrendous. Even Internet search does not help to find out how something trivial is done... The only way is to watch youtube videos.

dspillett• 2 years ago

For anything outside like-ing and post-ing, facebooks UI/UX is horrendous.

It isn't perfect for even that IMO.

Even Internet search does not help to find out how something trivial is done... The only way is to watch youtube videos.

That says more about where the web is heading than about facebook. Video is easier to monetise ATM⁰, and these days people don't put helpful stuff out there just to be helpful as much as they once did¹, so content creators are making them instead of simple web pages.

[0] Everyone wants to be the next big influenza who doesn't need a day job to get by.

[1] That sort of people are still out there, though they are somewhat drowned out as the signal-to-noise ratio heads inexorably towards “WHAT? WHAT?! I can't hear a single thing above the manscaping adverts!”.

bluetomcat• 2 years ago

Display a cute image that shows that the service is drastically failing (like a whale or something), and then people will know to stop trying to repeatedly log in.

Probably not so easy to implement in behemoth apps, consisting of 20'000 source files...

mkl95• 2 years ago

What do you mean my login flow sucks? Its time complexity is really good!

kossTKR2• 2 years ago

Cloudflare also having issues as of 1 hour ago:

https://www.cloudflarestatus.com

I wonder if it's connected to this. Seems like most large players are having issues now.

magk• 2 years ago

Cloudflare reporting that they were implementing a fix for something SSO-related at 16:02.

https://www.cloudflarestatus.com/

alecatoric• 2 years ago

This is also happening to me. When I try to reset my password using their password recovery I get "An unexpected error occurred. Please try logging in again." For some reason my phone number is not working as a way to get a recovery code, same error. I can't get the recovery code sent from the app or from my browser on my laptop, but when I use my chrome browser on my pixel it sends me a recovery code, which results in "An unexpected error occurred. Please try logging in again."

daniyelcr12122• 2 years ago

My service is down

annoyingnoob• 2 years ago

I don't use Meta's services but I have a number of co-workers that are all convinced that their Meta accounts have been hacked.

daniyelcr12122• 2 years ago

My Facebook is down

amyamy1• 2 years ago

I reaceived a highlight from a friend on fb tagging me in a kfc promotion, i clicked the add to the promotion and liked , followed and text @highlight, after that i was completely logged out from fb and instagram. With passoword being rejected etc.

oldmanfrom1900s• 2 years ago

Glad I'm not the only one!

AtNightWeCode• 2 years ago

SSO in Cloudflare also seems to have problems.

hsuduebc25• 2 years ago

It looks like AWS is down if we can trust downdetector which is reason why nothing works. Almost fifty percent of these services runs on this cloud.

sammy2255• 2 years ago

We can’t trust downdetector. Its just user reports. And users dont know anythig

cdme• 2 years ago

Eh, leave them down.

chhailavi_27• 2 years ago

i can't log in my account nor create one

shoebham• 2 years ago

hackernews is also v slow for me

xseparator• 2 years ago

You know shit's bad when even their status page is down.

mlucas823• 2 years ago

Frankly, I don't buy this explanation - technically or logically both from a devops,and systems architecture level - There is no way in hell a company like Meta is pushing database design changes this significant to production? We all know how many times these database architecture changes get run in staging,then even production subsets before rolling out to production at large.

Should we assume the teams working to ensure Business Continuity & Applications Resiliency redundancies feel asleep at the wheel?

Also, to assume that no down or outage messaging go out during a fairly routine maintenance based outage?

I call BS

Lol I can't find scenario where this happens, at this scale at a company of Meta's scale

kerirenee217• 2 years ago

I don’t buy it either! Something’s fishy…

mlucas823• 2 years ago

or phishy or both for sure

hsuduebc25• 2 years ago

Well if everything down maybe isn't problem at services itself but at the DNS side?

vb-8448• 2 years ago

Seems like even the status page doesn't work ... i'd say typical for this type of situations :D :D :D

mlucas823• 2 years ago

Should we assume the teams working to ensure Business Continuity & Applications Resiliency redundancies fell asleep at the wheel?

Also, to assume that no down or outage messaging go out during a fairly routine maintenance based outage?

I call BS

Lol I can't find scenario where this happens, at this scale at a company of Meta's scale...

buserror• 2 years ago

I can confirm that IRC still works! Pfewwwww..... /me takes his coat.

op00to2• 2 years ago

unless your bouncer is offline!

corobo2• 2 years ago

buserror has quit (*.net *.split)

alliao• 2 years ago

/me put on my wizard robe

guhcampos• 2 years ago

Yeah so much for graceful feature degradation.

jawns2• 2 years ago

Just last week, the Meta paper about their Defcon system landed on HN front page.

https://www.micahlerner.com/2023/07/23/defcon-preventing-ove...

nelle• 2 years ago

Both facebook and WhatsApp is not working globally, at first i thought someone hacked me

bookofjoe3• 2 years ago

I find it remarkable that Typepad, my blog host, not only isn't down but also is MUCH FASTER than its usual slothlike, approximately 5 seconds response time.

Rudraksh_15143• 2 years ago

It's back up here in India now.

peterleiser3• 2 years ago

Fixed. I can login to facebook.com now.

nuz2• 2 years ago

HN also seems unusually slow. Wonder what's going on

Deukhoofd• 2 years ago

HN is always slow when everything else goes down, tech people swarm here to check for information, and it overloads the site.

Etheryte• 2 years ago

Happens every time a major outage happens, everyone pours onto HN to discuss it which gives the one-core server HN runs on a really hard time.

belter2• 2 years ago

Probably DNS ...it's always DNS....

thinkingemote• 2 years ago

BGP is my guess

joejoesvk• 2 years ago

good luck to those on call

btbuildem2• 2 years ago

Hope it stays broken. Imagine? People would blink, confused, and slowly start to wake from a decade-long stupor.

kome2• 2 years ago

how many people, realistically, use it daily? I think not a lot

explain• 2 years ago

Really?

lenerdenator3• 2 years ago

Boomers probably do.

thehappypm5• 2 years ago

Like, everyone I know is on TikTok all the time and Reels is right up there

gpvos• 2 years ago

Lots and lots. Look at how many people are in this thread.

vincnetas2• 2 years ago

couple billions.

randomnumber314• 2 years ago

Facebook: global daily active users 2023.
Statista https://www.statista.com › statistics › facebook-global-dau Feb 9, 2024 — During the fourth quarter of 2023, the number of daily active users on Facebook reached 2.1 billion, a minor increase on the previous quarter.

So roughly 28% of the planet.

cmpxchg8b• 2 years ago

Enough for Meta to be a $1.2 Trillion company. What a silly comment.

rafark2• 2 years ago

I think not a lot

You have to be kidding me right?

baby6• 2 years ago

Man I would lose contact with a lot of friends if that were to happen

mikrotikker3• 2 years ago

Are they really your friends then?

pdabbadabba• 2 years ago

...and then install TikTok.

bookofjoe3• 2 years ago

My Meta Ray-Ban Stories glasses are working just fine FWIW

kilbill• 2 years ago

greeting from indonesia,Here too, the same error, lots of people asking what is wrong with FB IG error, lots of speculation saying update, but the repair process is taking too long without any confirmation from meta, something strange is happening

Generalist3001• 2 years ago

I can access what I call my family FB account, which I run on Firefox.

When I try to access my general purpose account I'm forced to log in again. And when I try to change my password I get the "An unexpected error occurred. Please try logging in again." message.

I suspect that one of the password servers has been compromised.

bratwurst3000• 2 years ago

lol how ?

https://en.m.wikipedia.org/wiki/2024_Facebook_outage

kome2• 2 years ago

I am surprised by how many people still use Facebook around here, as my social circle deleted or stopped using Facebook a long time ago.

belter2• 2 years ago

Internet Outages Map - https://www.thousandeyes.com/outages/

worldwidelies• 2 years ago

Is it coincidence that Meta is trying to get portions of NSOs source code via the court system. Also today is apparently a big voting day. The techno-apocalypse is near.

C4RN12• 2 years ago

Who thinks Meta pushed to prod without testing?

didip• 2 years ago

Laid off too many people? Critical services no longer have owners?

ur-whale2• 2 years ago

Since we're in full-on speculation mode ... Russian cyber-attack ?

KoalaKing1• 2 years ago

Glad, I found this. I literally just told me boss I spilled my coffee on my desk so I could have a break and figure this things out. F'ing FB... Thank you to whoever started this post. I tried resetting my password many times and also thought I was HaX0red.

tinycoder25• 2 years ago

Status page show some services are recovering. I wonder why ad services is the first service to be mentioned as recovering instead of login.

hengistbury• 2 years ago

No reason to allow people on the platform if you can't monetise them, especially after monetisation has been paused for a few hours

ricardobayes• 2 years ago

Oh, that's why I was logged out then.

solfox• 2 years ago

Seeing such a large web property go down like this is fascinating.

It's like a power plant grid failure, except for attention instead of energy.

When meta is down, a hoard of internet users desperately seek somewhere else to place their attention... But the system is designed with the expectation that meta will take all that traffic... And boom! everything starts falling over. Wild.

alwaysrunning• 2 years ago

It's DNS

nayuki• 2 years ago

Note that this web page only covers the status of Meta's offerings for business users. This doesn't track the downtime of Facebook, Instagram, WhatsApp, Messenger, etc. as normal users experience them.

jeanlucas• 2 years ago

Curiously, WhatsApp did not suffer an outage in Brazil.

FerretFred• 2 years ago

Ads Manager - Recovering from disruptions

LOLLLL .. oh dear

fuzzfactor2• 2 years ago

At least it's not a mission-critical website . . .

srvmshr• 2 years ago

I have an active Instagram account. Today, coincidentally was the first time I thought of promoting my best few posts as an ad to improve reach to fellow photographers. Bad luck! The app now seems to be back online but my ads which were supposed to run for 48-72 hours, show <Disabled>. And there is a new "Pay Now" link, even though it was paid for and seems ad-spend is already showing it used some of that payment.

As an individual, this is pretty confusing. I don't have much to lose. I am glad I spent 10-15 bucks on favorite 2-3 posts only. I can imagine many others to be more affected. What is normally to be expected for SME users? Does Meta resume the ads automatically? Do they make good for the lost time since the clock seems to be ticking -- although no one saw any impression.

Edit: I submitted a ticket to Instagram Help and they responded by asking for a screencast video. The first time I sent, the video bounced. I have re-sent this by trimming the video.

Out of curiosity I want to know firsthand how Meta handles the small customers.

Babadas2• 2 years ago

Everyone Beware, The “Black Hat Hackers” that have accomplished this massive Global hack represent a Clear and Present Danger, immediate and significant threat/danger to National Security, as well as all People’s personal and financial Security

Babadas2• 2 years ago

Everyone beware, The “Black Hat Hackers” that have accomplished this massive Global hack represent a Clear and Present Danger, immediate and significant threat/danger to National Security, as well as all People’s personal and financial Security

Ajay-p• 2 years ago

Eesh I thought it was a hacking situation.

kosolam• 2 years ago

Why the outage?

ragmondo• 2 years ago

Doesn't this kind of thing happen in every film regarding an AI becoming sentient ??

uselpa• 2 years ago

For now, some countries preparing a larger war seems more likely.

YeBanKo4• 2 years ago

So I guess this is what that national security warning and purchase of Palo Alto Networks stocks was all about in mid February.

WobblyTyre• 2 years ago

Can you elaborate further?

Razengan• 2 years ago

Since my dedicated post about this got no response, I want to use this opportunity to ask:

WHERE the HELL does Facebook store tracking data on my iPhone?

It shows my previous account even after I delete the app, clear the cache and KeyChain, disable iCloud Drive, AND sign out of iCloud??

Why can't I see where this data is stored? Same for TikTok.

WHY does Apple, parading around as a pompous paragon of privacy, allow this bullshit?

ak_1114• 2 years ago

I have a very tangential question to this situation prompted by some conspiracy theories linking the outage to today being Super Tuesday election day in the US.

Why does Facebook need sharing news media or political ad targeting on its platform from a business point of view? Am I being naive or is it really such an important revenue driver to the business or the core experience of the app?

I think it is the source of enormous reputational damage and risk, that if I was running the company I would even happily trade 10% if not more of the market cap to removing any feature that enables news sharing on the platform.

Actually if you remove these two things (political ad targeting and commenting on news media) I struggle to find any other issue that would make facebook a "political" target, they can literally shutdown the fake news division that employs 10,000 people...

cheeseomlit• 2 years ago

The ability to affect the outcome of elections is very much worth the political flak for them

dtdynasty• 2 years ago

My guess would be that they would lose a significant amount of their user base and engagement without politics and news media.

stemlord• 2 years ago

And nothing of value was lost

perlpimp• 2 years ago

probably some sort of BGP flood.

m3drano• 2 years ago

wild speculation: they were rolling out enforcement for DMA.

baby6• 2 years ago

this is the message I'm getting on X now:

https://www.threads.net/@cryptodavidw/post/C4JRaX9vZYE

seb1204• 2 years ago

So why is this site metastatus.com and not status.meta.com as mentioned in an earlier article here the east amount of domain names these big corps have is not helping with making sure this is not a scam. So why metastatus.com and not metastatuscheck.com? Sure that the buffs at Mets could come up with an independent machine that can show the status even if everything else is down.

pvg2• 2 years ago

Edit: I'm leaving the comment as penance but as https://news.ycombinator.com/item?id=39610130 points out, this is an old message about an old update, not the current one.

A short update from meta with some initial high level technical details:

https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A...

HenriTEL• 2 years ago

So it's a bug of the "automated remediation that makes things worse" family.

This system that checks for invalid values in the cache looks like a very bad idea in the first place as in my understanding it checks things beyond "is the cache up to date?".

gene313372• 2 years ago

That's from 2010.

edit: https://www.cnn.com/2010/TECH/social.media/09/24/facebook.ou...

pvg2• 2 years ago

You're absolutely right, thanks, my mistake trusting random URLs on HN/the internet!

pilgrim0• 2 years ago

Interesting how such large scale outages can ONLY happen because of human errors, in this case by a poorly thought out heuristics. Like, a literal explosion could hardly achieve this level of disruption when there’s physical failover protections everywhere.

mikrotikker3• 2 years ago

What about cyber attack? It's super Tuesday...

buttofthejoke• 2 years ago

600th comment

thecleaner• 2 years ago

So not really an outage ? Just meta

gck1• 2 years ago

In my country (Georgia), I received dozens of reports from people saying that after the outage, they have been offered to sign in to other users' Facebook accounts and were successful in doing that. I can't confirm it, but it appears that this was happening to accounts that were co-admins of Facebook pages.

husamia• 2 years ago

I wonder if this is related to 3 Red Sea data cables cut as Houthis launch more attacks in the vital waterway https://www.washingtonpost.com/business/2024/03/04/red-sea-u...

Vilian• 2 years ago

The cable cut only gonna after the arab world, or maybe india, but eropean to india communication don't depend on them that much so it's unlikelly, so probably not

mrzaggy4• 2 years ago

Anyone having luck resetting passwords yet?

My Messenger and FB on my phone magically logged back in a day or so ago, but on my PC, no luck - and that PC doesn't seem to recognise the password I thought it was before the outage.

Meta outage

Related Stories