Roblox has been down for over 24 hours

https://status.roblox.com/

by intunderflow • 5 years ago

238 117 5 years ago

status.roblox.com

There's probably a lesson in here about managing incidents, very few updates or acknowledgments there's even a major problem.

Will be interesting to find out what caused an incident big enough to kill their entire platform for so long (if they release details or they come out through the grapevine)

stingraycharles• 5 years ago

Seems to me that a routing / networking issue would have been resolved by now, and application bug would have been rolled back.

If I were to speculate, I would say that it must have something to do with databases / storage. Something must have gone wrong and broke some database, and it’s difficult to restore.

intunderflow3• 5 years ago

They had an incident in the past where they returned nil for all get operations to their database, that caused some chaos at the time so it's not unprecedented: https://devforum.roblox.com/t/update-datastores-incident-dat...

polack• 5 years ago

I agree that DB is the prime suspect here. They said the problem was identified over 24h ago, cant imagine it being something other than a data issue that takes that long to resolve.

deathanatos3• 5 years ago

The status page acknowledges there is an incident, that it's pretty bad — everything is marked as down, and it's in red! What more could you want? (/s)

Compare to Azure: most incidents never see the status page. Hell, getting support to acknowledge that an incident exists even once your internal investigation has reached certainty on "oh, yeah, it was them" is hard. There was an AAD outage earlier this year (?; IDK — I've lost track of the passage of time in the pandemic…) and the status page was down, and even once you managed to get the status page (IIRC you could hit the IP if you knew it magically which the Twitterverse did) … most services were still green, even if completely offline as far as one could tell by issuing queries to the service…

And I'm comparing a kid's game with a "major" cloud PaaS…

I'm definitely suffering from Stockholm syndrome.

aliswe2• 5 years ago

a kids game yes, but worth 50B

KronisLV4• 5 years ago

Compare to Azure: most incidents never see the status page.

That sounds like a conscious decision on their part. Everyone always talks about disclosure being the best policy, but at the same time there's plenty who believe that not informing anyone about an outage or even a breach is the correct thing to do, since then they'll probably get into less trouble themselves, or at least will create the illusion of not having as many outages as "those other guys".

After all, informing everyone about an outage that will noticeably affect only some probably has a larger impact on the company's reputation than having it be dragged through mud in smaller communities for its dishonesty. Then again, with many of the larger services, it's not like you have much of a choice of using or not using it - you just get a corporate policy passed down upon you and that's that.

Thus, sweeping problems under the rug and pretending that they don't exist is a dishonest, yet valid way of handling outages, breaches and so on. Personally, i'd avoid any service that does that, though it's not like that's being done just because of incompetence.

zinekeller• 5 years ago

Compare to Azure

I'll honestly change that to all major cloud providers - AWS has some small hiccups and then that were never recorded, and Google Workspace describes "a problem to a subset of users" when it clearly worldwide.

numbsafari3• 5 years ago

Interestingly... the stock is up 3% over the course of this outage so far.

jasoncartwright• 5 years ago

Sounds like when Fastly went down - the stock went up. Everyone suddenly realised how much of the internet they powered?

numbsafari3• 5 years ago

or because of ...free chipotle burritos?

https://www.ign.com/articles/roblox-servers-down-chipotle-fr...

gruez2• 5 years ago

roblox powers much of the internet? I find that unlikely. If you don't have kids, this is probably the first you heard of it. I guess there's the "my kid is complaining to me that it's down and now I know about it" angle, but it's hard to draw any conclusions from your kid's purchasing habits alone.

cma• 5 years ago

Probably because of Facebook turning themselves into a Roblox, validating them.

scottwi• 5 years ago

Metaverse overload

sudosysgen• 5 years ago

I'm guessing because many investors found out about Roblox from their kids when it went down.

carlosdp• 5 years ago

Yea, this is definitely it, wow

indigodaddy• 5 years ago

That does actually make a lot of sense.

raspasov• 5 years ago

Hilarious and possibly true.

Zeebrommer• 5 years ago

I wouldn't have thought "any publicity is good publicity" would hold even in this case.

Bombthecat2• 5 years ago

I could imagine something like this:

Dad! Roblox is down!!

Dad: whats roblox?

Looks at credit card over the years: fuuuuuu!

Goes to buy shares :)

baby2• 5 years ago

How is that interesting? Markets are irrational.

aliswe2• 5 years ago

sometimes things are interesting because they are counterintuitive, however irrational

rvz• 5 years ago

Well everything can be taken 'down', from blockchains to even real life by governments.

For the metaverse to succeed, it should never be 'down'. Good luck with that if it is on a single platform or a single point of failure.

Perhaps an interoperable metaverse / hyperverse then?

throwaway1777• 5 years ago

I think you’re right. Even Fb/Meta said interoperability is a key feature of the “metaverse”.

kwertyoowiyop2• 5 years ago

With everyone plugging into the Meta hub, of course.

ceejayoz• 5 years ago

Sure. Interoperability between Facebook, Instagram, Messenger, and WhatsApp.

kwertyoowiyop2• 5 years ago

Perhaps some sort of MetaChain…of MetaBlocks…is needed?

jpindar• 5 years ago

The various Open Simulator grids (known collectively as the hypergrid) are interoperable like that. As long as the one you created your avatar on (which can easily be a self-hosted one) is up, there are many worlds to teleport to.

EastOfTruth2• 5 years ago

Wow... seems like a good time to sell Roblox stocks anyways... because most of Roblox' popular games appear to haven been cloned into Minecraft... earnings are coming up real soon though... I guess we'll find out in less then a week.

edit: I edited this comment a bit to make it clearer because HN would not let me reply to the child comment.

ALittleLight• 5 years ago

I would think the opposite. This is a short term thing that presumably won't affect the business in any meaningful way. They will (probably) recover and be fine and nobody will be too worried that a kid's game has some, limited, downtime from time to time. If this outage causes a drop in share price, it's probably a good time to pick up a bit at a discount.

dylan6042• 5 years ago

nobody will be too worried that a kid's game has some, limited, downtime from time to time

except for the parents of kids addicted to the game

dang• 5 years ago

Recent and related:

Roblox Service Disruption - https://news.ycombinator.com/item?id=29034909 - Oct 2021 (40 comments)

fredley• 5 years ago

An insider told me that their internal secret store became overloaded, which caused every other service to suddenly fail to have access to any credentials.

lexandstuff• 5 years ago

Seems surprising that it would cause 24+ hours of downtime. My guess is they're recovering data from backup.

justapassenger• 5 years ago

Not surprising to me at all.

No idea about their infra, but for complex enough, built in Silicon Valley move fast style (and they are likely in that bucket) restarting after such a failure can be a huge problem. You can have thundering herds, that you may not have levers to control, infrastructure needed to build and deploy fixes maybe down, you can have cyclical dependencies sneaking in the past, that makes it impossible to startup some systems, etc, etc

mike_d• 5 years ago

Not if they managed to drop all the secrets.

sidlls5• 5 years ago

It points to it being a serious infrastructure issue really. You don't have a hugely popular platform be dead for two days unless there is some fundamental issue with the infrastructure.

Likely some stupid misconfiguration in some overengineered "cloud" architecture is the root cause, and they're mired in trying to debug and resolve the problem, fighting the complexity of the system the whole way.

ashtonkem3• 5 years ago

It can sometimes be very hard to cold start in such circumstances, because the demand from all the services trying to access their secrets at once can create a “thundering herd” that knocks the recently restored service over again.

My guess is that they’ve had to shut most services down, restore the secret store, and then bring up internal services in a careful sequence to avoid a thundering herd. At $JOB even understanding the correct sequence to cold start our organization would probably take more than 24hrs.

baby2• 5 years ago

I find it surprising when companies recover in less than 72h personally

encryptluks2• 5 years ago

Wonder how they would handle purchases lost and what not. All those kids having to prove they lost money they spent on their Roblox gift cards.

dilyevsky2• 5 years ago

$20 says dependency cycle

Corvus• 5 years ago

We had a situation like that once because of circular dependencies; services that needed another service to start. “How is that possible?” you ask. Code reviews; “take that code out and get the data from an existing service.”

We hard-coded data into a base service, pushed it to production, restarted services that depended on it, then services that depended on those services, etc.

slapula• 5 years ago

Roblox is pretty open about being all in on Hashicorp stuff so that would imply major issues with their Vault implementation. Going to be a very interesting postmortem...

aderdale• 5 years ago

Some interesting info on their infrastructure here

https://www.hashicorp.com/case-studies/roblox

vjacheslav• 5 years ago

imho, not Vault as such, but Consul which delivers too many unexpected troubles under slight pressure

kaidon• 5 years ago

Nomad makes things interesting...

johnmarcus• 5 years ago

I interviewed with their ops team there and turned it down. Just had a bad smell. I think they hire 'SRE' that can write good leet code. The problem with those folks for ops is that they over engineer complex solutions, and then completely forget the simple but very important stuff. I wish them luck in getting it back together.

ripper1138• 5 years ago

Leet code style questions for an SRE is so dumb. SRE is all about solving unexpected, unique problems. Opposite of leet code!

sjg007• 5 years ago

Worse the the FB outage for parents… help!

Wowfunhappy• 5 years ago

Completely unfounded speculation—what if they were hit by a ransomware attack?

I wonder only because, well, after a full 24 hours... a purposefully malicious attack starts to seem more plausible to me than a simple freak accident.

cale-• 5 years ago

Amusing, John Carmack mentioned the metaverse will be something like Roblox https://arstechnica.com/gaming/2021/10/john-carmack-sounds-a... .

FB (meta) taking drastic action to shutdown the competition

petermcneeley• 5 years ago

I think this is a pretty strong statement to make without any citation behind it.

derwiki• 5 years ago

Not GP but I would interpret that as sarcasm

xeromal• 5 years ago

I think the poster was joking

bemmu• 5 years ago

I'm curious where the traffic is now going that isn't going to Roblox? Steam player count doesn't seem to be jumping much. Maybe just mobile games?

sgc• 5 years ago

Youtube kids videos.

empressplay• 5 years ago

Probably. Roblox's original value proposition of 'kids making games for kids' has devolved into 'adults making dopamine slot-machines for kids' which is pretty sad :(

Bombthecat2• 5 years ago

For free!

midwestemo• 5 years ago

Steam is usually targeted 13+, since most users are under 13 I doubt most are on steam. Probably mobile games (as you said), Minecraft, YouTube, etc.

bonergarage• 5 years ago

Can confirm. They are still picture-in-picture on YouTube, but playing Minecraft instead of Roblox.

mensetmanusman5• 5 years ago

The kids are getting sleep they need?

https://www.healthline.com/health-news/children-lack-of-slee...

Parent advice: Get your kids to sleep, it gives them a huge unfair advantage in school :)

https://news.mit.edu/2019/better-sleep-better-grades-1001

hnarn• 5 years ago

How is it an “unfair” advantage? It’s just an advantage. It’s not like getting a good nights sleep is something that requires an exorbitant amount of money.

vl• 5 years ago

Minecraft

watwut• 5 years ago

My kids play roblox and don't have any access to steam. Which is probably very common.

Also, a lot of roblox playing is with tablets and phones. Those kids can't move to steam even if parents allowed it.

farmerstan• 5 years ago

Can someone please do the same for Minecraft? Maybe my kids will finally read a book

shmde• 5 years ago

Maybe your kids are already reading books from The Uncensored Library server in Minecraft :)

Edit: son to kids.

anticensor• 5 years ago

Guilded integration?

lexe092666• 5 years ago

That’s weird

Roblox has been down for over 24 hours

Related Stories