Reddit crashed because of the growing subreddit blackout

https://www.theverge.com/2023/6/12/23758002/reddit-crashing-api-protest-subreddit-private-going-dark

by bgc • 3 years ago

190 140 3 years ago

I have not read a reddit devblog or anything like that. My guess is they are doing some kind of caching of posts via elasticsearch/redis or something similar. It must depend on the privacy settings of a sub.

So when 1000 subs go private, they potentially had to update hundreds of millions of comments.

Anyone else have a theory?

hospitalJail• 3 years ago

The subreddits that are most popular are on high performance servers while the subreddits that are less popular are on lower performance servers?

philip1209• 3 years ago

Cache invalidation makes sense.

Could also be some kind of fanout query that assumes the top subreddits are open and wastes CPU cycles.

Scoundreller• 3 years ago

I’ve noticed that my comment history in privatized subreddits are now unavailable, so I wonder if it’s related to that.

kevincox• 3 years ago

It could even be that lots of queries to generate pages just fetch private posts and comments. The assumption was that users will not be subscribed to lots of subreddits that they have access to so skipping these posts is typically negligible and not worth using a more complex query for. If the fraction of skipped items becomes significant these queries will be doing lots more work. Its pretty believable that some very common query getting 25-50% more expensive will be more server capacity then Reddit has available.

Another possible explanation is that operations on private subreddits are not as optimized as it is expected that they have a small number of subscribers and take up a small amount of total resources. These assumptions would be flipped.

delecti• 3 years ago

I'm skeptical. Most of the outages I've been involved in have been for way dumber reasons than that. Based on my dumb experience, my guess is that the handling for a user trying to access a private subreddit stressed some system that doesn't make sense to even exist.

2bitencryption• 3 years ago

the cynic in me says, they're being DDOSd (I mean, they pissed off millions of internet nerds who have their own definition of justice), and instead of admitting it they're using the opportunity to blame the subreddit mods.

then again, not sure why that messaging would be any better for them than admitting to a DDOS...

manicdee• 3 years ago

Blame the user makes sense because they don't want to show how fragile Reddit is while they're trying to attract investors for the IPO.

ElevenLathe2• 3 years ago

That makes sense to me. It would be one variant of a larger theory of "a subreddit going private is an expensive process that is poorly optimized because it's normally so rare". Another variant would be that clicking the "go private" button might do something like start a huge cascade of queries/calls to mutate the status of individual posts and comments (and possibly hence also trigger the massive cache invalidation you mentioned). Possibly also "going private" infra, whatever it is, is some bespoke snowflake that doesn't autoscale (since it's never needed to and there was always something better to be doing with engineering time, or people just forgot).

One theory I have seen floated but which seems unlikely to me is that this was some kind of internal sabotage. I could maybe buy this if the protest was about a war or human rights issue or something, but I really don't think any IT pros would be willing to risk jail time for this one.

2OEH8eoCRo0• 3 years ago

Would a better protest be rapidly switching subreddits from public/private?

commandlinefan• 3 years ago

to mutate the status of individual posts and comment

AKA why you should normalize your data models.

putnambr• 3 years ago

My theory is that some middleware applications have policies to scale to zero or one based on traffic patterns, when scaled down due to lack of traffic things start breaking.

jedberg3• 3 years ago

Posted elsewhere, but yeah that's my theory too (having actually helped built it in the first place). My guess is the caches got blown out because the entire system is built on the idea that most people are looking at the most popular content.

When you close down all the most popular content, you have to dig deep into the long tail for fresh content.

Also, my guess is that the code for building homepages is not optimized for having a lot of skips due to private Reddits, since most people have probably never been subscribed to a private reddit, or if they have it wasn't for very long, or even if for very long, never more than one or two at at time.

biesnecker2• 3 years ago

Makes sense to me. I remember several lunch conversations at FB about whether or not we even could restart the site with totally cold caches. I’m sure it was possible but it wouldn’t have been clean or pretty.

EMCymatics3• 3 years ago

Well that is how wallstreetbets freezes the subreddit to log all of the potentially actionable comments.

amyjess• 3 years ago

My theory is that the algorithm that produces the front page relies on the largest subreddits all being public. But if it can't draw on posts from the largest subs, it'll have to dip into a bunch of smaller subs, and the part of the algorithm where it delves deeper and deeper into less popular subreddits is terribly optimized and puts a huge strain on the servers.

micromacrofoot• 3 years ago

Twitter was like this at one point, flipping over from public to private was one of the most resource intensive things an individual user could do... especially with a long tweet history, because it involves changing every tweet.

tester457• 3 years ago

Here's a guess from a former reddit dev.

(I used to work as a backend developer at Reddit - I left 6 years ago but I doubt the way things work has changed much)
I think it's extremely unlikely that this is deliberate. The way that Reddit builds "mixed" subreddit listings (where you see posts from multiple subreddits, like users' front pages) is inefficient and strange, and relies heavily on multiple layers of caches. Having so many subreddits private with their posts inaccessible has never happened before, and is probably causing a bunch of issues with this process

https://tildes.net/~tech/163e/reddit_appears_to_be_down_duri...

ramgine• 3 years ago

Unrelated, but any way to get a tildes invite these days? I asked on Reddit but then shut my sub down and have avoided going back

synicalx• 3 years ago

That was my thinking as well, and assuming they're still cloud-heavy (it's been years since I've checked) this could potentially be an expensive event for them. That opens up a very interesting for of protest/attack that reddit really won't like; "privating" then "un-privating" a subreddit several times a day.

Solvency• 3 years ago

I thought it was a cloudflare outage.

almost_usual• 3 years ago

Pretty sure Reddit uses Fastly as a CDN.

stillbourne• 3 years ago

How long before they admins swoop in and change ownership of the blacked out subreddits?

EA-31676• 3 years ago

What does that really accomplish? Reddit mods are often bad enough as is, imagine what Reddit scab mods would be like.

It would also be a huge invitation for the displaced the rest to make life unbearable for those new mods and the handful of admins overseeing them. Reddit as a community would make for one hell of an intellectual DDOS.

bitshiftfaced• 3 years ago

I seem to remember someone saying something about how a small number of users mod the vast majority of Reddit in terms of the number of users within the subreddits they control. I wouldn't be surprised if there was a black market for mod status. Might not be such a bad thing if the establishment was totally uprooted.

falcolas8• 3 years ago

What does that really accomplish?

To borrow from another comment buried below: I think the company cares more about the appearance of "business as usual" in the face of an IPO than the opinions of the users.

In their eyes, this is all nothing more than a tempest in a teapot; a small pothole on the road to public investor dollars.

falcolas8• 3 years ago

I personally believe they will wait to see which ones remain shuttered. 48 hours is not very long to wait to maintain your free civilian labor.

After that, they'll definitely reclaim the shuttered subreddits with new moderators.

phpisthebest3• 3 years ago

Reddit is odd.... while most communities hate the mods, they hate admins far more. Admins taken over a subreddit would kill it in the eyes of the users.

duxup2• 3 years ago

That would seem to setup a situation where Reddit has to fill a gap of a huge amount of moderation work that they previously didn’t have to do / people did that for them for free.

Longlius• 3 years ago

The main reason why reddit works as a platform is because its mods and subreddit regulars do a lot of unpaid work. Where is reddit going to find enough people to work for free on such short notice?

falcolas8• 3 years ago

They don't have to be good mods. There's precedent for a handful of mods to moderate a huge swath of subreddits.

jstarfish• 3 years ago

Where is reddit going to find enough people to work for free on such short notice?

Enter: an army of scab accounts ending in _GPT.

this_user• 3 years ago

A long time IMO, because that would just escalate the situation. At this point they are going to wait and see in the hopes that the majority of people lose interest after a few protests. Then they can just push through the changes.

If they take over these subs, you just radicalise the users, and you now have to find new moderators for these subs. In a worst case scenario, users actually start migrating to another platform.

graeme3• 3 years ago

Finding new moderators would be a incredible challenge. If someone wants to be a moderators badly, that is usually disqualifying.

You get an immense amount of power with very little accountability except to the community. Put that in the wrong hands and you can destroy a community incredibly quickly.

Or, put power in the hands of a dilettante, and you get overrun with spam and insults very fast.

superkuh• 3 years ago

Nah, there's plenty of people just waiting to take over even the smallest of niche subs. I saw it firsthand back during the 2nd wave of gentrification in spring 2018 when a bunch of commercially uncomfortable subreddits like r/gundeals and their users (like my 2008 reddit account) were banned. The subreddits I moderated like r/radioastronomy were given to other users to moderate within a month or two. Reddit eventually admitted it was wrong about banning r/gundeals and reinstated the sub but my account was never unbanned.

And any sub with serious activity will have power mods submitting applications for take-over so they can commercialize it.

vkou• 3 years ago

Oh, that's easy, if they are looking for scabs to mod a million+ sub, I volunteer myself immediately. I've got a lot of great ideas that I'm really eager to try out.

The only thing that I'll be able to guarantee is that you'll really not like me as a mod.

cdme• 3 years ago

What happens if the mods were to coordinate cycling subs from private to public?

ch4s3• 3 years ago

Probably rolling outages

TheAdamist• 3 years ago

Cfaa charges.

cryptonector3• 3 years ago

What's the theory?

anoonmoose• 3 years ago

They'll get replaced faster than they otherwise would have

seydor3• 3 years ago

Reddit hasn't replaced them for 10+ years. Free labour is not easy to find these days

PUSH_AX2• 3 years ago

This would be chaos. Most of the big subreddits have a combination of bespoke bots and complex automod configs that even the current mods struggle groking. Reddit would lose a ton of quality through poor moderation quickly if you put everything up for grabs.

I think this would be a poor move.

freespirit77• 3 years ago

Could be due to home feeds not having enough cached content as a lot of the first page posts went private and had to be skipped? Would have led to have to go deeper in lists of posts to fill the first page and causing more cache misses and db reads and performance issues.

derefr• 3 years ago

Very likely; before the crash, I noticed that the home page failed to load, but individual subreddits continued to load.

klodolph• 3 years ago

The way Reddit displays content is basically all caches, as far as I can tell. Every query you do is supposed to directly hit a cache of 1000 objects (like comments or posts). You can’t go past 1000, either through the web interface or through the API.

Kinda frustrating design, IMO, speaking as an end-user.

kenhwang• 3 years ago

It was always obvious when you scrolled past the cached content because errors or timeouts would start happening pretty frequently.

sofixa2• 3 years ago

A significant number of subreddits shifting to private caused some expected stability issues, and we’ve been working on resolving the anticipated issue,”

Calling bullshit on that one. If they had expected stability issues, and knew exactly which subreddits will go private and when, how is it possible they had so much "anticipated issues"?

schaefer• 3 years ago

The world has finite resources... such as man-hours.

sofixa2• 3 years ago

Such a critical moment for the future of the company (mass boycott after controversial changes, presumably on the path to IPO; they really don't need any more bad publicity) should be an all hands on deck situation. Again, they've had advanced notice, and knew the full list of subreddits. They should have prepared better.

falcolas8• 3 years ago

I imagine a previously not-stress-tested component that has to do with searches/fetches with private subreddits went kaboom.

Not too much of a stretch, IMO.

A fantastic Schadenfreude generator, regardless.

jedberg3• 3 years ago

My guess is that they knew the caches would get blown out and would have to refill with content from non-blacked out subs, and decided to just take the hit and let it ride, knowing it would recover itself soon enough.

I mean, honestly that's probably what I would have done if I were still in charge.

RSHEPP• 3 years ago

And in this case, use it as an opportunity to learn more about the system. Treat it as an experiment.

realusername• 3 years ago

They have an outage every two weeks maximum so I also call bullshit on that one, Reddit hasn't exactly been the most stable website before.

jklinger4102• 3 years ago

and knew exactly which subreddits will go private and when

As someone who is close to a reddit mod, they did not know which subs would go private

TheNewsIsHere• 3 years ago

I don’t say this snidely, but more than 7,000 subs openly committed to going private in advance of today.

I’m sure someone somewhere at Reddit was surprised by sub X or Y going private, but if they were surprised at half the largest subs that went private, then that would be as sad as it is amusing.

Edit: typo

gfv• 3 years ago

When operating a complex system like Reddit, you know that something is going to happen during a significant event, but you generally don't know exactly what will break first.

What's worse is that the system evolved as a response to the previous similar event, and what broke now is probably not what had broken in the past.

subarctic• 3 years ago

It'd be funny if the engineers at reddit decided now was a good time to upgrade postgres or do some other kind of scheduled maintenance, and just not worry about downtime because the blackout is on anyway.

AH4oFVbPT4f8• 3 years ago

Deniability is a powerful tool, and considering their track record this year, this decision wouldn't rank among their worst.

jklinger4102• 3 years ago

Unless there is some serious damage control happening from corporate which makes them unable, I think a team of engineers would talk themselves into doing this 99% of the time.

joeross• 3 years ago

I don't see any damage control... they're still committed to the "Apollo sucks" line:

Mr. Rathschmidt added that some apps are more efficient and require significantly fewer A.P.I. calls and that “Apollo is notably less efficient than other third-party apps.”

Source: https://archive.ph/y92P7#selection-681.0-681.173

Tao33005• 3 years ago

Or rolling out the big blackout override switch

poyu2• 3 years ago

Easy:

``` UPDATE subreddits SET is_private = FALSE WHERE is_private = TRUE AND updated_at BETWEEN 2023-06-11 AND 2023-06-12; ```

Bonus: Run it every hour

throwaway48373• 3 years ago

My unpopular theory is that the Reddit blackout increased Reddit usage, having the opposite of the intended effect.

falcolas8• 3 years ago

It would certainly change usage patterns. And there are enough popular subreddits who didn't go in on the outage, so there's content to be had.

The adage "there's no such thing as bad press" could definitely apply, given how broadly covered this is (I mean, hell, I saw it show up on NPR).

pkju62dqmx• 3 years ago

Currently over 90% of subs are private or restricted.

https://reddark.untone.uk

Re-hosts of that site have the figure even higher.

Tao33005• 3 years ago

there are enough popular subreddits who didn't go in on the outage

Yeah, this morning I noticed only one sub that I follow was actually private. I was going to mention it here, but I forgot which one it was - and now I can't find the one that was blacked out.

throwaway48373• 3 years ago

That's my point. It's so highly covered that it might actually bring more people to Reddit today than the number of people who fled. Imagine all of the people who don't use Reddit who saw this press, and visited Reddit because of it. That number may possibly outweigh the number of users who decided not to use Reddit because of the "strike" today.

EMCymatics3• 3 years ago

It is nice to see all of the toxic subreddits delete themseles.

cpncrunch3• 3 years ago

I operate a certain health condition subreddit. The main subreddit for this condition is highly toxic (which is why I created an alternative a few years ago).

Anyway, it turns out the primary (toxic) subreddit for this condition is taking part in the strike, so a lot of users have come over to my alternative subreddit, and they are glad to have found it because they were getting fed up with the toxicity of the other place.

Unfortunately I'm sure it will be back to normal in a couple of days.

speedgoose• 3 years ago

That's also my first bet. The strike was well covered in many medias.

Looking at the google trends, it was a big spike today compared to the classic daily workload: https://trends.google.com/trends/explore?date=now%207-d&q=%2...

Salgat• 3 years ago

This is more indicative of people just searching about reddit, possibly to figure out why the site they normally use is having issues.

throwaway48373• 3 years ago

The US trends are very telling... https://trends.google.com/trends/explore?date=now%201-d&geo=...

It coincides exactly with when Down Detector shows the Reddit report spike... https://downdetector.com/status/reddit/

I'm gonna go ahead and say my theory is probably right.

Anonasty• 3 years ago

It seems that Reddit will go down like Digg did.

willio58• 3 years ago

Unfortunately, I've been hearing this for like a decade and it doesn't seem to ever happen.

wmeredith2• 3 years ago

Digg.com’s implosion was so dramatic because Reddit was there to immediately start serving the same users just about the same thing. There are obviously plenty of other things Reddit users can do with their time, but there isn’t a monolith solution waiting in the wings to very closely replace Reddit with almost the exact same thing. Its destruction will be much more prolonged.

sebazzz• 3 years ago

Exactly the same comments have been made about Twitter, and Twitter is still here.

PUSH_AX2• 3 years ago

More likely that people will simply forget in 3-4 weeks, like literally every other issue that can sort of blow over.

mvdtnz• 3 years ago

I see Reddit themselves confirmed that the blackout caused the outage, but just to be clear Reddit is a very unstable website and outages are commonplace. It's clear their engineering team is not exactly top notch (or have been poorly resourced).

mkl95• 3 years ago

They should disclose what happens in their system when a subreddit goes private, and how it scales with subreddit size. Otherwise, I call BS.

seydor3• 3 years ago

I think reddit will never be the same after this, not because of the API thing (I dont agree with the blackouts), but because it shows how much they neglected their website. Over the years they have made it impossible to compete with the established subreddits. They have allowed a single uniform mob to take over the site, establish themselves as permanent tyrants and grow their ban lists to enormous scale. Even today with the blackouts, you can't find alternative subs to the closed ones, because new subreddits are hidden in search results and popular lists. Without this competition, how do they expect their subs to improve?

It's as if their mismanagement has come back to bite them. Mods have been allowed to run the show and ruin communities for years and years.

Very few people are using 3rd party apps but now they are causing inconvenience to millions for their own capricious reasons. I knew that the site was ran by a small powertripping group of mods but now that they make it so obvious it has kind of chepened the whole medium in my mind

btmiller• 3 years ago

The subreddit experience is bifurcated into (at least) two camps: big/popular/default or small/curated/niche. The tyrannical behaviors are most often found in the former camp where being a moderator is synonymous with “internet power”. The latter is populated by enthusiasts tending to their communities.

I agree that Reddit won’t be the same after this, but there’s just not a ton of remaining value in the big defaults. They’re already plagued with low value, low effort content, low cardinality content. The true damage that’s being done right now is to the small communities which represent(ed) the best of Reddit. Out of the spotlight but deep in quality content for those that care about the topic. The inherent risk of stability is more certain now with Reddit’s new policies. And I suspect now with many subreddits weighing indefinite shutdowns that many users (perhaps more importantly: the best content creators) will scatter to the wind for greener pastures. Those that are left are given crappy apps, lots of ads, and hollow communities eagerly looking for something better.

phil21• 3 years ago

The tyrannical behaviors are most often found in the former camp where being a moderator is synonymous with “internet power”.

What's old is new again, I suppose!

Similar happened on IRC back in the day. All the big channel wars and drama were over the big popular channels for internet points. I certainly participated in my fair share.

Then you had the small channels of folks who were just pretty chill chatting about whatever random topic of interest.

Might be nostalgia, but I still think the best balance of this was the phpbb/vbulletin era. The amount of "reference material" lost behind the walled gardens is crazy.

faeriechangling2• 3 years ago

I think the problem as somebody creating content for or moderating one of those small communities is - do you really want your users to be forced to be used a community as abusive as Reddit towards its users? Do you really want to have your labours exploited by Reddit for their profit?

I’m not sure what the alternative is though. Independent websites? Where are dejected Redditors going to go?

chongli• 3 years ago

This isn’t a unilateral move by a small group of moderators. Regular users support the blackout in droves. We don’t like Reddit corporate deciding to shut down 3rd party developers.

Reddit is extremely entitled. They as a company create nothing. The users create everything and do all the work of moderation. Spitting in the eye of your all-volunteer staff on which you depend completely for your livelihood is corporate suicide.

faeriechangling2• 3 years ago

Reddit was known for having lots of developers on it in the first place. It should be on some level unsurprising their users throw a fit over third party devs being thrown aside the moment they don’t help reddit’s IPO.

d10486fa91eb46• 3 years ago

The problem for me with the different subreddits and mods is the inconsistency between them. I am shadowbanned from some instantly before a single comment, I can't comment on a subreddit without providing a registered email, in so doing compromising my privacy, (looking at you /r/linux), appeals to mods go unanswered, and the universe help you if a mod disagrees with something you have said. It is a completely frustrating experience to try to opine on topics you find interesting without being herded through the tunnel like cattle to conform.

However, I do agree with the blackouts as it is a coordinated action to a larger problem, not some power obsessed individual subreddit with arbitrary rules also applied arbitrarily.

zdimension• 3 years ago

The blackout is as much about a small number of users using third party clients for personal tastes as the American civil war was about states rights. The real issue, as can be read in the long PSAs pinned to some of the restricted but public subreddits, is the impact on moderation and accessibility that forbidding third party apps will incur. If moderators can't do their job, Reddit is worthless. Calling them tyrants is... a peculiar way to frame it

WXLCKNO• 3 years ago

I'd love to see an API only version of Reddit where people build the clients on top of it but are forced to serve ads from the global provider, from which they get a generous cut.

Reddit crashed because of the growing subreddit blackout

Related Stories