Reddit comments and submissions collected by Pushshift

I'm going to miss pushshift, their service was valuable for catching reddit moderators performing underhanded censorship of posts they didn't agree with. Reddit is walking a thin line between becoming a dumpster fire and being the one stop shop for a good portion of the worlds interaction. I would personally prefer to see reddit kick all of their moderators to the curb and stop catering to the snowflakes of the world.

MisterBastahrd2• 3 years ago

This sentiment will forever fall on deaf ears because starting new subs cost you nothing but time and energy. I used to be a moderator on a fair number of subs with 30K+ members that I started from scratch based on that exact mentality. Nobody owes you an audience.

lmm2• 3 years ago

Doubt. Whenever a sub gets big enough the same cabal of powermods appears in the moderator list, as if by magic. They don't always replace the existing moderators but they do get added.

ericmcer2• 3 years ago

Is it going away?

CameronNemo3• 3 years ago

AIUI changes to reddit's API have made Pushshift unsustainable.

Edit:

see here for the full details, from the reddit admins' POV: https://old.reddit.com/r/modnews/comments/134tjpe/reddit_dat...

nanidin2• 3 years ago

Yes, it is no longer available from pushshift (presumably due to pressure from Reddit.) The historical data is now only available via torrent.

r721• 3 years ago

https://old.reddit.com/r/pushshift/comments/13mhuzq/api_has_...

Dalewyn2• 3 years ago

It became a dumpster fire well over a decade ago and has never recovered since. Even the site administration is in on the act, lest we forget spez editing the database to modify users' comments with no indication.

Lammy2• 3 years ago

It became a dumpster fire well over a decade ago

Relevant: “Containment Control for a Social Network with State-Dependent Connectivity” (2014), Air Force Research Laboratory, Eglin AFB: https://arxiv.org/pdf/1402.5644.pdf

…a.k.a Reddit's "Most-Addicted City" of 2013: https://web.archive.org/web/20160604042751/http://www.reddit...

Zamicol7• 3 years ago

This is why user comments need to be cryptographically signed. Cryptography prevents this sort of nefarious and hidden corruption.

menus• 3 years ago

Reddit has always walked "a thin line between becoming a dumpster fire and being the one stop shop for a good portion of the worlds interaction" (1) (2). It is up to you to figure out how you want your experience to be.

(1) https://slate.com/technology/2014/04/reddit-and-the-boston-m...

(2) https://www.vox.com/2015/6/11/8767035/fatpeoplehate-reddit-b...

I would personally prefer to see reddit kick all of their moderators to the curb and stop catering to the snowflakes of the world.

I'm sorry about your experiences with whatever moderators you have a beef with. My guess is you are talking about large subreddit(s). Large subreddits have their issues with transparency I agree but what are you trying to suggest? Reddit-appointed moderators? If so, you are in for a rude awakening.

I would instead spend effort in stopping them from charging third party apps for API access that are objectively better than the official apps and ensuring that they dont kill old.reddit.com (3) like they killed i.reddit.com

(3) https://old.reddit.com/r/reddit/comments/12qwagm/an_update_r...

senttoschool6• 3 years ago

Large subreddits have their issues with transparency I agree but what are you trying to suggest? Reddit-appointed moderators? If so, you are in for a rude awakening.

I would create a system where moderators have far less power, and that Subreddit members can vote on rules, posts being deleted, etc. The system would be much more democratic. Members would also be able to challenge moderator actions or even reverse moderator actions.

As of right now, Reddit has a problem where moderators have too much power. On top of that, moderators aren't being paid so they can't be controlled.

PragmaticPulp• 3 years ago

It is up to you to figure out how you want your experience to be.

No, it's up to the subreddit moderators. They are the hidden hand that shapes the conversation.

Many of the popular subreddits have been taken over by groups of full-time moderators who are pushing various agendas. Their actions can't even be seen. We can only see the posts they've chosen to allow in their subreddits.

Dylan16807• 3 years ago

I have an easy suggestion. Something like: a single person can be a moderator on 5 subreddits max

Edit: Uh, why is this so unpopular it's getting multiple downvotes? Like, there are people that moderate hundreds of different subreddits and I don't think that's how the community self-moderation is supposed to work.

technick4• 3 years ago

My guess is you are talking about large subreddit(s). Large subreddits have their issues with transparency I agree but what are you trying to suggest?

I want unpaid mods removed from the equation and for all content to be judged on the same scale and have full transparency to why moderation actions were taken, by who, and easy links to appeal. I also want clear rules for every subreddit, and if it's not a rule, users cannot be banned for it.

I've experienced many types of censorship, everything from mods silently deleting my posts without any reason or notification, then I occasionally get things removed with notes that the audience is too sensitive to have a real conversation (The topic was about dogs attacking people in my city). However all of my posts do not violate any of the published rules of the related subreddit. I argued more times with the moderators than I care to admit, till I just said fuck it, go ahead and ban me because I'll be back with another account in 10 minutes. I've gotten the game figured out on just creating many accounts and when one goes down, many others are available.

The dead accounts I can't use anymore actually make me some money, depending on the age and posting history. They can be sold for up to 50'ish dollars for a 11 year old account with 5k karma. However they won't be able to post in every sub, they're all banned from r/denver , r/askcarsales , r/denverbroncos , some are banned from r/linux because I called out some stupidity around RS, GNU and why it shouldn't expressly call out LGBT as an audience member allowed to use their software.

I'm up to 51 different accounts, about 20 of them banned from the previous subreddit... I try to birth 2-3 reddit accounts every month, with different verified emails.

threeseed5• 3 years ago

stop catering to the snowflakes of the world

Those snowflakes include Apple, Google and non-US governments who aren't interested in the "anything goes" version of free speech.

So if Reddit cares about having a mobile app or operating at all then they need moderation.

Or lest we forget Reddit's good old days of doxxing, child porn, snuff porn, drug dealing etc.

the_doctah9• 3 years ago

Unfortunately Reddit moved past "slippery slope" censorship and went to full on avalanche.

"Hate speech" censorship apologists are the worst, by the way.

the_doctah9• 3 years ago

Reddit has censored themselves into absurdity. I literally can't read the front page without being logged in anymore because it's actually just all propaganda from top to bottom. At least the site was usable during this cached period, but that version of reddit is totally gone now. The fact that they used to tout free speech is hard to believe these days.

teaearlgraycold• 3 years ago

Yeah. Absolutely loaded with Ukrainian propaganda. Granted, I want Russia to lose the war they started.

jimbob45• 3 years ago

On the other hand, there are entire communities that would entirely disappear if Reddit died tomorrow. Stuff like Marathon (the game) and ergonomic mechanical keyboards just don’t have enough of a presence online to be able to direct everyone toward a new gathering point.

Fuck Reddit though. It’s all propaganda and censorship at this point. Both US political parties too - r/conservative has rules just as abusive as r/politics.

threeseed5• 3 years ago

Reddit has never touted free speech.

They were just lax about moderation until things got out of control and the company was at risk of being kicked off the app stores and being held complicit in some of the illegal conduct that was happening on the site.

Zamicol7• 3 years ago

I'd love to see more systematic documentation of Reddit's censorship, since they have so much power over the opinions of so many. There is https://www.reveddit.com who has done a lot of work for anti-censorship.

shitlord• 3 years ago

Check out https://old.reddit.com/r/RedditMinusMods. It's dead now, but the number of popular posts removed for arbitrary reasons is insane. According to the graph on the front page, something went wrong in mid-2017, because that's when the number of post removals suddenly accelerated.

cantSpellSober• 3 years ago

This back in my day comment about reddit gets copy/pasted a lot, but what do you mean by "this cached period"? The data is from 2005 to 2022.

tmaly• 3 years ago

Mastodon is the open source Twitter alternative.

What is the open source Reddit alternative?

renewiltord3• 3 years ago

This is a total misunderstanding of the way Reddit works. If you don't like the moderation and enough people agree with you, you can just go start your own subreddit. All it is is an SSO and identity solution across multiple communities. If you don't like the moderators, you can start your own subreddit.

It's actually perfect because lots of people complain about "the power-hungry mods" but in the end, very few of those could do any better, which is why there are a few schism subreddits do well.

Ultimately, Reddit is the master of "show me the money". You think you could do better? Then do better. There's nothing stopping you. Just outcompete them in the marketplace of ideas.

You can start a subreddit and pay moderators. No one will get upset.

Zamicol7• 3 years ago

The systemic suppression and censorship on Reddit is overwhelming. It's not limited to a few "communities", it's the entire system.

Data scientist/activist Robert Hawkins has been working at quantifying the degree for which Reddit attempts to manipulate opinions globally. https://twitter.com/rhaksw (See his Youtube explainer about work: https://www.youtube.com/watch?v=aCadmiIfbcI)

For another wider perspective, see the article: "Leaked documents outline DHS’s plans to police disinformation" https://theintercept.com/2022/10/31/social-media-disinformat...

I'd hope after Snowden we'd know better. It's painful to see the same old tricks working so well.

wincy2• 3 years ago

Totally true! Which is why there’s a healthy conservative portion that’s on Reddit, and r/theDonald one of the largest thriving communities there is.

Oh wait, that’s not true at all.

These are just empty lies and niceties that don’t stand up to scrutiny. Reddit has weaponized information and is just another tool of propaganda.

fugalfervor• 3 years ago

the snowflakes of the world

Please enjoy. https://en.wikipedia.org/wiki/Poe%27s_law?wprov=sfla1

devd00d• 3 years ago

You are 100% correct. I know of hordes of people being banned for the most ridiculous reasons. And it's always a power-hungry mod. And it would be fine if they were just banned from a sub, but I've seen guys get banned completely for the most stupidest reasons.

lispm2• 3 years ago

I would hope that they censor lots of stuff, especially when they are legally required to do so.

There are a lot of more cases that just "snowflakes".

m-p-3• 3 years ago

ugh, back to scraping / web crawling I suppose..

ehPReth2• 3 years ago

yep, https://wiki.archiveteam.org/index.php/Reddit

HotGarbage• 3 years ago

Jan and Feb 2023 are also available from the same site. And the final March dump before Pushshift went dark is available from https://archive.org/details/pushshift-reddit-2023-03

jodrellblank2• 3 years ago

Looks like you can download the individual files in the torrent from Internet Archive instead of getting the entire 2TB torrent in one go, here: https://web.archive.org/web/20221022155442/https://files.pus...

(I've only tried a small one, not sure if they have all of them cached)

zhfliz• 3 years ago

while that may be useful to avoid dealing with torrents, many torrent clients support downloading only selected files from a torrent.

ehPReth2• 3 years ago

See also: Archive Team’s “Shreddit” https://wiki.archiveteam.org/index.php/Reddit

vosper2• 3 years ago

I'm confused by this link - it seems like a bit of a mish-mash of notes (to be fair, it is their wiki). Are they archiving Reddit, or not? It seems like mostly they were linking to Pushshift, and now that entry is struck through.

lrae• 3 years ago

https://archive.org/details/archiveteam_reddit

jb1991• 3 years ago

I’m curious how files like this comply with things like GDPR. For people who deleted their Reddit account, do individual user names show up in these files?

jojobas4• 3 years ago

The easiest way to comply with GDPR is not to be in Europe.

dgellow4• 3 years ago

That’s not correct, if your service deals with PII from European then you are subject to GDPR. It doesn’t matter where the company is located.

dgellow4• 3 years ago

Quite sure that’s a massive violation of GDPR. It’s almost guaranteed some European PII are in the dataset.

19h• 3 years ago

Yes, existing files are not updated.

zacharycohn• 3 years ago

Lotta commenters here clearly proclaiming they've never had the ~opportunity~ misfortune of moderating even a medium sized online community.

trollied• 3 years ago

Yeah. It’s loads more difficult without an easily searchable backup.

I was also quite active trying to detect and stop certain scammers, but it’s more or less impossible now.

NovaDudely2• 3 years ago

I have never done it but it seems like one of the most thankless endevours you can take on. If you do it right, from the outside it looks like you are doing nothing.

It is like the duck on the pond. Looks calm on top but the legs are frantically keeping them in place.

tudorw2• 3 years ago

my only experience led to 'delegate fast, most people are reasonable' I helped build Jolt.co.uk & Squadlinks, a large UK gaming site during the Quake/Counterstrike era, while we had issues with occasional individuals, on the whole our forums were nice places to hang out, we had legions of gamers who loved taking ownership of their sub, they'd then sub delegate moderation, being liberal with trust engendered trust and we maintained a virtuous spiral, I imagine because our audience were closely aligned it made things a lot easier.

sdflhasjd2• 3 years ago

I moderated a moderately sized gaming community forum during 2010-2015 as the "head admin".

It is a very thankless job, but harder than moderating the forum was moderating the mods themselves. Power corrupts, but the tiniest bit of lame internet power corrupts absolutely, and the only people that ever apply to be moderators are those that seek this lame power.

It starts off with reasonable people but slowly becomes a cabal of knobs with unquestionable authority who hire more of the same people until you've got an insular group of narcissistic, condescending mods that are just there to defend their own power.

boomskats• 3 years ago

It starts off with reasonable people but slowly becomes a cabal of knobs with unquestionable authority who hire more of the same people until you've got an insular group of narcissistic, condescending mods that are just there to defend their own power.

As a Serb living in the UK this hit me pretty hard.

account428• 3 years ago

Do you think anything could be done to keep moderators from going off the rails? Would making mod actions as transparent as possible help? How about user feedback that could see mods. Obviously you'd still need the dictator at the top to be benevolent but that seems to be more likely at least until the community gets big enough that selling out would actually pay.

voicedYoda• 3 years ago

I'm mod of a medium sized group (10k+), and not because i wanted it. I simply asked the mods why someone was being ignored, so they made me one...i don't go back with enough frequency to handle all situations, but as everyone else is asleep at the wheel, I've done what i can to make sure the group doesn't get bombarded with crap posts.

It is very thankless, but i do think it's neat what tools reddit tried to give us.

Just my .02

BeFlatXIII3• 3 years ago

IMO, the healthiest way to run a community is for the head admin (and top lieutenants) to be inactive. Let the deputy admin and mods handle the site. When they start power-tripping, it's time for the head to step in and clear the house.

This model is dangerous when no worthwhile applicants arrive after the call for new mods. Then the forum either stagnates, is overrun with spam, or the head admin starts becoming active (therefore putting them on the path to petty power trips that they just cleared the leadership board to be rid of).

jredwards• 3 years ago

Same. One by one the reasonable mods dropped off, until finally I couldn't take it anymore and I left too.

zaphirplane• 3 years ago

Sounds exactly like a software engineering hiring committee :)

s_dev• 3 years ago

How many subscribers is medium sized?

grt_thr• 3 years ago

I was number 3 admin for two defaults back when there were only 20 defaults. Power mods are a cancer that got all the regular mods banned and then ran the places to the ground.

mardifoufs• 3 years ago

It's a completely volunteer job. Why do it if it's so horrible? Especially for free, for a corporation like Reddit? Either it's not as horrible as it is being often portrayed or the mods get something out of it too.

hombre_fatal• 3 years ago

The day to day chore can still be a horrible experience even though you get something out of it. Whether it's a sense of charity (best case scenario) or a power trip (much more common), it doesn't mean the job is pleasant, but it may make the mission worthwhile for someone.

dang• 3 years ago

Recent and related:

Reddit Data API Update: Changes to Pushshift Access - https://news.ycombinator.com/item?id=35776848 - May 2023 (16 comments)

jokoon• 3 years ago

Important: is you want your data, just ask Reddit.

I realized this way later, after writing scripts to query pushshift.

molticrystal• 3 years ago

It is unfortunate that Pushshift's owner admitted he seemed overwhelmed working on the project as a solo individual and decided to stop communicating with everybody including the admins of reddit.

Since reddit is trying for an IPO and thus lawyers and execs looking to purge or exploit everything legacy it is hard to say how the negotiations would of went, but I'd imagine the terms would of impacted pushshifts functionality or been so unfavorable it would of shutdown anyway.

https://old.reddit.com/r/pushshift/comments/135cyzk/update_o...

guraf• 3 years ago

Would of doesn't mean anything.

You meant would have.

Reddit comments and submissions collected by Pushshift

Related Stories