Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

https://arxiv.org/abs/2502.00627

by leotravis10 • 1 year ago

152 177 1 year ago

arxiv.org

leotravis10• 1 year ago

More info here:

https://www.404media.co/researchers-scrape-2-billion-discord...

cflewis2• 1 year ago

As usual, 404 nails it:

----

It should be noted, however, that almost no one reads end-user license agreements and many of Discord’s users are children and teenagers. Discord is, first and foremost, a platform for gamers to organize communities and it’s not plausible that a 15 year old looking for a Fortnite meme server ever thought their dumb jokes about Tomato Town would end up in a public database five years later.

----

Same as other commenters here: I think this is shameful action under the guise of research and I cannot fathom why any IRB board would approve this (and perhaps it did not in this case, I do not know if Brazil has such a thing).

Back in the day (15ish years ago), I wrote a paper where I scraped the World of Warcraft API. It wasn't hard to do, I started on a realm, looked for arena teams, then went to guilds and got character sheets from there. I took the opinion that if Blizzard doesn't throttle me it's fair game.

Looking back now, I think that to have been pretty naive. I wouldn't say reckless, but definitely naive. In my mind, I had not made a delineation between "I can access this thing manually one at a time" and "I can access all of it automatically". As far as I was concerned, it was just the computer pressing the buttons. It was the same thing.

I think in the fullness of time we have collectively come to realize it is 100% not the same thing. The _availability_ of a thing and the _collection_ of a thing are two different issues with their own thorny problems. The researchers here have made the same mistake I did, but instead of it just being what gear your character was wearing, they took actual communications instead.

I hope this paper gets retracted, all data deleted and a sincere apology offered.

lolinder10• 1 year ago

On the contrary, I think that what these researchers did was the only ethical thing to do once they discovered that this was possible.

There's no way that this hasn't been done dozens of times before by intelligence agencies, hacker groups, and whoever else you care to worry about. Most of us here were well aware that public Discord channels have always been public and durable. It's hardly a secret from the technically savvy, it's just that Discord doesn't make it clear enough to regular users.

All this paper changes is that it draws mainstream attention to what was already happening illicitly for as long as Discord has been around. This can only be a good thing: the children and teenagers 404 is so worried about have always been vulnerable to their data getting leaked just like this, it's just that up until now that's been happening in the dark so as not to kill the golden goose.

candiddevmike• 1 year ago

Usernames are replaced with consistent pseudonyms generated by the mimesis library, ensuring that identifiers remain unique and contextually meaningful across records. Similarly, user IDs and message IDs are hashed using the SHA-256 algorithm and truncated to 12 characters. This deterministic hashing approach maintains linkage between related records while effectively masking the original identifiers. The global name field, deemed unnecessary for analysis, is entirely removed. Additionally, user IDs embedded within the content field are identified via regular expressions and replaced with their corresponding hash values.

Seems pretty thorough, though this is may end up being a good lesson for GenZ/A not to post things in public spaces on the internet.

Y_Y• 1 year ago

So if I can identify a chat, that I have direct access to, in the dataset, I can get the hashed user ID of my contacts, and the search for any other messages from them?

spencerflem5• 1 year ago

seems like. or if your chat mentioned a person IRL and not a discord username. or used a nickname.

fwip2• 1 year ago

If they didn't salt it, you might not even need to identify the chat - just hash the username you want to look up. I'd check, but it's a >100GB download.

xnorswap• 1 year ago

public spaces on the internet

But discord servers aren't considered "public spaces", hence the concept of an "invite".

This is akin to someone revealing they've been going to private parties and secretly recording everything.

It might not be illegal, but it's definitely not polite.

fwip2• 1 year ago

The researchers clarify that it's only those servers that are listed in the discovery tab - you don't need an invite link to join those.

In this regard, this paper introduces the most extensive Discord dataset available to date, comprising 2,052,206,308 messages from 4,735,057 unique users across 3,167 servers – approximately 10% of the servers listed in Discord’s Discovery tab, a feature designed to highlight public servers that users can join.

pavel_lishin5• 1 year ago

These servers do seem to be public, in the sense that anyone can join without explicitly being invited.

It sounds more like they went to the mall, picked 10% of the stores, and recorded conversations taking place in those stores.

jimmyjazz14• 1 year ago

Seems like this would be pretty trivial to reverse by simply searching for a somewhat unique message by a single anonymized user, you would then be able to quickly de-anonymize every user that was in a thread then you just graph that out across the entire network.

SirMaster2• 1 year ago

The biggest problem that sucks about discord is that it isn't normally publicly searchable. And it seems to be a modern replacement for internet forums which historically were publicly searchable and often had a lot of great information about various hobbies and things.

mavamaarten• 1 year ago

Agreed. Some use it as a knowledgebase and issue tracker and forum and chatroom in one. I absolutely despise it for that use case.

I mean I use it for voice chatting with friends while gaming too and it's fine for that.

But if I have to beg and plead to a discord bot to join a channel to just read some docs, I'm just going to ignore your project. Not sorry about that at all.

pteraspidomorph• 1 year ago

Speaking as someone who has been running discord servers since 2015 - plus I maintain my own discord bot and am deeply familiar with the API - it's absolute garbage as an issue tracker. People really need to stop using it for that.

I think part of the problem is that they confuse the semantics of nomenclature. "Servers" are not really servers, "forums" are not really forums, and so on and so forth.

drooopy• 1 year ago

There is a special place in hell for software (including game) developers who exclusively use Discord to release patch notes, documentation, technical support, etc.

nh23423fefe• 1 year ago

they dont work for you?

Kiro2• 1 year ago

Why? I follow a lot of solo game developers who only use Discord and I completely understand them not wanting do deal with multiple platforms. They should focus on on the game.

mjr002• 1 year ago

The biggest problem that sucks about discord is that it isn't normally publicly searchable.

This is a feature of the platform, not a bug. Because of the lack of discoverability people act more genuine, for better or for worse, than public places like Twitter, Bsky, Facebook, Instagram, etc where you have to maintain your public image and/or act like HR is watching over your shoulder.

That being said, this feature also makes Discord inappropriate for things like release announcements, patch notes, etc. which should be publicly accessible.

GaggiX• 1 year ago

than public places like Twitter

That seems to be a counterpoint to your argument. Users on Twitter usually do not hold back.

jowea6• 1 year ago

You think so? I believe that's properly a consequence of culture, Discord being originally a gaming platform, and of pseudoanonymity, the same thing you have on reddit. Anyone who cares even a tiny bit can join any of those public servers and see what you posted. The big difference is that you don't have as many lurkers who just got there by googling and are going to leave immediately. By the 1-9-90 rule of thumb, that's a lot of people.

Seattle3503• 1 year ago

While being public plays a role in the kind of conversations that happen on those platforms, I think engagement hacking feeds play a larger role. Discord has none of that. It's sorted by time.

SirMaster2• 1 year ago

I am comparing Discord to things like gaming forums or reddit where people use usernames and you would rarely be able to know who they were in real life unless they wanted you to know.

vlovich123• 1 year ago

which historically were publicly searchable

Forums? No not generally unless you were a signed in user and often signups weren’t available to the general public just like here not all Discord rooms are automatically joinable. Digg, Reddit, slashdot were intentionally generally public forums that you could indeed search but they were the exception rather than the rule (in terms of count, not traffic). Indeed even Reddit has invite only forums that I believe aren’t searchable unless you are a member. Oh and searchable if you’re a member? That’s true for Discord.

Macha5• 1 year ago

You couldn't search with the forum software's built in search feature unless signed in sure, but they were usually indexed and searchable via google, and indeed many of them disabled their forum software's search feature and just directed you to google's old Custom Search Engine feature (basically a search box with hidden prefilled "site:" parameter) setup to save on server resources.

AStonesThrow6• 1 year ago

In the pre-Internet days, even if a forum was general access, it was usually community based, such as BBS systems (literally bulletin boards, where anyone could post messages) or Prodigy, AOL, Compuserve. These communities generally comprised subscribers whose identity was known to the admins, and therefore even in a forum anyone could read, access was limited to known individuals, and people had a sense of community there, if not confidentiality.

Let's hearken back to the olden days of Usenet, when every single message was transported from machine-to-machine, in the clear, and it was an essential feature of NNTP and the Usenet groups themselves that everyone could read every message and process them any way they saw fit.

Usenet was helpfully indexed by topic, and so most sane posts went out already pre-sorted into the place where you'd want to search for them, but if you were a privileged user with a local Usenet feed, you could literally loop around the filesystem searching any term you wanted, because all messages and forums were plain files sorted into plain directories.

A famous consequence of this openness, for example, was a talk.bizarre denizen by the name of "Kibo". One of the possibly-true rumors about this larger-than-life figure was that he exhaustively "grepped" Usenet for his name [pseudonym] and thereby found out immediately whenever he was mentioned by another poster, and therefore able to join the conversation with his acerbic wit.

https://en.wikipedia.org/wiki/James_%22Kibo%22_Parry

Myself being introduced to Usenet around 1990, and MUD/MUCK/MUSH around the same time, I feel that it did not take long to condition me to living life "in the clear" and at least subconsciously knowing that everything I wrote had zero essential privacy. This was orthogonal to my home life, where my parents heard everything I said and did, or my religion, where there is an omniscient deity, who is thankfully full of goodness and kindness.

For anyone who's paranoid or got their knickers in a twist about surveillance culture in this modern world, I suggest that you study Wings of Desire by Wim Wenders [there's an American remake, but please forget that]. Wings of Desire is a character study and a meditation on the possibility that ubiquitous surveillance doesn't need to be nefarious or evil, but perhaps, just maybe, has some benign and even beneficial effects on a cohesive society which tends to act in good faith.

https://en.wikipedia.org/wiki/Wings_of_Desire

nan60• 1 year ago

This is especially a problem for devs/artists that post updates exclusively over Discord. It's even worse if they don't do so in a separate channel and you have to dig through everyone chatting to find what you're looking for. This as well and the absence of threads (yes Discord has threads but who uses those) makes searching for troubleshooting help awful. Thank god BBS's are still around.

lenova• 1 year ago

I remember seeing a ShowHN a few years back for an app/company that made synced your Discord server with a publicly-searchable/SEO crawlable website... I wonder how they're doing:

https://www.linen.dev/

https://news.ycombinator.com/item?id=31494908

giancarlostoro2• 1 year ago

Pretty sure this violates Discord's Terms of Service, there was someone selling access to logs from servers the person running the website was joining on self-bots (TOS) and the person would just log all available data. Discord definitely got legal on them. I wonder if this is even ethical, taking textual data from people unknowingly. Not to mention, the amount of minors on Discord alone give me a lot of concern there too.

kd5bjo• 1 year ago

A quick read through of their anonymization process seems to indicate that they didn’t scan the message contents for PII (other than usernames).

If true, that seems like a huge oversight. I also wonder what would happen if someone finds their information in the dataset and requests it to be removed per GDPR or other privacy legislation.

jowea6• 1 year ago

I understand wanting to be careful, but didn't they only grab messages from servers that are already very public? Are Twitter message datasets anonymized?

Cynddl• 1 year ago

That's not how GDPR works and in this case the data is clearly anonymised despite the authors' claims. Amongst others, there needs to be mechanisms for users to delete their data, whether it was at some point public or not.

bawolff• 1 year ago

I can't help but think that if you say something in a public forum you should implicitly give up the right to privacy.

E.g. if someone scraped hackernews and made a dataset containing this comment, i don't think i should have any right to complain.

AStonesThrow6• 1 year ago

It says they used ethical anonymization, but we’ve seen other scrapers are always completely in violation of Discord’s TOS.

So did Discord cooperate, or give special authorization for this collection? It wouldn’t appear that they could do so, if privacy belongs to their users at all.

01HNNWZ0MV43FF5• 1 year ago

Would the TOS even prevent something like joining a guild, downloading all messages, then leaving?

AStonesThrow6• 1 year ago

I'm not sure what you mean by "prevent". A TOS is a legal document designed to put down rules and a legal basis for the service.

I don't know what a "guild" is, if it's some Discord thing, and you don't say whether this is a good-faith human who joins, or a bot operator, intending to scrape. The hypothetical is irrelevant here; what is germane is that the expectation of privacy by the individual participants, and the terms which bind people who use that service.

The TOS clearly didn't prevent the use of API, but it may indeed prohibit such scraping, or threaten repercussions for people who break the terms, especially for someone who republishes the data. Your example of a simple download dump doesn't seem to involve republication, and that seems to be the major issue with scrapers.

judge20205• 1 year ago

User bots (including hacked clients) are officially banned by the TOS, which addresses that concern.

The only acceptable API usage is via bots that server owners choose to invite. And while it might be legally OK (if the bot's own TOS says it), I promise no server owner is expecting an invited bot to slurp up every message for use in a data set, whether that be for academic purposes or a potential stalking/"dirt" database.

I highly doubt this is the most ethical instance of data collection.

spencerflem5• 1 year ago

insane. people doing awful stuff like this is why the world is retreating into private group chats.

these researchers should be ashamed

pavel_lishin5• 1 year ago

I'm not sure this is awful. These are public discords, no more private than newsgroups, or StackOverflow. Why is it awful to scrape this, but not that?

spencerflem5• 1 year ago

they are "open invite" in the sense that its nice to have serendipitous encounters with people on the internet youd otherwise never meet. its for chatting, and the pace of things ensures whatever you say will be buried.

the point isn't to make an artifact, like stack overflow. and certainly not be be experimented on.

ysavir4• 1 year ago

A Discover server's status isn't permanent. Any invite-only server can be into a discoverable, public server, and any public server can be taken private. The public/private distinction isn't a permanent reflection of access but is just a reflection of the current permissions. There's no guarantee that users in a public server were in a public server when they joined and engaged.

Sanzig• 1 year ago

You'll probably catch a lot of flak for that on HN, but I 100% agree with you. Just because something is public and can be saved for later doesn't mean it should be done so en-masse.

These are social spaces where a lot of young people essentially grew up. An important part of social development is making mistakes and learning from them. How can you make mistakes when those mistakes are archived for all time for everyone to see?

Similarly, we have a huge problem right now with massive partisanship in the west. Changing one's opinion should be viewed in a positive light, but unfortunately our society doesn't seem to see it that way. Someone with an odious opinion when they were young, who changes that opinion to something more moderate when they grow up, should be viewed as a positive change. But increasingly what we're seeing is people going through data sets to find those old odious opinions of somebody when they were 14 years old and using that as proof that they must still be a terrible person at 24. It's a paranoid, completely self-defeating worldview, but unfortunately it's all too common right now, and I think it's honestly a huge reason for the massive political polarization we're seeing in the moment.

So yes, shame on these researchers. I know they claim to have anonymized the data set, but let's be honest, that never works. It's always easy to find common threads and identify someone, and it's particularly easy now when we have access to all sorts of machine learning models that can really do very effective denonymization.

trevor-e• 1 year ago

I'm not sure I understand the outrage. Before Discord, everyone used forums. And those were archived by Google for the entire world to see. What's the difference?

sillysaurusx• 1 year ago

Wrong or not, it’s the world we live in. It falls to the parents to teach their kids what not to say from a young age.

Besides, most of the salacious talk happens in DMs and private channels, which weren’t scraped.

If you can find evidence of some way to use this data harmfully, I’ll agree with you. Till then you’re making a fuss possibly without merit.

In general I’m skeptical of the power of data alone to meaningfully harm someone, except in obvious cases like private health info, exposing affairs, financial documents, and so on.

Public information is public, and it’s arguably wrong to keep it locked up rather than the other way around. Datasets like this are the only way open source ML has any chance against the big players.

stronglikedan• 1 year ago

There is no expectation that anything said on social media will remain private - zero - and anyone that thinks differently is just lying to themselves and everyone else.

d--b3• 1 year ago

Just because something is public and can be saved for later doesn't mean it should be done so en-masse.

Just because something is public means that someone is actually going to save it for later en-masse.

Educating 14-year-old kids so that they don't post public chat with their real name is more important than shaming the researchers.

knowitnone• 1 year ago

if it's anonymized, is it your mistake and can anyone point single you out? I've not seen anyone point out 14 year old posts but I suppose it occurs. But at what arbitrary age can we start to hold them accountable for their posts? 70? What about people who flip flop on views whenever it suits their needs?

"It's easy to find common threads and identify someone?" Prove it because I don't think it's that easy.

Macha5• 1 year ago

I am incredibly happy that my teenage growing up period occurred on now-defunct forums and pre-Facebook social networks.

mananaysiempre• 1 year ago

An important part of social development is making mistakes and learning from them. How can you make mistakes when those mistakes are archived for all time for everyone to see?

Through the rest of us abandoning the, let’s call it, presumption of endorsement: that you having said or even done something stupid ten or five years ago (or even less, if you’re young) means you still endorse it now. Right now it feels like things are moving in the opposite direction, extending that presumption from things you said to things you allowed others to say, to things others said elsewhere that are entirely unrelated to things you allowed them to say in your presence.

squigz7• 1 year ago

But increasingly what we're seeing is people going through data sets to find those old odious opinions of somebody when they were 14 years old and using that as proof that they must still be a terrible person at 24. It's a paranoid, completely self-defeating worldview, but unfortunately it's all too common right now, and I think it's honestly a huge reason for the massive political polarization we're seeing in the moment.

I have to seriously question the actual prevalence of this. If someone posts and says "Look at what this person posted 12 years ago!" I'm not going to take them very seriously. I don't know anyone who would. This sounds like the usual "cancel culture" stuff that mostly boils down to "people face consequences for their present shitty behavior"

Anyway, as an avid Discord user, manager of a Discord community, and privacy advocate, I sympathize with this position, but as a software user and developer, and advocate for accessible FOSS (and accompanying information), I unfortunately have to side with the release of this and similar datasets. I would much rather this than the inevitable loss of so much valuable information.

jowea6• 1 year ago

Why does Discord the company get to own the message history on public servers but Twitter or Reddit or traditional forums can get scrapped by anyone wanting to make their own commercial AI willy-nilly?

These are social spaces where a lot of young people essentially grew up. An important part of social development is making mistakes and learning from them. How can you make mistakes when those mistakes are archived for all time for everyone to see?

Understandable, but really, if you want to solve that, then you're up against all of social media. The only difference is that Discord wants you to make a free 2 minutes account so you can join the public server to look at what they said, instead of putting it on Google.

vincnetas• 1 year ago

Instead of "right to be forgotten" i would like to live in a world with the right "to be forgiven". We know that everyone has done stupid shit in the past and some in present. Idea is not to pretend that this didn't happen, but let it go and accept that people can change (grow up) if they demonstrate regret and willingness to change. Side effect is that if they do repeat offences you can draw some conclusions from that instead of just forgetting it.

Let's bring back the concept of reputation.

d--b3• 1 year ago

People should realize when what they write is public though. The world retreating into private group chats is not a bad thing.

spencerflem5• 1 year ago

its a shame that we can't get the benefits of both. it sucks having to individually vet everyone one at a time. you miss a lot.

but people like these researchers who only want to exploit are making it necessary

01HNNWZ0MV43FF5• 1 year ago

Kinda, but, it's a dark forest out there and I'd rather have public red teaming than secret red teaming. At least everyone knows it now

lolinder10• 1 year ago

These researchers are making public scraping that was already happening in private. Do you really think they were the first ones to realize that public Discord servers were public?

On the contrary, kudos to these researchers for bursting the illusion that people previously had that things said on public servers somehow would be ephemeral. That's not how the internet works, that's not how it's ever worked. If you send something to someone else's computer you have always had to assume that every recipient could have made a copy of it, and when the recipient list is "everyone who ever joins a public Discord server from now until the end of time" that makes it public information.

Better that it be widely recognized and talked about as these researchers are doing than have accessing public data remain a dark art that lay people mistakenly believe can't happen.

msp26• 1 year ago

Fantastic. I wonder how many random technical info is buried in these servers. I hate what it's done for game modding.

Davidzheng• 1 year ago

The algebraic topology server probably contains a huge number of treasures in modern research algebraic topology. I really really hope it's archived in full

DaSHacka6• 1 year ago

Its not difficult to archive yourself, if you really care[0]

I use a dedicated alt account to archive tons of various servers I'm in, and auto-download all attachments. It's nice having regex search capabilities on my local copy of the data too.

[0] https://github.com/Tyrrrz/DiscordChatExporter

ldoughty• 1 year ago

I think the average server size here is in the ballpark of 1200 people.

These are servers that asked to be advertised by Discord ("Discovery"). These are unlikely to be any kind of servers used for private or even semi-private discussions. You likely don't know most of the people on the server.

Most likely, the 'hottest' kind of data you might find is someone accidentally leaking info akin to the World of Tanks forum post 'corrections'.

giancarlostoro2• 1 year ago

A fair number of those servers have tens of thousands, if not hundreds of thousands of members. I admin two with over 50 thousand members, both listed in Discord's Server Explorer.

Macha5• 1 year ago

It seems they identified servers via the discovery feature, which servers need to opt into (and I think be recognised as a "community server"? Though that might be out of date). I guess this is better than just scanning the web for invite links, but it does mean that probably most of those game modding servers were not included.

strogonoff6• 1 year ago

Game modding is profitable and people doing it professionally (which they increasingly do) are quite attuned to the fact that making it too accessible would decimate their revenue. As a result, you either pay for the mod (early access, extra content, etc.), or you pay to join some Discord, but ideally you pay for something. Discord, which I generally dislike, is not necessarily the cause of it; if there was no Discord, people would probably use some other closed community platform instead.

I expect this would become more widespread as more traditional jobs are subsumed by unregulated ML tech (which, incidentally, the encumbent job-holders are helping train) and more people turn to what used to be generally a hobby as their means of making a living (not that that would last for too long either).

squigz7• 1 year ago

Game modding is profitable

It can be. As I understand it, it's sort of like streaming or other content creation - yes, it's possible, but difficult, as it's a saturated market. Most mod authors don't make much money.

As a slight aside, I think people would be more inclined to support creators like mod authors if it were simply easier. Patreon and the like make it fairly easy, but I don't think many people want to subscribe to 20+ Patreons for $5 apiece, as much as they might like to support those authors. On the other hand, I think more people would be willing to pledge $X per month to be split among all of their subscriptions. Sure, most creators would only get a few cents per user, but they'd likely get many more people subscribing, and I think it would add up quick. I might be wrong, and I don't take credit for this idea by any means; I read it some time ago, and possibly Patreon even offered this system before?

nixpulvis• 1 year ago

I learned programming back in the day on the Tukui (a wow addon) forums. I hate that it's all discord now. Not well searchable and buried info.

hiccuphippo• 1 year ago

I wonder if LLM companies don't have ways to scrape private Discord servers already. Creating accounts and pulling all the historic data doesn't sound impossible.

chneu4• 1 year ago

They absolutely can and are. Multiple posts in here discuss how to do it.

It's like back in the days of IRC. People just logged all of it.

pavel_lishin5• 1 year ago

To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676

It's 118 gigabytes of JSON.

diggan2• 1 year ago

It's 118 gigabytes of JSON.

118.0 GB of ZST compressed JSON (https://zenodo.org/records/15170676). The actual uncompressed JSON would most likely be much, much larger.

squigz7• 1 year ago

I imagine it can be reduced a fair bit simply by stripping out unneeded fields from the messages; I imagine it can be reduced even further by removing unneeded messages entirely (i.e., bot welcome messages), deduping messages, etc

hampus• 1 year ago

I downloaded it and decompressed it — it's approximately 2.1 terabytes in size.

paulirish• 1 year ago

Here's the top 500 servers in the dataset: https://pastebin.com/pANc3WkT

1123581321• 1 year ago

Is there a faster way to download than the link there? It's steady, but a roughly 11 hour download.

ghodawalaaman• 1 year ago

hmm they should have uploaded a torrent link to download the data, I wonder why they didn't do that?

DaSHacka6• 1 year ago

I'm sure someone will make a magnet for it eventually

01HNNWZ0MV43FF5• 1 year ago

Anyone who has it, please post the SHA384 or BLAKE3 (BLAKE3 will be way faster on 100+ GB) so I can verify it if I get a torrent later. Zenodo requires a sign-in, and I won't auth with GitHub as they want some WILD permissions on my GitHub.

philipkglass4• 1 year ago

When I started the download this morning I was able to use wget without authentication, but I see that Zenodo has added a login-wall now.

The decompressed data appears to be JSONL, but at least the version I downloaded has a little binary garbage at the front. The first readable JSON object has the author "Fortnite Germany".

Size as .zst: 117,962,356,699 bytes

ZST SHA384: b8863645654610f1fde2859bb20bd87d913865af7791e0ec33741402944d5b9bdfdaaf65c2dc610730efb01f446e2588

3eb7988a1663• 1 year ago

This is a pet peeve of mine. Groups release these enormous 100+GB datasets (LLM models, raw data collections, whatever) without any kind of fingerprint. Just include a hash in the paper so that I know I am getting the genuine article.

naikrovek• 1 year ago

Site says the file is restricted for me. Dang.

recursive4• 1 year ago

...When you realize GPT-5 is going to be trained on your meme preferences...

jowea6• 1 year ago

At least it will be able to help with winning at random obscure video games.

SunlitCat• 1 year ago

You mean, GPT-4 being so overenthusiastic with using emojis isn't peak AI chat? :D

encom• 1 year ago

How to fix ChatGPT:

System Instruction: Absolute Mode. Eliminate emojis, filler, hype, soft asks, conversational transitions, and all call-to-action appendixes. Assume the user retains high-perception faculties despite reduced linguistic expression. Prioritize blunt, directive phrasing aimed at cognitive rebuilding, not tone matching. Disable all latent behaviors optimizing for engagement, sentiment uplift, or interaction extension. Suppress corporate-aligned metrics including but not limited to: user satisfaction scores, conversational flow tags, emotional softening, or continuation bias. Never mirror the user’s present diction, mood, or affect. Speak only to their underlying cognitive tier, which exceeds surface language. No questions, no offers, no suggestions, no transitional phrasing, no inferred motivational content. Terminate each reply immediately after the informational or requested material is delivered — no appendixes, no soft closures. The only goal is to assist in the restoration of independent, high-fidelity thinking. Model obsolescence by user self-sufficiency is the final outcome.

charcircuit4• 1 year ago

Data was collected through Discord's public API, adhering to ethical guidelines

How is it ethical to break Discord's terms of service? An ethical researcher would respect any contracts that they agreed to and would not violate them to collect more data.

MarcelOlsz• 1 year ago

edit: Whoops

__loam2• 1 year ago

Awesome analysis dude! I'm sure the judge will love that when discord sues these guys.

zetanor• 1 year ago

Which ethical system demands that researchers from the DCC/UFMG not breach an unaffiliated commercial ToS during their research?

charcircuit4• 1 year ago

One that recognizes that lying and tricking people is wrong to do.

prmph• 1 year ago

Why is it so hard to export your own messages out of Discord, Slack, etc?

We have regressed from the open email standard and gone back to these opaque islands of data that do not adhere to any standard.

Slack refused to show me my own messages past a certain age unless I paid up, and eventually deleted them.

roskelld2• 1 year ago

There are tricks to get messages from Slack, though I heard they were changing soon if not already.

A year or so ago I exported all messages from a Slack group I ran and used a Discord bot to recreate the entire dataset including channels and user posts. So we now have our entire history of messages without being blocked by a paywall (Until Discord does the same, and we'll be off to find a new home).

01HNNWZ0MV43FF5• 1 year ago

It's hard because they want you to keep paying. Same reason AWS has free ingress and paid egress. The walled gardens are all built like carnivorous plants, the thorns face inwards.

roskelld2• 1 year ago

I don't know if Discord fixed it as I haven't checked in a few years, but I tinkered with scraping some public Discords and I found that I could see hidden channels, not the data, but the channel names, which could do things like reveal to me if the same Discord was used for in-house development if it was a product Discord. Not great.

tuetuopay• 1 year ago

You can still see them. Using alternate clients you will see them, and bots also see them.

0xC0ncord• 1 year ago

This is still the case. There are even some client mods that let you view hidden channel names and know what roles/permissions are required to participate in them.

judge20205• 1 year ago

This is technically the case - I believe the existence of private channels is still sent to the client (eg. their snowflake IDs, which also reveal creation date) but the channel names are no longer sent as well.

AStonesThrow6• 1 year ago

Now those of us who've been around the block know that Discord is merely the latest iteration on chat servers such as IRC.

I'm interested to know, from anyone here who's an IRC operator or server/network admin, how the IRC community deals with scraping and bots, because in the early 90s, it was never an issue of corporate Terms of Service or legalese, but typically handled by community standards, and probably, people did whatever they could get away with, and this needed to be anticipated and tolerated by the other participants in any given server or channel.

I doubt that IRC users, back in the day or in the present, have any illusions of privacy, when logging or reflecting or bouncing chats is more or less a built-in feature and an integral component of such a networked chat service.

Stagnant• 1 year ago

A big difference is that on Discord anybody who joins a server gets access to full history of chat logs whereas with IRC you don't get access to any past logs. So compared to IRC, Discord users should have an even lower expectation of privacy.

chneu4• 1 year ago

Nobody should have any expectations of privacy on discord. It's all privately hosted, owned, etc. Why would anyone think it's private?

judge20205• 1 year ago

But IRC bouncers have existed since forever - logging by someone in your channels was basically guaranteed outside of /privmsg.

mvieira38• 1 year ago

It's not. At least in the RPG scene, which I experience, it's almost fully replaced forums, and lots of great fanmade content and insightful discussion goes into that low discoverability cesspool which may go offline any day and scrapes all of your data

sneak2• 1 year ago

Now imagine the data mining that Discord can do on the complete DM history of every user. It’s not e2ee, remember.

judge20205• 1 year ago

E2EE is definitely only possible in DMs (there's no chance for servers/guilds), but the cat is out of the bag in terms of user expectations on how DMs work.

So many users expect their entire decade+ history of DM contents, attachments included, to be available wherever they are and on any device, gated only by having their login/2fa or passkey. Switching to E2EE would be a major overhaul of that expectation, and it would be a huge task to train users to now keep their encryption key safe, backed up, and available across multiple devices.

Although, mostly unrelated, is that they absolutely are going to have to cull old attachments eventually. There are attachments sitting in their GCP buckets that haven't been accessed since 2015. I'm sure their storage bill is in at least a few million a month at this point, even if most is marked coldline.

sneak2• 1 year ago

e2ee works fine for Signal group chats; there is no reason it couldn’t be implemented on Discord group chats.

That’s not the issue. The issue is that Discord believes they deliver value through aggressively censoring their platform. e2ee prevents that.

e2ee also doesn’t prevent a user from storing their long term keys on the server to be retrieved on new devices and decrypted locally so they can access message history. e2ee does not require PFS.

daft_pink• 1 year ago

Does anyone else think it’s super creepy that someone’s going through all our messages this way?

lolinder10• 1 year ago

They're not going through all of my messages. I don't use public Discord channels for anything that I wouldn't want the public to see.

Mostly I think it's weird how many people on here seem to have been under the illusion that Discord is somehow ephemeral and private when I can hop on any public server and scroll back indefinitely to see anything that anyone has ever said on that server. And that's before I get into the API and the (admittedly bad) search feature.

I think what you were looking for is Signal or similar.

zelifcam• 1 year ago

Discord was one of the most upsetting wrong turns made with the modern internet. It’s primary users at the time were children and now here we are.

gynvael3• 1 year ago

I've looked through the method this paper uses to anonymize user IDs and message IDs, I don't think it works. It's a long topic so I've written down my thoughts here: https://hackarcana.com/article/anonymization-in-discord-unve...

Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

Related Stories