How Discord stores trillions of messages (2023)

https://discord.com/blog/how-discord-stores-trillions-of-messages

by jakey_bakey • 2 years ago

435 231 2 years ago

Anyone else reading this and being quite happy that they're not working at this scale?

Any time I read anything about any web-adjacent technology I'm incredibly thankful that I don't work anywhere near that industry.

Embedded can be complex, but web stuff is just a Lovecraftian nightmare in comparison

milesvp• 2 years ago

I have stared into the abyss and seen the eyes of cthulu. I am much happier writing embedded drivers than I was trying to make sense of why previous devs thought it was a good idea to move bounded tunable server side api calls to the client, allowing it to effectively write arbitrary sql calls across multiple databases.

Twirrim• 2 years ago

But that's where the really fun and complicated problems are. The ones that really make you stop and think, and not just think, but be creative.

95% of the work is still the same "treading in well trod paths", same old same old tech work, but that 5% is really something.

pavel_lishin2• 2 years ago

I think I can understand the appeal, but it's just not there for me. I have enough complicated problems outside of work, some of which are even fun to solve.

Olreich• 2 years ago

This was a “double-pump” migration to a faster database and building a caching service. There’s nothing particularly fancy or creative about their solutions. The migration efforts and working out issues with the reverse table scan were probably way more creative, but they didn’t get into that unfortunately.

wavemode• 2 years ago

I don't mind scale. I mind the bureaucracy and promotion-driven-development that comes with working in a bloated engineering org.

pm90• 2 years ago

+100

Many companies have products that operate at “scale”. They manage to do so with pretty boring techniques (sharding, autoscaling) and technologies (postgres, cloud storage).

Because of the insane blog driven tech culture, many of these teams get questioned by clueless leadership (who read these blogs) and ask why the company isn’t using cassandra / some other hot technology. And it always causes much consternation and wastage.

est• 2 years ago

I am happy that I dont have to deal with this.

I am sad that my business aren't as big as this scale.

Aeolun6• 2 years ago

Honestly, 77 nodes doesn’t sound like a terrific scale? The more I scale things up, the more I realize that the tone of the problems doesn’t really change. You just get more layers to your data structures.

twelve40• 2 years ago

I'm happy I'm currently not working at this scale. I'm not happy when idiots (including one of our self-important ex-Google VP's) set this as a benchmark for backend interviews (for careers that 99% likely will never come close to such problems).

leetrout• 2 years ago

Needs (2023)

That services layer reminds be of a big, fancy, distributed Varnish Cache... they don't mention caching and they chose the word coalesce so I assume it doesn't do much actual caching. But made me think of Varnish's "grace mode" and it's use to prevent the thundering herd problem (which is where I first heard of 'request coalescing') https://varnish-cache.org/docs/6.1/users-guide/vcl-grace.htm...

Also love to see consistent hashing come up again and again. It's a great piece of duct tape that has proven useful in many similar situations. If you know where something should be then you know where everything is gonna come look for it!

loloquwowndueo• 2 years ago

Coalescing and “origin shielding” tend to be more common terms for that - I’ve never heard of “grace” until today :)

atombender2• 2 years ago

Varnish does call it coalescing. Grace is used for a specific situation: When a previously cached object has expired, Varnish won't evict it from the cache immediately, but will continue to serve the old content, while sending exactly 1 request to the background to refetch. How long an object can live after expiring is called the grace. The HTTP standard calls this behaviour "stale-while-revalidate".

dang2• 2 years ago

Year added above. Thanks!

hinkley2• 2 years ago

Nginx always more businesslike.

    proxy_cache_use_stale updating;

mnutt• 2 years ago

Grace mode itself doesn’t prevent thundering herd; varnish coalesces all requests automatically and grace mode is used to increase the likelihood of clients receiving cached (albeit stale) responses.

tonetegeatinst• 2 years ago

My love of embedded stuff is growing. I'm self teaching C and assembly....to get better at low level programming and interactions with hardware but it all seems much simpler than the big data systems. Granted I'm sure it call be broken down into steps and issues to solve like any programming issue but I'm happy focusing on low level stuff for now.

codexon• 2 years ago

The ScyllaDB team prioritized improvements and implemented performant reverse queries, removing the last database blocker in our migration plan.

I wonder how much they paid ScyllaDB to do this before even using ScyllaDB.

jsnell• 2 years ago

The article says they were using ScyllaDB for everything except the message store two years before they did the migration for messages.

awesomeMilou• 2 years ago

My takeaway from this is maybe somewhat different from what the authors intended:

The last one? Our friend, cassandra-messages. [...] To start with, it’s a big cluster. With trillions of messages and nearly 200 nodes, any migration was going to be an involved effort.

To me, that's a surprisingly small amount of nodes for message storage, given the size of discord. I had honestly expected a much more intricate architecture, engineered towards quick scalability, involving a lot more moving parts. I'm sure the complexity is higher than stated in the article, but it makes me wonder, given that I've been partially responsible for more than 200 physical nodes that did less, how much of modern cloud architecture is over engineered.

nicholasjarnold• 2 years ago

...how much of modern cloud architecture is over engineered.

I would wager a good majority of it is. The Stack Overflow architecture[0] sticks out to me in this regard as an example on the other end of the spectrum.

[0] https://news.ycombinator.com/item?id=34950843

romanhn• 2 years ago

They are talking about 177 database nodes, which is not an indicator of architecture complexity. I assume they have dozens/hundreds of services consisting of multiple highly available nodes each across various geographies.

Having seen a much smaller set of Cassandra nodes used to store billions (rather than trillions) of records, I can say that Cassandra was definitely a total PITA for on-call, and a cause of several major outages.

hiyer2• 2 years ago

Also bear in mind that they're now doing the same with just 72 nodes.

tcfhgj• 2 years ago

Storing is one thing. Performing data mining on them is another

CamperBob22• 2 years ago

Also, people need to keep in mind that those trillions of messages are archived nowhere. Thanks to the walled gardens we're obsessed with building, far-future anthropologists will know more about Pompeii and Machu Picchu than San Francisco.

xboxnolifes• 2 years ago

I don't think every single individual message ever needs to be archived. Every text, every email, every post-it, every poke, every emoji, every reaction GIF...

daedrdev• 2 years ago

For many people the fact that discord is not easily discoverable is a benefit, just like in many other messaging services

squigz3• 2 years ago

Firstly, no they won't. That's silly.

Secondly, how would such an archive work? Who would pay for it? How would it be safeguarded in such a way that it can be read by 'far future anthropologists' but not the people paying for the storage?

philipwhiuk2• 2 years ago

That's a separate problem with hugely different latency concerns, likely done on a separate copy.

zombiwoof• 2 years ago

I think it’s annoying they interview engineers like they are Google and reading the blog they made it up and learned some basic “pitfalls” as they went along

foobazgt2• 2 years ago

This blog post seems to blame GC heavily, but if you look back at their earlier blog post [0], it seems to be more shortcomings in either how they're using Cassandra or how Cassandra handles heavy deletes, or some combination:

"It was at that moment that it became obvious they deleted millions of messages using our API, leaving only 1 message in the channel. If you have been paying attention you might remember how Cassandra handles deletes using tombstones (mentioned in Eventual Consistency). When a user loaded this channel, even though there was only 1 message, Cassandra had to effectively scan millions of message tombstones (generating garbage faster than the JVM could collect it)."

And although the blog post talks about GC tuning, there's mention here [1] that they didn't do much tuning and were actually running on an old version of Cassandra (and presumably JVM) - having just switched over from CMS (!).

  0) https://discord.com/blog/how-discord-stores-billions-of-messages
  1) https://news.ycombinator.com/item?id=33136453

Aeolun6• 2 years ago

But then it’s still nice that they’re using ScyllaDB and now it’s not a concern at all right?

Even if they were using their original solution wrong, I think the solution that cannot use wrong is superior.

ericvolp125• 2 years ago

The funny part is ScyllaDB still uses tombstones for deletions, though they do have configurable compaction strategies and iirc Discord uses Scylla's Incremental Compaction Strategy that I suppose solves the specific issue they were dealing with. iirc that compaction strategy will trigger a compaction once a certain threshold of a partition is tombstones and then the table is rebuilt without the tombstoned content (which effectively pauses writes on that specific node and that specific table and partition for the duration of that process). Compacting a massive partition is really expensive. Scylla defaults to warning you that a partition is too large if it has at least 100,000 rows in it. My guess is when they moved to ScyllaDB they also adopted a new strategy for partitioning messages in a channel that keeps partition sizes reasonable so compactions don't take a super long time.

roenxi• 2 years ago

I don't see anything here that looks untoward. They increased their data storage by 3 orders of magnitude and decided to use a different DB system. Fair enough, maybe they've learned more about the nature of their data.

But that logic isn't sound. When dealing with huge amounts of data there are going to be trade-offs. Picking a system that makes different trade-offs to an existing system is not automatically helpful. Yes you don't have the old problems. However, you are about to discover new problems. There is always something of a gamble around which will be more of a problem to your business.

frr149• 2 years ago

What's the problem with Scylla? Honest question, BTW

vips7L• 2 years ago

having just switched over from CMS (!)

This is really interesting. CMS was removed in Java 14 after being replaced by G1GC in Java 9. They were probably running an antiquated Java 8 or 11 runtime. So that means that in 2022 they were either running a 4 year old Java 11 runtime or an 8 year old Java 8 runtime. They were really leaving a lot of performance on the table.

gorset• 2 years ago

They could also have gone the commercial route and gotten Zing with their pauseless GC. It’s been around forever and they even cover Cassandra in their marketing.

https://www.azul.com/technologies/cassandra/

jaimehrubiks• 2 years ago

Until they don't, or they can't, and they need to start deleting.

(Not trying to undermine the engineering efforts, or the welcoming engineering blog posts though! I really think all these is needed)

cynicalpeace• 2 years ago

Is there a fundamental reason you wouldn't use postgres for something like this? Scale certainly wouldn't be it.

cowthulhu• 2 years ago

I’m not sure if Postgres would have enough horizontal scaling to accommodate the insane volume of reads and writes. I would be super interested to be proven wrong though… anyone know of a cluster being run at that scale?

riku_iki2• 2 years ago

Scale certainly wouldn't be it.

vanilla postgres can't scale to such size, you need some sharding solution on top, which likely will be much harder to maintain than ScyllaDB..

justnoise• 2 years ago

I'd guess that Discord's storage systems lean towards processing a lot more writes than reads. Postgres and other databases that use B-tree indexing are ideally suited for read heavy workloads. LSM based databases like Cassandra/Scylla are designed for write intensive workloads and have very good horizontal scaling properties built into the system.

Aeolun6• 2 years ago

Would you actually have more writes than reads? Are messages read by fewer people than post them?

jhgg9• 2 years ago

Read traffic is much higher than write traffic due to mobile clients needing to sync chat history more often as their sessions are much shorter lived. Also search queries execute 1 query per result. And don't forget people doing GDPR data dump requests. It adds up.

ericvolp125• 2 years ago

ScyllaDB scales horizontally on a shard-per-core architecture with a ballpark throughput of 12,500 Reads and 12,500 Writes per second per shard. If you're running Scylla across a total of 64 cores (maybe on 4 VMs with 16 vCPUs each), you can get up to 800k Reads 800k Writes per sec of throughput with P99 writes of <500us and p99 reads of <2ms.

You will not be able to get that performance out of Postgres and the write scaling will also be impossible on a non-sharded DB.

If you're a company like Discord and are running dozens (70-something?) of ScyllaDB nodes, likely each with 32 or 64 vCPUs, you've got capacity for 50M+ reads/writes per second across the cluster assuming your read/write workloads are evenly balanced across shards.

ryanjshaw2• 2 years ago

Okay but this is where I get confused. Why does Discord need a single database system when discord servers are independent, right?

And the volume of traffic per Discord server must be human-processable or what would the point be? A Discord server doing 800k writes per second makes no sense.

So why not a RDBMS per Discord server, and if you want to ship all that out to a warehouse for analytics you do that as a separate problem?

Or is it that spinning up a Postgres instance per Discord server ends up being significantly more expensive than these mega distributed database systems?

jhgg9• 2 years ago

Fwiw the benchmarked numbers are for writing very small rows. When doing the messages migration, with no read traffic, and the cluster/compaction settings tuned for writes we only managed approx 3m inserts/sec while fully saturating the Scylla cluster.

riku_iki2• 2 years ago

You will not be able to get that performance out of Postgres

if writes are batched, I get this and higher performance from postgres. If 800k on 64 cores is Scylla's best result, it is not that impressive.

But also you probably mean writes/reads to indexed table, then it is another story.

pawelduda• 2 years ago

Pretty fun read, even tho I'll never work at such scale lol

KaoruAoiShiho• 2 years ago

Did they go with ScyllaDB just because it was compatible with Cassandra? Would it make sense to use a totally different solution altogether if they didn't start with that.

jhgg9• 2 years ago

Yes, we wanted to migrate all our data stores away from Cassandra due to stability and performance issues. Moving to something that didn't have those issues (or at least had a different set of less severe issues) while also not having to rewrite a bunch of code was a positive.

ericvolp125• 2 years ago

Did you guys end up redesigning the partitioning scheme to fit within Scylla's recommended partition sizes? I assume the tombstone issue didn't disappear with a move to Scylla but incremental compaction and/or SCTS might have helped a bunch?

hiyer2• 2 years ago

Very well-written article. I'm happy for them that part of the solution was switching from Cassandra to drop-in replacement Scylla, rather than having to deal with something entirely different.

m-hodges• 2 years ago

Fun article. Also fun to think about how many people have decided to document their crimes in these Cassandra nodes.

dang2• 2 years ago

Discussed (a bit) at the time:

How Discord Stores Trillions of Messages - https://news.ycombinator.com/item?id=35048410 - March 2023 (10 comments)

xyst• 2 years ago

Having used discord in the past. Most of the conversations were just shit posts. Nothing serious. Why even bother storing a trillion messages of garbage in the first place?

aurareturn• 2 years ago

How do you differentiate shit posts vs quality ones if you’re Discord?

adzm• 2 years ago

Because that is literally what Discord is for

hypeatei2• 2 years ago

That's why I laugh when people say discord content needs to be indexed on the web so things are more discoverable. 99% is garbage and the useful messages are scattered across channels.

retsibsi• 2 years ago

I'm not trying to be a smartarse but doesn't this describe the entire internet? The good stuff is rare and scattered, and that's why search is so important.

huimang• 2 years ago

Many people within niches have discord servers for researching and discussing specific things. There is a large wealth of information locked away behind them that can be lost pretty much whenever discord decides to start pursuing different monetization strategies.

squigz3• 2 years ago

That sounds like it was a problem with the communities you engaged with.

jerryspringster• 2 years ago

How do you sort the good from the bad? I'm sure most of my conversations were shit posts aswell but some weren't, especially when it figuring out how something new worked or how to fix a problem.

robertclaus• 2 years ago

Very cool that even at this scale the right vanilla SQL database just works. No fancy document store, map-reduce, or GPU implementations needed.

hinkley2• 2 years ago

It annoys me sometimes how effective B-trees are.

Every decade has some cool breakthrough in compression, and a handful of other disciplines. But OLTP databases are still basically better B-trees.

menaerus6• 2 years ago

LSM trees? ScyllaDB is LSM-based storage engine. RocksDB as well.

salomonk_mur• 2 years ago

How is ScyllaDB (the solution used in the article) a vanilla SQL DB? Its the complete opposite!

melodyogonna• 2 years ago

The syntax is SQL

asjfkdlf• 2 years ago

Aren’t they using a NoSQL store? They migrated from Casandra to Scylla DB

andrewstuart• 2 years ago

When you get to scale like this, I wonder if the access patterns of the application and its data might be best served by a custom data retrieval and storage application.

I may be wrong but I just wonder if efficiency is lost to the generalized nature of any data storage system.

The other question that comes to mind is, to what extent have the developers made a systematic effort to optimize how data is stored and retrieved? If you’re building a gigantic back end system and simply accepting that the system load is what it is then you might be missing a chance to dramatically impact the size of the task of managing that data.

lyu07282• 2 years ago

They did give one example, if someone does a @everyone in a big channel, they specifically optimized their architecture to make that efficient using their custom data services.

akimbostrawman• 2 years ago

in cleartext

7bit2• 2 years ago

The blog posts shows how great the technical expertise is at Discord. I work in IT and in my company devs are so incompetent, they don't even know how to create an M365/Azure dev tenant and constantly request *.Read write.All to our production tenant. I'm so envious!

On the other hand, the HOME/END keys jump to the beginning of the input field rather than the line and the frontend devs are unable to fix this non-default behaviour for years, which makes it a fucking pain in the ass to use the Posts feature within a Discord channel. I believe the budget for the backend geniuses meant that frontend had to be juniors only.

fastball• 2 years ago

In their defense, Azure is terrible.

7bit2• 2 years ago

I haven't found a difference to AWS, for example. They are all terrible in their own ways. But if one or the other is what you earn your money with, then at least put in the effort to be proficient with it, and not a complete dumbass. (Not you as in "you!")

crop_rotation2• 2 years ago

Hiring good is probably the most important thing for a company and also one of the hardest problem. I have seen a team of competent engineers outperform their sibling teams by 5-10x as long as each member of the team is good enough. Just 2 bad hires will slow down a team drastically. One terrible hire can do -5x work of a normal engineer.

aaptel• 2 years ago

This whole problem wouldn't exist if we used distributed chat protocols which have been around for over 40 years (IRC). With the added benefit of having an open specification and multiple implementations. No walled gardens.

And if you think IRC is too old for the modern world take a look at matrix or xmpp.

How did we let discord take over is a mystery to me, or rather a tragedy.

Krasnol2• 2 years ago

Usability did it.

You download an exe, install it, make an account and it runs. Just like that. Everybody can do it.

There are tons of useful and great software out there. Most of it is not easy for the public. Some (most?) of it doesn't even have an GUI. People rather sell their identity and even pay than suffer through too many hops.

Intralexical3• 2 years ago

Not even a EXE. The web version is feature-complete, so you only need to click a link.

maccard• 2 years ago

If you want to know why, look at the App Store reviews for discord and tea speak and compare them.

Discord just works.

dewey2• 2 years ago

I’m a huge IRC fan and I dislike Discord, but all these other services are way too clunky and IRC is really only usable through IRCCloud that has a relatively okay mobile app these days.

Recently a very technical group I’m part of migrated from Telegram to Matrix and the user experience is just not very good. The apps are buggy, don’t look good, then in the new “Element” app SSO isn’t supported so I can’t use my account with it. There’s lots of paper cuts that are okay for someone like me who likes to figure it out but I’d never try to convince my friends to use it.

nunobrito3• 2 years ago

For telegram refugees then maybe SimpleX is an option, except it has no bots nor other options for clients at the moment.

What I personally use is the nostr protocol through a client like Amethyst or OxChat. Messages and groups can be E2EE private, or you can just use the public groups.

The biggest advantage is that you are joining a bigger community of apps and services built on top of the same protocol, rather than joining some isolated island (again).

lofaszvanitt• 2 years ago

Discord wrapped irc in shiny paper.

elcomet• 2 years ago

IRC and distributed protocols un general had a big issue : you loose history every time you disconnect

menaerus6• 2 years ago

In the age we are living this starts to sound more like a feature to me.

rollcat4• 2 years ago

IRC does not store messages, it only relays them to clients. You need an add-on solution to store chat history, something we've been taking for granted for ~30 years.

IRC all but requires using a bouncer to follow a conversation from more than a single device.

IRC does not encrypt messages, only (optionally) the client<->server connection. Without E2EE, you have no privacy against the server/operator, which is an easily targeted SPOF.

Matrix (the protocol) is still in flux, and the implementations are lagging behind the spec. If you're not using Element, you're behind on features and security.

XMPP is (similarly to IRC) relying on optional protocol add-ons for basic things, like E2EE, which clients may or may not support fully or correctly.

I recommend reading these breakdowns by soatok: https://soatok.blog/2024/08/04/against-xmppomemo/ https://soatok.blog/2024/08/14/security-issues-in-matrixs-ol...

2013/Snowden happened 11 years ago. E2EE should by now be considered a basic feature, a commodity, something we should be calling for as relentlessly as we did for HTTPS. (Discord of course does not implement E2EE.)

voidnap2• 2 years ago

IRC does not store messages, it only relays them to clients.

Some people consider this a feature and prefer using IRC bouncers to discord.

OMEMO solved encryption for XMPP a decade ago. I haven't seen it on IRC yet though.

AnonCoward42• 2 years ago

IRC does not encrypt messages, only (optionally) the client<->server connection. Without E2EE, you have no privacy against the server/operator, which is an easily targeted SPOF.

Same as Discord.

Matrix (the protocol) is still in flux, and the implementations are lagging behind the spec. If you're not using Element, you're behind on features and security.

Discord also only has one reference client, but for me even with that client Matrix/Element was not as reliable. I still use and like it, but it's not a like for like in that regard.

XMPP is (similarly to IRC) relying on optional protocol add-ons for basic things, like E2EE, which clients may or may not support fully or correctly.

But if you use current clients like Conversations or Dino or the likes it does work. There is no point in counting the clients that don't support it if these aren't the reference or biggest ones. The problem here is more that it's not meant to be used like Discord in any way. Not for big group chats/channels nor for big voice chats (not even sure this possible).

grishka2• 2 years ago

Truth is, E2EE isn't a "basic thing". It's an add-on feature that most people don't want. It is impossible to have E2EE that doesn't leak into the UX, and most people would rather have a streamlined UX than deal with key management. It is also much more complex to have robust E2EE in a group chat.

The thing that sets E2EE apart from HTTPS is that HTTPS requires nothing from the end user. It just works. And as a site owner, you just set it up once and forget about it.

Zambyte• 2 years ago

IRC does not encrypt messages, only (optionally) the client<->server connection. Without E2EE, you have no privacy against the server/operator, which is an easily targeted SPOF.

FWIW this point isn't relevant to the IRC vs Discord discussion, since Discord is also very not E2EE. That said, XMPP my preferred protocol that checks all of the boxes.

crtasm• 2 years ago

Nothing stopping a server also acting as a bouncer and storing messages: https://ergo.chat/about

timeon• 2 years ago

IRC does not encrypt messages

Wasn't SILC later used for this instead of IRC?

tannhaeuser• 2 years ago

There’s no lack of open chat protocols and federated services but those have mostly torpedoed themselves: by usability and discoverability problems, holier–than–you attitudes, and plain nerd attention wars. Such as XMPP (used a lot until around 2010 but easily dragged into the mud because XML and overengineering), Mastodon (saw a surge as twitter was faltering but then seemingly stopped to be everyone‘s darling as its limitations became obvious, among them Mastodon admins taking their audience hostage; also ActivityPub fans going around advertising it for each and everything when RSS is just fine for web sites, damaging news feeds alltogether in the process).

Where spamming, or the systematic exploitation of digital communication by the „ad industry“, was killing it in the past (Usenet, and arguably the web), today there‘s also the problem of being consumed by LLMs to push non-public messaging. Though I‘m not sure the latter is really a concern for many, as developers not only are giving away their code, but their entire activity log/issues and their solutions on github such that they can easily be digested and replaced by coding assistant LLMs, git being a distributed system in the first place.

Terr_• 2 years ago

among them Mastodon admins taking their audience hostage

I was excited first hearing all the "fediverse" stuff, but having to hand over control of your online identity to a particular node forever felt a little bit like "old boss, same as the new boss."

(Yes, I know some folks are working on the identity issue.)

high_na_euv2• 2 years ago

How did we let discord take over is a mystery to me, or rather a tragedy.

Orders of magnitude better product than anything competition had at the time?

doublerabbit• 2 years ago

Orders of magnitude better product than anything competition had at the time?

Nah, it just comes down to non-techy folk wanting to play/chat with their friends in a just-work configuration.

Mumble, TeamSpeak were always janky, needed a hosted server. IRC is multiplayer notepad.

Geeks care about E2E, and all that glory but these folks don't. And that's what Discord dishes; as did Y!M, MSN, ICQ, AIM back in the day.

All discord has done is replaced those above as GitHub has replaced SourceForge.

We didn't care if the message were encrypted or not back then. Why do we now?

EGreg2• 2 years ago

I keep writing about this tragedy, but few people care. Even on HN:

https://cointelegraph.com/news/how-a-web-that-lost-its-way-c...

and

https://community.qbix.com/t/the-debate-about-end-to-end-enc...

philipwhiuk2• 2 years ago

Own this piece of crypto history

I would argue that the web lost it's way as much with "web3" as with the platforms of web 2.

Intralexical3• 2 years ago

How did we let discord take over is a mystery to me, or rather a tragedy.

I think I'm reasonably technically competent, and I also dislike Discord's issues with privacy, data sovereignty, siloing information away from the open web, etc.

But you know what I think whenever I click a Matrix link, or IRC? I just don't want to deal with it. You get a list of apps you've never heard of, some of which may not be feature-complete, some with more than one version, some which are advertised using words like "GNOME", "Rust", "Qt5", and "C++" that have no meaning or relation to actually using them as a chat app, and all of which I guess are different and would need to be tried and learned separately. Then picking and clicking one tries to open an outside program which probably isn't installed and I don't want to install because I don't really know/care what it is. And if at that point, out of the dozen or so app options it showed you, you happened to choose one with a web version like Element, and you figure out you can click the "Continue in your browser" button out of the four or five unexplained buttons that pop up as a result ("XDG-Open", "Cancel", "FlatHub", "Download", and "Continue in Browser")— You get a static screen that shows just enough message history to not be useful, with a confusing UI you can't seem to interact with, hidden behind a login wall that still hasn't really explained what in the Internet tubes you're actually looking at.

E.G.: https://matrix.to/#/#invidious:matrix.org

If you try to Google "What is Matrix"— You get pages about math. So then you Google "What is Matrix chat". And all the results harp on using words like "open network", "decentralised", "protocol", "real-time communication", "open standard", "federated"— Which, again, may be technically interesting if you're into that, but doesn't actually have anything to do with how it directly serves the user as a chat app and how you can use it or sign up for it.

It takes way too many clicks, and you get bombarded with way too much information… To still not end up using the app, and in fact end up more confused than before about what a "Matrix" even is. Let's say you lose 15% of incoming users at each step. That rapidly scares off most of the mainstream, before they've even tried it. Maybe Matrix and Element are great. But it just seems like such an ordeal.

Compare that with Discord. You click a link. And then either you're already in the server, or it has a single text box and a single button you click to funnel you through making an account and joining the server.

It doesn't try to convince you to install a Desktop app until you're already fully using it in the web version. You get clear answers and reasons to use it if you search "What is Discord" or go to the website. It doesn't overwhelm you with options and then hound you with technical explainers that you didn't ask for.

IRC goes the other way in usability. People want voice chat, message history, different channels in the same "server", PM channels, etc.

/rant

Ecoste• 2 years ago

How did we let discord take over is a mystery to me, or rather a tragedy.

The fact that you're baffled why discord took over is exactly why it took over. You can't even acknowledge that the user experience is 10x better and it's suitable for a general non-technical audience.

mystified50162• 2 years ago

New quest available! Buy nitro for stickers! Buy nitro as a gift! New quest available! New quest available! Restart to update. New quest available! Look at the new emojis you could use with nitro! New update available! New update available again! Third update today! New quest available! Look at these profile decorations you could use with nitro! Boost this server! *NEW QUEST AVAILABLE*

throw161803392• 2 years ago

How did we let discord take over is a mystery to me, or rather a tragedy.

Anyone can set up or join a Discord server. If you give users the choice between a complex open platform and an easy proprietary solution, they will pick the latter every time.

RadiozRadioz• 2 years ago

There are loads of comments exactly like OP's, and they always make the mistake of mentioning IRC alongside XMPP and Matrix. Inevitably repliers can't help themselves and spend their replies discussing IRC's unsuitability for modern IM and how it's not federated. When IRC is mentioned, commenters ignore XMPP and Matrix and attack the point in terms of IRC. (Though this thread in particular is better than average).

Matrix and XMPP are the far more appropriate competitors for Discord, we need to steer the conversation toward them. I deliberately never mention IRC when I make these types of comments so people don't latch onto it and ignore everything else I said.

weaksauce• 2 years ago

because the voice chat function is so leaps and bounds better than anything out there and it was primarily used for that to game in real time. the text was an afterthought for gamers.

urza• 2 years ago

100% !! It's so sad :(

dean2432• 2 years ago

They make it literally impossible to delete your old messages. It's a privacy nightmare and I wonder why the EU hasn't stepped in.

Xen92• 2 years ago

In Discord culture, indeed, users usually share a shit-ton of PII in "introduction" messages from images to specific hobbies to medical information (EG "support" communities).

The problem from GDPR perspective is that Discoed makes it impossible to delete those, since once thet detect your interest in trying to delete any of your accounts' data, they will try to get to "anonymisize" it. Then at least publicly your username isdisconnected from thos messages, but they can still be traced back to specific persons. Now if this also is done server side, then they would be in a situation where you'd either have to go through ton of messages or to bulk delete past messages of all to enforce the GDPR demands of an user wanting their PII deleted.

EU Parliament is not a real Parliament in the sense that ONLY the Comission can propose new laws, and the elected parliament basically just votes on those. Who controls the Comission if not the people? The US State Department. Newsguard and non-Musk US bigtechs including Discord are in the same poli-financial bed of the establishment here. And they are full of previous state department workers.*

Unless there is public outrage, the EU-level bodies at least will probably be owned. But Public opinion is controlled by the cyberpunk establishment that trains their LLMs & targets their campaign ads using that illegal Discord data to get political advantage.

You in my view ought to "worry" about the fact that it's possible there will sooner or later no longer be escape from a permanent establishment, Orwell-style. Goes along with the theme that "cybersecurity" is the United States government level has been "war against hate speech" for years, and of course "hate speech" meaning "censorship of internal and external enemy speech."

Budd Dwyers if I recall correctly shot himself in TV after writing to Biden (???) that under some conditions (that became true), the Department of Justice should have "Justice" removed from its name.

---

Most of this I hold only at 50+% confidence of being broadly correct. Take with lots of salt.

r3d0c• 2 years ago

incoherant babbling

Intralexical3• 2 years ago

I do think there is a balance to be struck, because directed communication means the recipients of old messages are also stakeholders, such that maintaining a consistent record by default is a fundamental part of the "service" they offer. The message contents are different from e.g. secretly hoovering up click patterns. Matrix had some thoughts when they faced the same questions:

  The key question boils down to whether Matrix should be considered more like email (where people would be horrified if senders could erase their messages from your mail spool), or should it be considered more like Facebook (where people would be horrified if their posts were visible anywhere after they avail themselves of their right to erasure).

  Solving this requires making a judgement call, which we've approached from two directions: firstly, considering what the spirit of the GDPR is actually trying to achieve…

https://matrix.org/blog/2018/05/08/gdpr-compliance-in-matrix...

intelVISA• 2 years ago

Given the sheer size and extent of the user data collected and processed one imagines the EU is working on a big case... quietly.

dorlaor• 2 years ago

Some additional nuggets by ScyllaDB co-founder: - Discord couldn't complete repair with Cassandra. Not the case with Scylla - Scylla has a lot in common with Cassandra, from a good reason, like the LSM tree, compaction etc. However, Scylla has a unique CPU&IO schedulers which allows us to prioritize the queries over compaction, and defer compaction to the half milisecond where we have enough idle bandwidth. We have plenty of articles about it - Scylla has a new (1.5 years) tombstone_gc=repair - a much safer mode - Scylla's new architecture of Raft and tablets was recently launched and is the next big thing for our users. Watch the cool youtube video of those tablet load balancing

gigatexal• 2 years ago

What a fun write up and a huge confidence building post for me in ScyllaDB.

yas_hmaheshwari• 2 years ago

Does this article imply that don't use Cassandra. Use ScyllaDB when you think you want Cassandra

znpy2• 2 years ago

Interesting read on one had, a bit disappointing on the other: when the solution is just "we moved to this other product" it smells of lack of serious and rigorous investigation.

Also, having worked with the JVM and with GC issues I don't buy the "GC problems" point: there are a number of improvements in recent JVM release, the main being ZGC (and generational ZGC in particular).

ZGC is great, I've personally witnessed sub-millisecond GC pauses (and i mean sub-millisecond stop-the-world pauses) on machines serving millions of requests per second. Garbage Collection is largely a solved problem in the industry as of today, thanks to ZGC.

Other than this, also comparing latencies for machines with 9TB disks rather than 4TB disks is a bit like comparing apples and oranges: we will never know if issues at the storage layer were affecting tail latencies. Were the node having, i don't know, filesystem fragmentation issues? Does the 9TB storage configuration deliver higher iops than the previous 4TB storage configuration? Is the same kind of hardware underneat (same disk type? same disk bus? or are we talking ssd vs nvme?).

As somebody that's been doing performance engineering for work, this piece is a bit appalling.

Glad to see they've solved their issue though!

ozgrakkurt• 2 years ago

GC is a problem, and it always will be at some level. You can improve it but that doesn’t mean it is not a problem. Memory allocation and management is a problem even in c/c++ problems if you want to optimize your program, there is no universe where gc is not a problem

dobin• 2 years ago

So the TL;DR is: Cassandra and ScyllaDB have bad performance when reading. So they put a cache in front.

jhgg9• 2 years ago

No cache. Just read coalescing. There is a big difference. Coalescing just ensures that while a query is executing if an identical query arrives, rather than sending the same query as an already executing query to the database it will wait for the existing query to complete and duplicate the result. If after this the same query arrives again, it will be issued against the database.

This means we don't have to deal with cache invalidation/consistency issues while also being able to handle thundering herds, for example a large server pinging @everyone and having a bunch of people click into the channel or launch their apps in response.

robmccoll• 2 years ago

Cassandra is essentially an append-mostly distributed fault-tolerant hash table. If you need specifically that with high write throughput, it's a good choice. I don't understand why people use it as a database. You run into it's limitations immediately and the pain of trying to use it like a database only gets worse with scale.

mianos• 2 years ago

All the Casandra documentation and web site say it is a database. You can't blame anyone from getting confused. In my experience, I have never seen a project that started to use it, continue to use it after a year or so it may take a year to run into its limitations before having to replace it, with a database, like Postgres.

LeifCarrotson• 2 years ago

FTA:

In Cassandra, reads are more expensive than writes.

This makes it insane as a message store for a chat server to me. It seems appropriate for a logging destination for a distributed system, one where you want lots of clients to dump data but most of the time you don't even need to audit the logs, so the number of reads for a given item is less than one. This is obviously not true for Discord messages.

Squeeeez• 2 years ago

Not too sure - I would have guessed that most of the messages are written once, read by the constant number of participants (say 1-100 or so) and then they disappear off the screen and are never accessed again, ever. Maybe a few people will scroll or search, or use some custom extension to load and export the history, but very rarely.

atombender2• 2 years ago

The sentence makes it sounds like Cassandra and Scylla are slow for writes, which isn't the case at all. It's just that writes require a bit less I/O. Reads are still very fast. If reads were slow, nobody would use Cassandra and Scylla for the purposes that they're being used for.

bofaGuy• 2 years ago

I’m lost at why a DB (Cassandra) with better write performance than read performance was ever selected for a messaging system. I feel like it’s obvious that a message will be read more than it is written (once).

SpikeMeister• 2 years ago

While it’s true that messages are read more, reading can be cached so not every read necessarily results in a DB call.

axelthegerman• 2 years ago

Which seems something they added recently but was not part of the original design of using Cassandra

remram• 2 years ago

The fact that it has better write speed than read speed doesn't mean that it has bad read speed. It just happens to have even better write speed.

It's like how I connect my phone to my home's cable connection to send a big file. It is better at downloading than uploading, but that doesn't mean it's not the best solution for uploading.

PaulHoule• 2 years ago

How is they just can’t shard the thing? Isn’t each Discord ‘server’ isolated from the others (can’t send a message from one to the other?) Why can’t they address trillions of messages by having thousands of shards that each handle billions?

hun3• 2 years ago

Last time I checked the Discord bot API, it had explicit provisions for sharding.

DylanSp• 2 years ago

The partition key included the channel ID, and they were still having problems with hot partitions even with that fine-grained sharing.

qntmfred2• 2 years ago

i usually start projects with postgres this days. i have reached the tens of millions of rows threshold without breaking a sweat, but is there any good reason postgres can't handle into the billions or trillions? any well known products at that scale that are known to use postgres?

mxscho• 2 years ago

Just the raw amount of data is not enough metrics to judge whether postgres is "enough". They seem to value horizontal scalability e.g. in terms of write throughput, which is easier to handle with something like their solution compared to postgres.

bastawhiz• 2 years ago

Postgres can pretty easily scale to billions or trillions of rows. It forces you to think carefully about how you query that data, though, and I think most beginners would find themselves in deep trouble jumping into the deep end.

qntmfred2• 2 years ago

most beginners would find themselves in deep trouble jumping into the deep end

probably true for any database platform. postgres probably easier for beginners than cassandra

postgresfan• 2 years ago

The problem with Postgres is that you have to read the doc (boring), sometimes read database books beyond chapter 2 (lmao nerds).

This filters out 99% of software "engineers".

So it's better to use KookaburaDB, version 0.2 just got released. It's written in Rust, and it's modern of course (whatever that fucking means: config written in YAML I guess? Complicated build and deployment?)

consteval• 2 years ago

Thank you prostgresfan, it's nice to see a completely unbiased source on database technologies.

Jokes aside, you're largely right. Postgres really does cover 99.9% of database usecases. But, I think Discord might still fall outside of that.

The problem is that Discord's scale pretty much requires heavy sharding. While you can make this word with Postgres, you can tell it was never designed out of the gate for this.

IMO, Discord isn't even taking it far enough. Working under the assumption every server is its own isolated pocket, I see no reason not to have 1 database (or database-like thing) per server. Then it's truly a distributed system, which matches Discord's business use cases. I often find matching business use cases to technology like this can greatly simply architecture and reduce friction.

airocker2• 2 years ago

Just wondering if anyone considered using Postgres or another relational db. I understand it won’t do multi master replication as well but it is much more stable and predictable if you give it right amount of traffic. I guess the team had to do that part anyways for ScyllaDB

crop_rotation2• 2 years ago

I don't think anyone runs Postgress at that scale (unless very specialized sharding setup). Given the choice between using ScyllaDB like everyone else and using Postgres in a super specialized best in the world setup, the choice becomes clear. Also keep in mind that Discord is not a huge super profitable company, so for them to develop something like vitess for Postgress would not make sense. For a small company with huge data like discord, using existing data solutions makes a lot more sense.

airocker2• 2 years ago

They could use vitess, citus or alloydb. They could use read replicas for read operations and single master in a shard for write. They would get many SQL features (upgrades, referential integrity etc) for free. It would allow them to extend their business logic considerably.

crakhamster01• 2 years ago

Interesting technical read, but I appreciated the lighthearted jokes/comments the author threw in as well. Felt like they struck the right balance - nice work!

molszanski• 2 years ago

In an afternoon, we extended our data service library to perform large-scale data migrations. It reads token ranges from a database, checkpoints them locally via SQLite, and then firehoses them into ScyllaDB. We hook up our new and improved migrator and get a new estimate: nine days!

How many machines this migrator was running on? One? :D Sounds absurdly amazing!

How Discord stores trillions of messages (2023)

Related Stories