How Discord Stores Billions of Messages (2017)

https://blog.discord.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7

by ibraheemdev • 5 years ago

696 362 5 years ago

blog.discord.com

tester7566• 5 years ago

Discord had like $300M invested and they created unparalleled piece of software that ate whole market, damn.

One of the most impressive softwares that I've seen and use after years of using ventrilo/mumble/teamspeak.

dvt• 5 years ago

ventrilo/mumble/teamspeak

To be fair, Mumble is FOSS, and Ventrilo and Teamspeak have literally not iterated since 2005. Discord is pretty mediocre software (remember when they accidentally allowed iframe XSS RCE attacks? A very amateurish mistake), but the incumbents were an absolute dumpster fire.

the_duke• 5 years ago

and Ventrilo and Teamspeak have literally not iterated since 2005

True, but to be fair: the next iteration of Teamspeak will be based on the Matrix protocol, which is quite a big iteration. See https://news.ycombinator.com/item?id=25743874 .

Scaless• 5 years ago

For Mumble in particular, the devs had their heads in the clouds for so long that it is no surprise that it is no longer relevant.

If you had a mic that had issues in any way (buzzing, volume, balance), "The Wizard" and "AGC" were supposed to fix it for you. Do not fret little one, for you do not need nor want to manually fiddle with settings, The Wizard will make everything right [1]!

The pivotal feature that was the reason so many people I know stopped using it is the ability to change the volume of an individual person [2]. It has been a requested feature since the beginning of time, yet it took until 2016 to implement in dev branch and didn't actually make it into a release version until 2020! Too little, too late.

[1] https://web.archive.org/web/20200223143654/https://wiki.mumb...

[2] https://github.com/mumble-voip/mumble/issues/1156

yelnatz• 5 years ago

Well it solved a lot of pain points with the target market.

I remember my friends and I kept bickering who would pay for this month's bill for the vent/mumble servers. That kept on for years until I had enough and hosted my own in a droplet in digital ocean. None of my friends knew how to do that since they're not very technical.

Discord you just had to click a couple buttons and its free.

m4rtink• 5 years ago

It works nice and is free, but for how long?

How long can they keep paying for that bandwidth and message data storage while keeping the thing essentially free?

nnncdrj• 5 years ago

I don't get it, you can run mumble on any random Linux box in your house, you don't need to pay to have it hosted somewhere. Works find running on any box on your desk.

Discord makes you the product. It's gratis in exchange for letting them spy on you. If you don't know why that's bad...

caymanjim2• 5 years ago

It's free for the same reason everything is free these days. VC funds anything that will attract a lot of users to mine data from so they can sell the data. Discord didn't do anything that was groundbreaking or even solve a problem that had no solution; they just came along during a time when investors are willing to fund a company operating at a loss for a decade until FANG buys them.

Discord's a pretty good product, and they've got the engineers and money to get better, but the only reason they won is because of timing. Same for Slack; there were identical products to Slack that tried for decades to gain traction, but they weren't free, because that business model didn't exist at the time.

sascha_sl4• 5 years ago

Discord's smart move was emulating the concept of servers (including all the teenage drama coming from having administrators and moderators more interested in (ab)using their power than community building) while making them accessible to anyone without technical knowledge.

But it's important to remember that Discord is not that. Discord holds all your data, in luxurious detail, with no option to delete. They go as far as ignoring GDPR when people ask for their messages to be deleted. "Deleting" your account will not even anonymize your ID, it unsets your avatar, renames you, kicks you from all guilds and disables logging in. That's it. And if they ban you there is no place to move on to.

tester7566• 5 years ago

and you aren't going to get DDoSed if you join server of not the nicest people :)

kccqzy• 5 years ago

I've also noticed that in a lot of tech-related social circles people are increasingly choosing Discord over Slack. That's a trend I totally didn't expect: at least until a few years ago it was clear that Discord was for gamers and Slack was for work and everyone else. That changed quickly. Impressive indeed!

Nadya• 5 years ago

At my work we use Discord to have virtual "desks" (really just audio channels) so people can drop by and chat while you are at your desk. If you're busy or don't wish to be disturbed you can 'lock' your desk to prevent people from joining it (it limits the room size to 1, aka just you).

It really has helped the social factor of moving nearly everyone in the office to remote working. Every department that has adopted the "virtual office" Discord setup loves it over Slack and basically never uses Slack anymore. It's way less awkward to call people, it's easier to not incidentally disturb them when they're busy, during breaks/lunch you can go to the "breakroom" and hang out and chat with everyone else. And it was all very easy to setup and with the Discord server template stuff we can even clone it for each department with very minimal work (renaming channels to that departments' people).

vxNsr• 5 years ago

The free discord tier is better than the free slack tier. That’s honestly why 90% of ppl use discord over slack.

Also paid discord is 100x cheaper than paid slack, for non-corporate entities. You can get top tier discord for like $100/m while slack price goes up with each user. Not to mention that discord allows users to easily assist in upgrading your server while slack doesn’t have that functionality at all.

Xavdidtheshadow• 5 years ago

I think which fits your community is really dependent on your use case.

Discord has great tools around moderation and membership tiers; it's designed for users you don't trust.

Slack is much more for a community where everyone knows each other (or at least trusts each other a bit, like you'd trust a coworker).

craftinator• 5 years ago

it was clear that Discord was for gamers and Slack was for work

One of the best hammers I own is a screwdriver.

nonbirithm• 5 years ago

They also pivoted their marketing message away from being "for gamers" and towards anyone who wanted "a place to hang out," like developer groups or high schoolers.

mciancia• 5 years ago

Well, slack is extremely expensive IMO, so I'm not surprised

atom_arranger3• 5 years ago

Slack uses a language that is like markdown, but is not markdown, which is annoying for engineers.

Slack does not support syntax highlighting of code blocks.

Discord uses proper markdown and supports syntax highlighting.

These are two things that make me think Discord is better specifically for engineers, aside from it just being generally way better.

munchbunny• 5 years ago

Discord hasn't gotten bloated yet the way Slack has, which makes it much more pleasant when all you actually want to do is chat and maybe hang out on voice and sometimes with a screen broadcast.

Once you're in an enterprise space, Slack's features become actually useful.

RHSeeger• 5 years ago

Honestly, I find discord super frustrating. Can't have multiple chats open at the same time, can't close the right rail, etc. It's UX is subpar in almost every way that matters to me. I use it because _everyone_ uses it, not because I want to.

jonshariat• 5 years ago

Ditto, feels fast and has a lot of features but to me the UX is very confusing.

herodoturtle• 5 years ago

I second this.

Find the UI very confusing. Perhaps I'm just old; but damn, my intuition in using discord's interface constantly lets me down.

derefr3• 5 years ago

Operyl• 5 years ago

Going to be the dissenting voice here and say I find the UI usable, and have no major gripes. The Member List can be hidden by the user icon to the left of the search field, and there's a convenient "Compact mode" to make things more dense information wise.

Kaze4048• 5 years ago

I can't think of many reasons Discord "ate the whole market" besides smart marketing, honestly. It does audio rooms incredibly well, but everything else (even their developer support team) is just terrible.

masklinn4• 5 years ago

Samesies.

It's not bad per se but there's plenty of crap in there.

The shortcuts situation is absolutely dreadful for one, I don't understand how gamers can cope with it:

* there are all of 5 actions you can bind to custom shortcuts

* discord defines dozens of built-in shortcuts you can not rebind or disable, if any of those conflicts with something you need you better hope the OS has a way of overriding it

Large chatrooms as well, the moderation tools seem rather limited, maybe it's better for administrators but as a user all you can do is block someone and you still have to see that they're posting comments. It' incredibly frustrating.

Then the linking and jumping to old message works half the time, maybe, search is absolute dogshit, and I've rarely seen a less reliable @-autocompletion, half the time I have to find old messages of the person I'm trying to ping before discord remembers they exist and lets me actually @ them.

And I don’t think support actually exists. You just post into the black hole that are tte support forum thing.

CoryAlexMartin• 5 years ago

You may enjoy Ripcord if you're not happy with Discord's UI. I've been using it for a few months, and it's made Discord enjoyable to use.

I do have to open the official client whenever I do voice calls though, because there's currently an issue that can cause incoming audio to sound terrible. But for text chat, it's great.

https://cancel.fm/ripcord/

Rd6n6• 5 years ago

In indie game dev, everyone’s on discord and so are your extremely-important potential players. Whether you enjoy discord or dislike it, you’re going to be on there. It has really strong networks effects

tsjq2• 5 years ago

discord is maha complicated UX. close to nightmare

mech422• 5 years ago

I'm no fan of discord or Slack...but the fact that companies are using slack, and the slack 'threading' is sooo bad, drive me bonkers...

jokoon• 5 years ago

The top problem I have with discord is that you cannot have a list of favorite channels.

I really prefer IRC.

zitterbewegung• 5 years ago

IMHO it is even better than Slack.

5faulker• 5 years ago

If that's 2017, imagine what it would be in 2021.

tschellenbach• 5 years ago

Better engineering than Slack, but not as good of a business

brightball• 5 years ago

Elixir ftw

mrdoops• 5 years ago

Want to build a real time chat app? Best build it on the BEAM.

PHGamer• 5 years ago

its just slack for gaming. the ui is ripped off as suck. it is better than ventrilo but its not like they are that much better, just they realized a good concept and took it.

Aeolun3• 5 years ago

I don’t know about your definitions, but 300M is a fuckton of money to me.

paulryanrogers• 5 years ago

TLDR MongoDB then Cassandra

typon• 5 years ago

Did they make the move to ScyllaDB, as mentioned in their "Future work" section?

misframer• 5 years ago

Their jobs pages (e.g. [0]) mention "ScyllaDB/Cassandra".

[0] Senior Site Reliability Engineer: https://discord.com/jobs/4004051002

Sikul4• 5 years ago

We've moved quite a few datasets from Cassandra to Scylla, but not messages. I think we're planning to make a blog post about our experience with Scylla at some point.

yeswecatan3• 5 years ago

Still a good read if you're curious about Cassandra

gregoriol• 5 years ago

2017 is pretty antique though now: the scaling and the ecosystem change fast

PeterCorless21• 5 years ago

...then [post this blog] Scylla.

https://www.scylladb.com/press-release/discord-chooses-scyll...

jamesdwilson4• 5 years ago

With privacy concerns, companies should be shamed for storing billions of messages.

losteric• 5 years ago

That is a fair concern. However, as a customer - searching my message history is a desirable feature. I would rather see meaningful individual and corporate accountability for privacy breaches. The threat of jail and/or 100MM's in fines should motivate better data handling.

jamesdwilson4• 5 years ago

We've seen this happen over and over again:

- a company amasses a large trove of sensitive information

- it is exposed to adversaries or political enemies

- the information is used against the people

colesantiago3• 5 years ago

With privacy concerns, companies should be shamed for storing billions of messages.

should we shame ycombinator for storing the messages, accounts and comments on hacker news then?

I am still unable to delete my account here even though the CCPA and the GDPR exists. But here we are.

mishafb• 5 years ago

Your comments are a public thing you choose to publish to the world, but many (most?) discord messages are not. You'd want them to get deleted.

antoinealb• 5 years ago

Honest question: could it be argued that HN does not store any PII and therefore does not need to let you delete your account ?

jamesdwilson4• 5 years ago

yes. whataboutism.

NotAnOtter• 5 years ago

This is like a masterclass in how to answer system design questions. Maybe a bit verbose. They cover requirements, how to answer those requirements, relevant tech for the problem, implementation, and techniques for maintenance

vortico3• 5 years ago

I'm curious of the 2021 measure of total disk space that Discord consumes. Servers that I'm in share images every few minutes, which must add up pretty quick.

latchkey• 5 years ago

Unique images or just copied from elsewhere?

objektif3• 5 years ago

How does that matter? Do they keep track of all images on the internet?

snak2• 5 years ago

The Casssandra cluster mentioned (12 nodes, 1TB each) only handles text, as far as the article goes.

jhgg3• 5 years ago

We're well over 12 nodes in current year :P

SilverRed6• 5 years ago

I'd imagine that the images are not part of the main database and that they are in some kind of s3 like file storage system.

klaussilveira• 5 years ago

Did they ever make it to ScyllaDB?

jhgg3• 5 years ago

Yea. Everything but messages is on Scylla.

res0nat0r4• 5 years ago

It's unfortunate Discord is still requiring relocation to SFO, the product is amazing and it looks like some awesome engineering behind the scenes that would be fun to work on!

_ycxf• 5 years ago

Why not messages?

mikesun3• 5 years ago

Yes. And we're hiring storage engineers! https://discord.com/jobs/5411664002 https://discord.com/jobs/5426301002

auchtopus• 5 years ago

Does Discord offer undergraduate internships? I use the platform daily, enjoyed reading the thought process that went in to this design, and would love the opportunity to intern there.

If y'all are, I'd like to get in contact! My email is: anthony75025[at]gmail[dot]com, and my resume can be found at https://www.anthonyjiang.com/pages/resume.html

preetamjinka• 5 years ago

Looks like they migrated (at least partially) to Scylla: "Discord Chooses Scylla as Its Core Storage Layer" (2020)

https://www.scylladb.com/press-release/discord-chooses-scyll...

PeterCorless21• 5 years ago

Yes. They started experimenting with Scylla earlier, and made the switch in 2020. Here's more on their logic — they like "opinionated systems":

https://www.scylladb.com/2019/03/20/discord-on-the-joy-of-op...

tomnipotent• 5 years ago

Scylla is really impressive. The only complaint I've heard is about the shard-per-cpu approach, causing some issues when data has bad distribution.

umvi• 5 years ago

Discord is so good. I just can't imagine it can stay this good forever. My fear is that eventually it will be bought out and aggressively monetized.

alpb• 5 years ago

If you're paying for Discord every month, it's actually fairly expensive. A lot of the good features unlock once people start boosting servers with Nitros and those aren't cheap either. So I'd assume they aren't bleeding cash left and right on infra costs. They might actually breaking even on the infra costs at least.

helen___keller• 5 years ago

For years I've had a little bet going with friends about who ends up buying them to subsidize all this. My money was on amazon, because it could work so well with twitch + amazon prime.

dancemethis2• 5 years ago

It's not good. It's user hostile software.

knownjorbist• 5 years ago

Only out-of-touch tech elitists on HN think this.

throwaddzuzxd• 5 years ago

As someone who loves Discord, I'm curious why you'd think that, can you elaborate how it's user hostile?

kerblang• 5 years ago

I've used cassandra quite a bit and even I had to go back and figure out what this primary key means:

    ((channel_id, bucket), message_id)

The primary key consists of partition key + clustering columns, so this says that channel_id & bucket are the partition key, and message_id is the one and only clustering column (you can have more).

They also cite the most common cassandra mistake, which is not understanding that your partition key has to limit partition size to less than 300MB, and no surprise: They had to craft the "bucket" column as a function of message date-time because that's usually the only way to prevent a partition from eventually growing too large. Anyhow, this is incredibly important if you don't want to suffer a catastrophic failure months/years after you thought everything was good to go.

They didn't mention this part: Oh, I have to include all partition key columns in every query's "where" clause, so... I have to run as many queries as are needed for the time period of data I want to see, and stitch the results together... ugh... Yeah it's a little messy.

habibur• 5 years ago

Reading the article I was right now visiting Cassandra site to figure out what the catch is. Surely there should be a catch.

Well, here it is. The partitioning in manual upto the SQL level.

PretzelPirate• 5 years ago

The bigger catch is that when your partition grows too big and your nodes are hit by the OOMKiller, you have very few options other than create a new table and replay data, or use a cli tool to manually partition your data while the node is offline.

Using Cassandra tends to mean pushing costs to your developers instead of spending more money on storage resources, and your devs will almost certainly spend a ton of time fixing downed nodes.

Apple supplied some of the biggest contributors to Cassandra who were optimizing things like how to read data in a partition without fully reading the partition into memory to avoid the terrible GC cost. They put in a ton of engineering effort that probably could have been better spent elsewhere if they’d used a different database.

ChrisArchitect• 5 years ago

plenty of discussion when this was news:

https://news.ycombinator.com/item?id=13439725

dang• 5 years ago

Thanks! Macroexpanded:

How Discord Stores Billions of Messages Using Cassandra - https://news.ycombinator.com/item?id=13439725 - Jan 2017 (155 comments)

ryanianian• 5 years ago

TFA states:

we knew we were not going to use MongoDB sharding because it is complicated to use and not known for stability

But then goes on to describe using Cassandra and overcoming sharding and stability issues. I.e., changing the key, changing TTL knobs, adding anti-entropy sweepers, and considering switching to a different cassandra impl entirely.

Are these issues significantly harder to solve in MongoDB than Cassandra?

PeterCorless21• 5 years ago

At the time MongoDB's sharding story wasn't great. They've gotten better since, but still have a primary-replica set model that has a single point of failure/failover. Cassandra (and Scylla) are leaderless, peer-to-peer clustering. Any node can go offline and the cluster keeps humming. Cassandra shards per node. Scylla goes beyond that and shards per core.

Cassandra and Scylla also use hinted handoffs so if a node is unavailable temporarily (up to a few hours) you can store "hints" for it when it comes back online. Handy for short admin windows.

calmoo• 5 years ago

A MongoDB shard isn't necessarily a single-point-of-failure since a shard is usually deployed as a replica set. If a shard's primary node goes down, a secondary node in the replica set is elected as a primary and takes reads + writes. Similar to what you mentioned for Scylla - a node can go offline on a shard in a MongoDB cluster and it keeps humming.

trashcan5• 5 years ago

MongoDB has the equivalent of hinted handoffs. Changes are streamed to secondary nodes via the oplog, and the secondary just resumes where it was once it is back online. There is a limit to how long it can be offline (based on the size of the oplog), but that is the same limitation as hinted handoffs.

spmurrayzzz• 5 years ago

Its hard to say because they're not explicit about this but, despite being a decade-long Mongo apologist myself, I'd totally believe that they liked the linear scale story for Cassandra more from an infrastructure/config perspective.

Increasing top-end write throughput or replication in Cassandra is just adding more nodes, where in Mongo its not just adding nodes, its adding replica sets (which consist of 3 or more nodes). So there's a few more layers of complexity to that story. You need more replica sets to increase write throughput and need more nodes in replica sets to increase replication.

Im hand waving some details here, but I've worked with both platforms can definitely understand the choice at least from a pure infra lens.

jgilias• 5 years ago

Can anyone share experiences with using Discord as a communications tool in a workplace? We're currently on Google Chat because it comes with the package that we pay for anyway, but it's pretty lame. So from time to time we consider jumping to Slack. But then, why not Discord?

bhouser2• 5 years ago

Discord would be awesome for work ... I want to use it so bad. But the terms of service are completely unpalatable. Discord basically gets a perpetual license to anything you post.

bhouser2• 5 years ago

The reason for Discord over Slack, BTW is voice & video. Otherwise Slack is working great.

SilverRed6• 5 years ago

Isn't this the same for literally every service you use since they need a license to use it just to display it in app?

kroltan2• 5 years ago

We used Discord for a while for our team at work for over a year. Stopped using because company policy changed (we had no centralized chat program, some teams were on Skype, some on WhatsApp, then Slack was instituted). Context: 10-person, mostly technical, project-oriented, game development team.

---

It was a joy to use, we created channels left right and center and knew everyone needed would be in them thanks to the centralized "role-based" permission system. (We would create project-specific channels and an accompanying role, or client-specific roles for the few high-throughput clients that had lots of small projects)

At the time it did not have threading, which was one of the biggest pain points on the text-chat front.

---

The voice chat is very good, and having dedicated voice channels means you can emulate meeting rooms or desks and have people join as desired/needed. You could be working and idle at "kroltan's desk" voice channel, but even if you weren't, joining one is trivial (a single click, can be done independently by many people) compared to Slack (find the call button somewhere different each time because they redesign the UI every week, then wait for your peer to join the call).

Screen sharing is 720p on the free plan, so for meetings, it was hard to read documents, requiring zooming and whatnot. At the time there was also no setting to optimize for framerate or definition, so even 720p felt closer to 480p. Nowadays you can lower the framerate and also select the desired optimization, so you can ask Discord to optimize the stream for quality which is much better for documents, even in 720p.

---

The client is also much more responsive than other Electron-based chat programs, especially with big workspaces with close to a hundred channels (yes, for a 10-person team, we sure type a lot), search is basically instant and has very useful filters, mentioning roles is great and the notification settings are fine-grained enough to please everyone.

onkoe• 5 years ago

It's everything you could want, but this a lot of asterisks. A lot of things are limited (check other comment), but most importantly, their policies force you to hand over any text you write on their platform. But frankly, I'm not sure if there's anything near a perfect solution. A lot of companies use a big clump of services including both self-hosted and "rented". I hope to one day see a better, more comprehensive solution in line with the future of online work.

Ansil849• 5 years ago

Nothing about any sort of encryption.

imdsm• 5 years ago

That's an interesting point too. They talk about not being a blob store, not wanting the serialisation cycle to hamper performance but makes you wonder how exactly they're storing the data. I'd guess it's not encrypted at all.

ETA: Going back to the original thread, the whole question of encryption seems to be dodged and that usually means the answer isn't the one people are looking for: https://news.ycombinator.com/item?id=13440921

PeterCorless21• 5 years ago

Scylla, Discord's replacement for Cassandra, supports both encryption in transit (server-to-server within the cluster; client-to-server) and encyption at rest for stored data.

More on the latter here:

https://docs.scylladb.com/operating-scylla/security/encrypti...

mikesun3• 5 years ago

We're actively hiring for our storage infrastructure team (SF or remote) ! If this sounds interesting, check us out!

https://discord.com/jobs/5411664002

https://discord.com/jobs/5426301002

c7DJTLrn2• 5 years ago

They wouldn't need to store so many if they actually let people delete their messages on account deletion. Instead, they ban many people who attempt to do so via automated scripts.

anigbrowl• 5 years ago

I don't think they delete anything really. I've retrieved stuff from servers that were ostensibly deleted over a year ago.

c7DJTLrn2• 5 years ago

Well, data is their greatest asset I'd assume. They don't want that precious user data going anywhere.

simonw2• 5 years ago

Deletion of data at scale is a really difficult technical problem, unfortunately.

I'm not saying they shouldn't do that though - especially given regulations like GDPR. Designing systems for deletion is important! But it's also really hard, especially if you didn't design for it from the start.

There's also no way the tiny fraction of users who want to delete their data would make up a significant enough proportion of the messages that it would impact their scaling strategy.

coldblues• 5 years ago

One of the big reasons I refuse to use Discord. Deleting your messages is a right every user should have. Whether it be individually or in bulk. The way it's done now just makes it more susceptible for users to be open to malicious attacks. Whether someone archives your content before you delete it, that's not of importance, that can happen on any internet medium.

dave_sullivan• 5 years ago

Maybe I don't know enough about databases, but could you really not have done this with postgres?

simonw2• 5 years ago

I'm very much in the "use PostgreSQL unless you have absolutely proven to yourself that it won't work for your project" camp but in this case it really does look like moving to Cassandra was a good choice.

NoSQL scalable stores like Cassandra basically only work well if you have a very strong model of the queries that you will need to make.

In this case, that's exactly what they had: they knew what their read/write patterns looked like and they knew that they would be growing at hundreds of millions of rows per month, so easy horizontal scalability was a hard requirement.

The biggest weakness of classical relational databases like PostgreSQL come when you have a super high volumes of inserts (as opposed to updates) which will continue to grow your database over time, and you need to keep all of that data accessible for real-time queries.

They might have been able to achieve something like this using a PostgreSQL extension such as Citus, but it really does look like what they are doing fits Cassandra's sweet spot.

chillacy3• 5 years ago

Afaik you could up until the data exceeds your capacity to fit in one machine, at which point you have to figure out how to split your data up in a way which lets you preserve all the strengths of sql (strong consistency). At that point you run into a lot of complexity with managing your shards.

Quarrelsome• 5 years ago

do they ever explain what their "anti-entropy" processes are for and what they do?

PeterCorless21• 5 years ago

There are a few:

1. Hinted Handoffs - if a node has a transient failure, the other nodes store up messages, like your buddy might take notes in class if you had to go to the bathroom. They'd pass you those notes when you got back. "Here's what you missed." When the node comes back online it processes all new operations and works through its backlog of hinted handoffs to get caught up. Because of the backlog it creates, hinted handoffs are only stacked up for a few hours. If the node never comes back up, or comes back after that window...

2. Repairs - in an eventually-consistent database you might miss an update or two over time. Or maybe you're a replacement node that has to fill in for a failed node. The replacement will get streamed data from the other replicas to get it started, or you might restore sstables from a backup, but then you should run a repair job to make sure all your replicas are properly in sync.

(That's my understanding. Let me know if that sounds correct from the hands-on experts.)

andrewstuart• 5 years ago

I'd first reach for Postgres to do this. Anyone have any idea how Postgres would stack up in a similar challenge?

swman2• 5 years ago

At discord scale (global), I'd assume keeping things in sync even with R/W + global replication needs would be quite difficult.

chillacy3• 5 years ago

Hmm... have not tried this myself, but just brainstorming. You could shard based on discord server or chat room, which would give "read before write" consistency since writes can lock, but then you'd have to manage shards to account for varying loads like servers/rooms which grow rapidly, and deal with hot shards which might outgrow the capacity of a single db server.

Given that they said their requirements were "linear scalability, automatic failover, low maintenance, predictable performance", I don't think I'd go that route.

aeyes2• 5 years ago

I see several issues:

- No out of the box horizontal sharding, according to the post they had 4TB (compressed) data in the cluster in 2017. Looking at their growth I think it is safe to assume that today they would have >50TB which can't be done on a single node. You could use Citus but this is not exactly vanilla Postgres anymore. For such a simple data model wasting time implementing your own sharding solution and (more importantly) shard migration makes no sense.

- Discord is storing text data, in Postgres this will be stored in TOAST tables which has some drawbacks.

- Their workload is mostly inserts, almost no updates. Vacuum only operates on complete tables so you would wast I/O and CPU processing data which you don't even touch. You can partition tables but it's a manual process and you have to make compromises. In 2017, Postgres partitioning still had many performance drawbacks.

- No out of the box redundancy.

- Once your data doesn't fit in memory, Postgres performance becomes unpredictable.

Personally I would have chosen ElasticSearch for this project.

emptysea2• 5 years ago

Why would the text data be stored in TOAST?

My understanding was PG only uses TOAST when the data is too large to fit in the row, and since PG compresses data before inserting wouldn't user messages be fine?

theptip• 5 years ago

Do you have any case studies for ElasticSearch that you can recommend, in projects similar to this? Would be very interested in seeing what that option would look like.

imadeamistake5• 5 years ago

You'd be screwed if the DB went down, for one.

secondcoming• 5 years ago

They should check out Scylla if they want even faster queries and - depending on their workload - fewer instances.

Edit: It seems they have moved to Scylla

kaladin-jasnah• 5 years ago

They already did—https://news.ycombinator.com/item?id=28293097, main article: https://www.scylladb.com/press-release/discord-chooses-scyll...

josephd79• 5 years ago

oh great, here come the mongodb haters.

I do use discord for a few groups, too bad they will not allow 3rd party clients because a discord terminal app similar to irssi would be awesome

PeterCorless21• 5 years ago

MongoDB is great for developers. Very facile to get started. However, it tends to fall over when it hits scale — which could be in total data set size (like, >TB scale), transaction scale (>100k ops) or in low latencies (submillisecond to single-digit millisecond).

In any of those domains, if you are trying to solve your problem with MongoDB you are in for a world of hurt.

That's generally when people start looking at other options. Whether an in-memory system for pure speed, or a horizontally scalable system for raw size or throughput.

legerdemain3• 5 years ago

We took a big bet on Cassandra, and then on an opinionated wrapper around Cassandra at $PASTJOB. The use case was a text search engine for syslog-type stuff.

The product we built using Cassandra was widely known as our buggiest and least maintainable, and it died a merciful death after several years of being inflicted on customers.

We didn't have a good handle on the exact perf implications of different values of read/write replication. Writing product code to handle a range of eventual consistency scenarios is challenging. The memory consumption and duration of compactions and column/node repair jobs is hard to model and accommodate. It's hard to tell what the cluster is doing at any given moment. Our experience with support plans from Datastax was also pretty dismal.

Maybe the situation has changed since 2016. In my experience with several employers since then, it seems like every enterprise architect fell in love with Cassandra around 2014-2015 and then had a long, painful, protracted breakup.

yukinon• 5 years ago

If you had to do it again, what would you choose instead?

legerdemain3• 5 years ago

These days? Redshift or B̶i̶g̶Q̶u̶e̶r̶y̶ BigTable (thanks for correcting my think-o!). Back then? Maybe HBase.

trashcan5• 5 years ago

I've used Cassandra at two companies, and had the exact same experience as you at the first company. At a much bigger company that had some very, very highly paid Cassandra DBAs it was actually a relatively smooth experience.

objektif3• 5 years ago

How much is very very high pay?

PeterCorless21• 5 years ago

We just benchmarked Cassandra 4.0, which is brand-spanking new.

The good news: C4.0 is a far better performing database than C3.11. The new GCs definitely get rid of the long tail latency nightmares:

https://www.scylladb.com/2021/08/19/cassandra-4-0-vs-cassand...

However, we also compared it to Scylla's latest release, and though C4 is better*, you can still find other CQL-compatible databases that outperform it. Especially around compactions and topology changes:

https://www.scylladb.com/2021/08/24/apache-cassandra-4-0-vs-...

Just published these numbers today.

imadeamistake5• 5 years ago

I like Scylla. I've been working with it for a while now, and it's a good alternative for transactional loads. It's a hell of a lot faster than Cassandra, and much much much much cheaper than DynamoDB. Cassandra has always felt like improvements came in fits and starts. I work at a Fortune 50 company, and Amazon quoted us ~$2-3 million a year to run our load on Dynamo (we were paying $350k/year for Aurora). With Scylla, we're looking at > $100k to get an order of magnitude better performance and a nice stable system that doesn't wake me up in the middle of the fucking night.

I was following Scylla since the beginning (only because I like their mascot), and it's actually sort of interesting to see what's going on with the company. I've spent the past few years designing things where transactional systems backed by Cassandra. This is the first time I've been able to use Scylla on someone else's dime, though. The unpleasantly big company I'm at right now is looking to replace a bunch of infrastructure with ScyllaDB (Couchbase, Cassandra, Elasticsearch, DynamoDB). It's catching on for sure, but it still doesn't return any results when I search Dice. It looks like Discord is hiring, though...

throwdbaaway3• 5 years ago

Maybe the situation has changed since 2016. In my experience with several employers since then, it seems like every enterprise architect fell in love with Cassandra around 2014-2015 and then had a long, painful, protracted breakup.

I think 2012-2014 was peak marketing from DataStax. There would be some new major feature with every new blog post, and it would mostly never work as expected. Between 2017 and now, things have settled down.

jitans• 5 years ago

I would have used CockroachDB, it has all the requirements listed and you don't need to know in advance the queries you will perform when deciding the database schema.

rnotaro• 5 years ago

On the day that this blog post was written, CockroachDB was only at beta-20170112 and didn't even had a production release yet.

v1.0 was released on May 10, 2017 [1], so I doubt it was even on their mind when they started working on the project.

[1] https://www.cockroachlabs.com/docs/releases/index.html

PeterCorless21• 5 years ago

Amazing. The whole industry has come quite a ways since 2017!

PeterCorless21• 5 years ago

This would not work at scale for a company like Discord, with its volume of traffic. Cockroach, being consistency-oriented would quickly become transaction-bound. You want a database like a Cassandra or Scylla that is more performance/availability oriented. Otherwise you are going to see a lot of lag and latency in the Discord chat.

Cockroach is very, very good for a distributed SQL database. But it's still performance-limited in its very nature.

More here on the difference between NoSQL/NewSQL performance, using Scylla (a CQL-workalike) as a point of comparison:

https://www.scylladb.com/2021/01/21/cockroachdb-vs-scylla-be...

jjice2• 5 years ago

I love Discord's tech blog. There are a few corporate tech blogs that are just fantastic. Fly.io is another one that has great writing and interesting topics.

dragonfax• 5 years ago

KKV databases (Cassandra and DynamoDB are good examples) have a common problem with hotspots or "hot partitions". The most common mistake is to use a timestamp of any kind in the range (cluster) column. Then, whatever partition represents "today" or "this hour" ends up being the hot partition.

The article mentions hot partitions becomming a problem with max partition size, but they're also a problem with scalability. Say, if your writing a very high throughput of logs into the table (contrived example), then your bottlenecked by the rate at which you can write to one partition.

Adding the bucket id (say, the current day or hour), is a common solution, and solves the max partition size issue, but not the scalability issue of hot partitions.

geenat• 5 years ago

Cockroach DB recently addressed hotspots on sequence/timestamp workloads with: https://www.cockroachlabs.com/blog/hash-sharded-indexes-unlo...

Does what it says on the tin for the primary key.

That said, hotspots are 100% the reason why Cockroach encourages UUID primary keys. The disadvantage to UUID is you want sequential data, you then need a secondary index which you'll have to bucket anyway.

welder4• 5 years ago

Two horrible choices of databases... first they used MongoDB then they migrated to Cassandra. I've used tons of databases [1] in production and those two are the worst.

[1] I've used RethinkDB, Postgres, MongoDB, MySQL, Cassandra, CockroachDB, TimescaleDB, SSDB, and others

kin• 5 years ago

Cassandra has been successfully deployed at many companies. Would you care to provide some insight into your experience and why you consider it one of the worst?

welder4• 5 years ago

Here's a bug I ran into a few years ago:

https://datastax-oss.atlassian.net/browse/PYTHON-891

With all the issues I encountered using in prod, it gave the impression of an overly complicated key/value store.

winrid• 5 years ago

What would you have chosen?

welder4• 5 years ago

An in-house data service built with RocksDB/LevelDB. [1]

[1] https://rocksdb.org/

anaganisk• 5 years ago

Im not questioning your intellect, but each database has its own use case for usage. You might be expecting wrong things from Mongo or cassandra. Using gazillion databases for wrong use cases doesnt mean nothing. From your child comment, you cooked up an in house solution, it may be best suited for you. But for others it would be horrible too. If Cassandra is suited for Facebook, I think the problem is with you making a choice to use it for something its not suited for rather than the database itself.

welder4• 5 years ago

Yes, you're right.

lmilcin17• 5 years ago

Well... we have 3 node MongoDB cluster and are processing up to a million trades... per second. And a trade is way more complex than a chat message. Has tens to hundreds of fields, may require enriching with data from multiple external services and then requires to be stored, be searchable with unknown, arbitrary bitemporal queries and may need multiple downstream systems to be notified depending on a lot of factors when it is modified.

All this happens on the aforementioned MongoDB cluster and just two server nodes. And the two server nodes are really only for redundancy, a single node easily fits the load.

What I want to say is:

-- processing a hundred million simple transactions per day is nothing difficult on modern hardware.

-- modern servers have stupendous potential to process transactions which is 99.99% wasted by "modern" application stacks,

-- if you are willing to spend a little bit of learning effort, it is easily possible to run millions of non trivial transactions per second on a single server,

-- most databases (even as bad as MongoDB is) have a potential to handle much more load than people think they can. You just need to kind of understand how it works and what its strengths are and play into rather than against them.

And if you think we are running Rust on bare metal and some super large servers -- you would be wrong. It is a normal Java reactive application running on OpenJDK on an 8 core server with couple hundred GB of memory. And the last time I needed to look at the profiler was about a year ago.

kevinsundar2• 5 years ago

Thats not really what this article is about. Their problem wasn't throughput. What's the size of all the data in your MongoDB instance? And what's the latency in your reads?

In the big data world the "complexity" of the data doesn't really mean much. It's just bytes.

lmilcin17• 5 years ago

What's the size of all the data in your MongoDB instance?

3x12TB

In the big data world the "complexity" of the data doesn't really mean much.

Oh how wrong you are.

It is much easier to deal with data when the only thing you need to do is to just move it from A to B. Like "find who should see this message, make sure they see it".

It is much different when you have large, rich domain model that runs tens of thousands of business rules on incoming data and each entity can have very different processing depending on its state and the event that came.

I am writing whole applications just to data-mine our processing flow just to be able to understand a little bit of what is happening there.

At that traffic you can't even log anything for each of the transactions. You have to work indirectly through various metrics, etc.

dpedu• 5 years ago

Sounds interesting. Does this "we" have any writings about this?

lmilcin17• 5 years ago

I have a plan to write a book on how to write reactive applications like that. Mostly collection of observations, tips, tricks, patterns for reactive composition, some very MongoDB specific solutions, etc.

Not sure how many people would be interested. Reactor has quite steep learning curve but also very little literature on how to use for anything non-trivial.

The aim is not just enable good throughput, but also achieve this without compromising on clarity of implementation. Which is where I think reactive, and specifically ReactiveX/Reactor, shines.

jrockway• 5 years ago

I'm not "we" but I have some experience in this area.

Computers are fast, basically. ACID transactions can be slow (if they write to "the" disk before returning success), but just processing data is alarmingly speedy.

If you break down things into small operations and you aggregate by day, you can always have big numbers. The monitoring system that I wrote for Google Fiber ran on one machine and processed 40 billion log lines per day, with only a few seconds of latency from upload start -> dashboard/alert status updated. (We even wrote to Spanner once-per-upload to store state between uploads, and this didn't even register as an increase in load to them. Multiple hundred thousand globally-consistent transactional writes per minute without breaking a sweat. Good database!)

apenwarr wrote a pretty detailed look into the system here: https://apenwarr.ca/log/20190216 And like him, I miss having it every day.

sumnole• 5 years ago

Just curious, what was the rationale for choosing MongoDB?

lmilcin17• 5 years ago

Idk. I wasn't the one making the choice. I am just the one trying to make the best with what I have inherited.

eismcc• 5 years ago

Arctic [0] uses mongodb so perhaps that was inspiration?

[0] https://github.com/man-group/arctic

bob10293• 5 years ago

A million sounds impressive, but this is clearly not serialized throughput based on other comments here. Getting a million of anything to fast NVMe is trivial if there is no contention and you are a little clever with IO.

I have written experimental datastores that can hit in excess of 2 million writes per second on a samsung 980 pro. 1k object size, fully serialized throughput (~2 gigabytes/second, saturates the disk). I still struggle to find problem domains this kind of perf can't deal with.

If you just care about going fast, use 1 computer and batch everything before you try to put it to disk. Doesn't matter what fancy branding is on it. Just need to play by some basic rules.

Primary advantage with 1 computer is that you can much more easily enforce a total global ordering of events (serialization) without resorting to round trip or PTP error bound delays.

t0mas882• 5 years ago

Trading is for multiple reasons ideal for this, one is that total global ordering is a key feature (and requirement) of the domain so this "1 fast big server" thing is good. It is also quite widely known that several of the big exchange operate this model, a single sequencer application and then using multicast to transmit the outcomes of what it sees.

The other thing that is helping a lot here compared to Discord: Trading is very neatly organized in trading days and shuts down for hours between each trading day. So you don't have the issue that Discord had where some channels have low message volumes and others have high, leading to having scattered data all over the place. You can naturally partition data by day and you know at query time which data you want to have.

giancarlostoro• 5 years ago

-- if you are willing to spend a little bit of learning effort, it is easily possible to run millions of non trivial transactions per second on a single server,

I got into programming through the Private Server (gaming) scene. You learn that the more you optimize and refactor your code to be more efficient, the more you can handle on less hardware, including embedded systems. So yeah, it's amazing how much is wasted. I'm kind of holding hope that things like Rust and Go focus on letting you get more out of less hardware.

naikrovek2• 5 years ago

Go and Rust won't help there, but Zig might.

tonymet• 5 years ago

Considering that a cpu can do 3 billion things a second , and a typical laptop can store 16 billion things in memory , it shouldn’t take more than 5 of these to handle “billions of messages” . I agree with you that modern frameworks are inefficient

adamnemecek2• 5 years ago

By 16 billion things you mean 16 billion bytes? If you are talking about physical memory, then no, you can't occupy the entire memory. If you are talking about virtual memory, then you can store more data.

ljw1001• 5 years ago

Curious as to how many days of data you have in your cluster. It seems like it could be ~1/2 billion records per day, 125 billion per year-ish. In a few years your 3 node Mongo cluster would be getting towards volumes I associate with a 'big data' kind of solution like BigTable.

thibauts• 5 years ago

This is my experience too. Millions of persisted and serialized TPS on regular NVMe with ms latency. Though it took a bit of effort to get to these numbers.

https://github.com/dataptive/styx

greenbeans19912• 5 years ago

Could somebody help me understand the reasoning behind

`timestamp = snowflake_id >> 22`

Thanks :)

karmanyaahm• 5 years ago

IIRC a snowflake ID's first few bits (or characters?) are by definition a unix timestamp.

greenbeans19912• 5 years ago

Ah it gets rid of the non-time related bits. Below is a description of the formatting

* id is composed of: * time - 41 bits (millisecond precision w/ a custom epoch gives us 69 years) * configured machine id - 10 bits - gives us up to 1024 machines * sequence number - 12 bits - rolls over every 4096 per machine (with protection to avoid rollover in the same ms)

thanks :)

Notanothertoo• 5 years ago

We use scylla for our IoT stream, bucket per day, with a date index for second resolution data. The current day is a hot spot of, but we throw that in redis. It's running one of the largest re insurance providers IoT deployments.

PeterCorless21• 5 years ago

That sounds like one of my favorite use cases, Meshify / MunichRE! :)

A little history lesson is in order:

https://www.scylladb.com/2019/02/01/meshify-and-scylla-an-in...

dancemethis2• 5 years ago

Well, very good for them and their "business partners". That data and metadata all for them to snoop into...

Can't wait for OpenFeint 2.0 to have its scandal lawsuit too.

tsjq2• 5 years ago

"Billions of messages"

that's a prime target for acquisition by big-tech / big-data companies

siculars• 5 years ago

Partition Key selection aside, if you want a better Cassandra, look at ScyllaDB. Much, much better engine, imho.

/disclaimer/ I used to work at Scylla.

tbarbugli• 5 years ago

If you can, use Scylla over Cassandra. The performance difference is tremendous in my experience and replacing can be trivial (easier if you start with Scylla on day 0)

anonymoushn4• 5 years ago

I am excited for the upcoming launch of full text search on discord.

How Discord Stores Billions of Messages (2017)

Related Stories