Scaling Mastodon in the face of an exodus

https://nora.codes/post/scaling-mastodon-in-the-face-of-an-exodus/

by NoraCodes • 4 years ago

213 95 4 years ago

Havoc• 4 years ago

Feels a bit like mastodon needs to come with load testing tools? If only to give smaller admins a quantifiable sense of what level their instance turns into a flaming ball

capableweb3• 4 years ago

Or Mastodon admins needs to publish a bit more numbers about what kind of instances they run on what kind of hardware, so people know what to expect at what stages.

NegativeK• 4 years ago

That's coming. The infosec.exchange admin has been tooting what's going on to some extent, but delayed until now the full documentation in favor of gaining headroom for the incoming growth.

WorldMaker• 4 years ago

Also, Mastodon needs a better handle on its "range" of scale: a lot of the core contributors of Mastodon get focused on the performance of the two massive "flagship" instances (mastodon.social/mastodon.online). Those have far different scaling needs than most of the smaller instances and sometimes the admin work needed for smaller instances gets overlooked because the "flagship" scale handles it just fine.

wmf• 4 years ago

Mastodon should probably throttle signups by default so new users can't pile on to servers.

sandyarmstrong2• 4 years ago

You understand there is no central authority for this, though, right? Many instances are growing, and account creation is a per-instance concept. The instance in the article is relatively small but it's impacted by activity coming from other, larger instances.

BitPirate• 4 years ago

I would rather encourage migration and let the admin provide links to similar instances.

egypturnash• 4 years ago

This has happened. Lots of servers, including the ones run by the Mastodon project, are now closed to new users. If you want an account on a Mastodon server then you pretty much need to set one up yourself, or know someone crazy enough to think joining the Holy Order Of Social Plumbers sounds like a good thing.

mdaniel4• 4 years ago

So this one is a fork, and infosec.exchange is a fork ( https://github.com/glitch-soc/mastodon ); are the mastodon committers opposed to incorporating those changes, or this fork situation is "as it should be" while these experiments run their course?

I'm all for experimentation, but since "the exodus" I've become acutely aware of how many instances are out there advertising their version number[1], so the first big rails (or mastodon itself) vuln is going to be some ooooonooooz

1: although in this case, it says a version but the "Source Code" link points to the wrong repo, so .. security by obscurity I guess?

huimang4• 4 years ago

I'm not sure about Glitch, but Hometown has several different features [1] that Mastodon may never opt to include as it changes the behavior a bit.

1: https://github.com/hometown-fork/hometown/wiki

sandyarmstrong2• 4 years ago

"By our best understanding, our major changes are not wanted by the Mastodon project, hence maintaining this fork instead of trying to commit the changes to Mastodon."

https://github.com/hometown-fork/hometown#readme

jdp23• 4 years ago

Mastodon lead developer has resisted including some of the glitch-soc improvements since 2017 -- like "local-only posts", which are valuable both from a privacy and anti-harassment (and are also in Hometown).

Kye• 4 years ago

I've gotten the impression Eugen has a very clear idea of what he wants Mastodon to be and isn't shy about rejecting features that he doesn't think serve that goal. Those forks are the model of good open source. Other people have strong ideas about how it should be, couldn't convince the main project's owner, and moved on to create their own version that suits their vision.

It's great. This is how it should be. And it's a huge improvement over open source projects that languish for years for lack of strong leadership with no one forking it because there's no one to say no to features. As long as the possibility exists, people are wary of forking. Remove the possibility, and people can feel confident doing their own take.

As for Mastodon, as far as I know all the major forks are maintained by experienced developers who incorporate security updates.

thwarted• 4 years ago

And federation is about the protocol and the interoperability, which is how software achieves stability and longevity (although neither is guaranteed). It's perfectly fine for multiple implementations, or forks of a single implementation, to bloom.

huimang4• 4 years ago

I'll use this opportunity to mention Pleroma [1], my preferred alternative to Mastodon & others. It's much more lightweight than Mastodon.

1: https://pleroma.social/

capableweb3• 4 years ago

That's neat. Do you happen to know if it's possible to migrate from Mastodon to Pleroma without having to recreate the account and setting up redirects? I already host my own Mastodon instance since a few years back, and don't want to do another redirect if I can help it.

huimang4• 4 years ago

To my knowledge you would have to create a new account and set up redirects, since it's a new server. Migrating is definitely a pain point in the fediverse. That's why I ended up going with a single-user instance that I selfhost.

mdaniel4• 4 years ago

My years of scar tissue with RabbitMQ have given me a healthy suspicion of OTP ( https://git.pleroma.social/pleroma/pleroma#otp-releases-reco... ), and I would never want to be responsible for keeping it alive and healthy. For comparison, while I'm sure people also have strong opinions about rails and sidekiq, at least there is a non-trivial body of existing blog and SO posts about it

huimang4• 4 years ago

That's fair. I selfhost Pleroma as a single-user instance, so I couldn't tell you about running it at scale. I went with it because it could run on a raspberry pi a few years back whereas mastodon was too resource-hungry.

pmarreck5• 4 years ago

I've been working on Elixir sites for years now and have never had an issue with OTP underneath, although none of them (yet) have been high-demand. The reliability and performance characteristics have been top-notch. Is it possible that with the mass influx of Elixir users into the Erlang space since the RabbitMQ days, OTP has made leaps and bounds of improvement?

throwaway894345• 4 years ago

I thought OTP was supposed to have a very strong operations/reliability story ("nine nines of reliability in telecoms" and all that)?

GordonS• 4 years ago

That's odd given OTP's focus on stability and reliability.

RabbitMQ has been my message bus of choice for over a decade. Configuration is a total PITA, but it's a one-time thing. Otherwise, it's been rock solid and I'll happily continue to use.

pmarreck5• 4 years ago

TL;DR: Mastodon’s Sidekiq deferred execution jobs are the limiting factor for scaling federated traffic in a single-server or small cluster deployment. Sidekiq performance scales poorly under a single process model, and can be limited by database performance in a deployment of the default Dockerized configuration.

Well, that in short explains why Pleroma/Elixir is a much better stack to build this in. You don't need Sidekiq (https://sipsandbits.com/2020/08/07/do-we-need-background-job...) and you aren't stuck with a GIL (Global Interpreter Lock).

sjustinas3• 4 years ago

Just to note, Pleroma also uses a job queue: Oban. https://hexdocs.pm/oban/Oban.html

krainboltgreene• 4 years ago

Well, that in short explains why Pleroma/Elixir is a much better stack to build this in

As someone who has extensive experience in both elixir and ruby, and mastodon itself, no this is doesn't.

faraaz98• 4 years ago

I like Elixir but as of the latest Ruby version, you can use ractors. They aren't bound by the gil

guywithahat• 4 years ago

I also prefer pleroma, plus they just have larger instances (like poa.st, which had more active users than mastodon.social before Elon bought twitter, although I'm unsure if that's the case now). Sites like Gab started off on Mastodon but found it was too hard to scale and moved off it.

sjustinas3• 4 years ago

Note also that recently a fork called Akkoma[1] sprung up. If one is looking to self-host, it might be worth looking into this vs Pleroma (I don't have a horse in this race, was just doing some research on different ActivityPub servers).

[1] https://coffee-and-dreams.uk/development/2022/06/24/akkoma.h...

mcguire• 4 years ago

How much memory/cpu does Pleroma need for a single user instance?

sjustinas3• 4 years ago

I spun up a test server of Pleroma recently. Following just a few people, 0 people following me AFAIK, I did join a few relays so my federated timeline is populated. It is currently showing 360MB RSS and the CPU load maxes out at ~50% of a single vCPU.

self• 4 years ago

I've run a single-user pleroma server since January 2019. I ran it on a digitalocean droplet with 2 GB RAM, and then an AWS instance (t3.small, also 2 GB) for the better part of two years; I recently switched to a t3.medium. Beam's current process size: 250MB resident, uses 5-7% CPU.

I discard remote posts after 120 days, and my database (using PostgreSQL 15, no RUM indexes) takes up just over 5.4 GB (as reported by `pg_database_size`). It's grown a lot faster in the past couple of weeks. I follow 245 accounts.

riffic• 4 years ago

More alternatives:

https://codeberg.org/fediverse/delightful-fediverse-apps

I'm partial to WordPress with the ActivityPub plugin and would highly recommend pushing for inclusion of core AP functionality into your content management system of choice (so everyone can just stand up and participate on their own domain name, like media/gov/your local school etc).

Mastodon itself is fun but the real value in the fedi is going to be diversity of interoperable (or even incompatible) implementations. There (on a long enough timeline) will be a successor to Masto's fedi domination.

adultSwim• 4 years ago

These technical solutions are great, but the biggest need is still organizational. We need infrastructure with transparent open governance, a la Debian.

Is anyone aware of an org that hosts an instance but is also putting work into the governance side?

gnramires• 4 years ago

From some non-exhaustive research, there are many: social.coop, chaos.social, social.tchncs.de, nerdica.de (friendica) -- seem to have governance. I believe, mastodon.xyz, fosstodon.org among others have transparent donations. (I believe mastodon.xyz uses the default source)

I agree that you should seek more open governance, at least transparent finances and active open source, (technically that's covered by AGPL), because otherwise the server could go at any time, and it violates the spirit of federation in the first place.

Here's a nice transparency report: https://hub.fosstodon.org/about/

squidbeak• 4 years ago

The most likely outcome here is that Twitter stabilises somehow, Mastodon increases its name recognition and even usage, but that its incoming migrants eventually skulk back to Twitter for its reach.

Ambolia2• 4 years ago

Twitter seems to have peaked on usage the last days, not decreased, despite the migrations.

pornel• 4 years ago

Twitter is very entertaining right now with everyone dunking on The Main Character and making cheap ($8) jokes.

But at the same time I think it's eroding Twitter's seriousness, and soon Elon's Twitter may be as uncool as Zuck's Facebook.

yborg• 4 years ago

I don't see it as either/or, choices are good, and if Twitter rights the ship before it becomes a hellscape I'm sure many people will continue to use it as a primary platform. Ideally, Fediverse will have established mind share so more people consider it as another channel for engagement, just as people have multple Twitter/Instagram/Tiktok presences.

alwillis2• 4 years ago

Probably not, unless someone (or a consortium) buys it from Elon out of bankruptcy, which Elon says could happen next year.

Or Twitter refugees migrate to Jack’s new Blue Sky service.

mdaniel4• 4 years ago

new hypothetical Blue Sky service

... since one cannot join a protocol spec. Although it's certainly possible that this Mastodon exodus will help Blue Sky (if|when) it does launch because NPR has already started airing stories about distributed social media

standardUser11• 4 years ago

I was extremely bummed about their sign-up process.

Mastodon doesn't have a snowflakes chance in hell of becoming the de facto Twitter replacement if they don't change that process to a normal, near-instant sign up process that users have come to expect from every other service.

Make the "choosing server" nonsense come after account creation. Ffs.

EDIT: Geez, I get it, there are a lot of Mastodon fanboys that have zero interest in the success of Mastodon as a Twitter replacement. Which is fine, since it never will be with it's current onboarding process.

LukeHoersten• 4 years ago

Email did alright with users choosing servers.

standardUser11• 4 years ago

Not sure what you are talking about. I can sign up for an email right now with a dozen different services and not one will make me complete a mysterious task like "choosing a server".

ProjectArcturis• 4 years ago

Email only took off among the non-nerd population when easy-to-use webmail sites like hotmail became available.

Ptchd• 4 years ago

too bad Mastodon requires an email to signup though

alwillis2• 4 years ago

In reality, the vast majority of email users don’t pick a server, they pick a service/brand.

Gmail, Outlook, iCloud… nobody picked a server.

And certainly there was no concern that if I pick the “wrong” one, I may not be able to receive email from one of the other big providers.

Oh, there certainly was no concern I might end on a email server with an ideology I don’t agree with.

kerblang• 4 years ago

There was a posting on HN yesterday about how original maintainers were kind of flooded & overwhelmed by Twitter refugees who didn't understand that Mastodon wasn't really meant to be the go-to Twitter clone.

VikingCoder8• 4 years ago

I go to Samsung to buy a "cell phone," and I thought it would be a normal, near-instant sign-up process that I've come to expect. But instead they're making me "Choose a carrier" or some nonsense. What is a Verizon, T-Mobile, AT&T, or uscellular, and why would I want one? Can't I choose one of those after I get my "cell phone number"?

standardUser11• 4 years ago

The pedantry is more than I can even believe. How is this genuine and not trolling?

weberer• 4 years ago

Make the "choosing server" nonsense come after account creation. Ffs.

Where would you like these account credentials to be stored? Imagination land?

standardUser11• 4 years ago

Users expect a company called "Mastodon" to store it, just like a company called "Twitter" stores their credentials for that site. I understand this is not how Mastodon works, so save your breath. The problem is almost everyone else on the planet does not know how Mastodon works and it will confuse and deter them.

yjftsjthsd-h• 4 years ago

Geez, I get it, there are a lot of Mastodon fanboys that have zero interest in the success of Mastodon as a Twitter replacement.

You're not getting downvoted because everyone who disagrees with you is a strawman (sorry, "fanboy"), you're getting downvoted because your premise is shaky and every time someone points out an existing ecosystem where federation is normal you claim it doesn't count.

pwinnski4• 4 years ago

The onboarding process is certainly inferior to that of twitter, but that's baked into the design of mastodon, and literally cannot change. It is definitely the single most challenging thing about mastodon, with the second probably being vocabulary choices made entirely by developers rather than anybody with UX experience.

On the other hand, as a result of this design, no number of oligarchs can run the mastodon network into the ground, so there are obviously trade-offs.

Many mastodon servers do, in fact, have a normal, near-instant sign-up process, although given the huge influx of users, many have switched to manually approve registrations in an effort to control load/costs and keep moderation manageable.

andix• 4 years ago

Shouldn’t you use a postgres connection pooler like pgbounce for such a scenario? AFAIK it can be switched to transaction mode, which allows to have way more connections from clients to pgbounce than available postgres connections.

dewey• 4 years ago

For those interested, the author of Sidekiq also gave some hints on Reddit where this link was posted: https://old.reddit.com/r/rails/comments/yse6no/_/ivz1clf

killingtime74• 4 years ago

I work on sharding at a bigcorp. Why not simply limit sign ups to a single Mastodon server which is then considered a shard in The overall fediverse. Stops one/some servers from becoming to powerful too

mdaniel4• 4 years ago

Well, according to that theory all servers are fungible, but having read over quite a few of the Mastodon /about pages over the past few days, they are for sure not. The audience and sometimes the rules vary wildly

Also, while yes federated etc etc, I for sure get a lot more enjoyment from the "Local Timeline" in my server than trying to keep up with the absolute firehose of "Federated Timeline" which is another reason to pick one where the local content would be interesting

OctopusLupid• 4 years ago

That is indeed usually what happens if the admin doesn't want to deal with a big userbase. I run an instance and I intend to close it off if it reaches about 100 monthly active users.

Scaling Mastodon in the face of an exodus

Related Stories