Matrix.org homeserver: Severely degraded performance due to database issues

https://status.matrix.org/incidents/r73l6ll51xns

by jeroenhd • 4 years ago

27 20 4 years ago

status.matrix.org

bfrog• 4 years ago

In the end matrix has the issue that federation seems less widely used than it maybe should be. It was a concern I had when looking at it, and when I spoke to the author of conduit (rust matrix server) i made the point that easy deployment should be a primary goal. The existing python server seems non trivial to deploy.

I think that’s still true, I’d you want a distributed setup like that it has to be as close to trivial to setup as possible. Email has shown that complex setups cause centralized solutions.

the_duke• 4 years ago

The Python server is pretty trivial to deploy.

Set up Postgres, tweak a hand full of settings in the config, that's pretty much it.

Has been running without issues for half a year.

Just a single person deployment though.

Arathorn4• 4 years ago

The existing python server seems non trivial to deploy.

This really isn't true (and hasn't been since Matrix exited beta in 2019). `apt-get install matrix-synapse` or `pip install matrix-synapse`, edit the config, and `synctl start` and off you go.

bodangly• 4 years ago

I launched my own matrix server for my company two years ago and I’ve never had to do a thing with it. It gets frequent use by ~10 people, so not huge, but it’s been quite easy to deploy and maintain.

DoItToMe81• 4 years ago

I've installed several. It's fairly simple. The hardest part of setup was reading the quotes for the config files, because there were very many and not all were written clearly back then.

Lack of federating is a real problem, though. I don't federate. It seems too easy for a malicious user to overrequest data and screw over your budget server, and I've heard of it leading to a lot of resource usage nightmares.

fsflover2• 4 years ago

Anyone can recommend a reliable alternative Matrix server?

IceWreck• 4 years ago

Dendrite, written in go. Its super easy to install, is fast and even runs on low specced systems

xyst2• 4 years ago

I loved using dendrite implementation but people should be aware if it’s limitations. Since it’s basically a rewrite of synapse in go, they are currently playing catch up.

All features of synapse are currently not there. For example, most media APIs are not working, like URL preview. Before 2 months ago, I feel like they were making great strides with the project. They were pushing out weekly updates.

Development has definitely slowed down, and it doesn’t help one of the key contributors/maintainers has left the company/project.

Hopefully the project gets a few dedicated maintainers and people to manage and fund it. It really is a great implementation but I ended up switching back to synapse.

Kab1r• 4 years ago

An alternative server instance or alternative implementation?

aesh2Xa1• 4 years ago

Do you mean an alternative server software or alternative homeserver instance?

On the latter case Mozilla runs one. There are tables of free ones on the Internet, and you can also host your own.

On the former it isn't clear that the matrix.org software was to blame at all.

fsflover2• 4 years ago

Thank you, this is what I was looking for.

Arathorn4• 4 years ago

I find it very surreal that the first performance degradation on the matrix.org homeserver in about 6 months warrants its own HN thread - I guess it's a testimony to how reliable people have grown to expect the server to be these days! Fwiw, this was very much a partial degradation over about 3 hours caused by a pathological database query, and most people didn't notice it. The biggest issue was that to-device messages were delayed, impacting new E2EE sessions.

Eitherway, apologies for the degradation. Obviously anyone on their own server won't have been affected anyway.

kop316• 4 years ago

most people didn't notice it

I am on a fair amount of channels on matrix.org and can confirm, if it wasn't for this news article I would have never known.

On a sidenote, thank you for the work you all do!

sschueller• 4 years ago

I have issues when I join large rooms with my tiny private instance. Does anyone know if there is some setting that can fix this or is this currently a flaw in the way matrix is federated? I don't need the history of the channel and I don't need stuff I missed when I wasn't present.

Arathorn4• 4 years ago

This is partially a flaw and partially a feature. It should be common sense that joining a massive room on a tiny server may overload the server: it has to talk to all other servers in the room (which is where most of the resources go), and keep track of the current state (ie key/value pairs such as who’s in the room).

It’s also a flaw in that this could be less heavy: it could be faster to join (which we’ve almost solved via Faster Joins - see https://matrix.org/blog/2022/12/25/the-matrix-holiday-update...). We could also speak to a subset of the servers in the room (at the expense of reducing the decentralisation of the room). We can also optimise state resolution further.

For now: if you want to participate in a room whose users are spread over 1000 servers, you’ll need a server beefy enough to talk to 1000 servers every time someone says something.

Matrix.org homeserver: Severely degraded performance due to database issues

Related Stories