Breaking the 100bps barrier with Matrix, meshsim and coap-proxy (2019)

https://matrix.org/blog/2019/03/12/breaking-the-100-bps-barrier-with-matrix-meshsim-coap-proxy/

129 30 5 years ago

Raineer• 5 years ago

This is so awesome. I'm thrilled to see work progressing at the other end of the bandwidth spectrum.

Arathorn6• 5 years ago

This work has progressed a lot since 2019, although the blog post here describes the original spurt of work on it. Nowadays Matrix has MSC3079 to formally spec low bandwidth transports: https://matrix.org/blog/2021/06/10/low-bandwidth-matrix-an-i... and we’ll be building it into clients & servers by default in future.

The final major unsolved problem is that the sync API doesn’t support pagination yet, so on a big account you can end up burning bandwidth (and cpu) syncing data for rooms or room members that the client doesn’t care about. This is the final piece of the puzzle, and the lowbandwidth team is working on it right now - and will probably be the most exciting and fundamental improvement in Matrix in years :)

hirako20003• 5 years ago

It's fascinating to see so much fundamental work taken on by the team at matrix.

One thing I still can't understand though, as a matrix user, I try to get on my circle onto it. But instead of hearing praises for the technological superiority of the network, all I get is hate and quick dump of its use.

I wondered why. And then realise their claim is justified. The market is dominated by centralised pseudo e2e encrypted services. And people stick to them for two reasons, network effect and the fact it just works. The official client is plagued with UX bugs. Push notifications works some of the time, sometimes the app would hang trying to open or access a channel, some message get stuck as the last entry with no means to delete them. It makes the network barely usable since clients can't keep up with whatever is going on at the network level.

I think technological advances shouldn't be made at the expense of the UX. Today's problem with matrix is adoption, and the slow adoption rate isn't due to not breaking the 100byte barrier. People would greatly benefit in getting an alternative car engine that doesn't burn polluting fuel, and matrix is delivering a propulsion engine to send us to Mars, at zero cost to consumers.

I still admire the project and its contributors, I just wish those hundreds of real issues on each of their github projects were being prioritised over cutting edge innovation.

foxfluff• 5 years ago

Not just UX.. one day Element just wouldn't connect anymore (and in a typical webapp fashion, no useful error messages). Nuking its config directory and starting from scratch fixed it (never needed to do that in 20 years of using irssi).

I've seen other issues like video previews getting their bottom half completely crippled. Can't make a video call just sharing the desktop with someone -- it demands a microphone. Can't disable read markers that are broadcast to everyone (bad for privacy imo). They were working on this when I looked into it but I've no idea if they've fixed it yet.

And yes I've seen UX bugs such as some UI setting getting stuck in a state when you try to toggle it in the settings.

I'm guessing these are more Element issues rather than Matrix issues (except for the read markers). But I get the vibe that we won't be seeing a client I like anytime soon; or it'll be a very minimalistic client with no support for all the fancy modern features, in which case I'd likely get the same or better experience with IRC anyway.

abnercoimbre2• 5 years ago

Thank you for writing this because I'm using Matrix for my engineering conference [0] and I believe in the project. However, no one in our online community (4k members) is willing to switch for good. Virtually no one likes the client's UX at all -- they see it as strictly a downgrade.

So we use Matrix briefly during the conference and then switch back to Discord.

[0] https://www.handmade-seattle.com

bscphil2• 5 years ago

The official client is plagued with UX bugs.

Agree. As a pretty knowledgeable tech person, I tried a few days ago to get an account working with Element, and I almost failed entirely. Different logins failing to sync with each other, one login which claimed that it could not verify ITSELF as a valid login, the "emoji check" button doing quite simply nothing at all. And on top of all that, the security aspects of Matrix are sufficiently non-obvious that even I am not confident that I have things set up securely, what the relevant trade-offs are (if any), and what security model the default setup is using.

It's shocking but 100% true that I feel more confident in my ability to communicate securely with someone using GPG than with Matrix at present.

I think technological advances shouldn't be made at the expense of the UX.

Disagree. Every other service out there, including Signal (especially Signal) has focused on getting UX right and making it easy to use. Matrix is trying to solve a hard problem with decentralized communication, and we need someone out there trying to solve that problem. They've deliberately set themselves up for UX difficulties by refusing to compromise on features. It would be a problem if everyone did that, but not everyone is doing that - Matrix is more or less going it alone. Matrix is best understood as an experiment; maybe it will work, maybe not. I think we just have to wait and see - though I agree with you that it's not usable yet for the general public.

pedroaraujo• 5 years ago

I don't know why people just keep comparing the core protocol to the client. They can both be developed independently, in fact, you don't even need to use the official client, there are many open source alternatives.

If you think something is broken, you are free to contribute to a patch, the project is open source.

Arathorn6• 5 years ago

The official client is plagued with UX bugs. Push notifications works some of the time, sometimes the app would hang trying to open or access a channel, some message get stuck as the last entry with no means to delete them. It makes the network barely usable since clients can't keep up with whatever is going on at the network level.

Speaking as the project lead, one of the controversial decisions we made with Matrix was to try to seed the ecosystem with effectively three entirely independent clients. Element Web, iOS and Android share zero code (other than the low-level E2EE library). The reason we did this was to dogfood the protocol, avoid a client monoculture, ensure that multiple independent clients would work well together, and also provide native SDKs for the three main platforms that others could build on or fork.

This strategy worked to some extent... but the major downside has been that it's more than tripled the dev efforts on the clients (relative to doing PWAs or React Native or Flutter or whatever), and each client has ended up with a completely different set of problems and bugs. For instance, the things you've complained about above all sound specific to Element Android (other than slow room joins, which is a serverside issue that we're working on right now: https://www.youtube.com/watch?v=z7tGU6vv6ZQ). It's rather sad to dismiss Matrix (or even Element) as a whole due to bugs on one specific app on one specific platform.

We've also ended up being massively distracted by building out the non-optional fundamental work such as E2EE and the core decentralisation bits of Matrix (as well as stuff like low bandwidth, which has been driven by paying customers who care less about UX and more about capabilities). This has in turn ended up undermanning the core app teams fairly catastrophically. Until this week there was effectively only one guy paid to work fulltime specifically on the core app for Element Android - and it's only been in the last few months that we've been able to put more than one person on Element iOS and Element Web. Everything else has been stuff which has indirectly landed via customer projects for people like the French govt or German Bundeswehr, or FOSS contributions.

So, the good news is that this is being fixed (better late than never) - there are ~5 people working on improving core UX in Element Web right now, ~3 on iOS and 2 on Android. The improvements on iOS & Web over the last few months should be really quite palpable - e.g. all the stuff on https://element.io/blog/introducing-voice-messages-and-so-mu.... Android should be starting to catch up quick too.

Meanwhile, if anyone likes Matrix and wants to help us improve the clients, we're frantically trying to bring on more people to work on it fulltime to fix this faster - see https://element.io/careers for details.

kixiQu• 5 years ago

As an admin and end user, I'm gonna tell you that the most exciting improvement was voice messages, which I am so happy to have again and definitely didn't get as much attention as they deserved. Wish I could buy y'all a beer. I made a bot [send the audio] to a speech-to-text API so I've got the feature I really missed from Google Allo.

The lower-level work is real interesting, but I hope people in this thread are also aware that the user experience is improving a lot all the time because y'all are working on features like that too.

[send the audio]: https://maya.land/monologues/2021/08/05/matrix-bot-transcrib...

pantalaimon• 5 years ago

There is also https://comatrix.eu which I think implements this.

Arathorn6• 5 years ago

That’s a different but similar variant of matrix over coap :)

meibo• 5 years ago

This is awesome. Hope this will be adopted eventually for their P2P project - would do wonders over slow mesh networks, additionally to stuff like Tor.

goodpoint• 5 years ago

...or just use Briar.

FullyFunctional2• 5 years ago

I'm sorry if I miss it, but no protocol in this millennium should be without mitigation for denial of service. DJB's curveprotect uses fixed sized packages (refusing small ones) to kill amplification attacks. I wonder if they considered such issues.

Arathorn6• 5 years ago

iirc curveprotect is effectively a replacement for TCP which uses EC crypto to encrypt each packet individually, and protects against DoS by each packet being cryptographically authenticated. Ultra low bandwidth Matrix does the same thing but using Noise instead.

FullyFunctional2• 5 years ago

No, that doesn't protect again amplification attacks by itself. Quoting https://curvecp.org/availability.html:

Blind amplification

Some protocols allow attackers anywhere on the Internet to generate a packet that will trigger a much larger outgoing packet from the server to a victim address selected by the attacker. The issue here isn't the availability of that protocol; the issue is that the protocol is amplifying the attacker's resources, damaging availability for the rest of the Internet. The worst offender at the moment is DNSSEC, which has set up a remote-controlled machine-gun pool containing more than 2000 servers with amplification factors between 30 and 95 and with an overall outgoing attack capacity estimated to be close to 50 gigabits per second.

With CurveCP, the first incoming packet from the client is padded so that it is as large as the outgoing packet from the server. If this padding is missing, the server won't respond. Subsequent packets from the client need to repeat server cookies and can't be generated blindly.

Breaking the 100bps barrier with Matrix, meshsim and coap-proxy (2019)

Related Stories