Disclosing E2EE vulnerability in multiple Matrix clients

https://matrix.org/blog/2021/09/13/vulnerability-disclosure-key-sharing

by Sami_Lehtinen • 5 years ago

333 106 5 years ago

This is not a protocol or specification bug, but an implementation bug which was then unfortunately replicated in other independent implementations.
We will also accelerate our work on matrix-rust-sdk as a portable reference implementation of the Matrix protocol, avoiding the implicit requirement that each independent library must necessarily reimplement this logic on its own

I like the idea of providing a core Rust/C++ library that can be called in any other runtime via FFI, compiling to WebAssembly, etc. I've used the same technique to ensure consistency in usage of a custom time-series compression format across an organization. Overall this technique seems in-line with the "DRY" principle, and the tradeoffs have been favorable in my experience. I think it will be a good step forward for the Matrix ecosystem.

Arathorn11• 5 years ago

The current way we're approaching this is to split the reference E2EE implementation into its own rust crate (https://github.com/matrix-org/matrix-rust-sdk/tree/master/ma...) which can be used with any SDK (e.g. we're almost finished embedding it into the Kotlin matrix-android-sdk2 client)

Separately, there's also the overall matrix-rust-sdk https://github.com/matrix-org/matrix-rust-sdk for clients to use as a "full fat" Matrix client SDK - as used by Fractal Next (https://gitlab.gnome.org/GNOME/fractal/-/tree/fractal-next) etc. We might end up using this in Element too in future (especially in Element iOS, where a Swift UI + matrix-rust-sdk backend could be quite a cute next generation architecture).

So while the first generation reference Matrix SDKs (matrix-js-sdk, matrix-ios-sdk and matrix-android-sdk) were completely independent implementations, each with their own bugs and increased audit surface, we're hoping that matrix-rust-sdk will simplify this a lot in future.

danShumway• 5 years ago

Quick comment, just want to show some additional support for the shared crate approach that Matrix is pursuing, it should give people a lot more confidence to try alternate clients.

I throw messaging apps in the same category as, for example, web browsers. It's really tough to try an alternate implementation if you're worried that they might have broken a complicated security feature. So having more shared crates mitigates some of that risk.

jaimehrubiks• 5 years ago

Any recommendation to get started with matrix? Like use cases, featured clients, interesting public chats, or similar? To someone whose close environment does not use it

zeroping2• 5 years ago

The reference clients and server are a good place to start. They're the place to find all the protocol features implemented and well supported.

The Element client (desktop/web/android) is electron-based https://element.io/get-started Synapse is the server to start with https://github.com/matrix-org/synapse/

From there, what's the first place to find people to chat? I don't have a good answer there, but the 'spaces' functionality is a nice way to group communities.

preya2k• 5 years ago

I assume you meant the “spaces” functionality to group rooms?

galbar• 5 years ago

I started by setting up a synapse server (with docker it is quite easy, although reading all the config options takes some time). I already had a server with docker services running there, so deploying an extra container was easy for me.

I started trying it, with the Element.io apps and convinced a techy friend to try it. Since then, he set up his own server and we have a group of friends that we mainly communicate through matrix. Also, I try to use it with my parents. They want to use it, but end up writing to me through WhatsApp out of muscle memory.

After some time I started looking at public rooms that I'd be interested into, available in matrix.org or matrix.feneas.org.

The easiest way to start, though, is to create a free account in the matrix.org public server using the webapp[0].

[0] https://app.element.io/#/register

kiliankoe• 5 years ago

If you're looking to get started using Matrix, but have no contacts to bootstrap it with, you can look into bridges to other networks or using something like beeper.com with hosted, well-supported bridges.

Disclaimer: I work @ Beeper

callahad4• 5 years ago

As for interesting public chats: Quite a lot of the free software community is on Matrix these days; check out projects you're interested in and see if they have a Matrix link.

For example, the NixOS community has a curated Matrix Space linked from https://nixos.org/community/index.html with lots of rooms to explore. GNOME, KDE, and Mozilla also have a significant presence on Matrix.

ptman• 5 years ago

I'm partial to the guide I wrote https://matrix.org/docs/guides/free-small-matrix-server/

mxuribe2• 5 years ago

The comments that others suggested would be good, or of course there's also the "Discover Matrix" page: https://matrix.org/discover/

unbad505• 5 years ago

I started with element and matrix.org, now I use element on IPFS and host the conduit homeserver behind a caddy reverse proxy.

fouc• 5 years ago

trick vulnerable clients into disclosing encryption keys for messages previously sent by that client to user accounts later compromised by an attacker.

Doesn't the fact there's some kind of mechanism or expectation of sending encryption keys willy nilly imply a flaw with the protocol perhaps?

pkulak3• 5 years ago

willy nilly

It's not though. The fix is to cryptographically verify requesting clients before sending.

aidenn0• 5 years ago

They address this in the last two paragraphs of the article.

Joe_Cool• 5 years ago

If "send key to all members of channel x" is enabled it would be good if that would work.

You can set it to "only other clients and people I have personally verified" but that is quite cumbersome for big channels that just don't want clear text.

pkulak3• 5 years ago

I'm a tad worried about getting rid of key-sharing. I seem to use this every once in a while. Pretty much any time I use a client I haven't used in a while. I guess I could just get the keys from the backup, but then I need to have THAT key, which is a bit inconvenient.

Arathorn11• 5 years ago

This is a great example of why key-sharing exists today... and why it shouldn't. In theory, you should be able to pick up and old client and decrypt new messages and history on it absolutely fine. However, in practice, Matrix's E2EE publishes 100 one-time keys (OTKs) on your server to let other devices establish secure 1:1 channels with you if you're offline - and if you go offline and that pool of OTKs exhausts, then new sessions won't get set up and you won't receive keys for new messages... giving the misbehaviour you're seeing.

Now, you're right that key-sharing is a useful way to fudge around that failure mode.

But an even better way to fix it would be to find a way to stop the OTK pool exhausting - and that's precisely what MSC2732 is: https://github.com/uhoreg/matrix-doc/blob/fallback_keys/prop.... This provides a last-ditch key which can be used to set up 1:1 sessions even if you run out of OTKs, which is marginally inferior to using a different OTK every time, but in practice really isn't a disaster (see the MSC for details).

However, fallback keys are relatively new and aren't implemented on all clients yet (matrix-js-sdk has them, but matrix-ios-sdk is implementing this coincidentally this week)... and so until they land, we still need keyshare requests to paper over this limitation.

But in future, hopefully it will be almost unheard-of to need a keyshare request, and we can change them to be an entirely manual or out-of-hand mechanism of some kind, and avoid classes of bugs like the vuln in question here in future.

pkulak3• 5 years ago

Thank you for writing that all out. I wasn't familiar with most of these details.

criticaltinker2• 5 years ago

100 one-time keys (OTKs) on your server

Is 100 a safe default for devices with limited memory, or are there other more salient constraints that I'm ignorant of?

e12e3• 5 years ago

In theory, you should be able to pick up and old client and decrypt new messages and history on it absolutely fine.

No? Assume the lost/stolen/forgotten device has outdated (or no) protection around its keys - and it should be rotated out as soon as possible?

I'd rather not have to worry about if my old phone is in my drawer at home, or in an mi6 office (or more likely, with an estranged spouse or a journalist)?

xiaomai• 5 years ago

What are people using for a self-hosted "slack" type messenger these days? I have looked at Matrix (which I am interested in but I think needs time to settle right now), Zulip (I really like their threading model but it has been confusing for other people I have tried it with), Rocket chat (only recently aware of this one). What else is there? Anyone have good/bad experiences with any of these?

cyberlurker• 5 years ago

Mattermost is okay

dbrgn• 5 years ago

Yeah, I have good experiences with Mattermost (bundled with GitLab) as chat for a hackerspace/makerspace. As long as you're OK with unencrypted push messages, building your own version of the app and keeping it up to date is horrible. (React native levels of horrible.)

mxuribe2• 5 years ago

...I think needs time to settle right now...

I hear this on occasion (though more rarer nowadays than in the past)...But curious, is there something specific that you're finding in matrix that is not yet settled? By all means, the beauty of open source stuff is that you get to choose whether you (and your circle of contacts) uses zulip, mattermost, etc...So, i encourage you to use whatever open source offering gives you (and your circle of contacts) what is needed/desired. But with me being such a fanboy of matrix, i always get curious. Would you care to share specifics?

agentdrtran• 5 years ago

That's basically it. A lot of options but they all have some flaw. Rocket Chat and mattermost both lock features behind an enterprise paywall (although Rocket Chat is more forgiving and has optional E2EE), Zulip is nice but still a small player and some people aren't fans of the UX, and Matrix lacks the polish of slack and moderation tools are not up to speed with Slack/RC/Mattermost, which is a problem for larger groups.

remram2• 5 years ago

Matrix supports the concept of “key sharing”, letting a Matrix client which lacks the keys to decrypt a message request those keys from that user's other devices or the original sender's device.

I'm a bit curious why you'd ever need to request those keys from somebody else, in addition to your own devices (or your homeserver, where they'd be stored encrypted by a key known by your own devices).

Youden2• 5 years ago

Imagine you're a "normal" person who has a phone and doesn't really use a computer. You lose your phone and want to retrieve the history of a chat you had with your friend on the new phone. You ask your friend to send the key to your new device.

remram2• 5 years ago

That sounds like a manual process that would not be part of the automated "key sharing" functionality as described.

Arathorn11• 5 years ago

The recommendation is that devices only accept requests to share (forward) keys from a) other devices belonging to that user, b) the device which originally send the message.

It's acceptable to gossip keys between your own devices (assuming you authenticate them correctly!) given... they're all your own devices.

It's also acceptable to request the sender to re-send the keys to you, given the sender knows for sure whether you were allowed to receive the key (given it made that choice in the first place).

It's not acceptable to request keys from other devices in the room, as they won't necessarily know whether you were actually allowed to receive the message key you're requesting - only the sender knows for sure whether it should have been sent to you; it was their key after all. Plus it would risk make vulnerabilities like the one in question here even worse, given anyone could try to exfiltrate messages from anyone else, rather than "just" the sender (as was the case here).

nightpool4• 5 years ago

With the recent dust-up around protonmail's "end-to-end encryption", it's hard for me to believe that the hammer isn't going to come down on Matrix in a similar way. Nobody has seriously turned their attention to the compromised-homeserver issue, even though that's the only threat model under which e2ee makes sense. Expecting users in hundred-person group chats to manually verify each other individual user's keys is nonsensical. And even a single unverified key means that their homeserver operator could be listening in. I feel like Matrix should move more towards emphasizing how easy it is to self-host and maintain control of your own Matrix data, making e2ee is a non-issue.

dkasak2• 5 years ago

Realistically, when would you be in a hundred-person encrypted group? Mostly this is the case when you're a member of some kind of organization, and there are ideas how to solve this case without pairwise verifying all participants (e.g. by delegating trust through a single trusted person such as the CEO, reducing the number of verifications necessary from N(N+1)/2 to N). Even without this, fully verified E2EE is still feasible and useful for smaller groups.

And even if you own the homeserver, you still want E2EE since you don't want the data to rest in plaintext server-side.

However, there is work currently being done to make it feasible for every node to also be its own homeserver, via P2P Matrix (https://matrix.org/blog/2020/06/02/introducing-p-2-p-matrix).

schmorptron• 5 years ago

At uni, it's common to have whatsapp groups for classes, which tend to be encrypted 100 to 200 people WhatsApp groups.

miloignis• 5 years ago

Even if it's not currently practical for 100 person group chats, verifying each other's keys is quite practical for 1on1 or other small groups of say 10 people or so, and you get a great benefit from doing so! Two normal users could use any matrix server, including the huge matrix.org one, and communicate privately in a direct chat with peace of mind.

Of course, I think self-hosting and decentralizing the servers is very important and good work too!

Youden2• 5 years ago

I feel like Matrix should move more towards emphasizing how easy it is to self-host and maintain control of your own Matrix data, making e2ee is a non-issue.

Matrix is way ahead of you. There's ongoing work to build P2P clients [0]. That is, each phone/device can be its own home server. The latest update was in May [1].

[0] https://matrix.org/blog/2020/06/02/introducing-p-2-p-matrix

[1] https://matrix.org/blog/2021/05/06/introducing-the-pinecone-...

progval2• 5 years ago

It still does not solve key distribution.

Matrix P2P removes the homeservers, but clients are still talking to each other through some medium that can impersonate them.

There's really no magic solution to solve this, you need either a trusted third-party (such as Certificate Authorities, who are expensive) or a Web of Trust (impractical for users).

vbezhenar2• 5 years ago

For me it looks very weird and defeats the whole purpose of Matrix. I mean: it was designed from the ground as a federated network, similar to E-mail. Turning it into a completely different topology will not end well. If they want a P2P client, they should design it from the scratch. Otherwise it'll end up as a bunch of hacks glued with tape.

m4rtink3• 5 years ago

That sounds very promising! Having two mobile devices with 5+ radios (2G, 3G, 4G, wifi, Bluetooth, NFC, ...) inside 5 cm from each other yet they can't easily talk directly to each other in a user friendly way is such a fail!

Like really, my Palm TX could do that via infraport!

So any effort to break this stalemate makes me very happy. :)

liotier• 5 years ago

Matrix should move more towards emphasizing how easy it is to self-host

They are already moving beyond that - towards P2P Matrix, with a network mixing traditional servers and individual nodes that collapse client and server: https://matrix.org/blog/2021/05/06/introducing-the-pinecone-...

zaik2• 5 years ago

Expecting users in hundred-person group chats to manually verify each other individual user's keys is nonsensical.

Maybe this could be helped if Matrix clients supported OpenPGP encryption which allows for key signing/web of trust? Some XMPP clients already have OpenPGP support, it would be nice to one day be able to send encrypted messages to Matrix users.

ajconway• 5 years ago

The possibility to perform manual or semi-automated (QR code) key verification is still useful.

1. Those who care about encryption are able to verify their keys.

2. The service can't implement mass-surveillance in secret if there are at least two random users who verify their keys.

COGlory• 5 years ago

What dustup around end-to-end encryption? I'm only familiar with ProtonMail's IP logging controversy. I was unaware it impacted E2E at all?

abnercoimbre• 5 years ago

Did you find further information? Would've been nice for the GP to point to a source or news item.

0xdeadb00f2• 5 years ago

I'm pretty sure OP is just misinformed. That, or they're talking about that one huge hole in PM's PGP implementation years and years ago.

nightpool4• 5 years ago

Right, the dust-up was "PM advertises their service as secure and end-to-end encrypted, but in practice that doesn't mean much because user access is non-anonymous and they're still a central metadata honeypot for any law enforcement agency to log accesses to" (compared to e.g. Signal which AIUI is designed as an anonymous dropbox so that Signal can't correlate between user access to their servers and individual senders)

garaetjjte2• 5 years ago

I feel like Matrix should move more towards emphasizing how easy it is to self-host and maintain control of your own Matrix data

The problem is, it isn't easy at all. Running homeserver requires gigabytes of RAM and significant CPU performance to work at usable speeds. It just isn't feasible to run it on some dirt cheap VPS, you need beefy machine.

Semaphor• 5 years ago

Why does it require that much? I run a Jabber server, and it barely uses any resources on my VPS. Unless you mean for significant users?

dkasak2• 5 years ago

You'll be thrilled to know this pretty much isn't true anymore, even with Synapse. My instance is running on a cheap Hetzner VPS with a not very powerful CPU and is currently using about 700M RSS and not much CPU. And I'm in a lot of rooms, some of them quite large and high in traffic.

I'm also not even using Synapse workers at all, just a monolithic instance. Splitting the setup into workers would buy me an additional speedup if things got overly slow.

nightpool4• 5 years ago

Yes, unfortunately Synapse is possibly one of the worst apps i've ever sysadmin'd for, and I'm not even the primary sysadmin for my homeserver. I still use it, regularly, but we're all really anxiously anticipating the release of Dendrite. I'm not the biggest fan of the protocol either, or the UI/UX of Riot. I think Matrix is a good idea, but there's a lot of historical baggage and I think the world probably needs one more "throw everything away but learn the lessons of the past" cycle before we get something really, truly good in the chat space.

delroth• 5 years ago

I'm not sure when you last tested, but my Synapse instance (v1.42) is currently using around 180MB RSS. I'm on a few rooms with 500+ users and multiple with 50+ users.

m-p-32• 5 years ago

Still haven't received the up-to-date version from F-Droid, which is still stuck at 1.2.0. They really should have some provision to expedite a release when a serious security flaw is discovered.

Not the Matrix team's fault, but still a bit worrying. Maybe they could implement their own F-Droid repo to skip the F-Droid build server?

progval2• 5 years ago

Or they could coordinate with the F-Droid packagers, like they did for other distributions. (Actually they might have, since they don't exhaustively list them)

callahad4• 5 years ago

Element did coordinate with the F-Droid packagers on behalf of both Element Android and SchildiChat.

Because F-Droid runs their own builds and requires published source code, there is no mechanism for pre-building and staging security releases for issues which are not yet publicly disclosed.

That said, the packager was extremely helpful and prepared the metadata updates in advance to ensure that both applications could enter the build queue as quickly as possible upon release:

- Element 1.2.2: https://gitlab.com/fdroid/fdroiddata/-/commit/76c8f5b87aa8df...

- SchildiChat 1.2.2.sc43: https://gitlab.com/fdroid/fdroiddata/-/commit/232b316f8affe0...

Arathorn11• 5 years ago

F-Droid do the builds rather than Element (or other Matrix client vendors), so the whole thing is at the mercy of the F-Droid CI process. The Element Android build has entered the queue apparently, but not landed yet. We alerted the F-Droid team last week to prepare for the release, but beyond that there's not much we can do.

m-p-32• 5 years ago

I understand that it's out of your control, and you even gave them a heads-up. Thank you for your work :)

j-james• 5 years ago

Have you thought about hosting your own F-Droid repository, like NewPipe does?

https://newpipe.net/#download

hrfbi• 5 years ago

The fdroid maintainers are... Less than competent. I recommend anybody not to use the main repo at all if they care about security, as any update could take weeks to arrive.

codetrotter• 5 years ago

Probably they are just overworked and understaffed. Possibly also lacking money for more/beefier build-servers?

SubzeroCarnage• 5 years ago

F-Droid maintainers are solely a volunteer effort. The CI helps a lot. The biggest slowdown in the chain is the manual offline signing step.

You can see here the history https://gitlab.com/fdroid/fdroiddata/-/commits/master/metada...

You can donate to support them here https://opencollective.com/f-droid or here https://liberapay.com/F-Droid-Data

yjftsjthsd-h• 5 years ago

That's a bold claim to make without something to back it up; care to share what warrants a rating of incompetent?

0xdeadb00f2• 5 years ago

The maintainers aren't really the problems - it's their, long and silly build process + schedule. Updates are _always_ at least three days behind, because their build schedule is set in stone and they refuse to change it.

vanous• 5 years ago

F-droid is a blessing and a curse. Many devs experience this 'F-droid catching up with releases' due to the manual signing process. The userbase is always confused about that. Reproducible builds are claimed to be the possible solution but almost nobody is doing that due to the complex setup.

smichel173• 5 years ago

Are reproducible builds the solution? Seems like the fdroid folks would still have to build ("reproduce") the apk to verify that it matches the one provided by the developer. Then they'd sign it and distribute the double-signed apk.

I was under the impression that reproducible builds guard against the "fdroid build servers got compromised" attack vector, (or "developer got compromised", as compared to dev-signed releases) and nothing else.

eighthave2• 5 years ago

If a project maintained reproducible builds, then they can have the best of both worlds: their own repo where they can publish builds as they like, then f-droid.org can later reproduce them and use the upstream developer's signature.

ncmncm3• 5 years ago

In other news, Nheko supports E2EE.

Sami_Lehtinen• 5 years ago

Quote from the original documentation: "The recommended strategy is to share the keys automatically only to verified devices of the same user. Requests coming from unverified devices should prompt a dialog, allowing the user to verify the device, share the keys without verifying, or not to share them (and ignore future requests). A client should also check whether requests coming from devices of other users are legitimate." - It made me smile, because should and recommended doesn't generally mean anything to developers. Only mandatory, must and must not, sections might get some attention.

jeroenhd• 5 years ago

As discussed above, key sharing was originally introduced to make E2EE more reliable while we were ironing out its many edge cases and failure modes.

Based on what I understand about this feature, I've had to rely on key sharing quite often (even requiring manually requesting keys a bunch of times) for E2EE to work reliably. I worry that my experience will be severely degraded if key sharing ends up being removed from the protocol.

Having the sending party re-encrypt messages for a new device require the other party to be online. I can see that requirement not being met in several ways, e.g. if the other party dies or uninstalls the application, or when the other end is arrested (I can imagine journalists losing access to activists' messages because of this).

Arathorn11• 5 years ago

I've had to rely on key sharing quite often (even requiring manually requesting keys a bunch of times) for E2EE to work reliably.

This is because it's papering over edge cases in E2EE.

Having the sending party re-encrypt messages for a new device require the other party to be online

We never make senders re-encrypt messages like this, and we never would.

Firstly, key-sharing from the sender does require the other party to be online already.

Otherwise, if you have the keys in an existing device, you could get them onto your new device by backing them up on the server - or using "dehydrated" devices; where the 'new' device you log into is actually one which is stored encrypted on the server, and "re-hydrated" into a new device when you login... and so already has your keys. These have security trade-offs obviously (what if your online backup gets pwned? what if your dehydrated device gets pwned?) but it's not obviously worse than exfiltrating the missing keys from the sender.

Disclosing E2EE vulnerability in multiple Matrix clients

Related Stories