Instant messaging: Protocols are "commons", let's take them seriously

https://www.process-one.net/blog/instant-messaging-protocols-are-commons-lets-take-them-seriously/

by mooreds • 3 years ago

88 93 3 years ago

edhelas• 3 years ago

It's time to give XMPP more love <3

XMPP in 2023 is having all the modern features that you can expect while being massively scalable and having a really nice ecosystem (there is still some quirks here and there but it really improved the past few years).

wyldfire• 3 years ago

But XMPP encoded everything in XML? This seems like a really high overhead.

f1refly• 3 years ago

Fortunately, there's an RFC for people concerned about this :) https://xmpp.org/extensions/xep-0295.html

geraldhh4• 3 years ago

the internet mostly works with lengthy markup languages, they compress well, no problem

rijoja2• 3 years ago

Absolutely yes, I have fond memories of the Pidgin client, which handled everything gracefully, and also libpurple.

This was ages ago, is pidgin still one of the best xmpp clients, or are there better ones now?

I remember that pidgin even used with facebook, unfortunately facebook dropped xmpp as someone else in this thread mentioned!

PaulHoule2• 3 years ago

Even though people have long experienced that you can call an iPhone from an Android or that a Vodaphone customer can call a Reliance customer, somehow people aren’t bothered at all that “messaging” programs don’t interoperate. Google can make 10+ messagers that are incompatible with each other and only a tiny fraction of people see that’s a problem.

And that’s a problem!

After about 30 years we see a clear pattern: something like AIM comes out, because it is locked in there is no competition for the client or the server so there is no motivation to improve either. The thing is the computing world moves ahead so the system goes bad like cheese as the environment it works in changes. A decade later there is a new client out and people are saying “Remember when AIM used to work? There’s a new service that works just as well as AIM used to work” and finally people are so burned out of AIM they are ready to switch but failing messaging systems can lumber on for years like that.

Emphasis on “just as well” because in this scenario the systems are not getting better, you’re not getting an upgrade by switching, you’re just treading water. What people aren’t realizing is that if we had interoperability we would have had competition and after 30 years we’d see 30 years of progress in messaging systems unlike the occasional churn but lack of progress we’ve had.

lotsofpulp4• 3 years ago

Emphasis on “just as well” because in this scenario the systems are not getting better, you’re not getting an upgrade by switching, you’re just treading water.

Modern messaging systems allow me to do much more than AIM. I can send and receive very high quality photos and other files. I can video and audio call all around the world at zero cost. I get very little spam on WhatsApp/imessage.

What people aren’t realizing is that if we had interoperability we would have had competition and after 30 years we’d see 30 years of progress in messaging systems unlike the lack of progress we’ve had.

We had interoperability with phone numbers, but I saw no improvement until AIM/Skype/MSN Messenger/Google Talk/WhatsApp/iMessage came along.

With the previous “interoperable” system, I had to pay for international audio calls and SMS/MMS, so I never did.

So interoperability does not seem like a guarantee of progress, and things might even stagnate due to interoperability requirements. Pros and cons for both.

paulryanrogers4• 3 years ago

Messaging and expectations thereof have plateaued. Moving to standardized protocols is unlikely to stifle innovation that isn't really happening.

naravara4• 3 years ago

People seem to have forgotten that AIM (and Google Talk) were not asynchronous. If you wanted to IM someone they had to be online at the same time you were. Neither of these protocols was particularly good at syncing across devices and working with the modern expectation that people be able to check in and check out from any device at any time.

They were always destined for being retired once smartphones became a thing.

yterdy3• 3 years ago

somehow people aren’t bothered at all that “messaging” programs don’t interoperate.

I used Trillian for almost a decade; it was the second best solution to actually being able to message Yahoo and MSN users from my AIM handle.

I think people have been bothered, but when an entire platform disappears, there's not much anyone can do. The coding wizards who are capable of doing something (in concert with some marketing wizards, maybe) don't seem interested.

jrochkind13• 3 years ago

The "coding wizards" are largely interested in doing what someone will pay them to do, of course. With some exceptions, but of course the bulk of the work that has been put into messaging apps and protocols over the past two decades, whether open or proprietary, has been by engineers and "coding wizards" getting paid by a for-profit company to do it.

So it's really about what those who have money they spend paying for development are interested in, I think, not the interests of "coding wizards", who might love making something different if they could be paid similarly (or at all?) to do it.

Shish2k2• 3 years ago

if we had interoperability we would have had competition and after 30 years we’d see 30 years of progress

Counterpoint: IRC

dijit2• 3 years ago

i dont see the counterpoint here, which company has poured even a fraction of a percent into IRC as Google has with its myriad of messengers or Whatsapp/Telegram ec?

One google engineer on his personal time introduced entirely new systems into IRC (RobustIRC) https://robustirc.net/ which is the most commercial support that exists.

The IRCv3 initiative is a very small number of people, none of them supported commercially.

bebna• 3 years ago

why counter?

still alive and well, survived takeover attempts, see freenode -> libera.chat and got a shiny new protocol spec with modern features

tycho-newman• 3 years ago

I used Pidgin/Adium back in the day to consolidate all my IM protocols. There was a hot second where XMPP was really taking off and even worked on Gchat. I think iMessage and WhatsApp made that moot because people got used to apps, and did not pay attention to the protocol.

All the cool messenger tech that I want to use is buried under user experiences that drive off normies accustomed to iMessage. I have made peace with it and use Signal instead.

alexwasserman• 3 years ago

iChat, Apple’s earlier messenger worked with XMPP too. There was a small period it looked like it would be a good standard app.

parineum• 3 years ago

Even though people have long experienced that you can call an iPhone from an Android or that a Vodaphone customer can call a Reliance customer, somehow people aren’t bothered at all that “messaging” programs don’t interoperate.
...if we had interoperability we would have had competition and after 30 years we’d see 30 years of progress...

SMS has been popular and interoperable for at least two decades. The progress we've seen is MMS and the promise of RCS.

The problem with building interoperability is that the innovators are innovating for their competitors.

timw4mail• 3 years ago

The question then, is what kind of protocol(s) should be used.

SMS is about as close as exists for open instant messaging, but it requires a phone number

IRC seems to have inspired Discord and Slack, with the closed variants improving on features and discoverability.

As far as I know, XMPP doesn't really have a cross-server ability like SMS and IRC, though.

jon-wood2• 3 years ago

XMPP absolutely has cross server support, it was designed from the ground up to do so. You can run your own server and advertise it via DNS, which will then allow other servers to communicate back and forth with users on your server.

BirAdam• 3 years ago

SMS is essentially just email. The phone number just ends up being part of the email address: 0123456789@att.net and whatnot.

Edit: there was also a time when AIM could send and receive SMS due to this structure, and at one point it was trivial to set up a script to email your phone number as SMS for alerts and such. This became more complicated as SMS providers started to require a lot of the same stuff as email providers to cut down on spam.

jon-wood2• 3 years ago

It’s really not. Lots of mobile providers did indeed run email to SMS bridges back in the day, but those were taking the body of the email (or sometimes just the subject) and converting it into an SMS for delivery.

Hakashiro2• 3 years ago

It is more than a protocol. It's an experience.

E-mail is totally intercompatible, but the experience for anything apart from "the equivalent of letters" is simply horrendous. Delta Chat tries to make e-mail more like a chat app, but it isn't perfect, because e-mail wasn't designed to be a chat application.

XMPP has other massive usability flaws. So does IRC, Matrix, and others.

Arathorn2• 3 years ago

None of Matrix’s typical usability problems are the protocol’s fault - they are the clients’ fault. The fact is that on the Matrix side we invested much more time on the protocol, spec process and even server implementations in the early days.

Meanwhile the clients Element built to bootstrap the protocol have been stuck paying off huge amounts of product debt, and are finally at the point now of surpassing mainstream apps - eg https://element.io/blog/element-x-ignition/ - but it took years longer than it should have to get there. Again, it’s not the protocol’s fault though, other than the extent to which building a protocol draws energy from building killer apps.

pocketarc2• 3 years ago

The problem is that there is absolutely no incentive for a company to do this. There is no reason for Apple to work with Meta to make iMessage work with WhatsApp, and for both of them to work with Telegram and Signal so they all work together and share everything.

It kills their ability to add differentiating features, it removes their networking effect moat, and it provides them with no benefit. It's so much more appealing for their answer to be "just download my app".

It happened for telephone and email because they were developed in a different era. It's not going to happen for IM because we're in the era of "why build it if I can't profit from it?".

lotsofpulp4• 3 years ago

It happened for telephone and email because they were developed in a different era. It's not going to happen for IM because we're in the era of "why build it if I can't profit from it?".

Who is going to pay for the bandwidth for 5MB to 100MB photos/videos/gif/files that are sent through the network?

paulryanrogers4• 3 years ago

The sending service should do the hosting so they can pass it on to the sender themselves. Receiving services can fetch thumbnails or full images on demand

zo1• 3 years ago

This makes no sense to me and is probably why we're so far away from a good solution - I pay my ISP for bandwidth to "the network", which includes other devices on "the network". This is the problem, not everything has to go through a "server" which then needs "bandwidth".

krrrh• 3 years ago

Signal runs on donations with a small staff and budget, and even still half of its infrastructure budget goes to sending a verification SMS on sign-up. You can run an open chat network as a tiny non-profit, and any hope of profits is in the same ballpark. For the incumbents it’s a loss leader for other services, and that complicates the whole analysis going on in this thread.

esafak2• 3 years ago

Interoperability does not benefit incumbents, but it does benefit entrants.

The cross-protocol clients back in the day used reverse engineering to work. That would not fly today because today's tech culture is less hacker friendly, more litigious. Such a company would not receive venture funding due to its risky nature.

I think there is room for improvement in messaging, but interoperability makes it more difficult. I think it is in everyone's best interest to first ask for better features, let its inventors make a profit, then worry about interoperability once competitors catch up. If the market is competitive, interoperability can be achieved through coalitions. IBM's MCA vs Compaq et al's EISA is an instructive example.

darklion• 3 years ago

it does benefit entrants.

Which is why you don’t see successful messaging companies rushing to open their systems.

What company wants to spend the effort to build a successful messaging platform, only to open it and see some nobody come in with feature parity _and_ the ability to reach/convert every single one of their customers?

Open protocols would have worked great if that’s what was mandated _before_ these established products were in the market. Now, it’s going to be an uphill fight all the way, and companies are going to do the absolute bare minimum—-not just what they can get away with technically, but what they can get away with legally (e.g., Apple adopting RCS so they can point to RCS as the common messaging denominator, thus avoiding opening up iMessage itself).

naravara4• 3 years ago

iMessage already works with SMS/RCS, which is low level enough that any messaging service could use it as a method to interoperate with anything else.

What iMessage can’t do is translate a lot of the QOL and UI features, each client will have to adapt to read those on their own.

user3939382• 3 years ago

Many have probably long forgotten Facebook ditching XMPP. For a moment there was hope we’d be moving open federated comms protocols into the new era.

rijoja2• 3 years ago

Yeah I remember this! I used to use it with pidgin, as I mentioned in another comment on this thread.

Google talk used to use xmpp as well back in the good old days, and there's maybe even some more!

It'd be nice if open source projects where to culturally make it a norm to use xmpp instead of slack, discord, and the like, sort of how everything used to be run over IRC in the even better old days!

bsaul2• 3 years ago

Seems like european union DSA will force this interoperability protocol to emerge quite soon, doesn't it ?

mikem170• 3 years ago

"The EU Digital Markets Act mandates that messaging services like iMessage are required to offer other companies some level of interoperability if they’re deemed to be big and important enough." [1]

The law has already passed and will apply to messaging services with more than 10% of the market. WhatsApp and Facebook Messenger will need to comply. After review it was determined that iMessage doesn't have enough users in Europe at this time.

[1] https://www.theverge.com/2023/9/6/23861030/imessage-bing-eur...

bsaul2• 3 years ago

yes sorry i didn't notice that at first read. Thanks for the info about imessage, i had no idea about that 10% market share..

rapnie• 3 years ago

Tangential. Here's the CCC talk [0] about RFC 9420 Messaging Layer Security, mentioned in the article, and More Instant Messaging Interoperability (MIMI) [1] at the IETF, mentioned in the video.

[0] https://media.ccc.de/v/37c3-12064-rfc_9420_or_how_to_scale_e...

[1] https://datatracker.ietf.org/group/mimi/about/

bitzun• 3 years ago

A developer bemoaned to me the threat of being legally obligated to make major chat apps interoperate because it would be too much effort and would stifle innovation, and I found that rather pathetic. If we couldn't figure out how to create protocols that connect these "modern" apps, especially given that they have mostly converged on the same set of basic features, it'd be a pretty strong indictment of the industry. To be clear, I hope it is mandated and if it is I'm sure it'll be achieved easily.

j45• 3 years ago

The delusion that one app can be the only centre of your life continues to litter our phones with 9 chat apps.

Not making protocols interoperable allows the app to attempt to build their own walled garden.

Interoperability is a very old problem, and it has been reasonably solved a few times (jabber pre Cisco/xmpp/etc) and most recently technologies like matrix/riot are pretty well positioned.

While tools like pidgin exist today, software like trillian of old allowed one interface to log into everything.

JohnFen7• 3 years ago

and would stifle innovation

I am getting increasingly concerned about the number of times aggressively antagonistic things are defended because addressing them would potentially "stifle innovation".

Innovation is important, but it's far from the only important thing. We need to stop sacrificing everything on the altar of "innovation".

geraldhh4• 3 years ago

we've come a long way in positioning interoperability as a differentiating innovation, thou

RugnirViking• 3 years ago

Not to mention how many things are posed (and believed by policymakers) as stifling innovation when the effect of literally only consolidating markets into as few providers as possible.

"Woe is me, this will stifle innovation" cries the megacorp, as their latest merger is blocked. The amount of people that are sympathetic to those cries are alarming.

gumby3• 3 years ago

Why did SMTP win? My first business card had my ARPA email address on it, which occasionally generated a question but which was usually ignored by those who didn’t have email. Then people started to list their compuserve or whatever address on their cards — often several addresses — which made me laugh, and then eventually an SMTP address wiped all those alternatives out.

I used to see the same thing with phone numbers: a stack of phone (desk phone), fax, pager, voice mail, mobile, all eventually collapsing into a single phone number.

Why did this not happen with messaging (which is more than 30 years old, despite what the article says)?

Anunayj• 3 years ago

What is a ARPA email address? I've never heard of it. Did they look different from normal SMTP email address used today? Or was it like your email server just supported ARPA and not SMTP.

gumby3• 3 years ago

I was gumby@MIT-AI.ARPA. By then mail used SMTP but the arpanet had not yet transitioned to IP/TCP.

Those .ARPA names were part of a lengthy transition to the DNS; before this I was just gumby@mit-ai or just gumby@ai

The ARPA. TLD isn’t used for hosts any more but has been repurposed mainly for network database lookups. Yes, the `.` after the TLD is there for a reason.

esafak2• 3 years ago

WhatsApp lets you use phone number as your ID, and it is very successful.

gumby3• 3 years ago

My point was that there is a single universal email system but no single universal IM system.

WhatsApp is another silo, even though it uses phone numbers as identifiers.

mike_hearn• 3 years ago

SMTP only won for a relatively brief period. Nowadays it's mostly used for newsletters and transactional mails. Personal messaging moved to IM and Discord, business messaging is now largely Teams/Slack.

I sent someone a business email the other day. They told me I got lucky they even saw it, as they now communicate entirely via Slack. Slack isn't even trying to be an email competitor and it still wins.

SMTP is still useful for cases where the additional feature-set of other platforms isn't important but a globally unique ID is.

tptacek2• 3 years ago

The burning question is, why not choose an open protocol that already works on a large scale, like XMPP, adding their value on top?

This never works. It's a recipe for disaster. The security model of a secure messenger informs the protocol from top to bottom. You try to retrofit cryptography onto a general-purpose messaging protocol and you get group membership protocols that allow servers to decrypt messages, and centralized databases of everybody who's talking to everybody else. Even good, purpose-built protocols have ended up with these kinds of problems (see Matrix). Never do this.

uhoh-itsmaciek• 3 years ago

Do you have more info about the issues with Matrix?

MattJ1002• 3 years ago

https://nebuchadnezzar-megolm.github.io/ has details of the main issues I recall (most/all fixed as I understand it).

Arathorn2• 3 years ago

He’s talking about https://nebuchadnezzar-megolm.github.io/ - and the fact that group membership in Matrix is currently determined by the server rather than a client. So if Alice is in a DM with Bob, Bob’s server could make up malicious devices for Bob, or fabricate an invite to add Charlie to the room. This is mitigated by encouraging users to verify devices, so Bob’s fake device would be recognised as an attacker - and for faked invites, relying on Alice and Bob spotting that an unexpected malicious user just joined their conversation.

However, it could be better: first we have been converging on a single Rust crypto impl (at last) - see the crypto section of https://matrix.org/blog/2023/12/25/the-matrix-holiday-update.... Then we’re shifting to TOFU (so unsigned devices never get encrypted for). Then, finally, we’re shifting to client-constrained group membership (unless MIMI gets there first, and then we may switch horses to their MLS-based solution).

edit: the other issues raised by the nebuchadnezzar paper were impl bugs in the 1st gen crypto impls, which were fixed at the time. also, the premise that you can’t add encryption posthoc to a protocol is flawed, imo. Matrix may not have nailed it, but as long as you consider the crypto as if it’s a new protocol and have the agility to evolve to it, everyone wins.

edit 2: there is no centralised db of who talks to who in Matrix. individual servers can see who shares a conversation which their server participates in, but this is very hard to avoid given you could always figure it out via traffic analysis (unless you go down a Nym style mixnet solution).

MattJ1002• 3 years ago

MLS addresses many of these issues and both Matrix and XMPP are working on incorporating it.

daft_pink• 3 years ago

1. I think security and privacy has a lot to do with it. If the services interoperate then other companies that make the services will get access to your personal data even if you don't use them.

2. Even though there are so many services, I find functionality to be lacking on many of them. I don't want to use facebook, because of privacy. I don't like signal because it doesn't give you your chat history across devices. I used to love hangouts, but google destroyed it. Many of the services are tied to your phone and don't work if it isn't on. There is a good chart in german comparing the different features. I just use imessage because it's end to end encrypted, i trust apple's privacy and it works.

Mortiffer• 3 years ago

Surprised to not see DIDcomm standard mentioned https://identity.foundation/didcomm-messaging/spec/v2.1/ https://didcomm.org/

JohnFen7• 3 years ago

That every IM service is an island of its own is the primary reason why I don't use any of them. I'm not about to install and keep track of multiple IM service apps. It's just better and easier for me to avoid them all.

Hakashiro2• 3 years ago

While this is commendable, the problem with this is that you're also refusing to interact with people who use those platforms.

For example, you can refuse to use Google Chrome, and instead choose Firefox or Vivaldi. Your web experience will be slightly different, but the most important parts will remain the same: You type an address, you wait for it to load, and you access the content.

On the other hand, refusing things like WhatsApp means there's a non-insignificant amount of people that use WhatsApp to communicate exclusively. This may not have impacts for you, personally (although I would be hesitant to believe that), but it definitely leaves out billions of people who communicate exclusively via WhatsApp.

A similar thing happens, for example, if you refuse to use YouTube, which is the largest Internet video platform on the planet: You will have to refuse to watch any content that is only uploaded to YouTube, or put up with frontends that use YouTube in the background, or perhaps even be forced to pirate videos, neither of those three options is good for different reasons.

JohnFen7• 3 years ago

the problem with this is that you're also refusing to interact with people who use those platforms.

That's not a problem for me at all, honestly. I interact with my friends and family through other channels. Nobody I know uses these services exclusively.

it definitely leaves out billions of people who communicate exclusively via WhatsApp.

I don't want or need to communicate with billions of people. I want and need to communicate with the people I know and care about.

Don't get me wrong -- I'd use an IM app for the convenience if there was one that served my needs. But as long as each IM app is it's own island, then none of them serve my needs at all. I'd have to have a half dozen of them, and that's unmanageable and ridiculous.

camgunz• 3 years ago

Mobile really messed us up. We had IRC but it just didn't work well in a mobile environment, and we let Google rug pull us with XMPP Google Talk. App Store monopolies seem like they're coming down, but stuff like having to use Google's push notification service and whatever is really limiting.

We do have common protocols: email, IRC, XMPP, Matrix, Signal, MLS, all of which are various good degrees of well-specified and open source. We have the tools, we just essentially need to be OK with saying "WhatsApp is just for interacting with my kid's teacher; my friends and family use Signal". I think nerds, geeks, punks, and artists forget we're the interesting ones. It's fine to use Threads for sports/normie stuff and Mastodon/Signal for our stuff. The separation is a gift.

Instant messaging: Protocols are "commons", let's take them seriously

Related Stories