davecridland

XMPP is routinely used on HF radio and SATCOM networks field-deployed by militaries. It's deployed with channel compression (so the links themselves are compressed), but it routinely operates down to way way below mobile. In general, the really bad links are used for federation (s2s) rather than client connections which demand lower latency for user interaction. I've personally watched it operate on HF radio modems down to single-figure bits per second - those are simplex links with a 30 second turnaround. It's slow of course, but it does work - and this is on the base standard. Any server can do this, there's three I know of in use.

WhatsApp decided, at some point in the past, that XML was too verbose and compressed it using a technique borrowed from WAP - essentially a fixed-dictionary compression. Newer WhatsApp systems don't ever bother decompressing to XML, and it's likely it can't be anymore, but it's fundamentally the same traffic patterns even now. Your WhatsApp account still has a jid (with a very short, fake domain) and so do your groupchats (also a very short fake domain). The WAP compression choice was unfortunate, as it only really had an effect back in GPRS/EDGE days, only on certain networks, and only on certain traffic patterns, because it's the number of packets that matter in most cases, not the number of bytes. There'd still be some rare benefit on 3G, but by 4G the benefit had all gone.

Meanwhile, clients like Siskin and Conversations work just fine on modern mobiles. Conversations (an Android client) does work better when given permissions to keep the sessions live rather than rely on push notifications, but testing has consistently shown that battery life in unaffected. There's a number of extensions that are widely deployed that help efficiency, for sure, and I'd not want to run without for preference - though it is possible, and not as bad as you'd think.

So I'd say that XMPP has no performance problems at all on "long thin" networks way beyond mobile.

To get the same performance out of anything based on HTTP, you need to recode to use the parallel protocols of CoAP and friends. Matrix have done this, but it means specialist clients and servers are needed, and as far as I'm aware there's only a single implementation.

There's an awful lot of "why not?" here. Remember, this is an Experimental XEP. The XMPP Council saw no reason to actively block it, but that doesn't mean we're all mad keen that everyone should rush out and do it.

There was an intense debate on whether it ought to be published as Standards Track or Humorous...

Nice write-up of it, though I disagree that you can (or should) "recover" from a database breach in that way. If you detect a database breach, it's likely considerably after the event, and you should enforce password changes (and TOTP resyncs).

Also, there's no mention of Channel Binding, which adds considerable protection to MITM attacks aimed at obtaining the ClientProof off the wire.

I'd really appreciate more detail, here.

And XMPP has always tried to be the email of IM. Similar model, address format, and so on. A sea of independent, autonomous domains openly connecting to each other.

Agreed, which is why XMPP does this.

Sure it is. https://xmpp.org/extensions/xep-0365.html

I don't buy this. XMPP uses no more than any other IM system, and much less than most. The military use it over HF links in theatre, for heaven's sake - this is not what you'd choose to be doing if the bandwidth was so high. (HF is STANAG 5066, and runs down to a handful of bits a second - this stuff is not fast).

1) XML really isn't that bad. Honestly. It can be (and is) parsed fast, without any schemas at all. The only thing you need to define about your namespace is the URI you're using. XMPP is not SOAP.

2) Wrong. RFC 6122 explicitly permits this. XMPP was designed for multiple devices, actually, but expectations changed over the past decade. Carbons (XEP-0280) does what people want now, and is widely deployed, in real clients and servers. Getting extensions to the "Experimental" state is easy, BTW - it's trickier moving them on from there because it's trickier to change them once they're widely deployed.

3) Bizarrely, you're mostly describing the client behaviour I have on every client I use. As for federation, that's what XMPP does. It doesn't do distribution, because doing that and getting autonomy for your security domain is hard (if not impossible).

4) Saying something is "objectively terrible" when it's clearly your opinion, and then rubbishing everyone who might hold a contrary one, is pathetic. The validity of your position is not helped by the XSF working on live documents for most of the time (the "Experimental" phase). Where it differs from the WhatWG is that specs move out of that phase once they're proven, and the documents become harder to change once they're Draft or Final, reflecting how hard changing all the implementations would be. The XMPP world sees people add extensions, try them out, and then move them to standards all the time. We see that from HipChat, Conversations, and plenty of others - it's a great way to work. Other folks like to get something to Experimental first, and that way get early feedback from the community. We think the way we work is a pretty good balance between stability and dynamism.

5) Absolutely not. That "X" stands for "Extensible" (I know, it was fashionable back in the '90's), and the XML namespaces you despise (and appear not to understand) mean that you can cheerfully slap whatever data you like in and alongside the standardized stuff. Trust me, the (many) IM services based on XMPP - and if it were as bad as you claim, that'd seem a poor choice - aren't refusing to federate because the OSS fans might not like it. Quite the opposite, actually.

Matrix will hit exactly the same problem in a few years. The solution is certainly not to restart with a clean slate every few years and fucks to you if you wanted compatibility. That's Google's approach, where every few years they trash an old service and - maybe - replace it with a new one that works in a different way.

There's no simple answers to this. I can tell you it's absolutely not a protocol issue - the protocol issues are ensuring graceful degradation remains possible during advances, which XMPP does well - but a political one. Profile specifications, which indicate groups of XEPs which the community expects to be supported, are part of this. Certification might also need to happen. Maybe monetary awards, even.

One of the biggest things currently driving server implementors to get the XEP support, and server maintainers to deploy it, though, is folk like inputmice driving the market with Conversations.

Oh, okay - in that case I slouch corrected.

A device-global push protocol isn't entirely useless, since the device OS can wake all the clients at once if it needs to, synchronizing network activity. But I agree it's a significant overhead if you're able to just hold open the TCP session, as well as concentrating yet more information into a single entity you don't get to choose.

So either XMPP is bad because it hasn't changed for years and is therefore old, or else XMPP is bad because there's new replacements for stuff that didn't work so well.

Honestly, there's some days I wonder if there's any way to win.

What? Openfire has had XEP-0136 for literally years. I think a little over a decade, actually. It's only recently implemented XEP-0313.

Also a later version at http://www.stateslab.org/HowToWriteUnmaintainableCode-Green0... which includes that crazy offbeat language, JavaScript (which was, amusingly, my contribution).

Actually a mixture. Some shortcomings have vanished because while they're problems, they're solved problems. And adoption etc, too.

I'd note that I was never against XMPP, though; I just don't see the need to pretend it's perfect. I do, however, object when problems are claimed for it that it doesn't have.

The Holleriths (later IBM Tabulators) were also used extensively in Bletchley Park and the Heliopolis station as part of the technique for cracking Italian Air codes - explaining why there's one or two knocking about in the National Museum of Computing in Bletchley.

Worth noting that two of the people fairly heavily quoted on the shortcomings of XMPP both serve on the XMPP technical Council at the moment. (One of them is me, the other is Philipp Hancke).

Thanks for picking up on the important content there. ;-)

If you want to be pedantic, Merlin - assuming he existed in the most likely timeframe - probably was aware of Angleland to his east, and would have been defending against its encroachment. (This based on the rise of Arthur as a popular name around 550AD as I recall, during the "Saxon" invasion largely carried out by Angles). The country he lived in was probably called something like Britain, though may have simply been closer to Cymru.

But I was largely paraphrasing T.H. White's Once And Future King, which in turn cribs the La Morte d'Arthur, which forms the basis for the "modern" Arthurian legend, and recasts it in an idealized England.

Really? The old Google Talk service used to do this, but I didn't think ejabberd did. There's a bunch of non-IM use-cases which need presence-less messaging.

You might want to read it. :-) The problem has mostly been that nobody's come up with a better JSON representation than one in a April 1st XEP (and they have been proposed - as ralphm says, they're mostly XML-with-curly-brackets, which makes them have the disadvantages of both).

On the other hand, both stanza.io and XMPP-FTW both use the same fixed-format (ie, inextensible) JSON format for XMPP just fine. They can do this because they drop unknown namespaces and don't intend to provide round-trip conversions, etc. So if you're after a JSON "interface" to XMPP, there is one, and it's pretty good.

Worth a look.

You'll be glad that someone got the IMAP reference... (Unless you're as niche as I, and were referring to ACAP).

I don't know how accurate Android's power estimates are, but I consistently have people telling me that they are switching back to proprietary messengers because every XMPP client they've tried drained their battery. Whatever causes it, there's a real problem here.

I honestly cannot help you. Could be that my usage patterns just happen to make the battery usage better (though I'm a heavy user of multiple accounts), or could be I'm a really serious Solitaire player.

I do know that over the past year or so the situation has really improved, though (mostly with the merging of aSmack and Smack, but Conversations too).

The problem is that theoretical future benefits are worthless if you never get a base implementation going. XMPP is just too hard for many developers - it's not a thing that you're going to understand well unless you work on it full-time, and that's not acceptable for a protocol.

I mostly agree; I'd hope that we could get to a place where the requirement to "know" XMPP was roughly that of needing to "know" HTTP. So simple use cases could be done without truly understanding the guts.

Hitting XML directly is almost always a requirement. Libraries are woefully incomplete due to the large number of XEPs, and to implement anything yourself, you must understand the XML structure.

If the intention of XMPP or XMPP frameworks was to abstract away the XML, then they have failed.

I'll confess to not being your typical XMPP developer, but I was under the impression that successfully hitting the XML with libraries like Smack was actually quite hard.

I experience constant issues with OTR, most servers don't seem to even support message carbons (nor do major clients like Pidgin have any useful indications of it), and frankly, I have never seen a single 'regular' user get synchronization set up correctly.

OK, so Pidgin is a multi-protocol client which means the UI is abstracted out... But... https://developer.pidgin.im/ticket/15508 I don't understand their logic there, I have to say.

For OTR, by the way, that just breaks with multi-device anyway. There's some experimental work with Axolotl underway which looks very promising.

This is really a pervasive problem in the XMPP ecosystem: everything is too hard. Little to no thought seems to have gone into the UX, and unless you are an 'experienced XMPP user', you really have no clue what is going on. The experienced users say "meh, works for me" and don't set out to fix it.

I won't disagree there. The lack of a consensus on UX considerations (in fact, the lack of any attempt to get such a consensus) means that different clients present even simple things like jids differently. That's something the community (or the XSF) could seek to address.

This is true for the frameworks, the clients, and most of the servers. It's just all-around user-hostile, and that needs to change.

And again, I agree. It's sad, especially, that you qualify servers yet not the clients - I'd rather see clients as really easy. I mostly think Conversations is there, mind, but I'm hardly the target audience here.

That said, I'm aware that one key problem faced by a federated network is that client on-boarding can't, so easily, create your account for you, and early attempts to streamline that process have left serious abuse problems in their wake.

I use Gajim on Linux and Conversations on Android, but the technology you're looking for is "XEP-0280: Message Carbons".

FWIW, I added the following to the Wiki page when I saw it was on HN, which might help give some context to what you're reading:

Hi. I'm Dave Cridland. There's a note just up there which says "(Draft blog post)", and that's been there from the start - but when I asked for some feedback from the public standards@xmpp.org mailing list, people posted it on Twitter, and then it got onto Hacker News. Which is all cool, it's not secret, but...

This has not been properly reviewed yet (though the community is starting to), and so may contain inaccuracies, errors, or simply bad wording that's inadvertently as misleading as some of the comments I'm trying to rebut, and so feedback is genuinely welcomed.

Also equally welcome is feedback of the kind that says "Yeah, but I think that XMPP is entirely made by KILLING KITTENS", or whatever, so I can either write a rebuttal, or agree with you and try to get the community to fix it.

This is, currently, my personal opinion, and may not represent that of the XMPP community or the XMPP Standards Foundation.

Just for the record, I don't think we kill any kittens.

The specific claims I was responding to when writing this (and it's still a draft) were an HTTP-based JSON protocol whose FAQ makes all these claims and more. And yeah, I think it's dumb to do things that way (both JSON/HTTP and making ill-informed and misleading comments in your FAQ).

As to your comments:

(1) I'm just not seeing this. Even without mobile-style push, apps like Conversations appear way down the list in Android. Note that because XMPP maintains a continuous TCP session (or WebSocket, or whatever), then you get "push" in a general sense; that is, there is no polling.

Using an XMPP client on your mobile will, of course, have an impact on battery life, but I don't think a good client will drain your battery. For me, Conversations uses less power than Solitaire according to Android.

(2) I'll agree that XML does make the development of XMPP libraries more complex, but we gain because it grants very simple permissionless extension capability, and that's hugely valuable. If developers working with (rather than on) XMPP are having to hit XML directly, that is absolutely a problem.

Finally, resources are the way they are for all sorts of reasons, but Message Carbons really does work for having conversation sync across multiple devices. I often walk away from my desktop and continue conversations on my mobile and/or tablet, and wander back to the desktop when I feel the need to continue with a keyboard.

Hope that helps.

http://matrix.org/docs/spec/#identity

""" Users in Matrix are identified via their matrix user ID (MXID). However, existing 3rd party ID namespaces can also be used in order to identify Matrix users. A Matrix "Identity" describes both the user ID and any other existing IDs from third party namespaces linked to their account. Matrix users can link third-party IDs (3PIDs) such as email addresses, social network accounts and phone numbers to their user ID. Linking 3PIDs creates a mapping from a 3PID to a user ID. This mapping can then be used by Matrix users in order to discover the MXIDs of their contacts. In order to ensure that the mapping from 3PID to user ID is genuine, a globally federated cluster of trusted "Identity Servers" (IS) are used to verify the 3PID and persist and replicate the mappings. Usage of an IS is not required in order for a client application to be part of the Matrix ecosystem. However, without one clients will not be able to look up user IDs using 3PIDs. """

OK, so aside from that last paragraph, that reads very integral. If it's just a case of "you have an identity at a domain" by default, and the 3PIDs are all an extension^Woptional-feature-that's-part-of-the-baseline, then what's the "strong identity system" you claim XMPP is lacking?

Actually if you read through that Twitter search, you see a bunch of XMPP folk getting annoyed at you for "half-truths, as usual", you referring to XMPP as a failure, you claiming that only the baseline counts, you claiming that MUC - universally and interoperably supported in every server (and every client that wants it) - is fragmented.

If this is you being supportive, I'd hate to see your actual disinformation campaigns.

Now, if you actually want a constructive conversation, that's great, but trolling just isn't the way to do that.

If you'd like a case where I suspect that Matrix models better, it's that ad-hoc, "ungoverned" multiparty chats work better in Matrix than XMPP.

That's because XMPP's multi-user chat model is (intentionally) based around IRC-channel-like models, where there's a single identity and authority. As I understand Matrix (and I don't claim expertise here), Matrix instead models a multi-party chat as simply a conversation involving multiple parties.

OK, so your website claims that identity mapping is an integral key feature, so if I'm spreading FUD I apologise for repeating what's on your website.

As to whether XMPP and Matrix are competitors, you make direct comparisons between them (with incorrect assertions that have been pointed out to you before), and since you attack XMPP constantly on Twitter et al, I take it you think it is a threat.

Specifically, from your website's FAQ: - There's around four or five XMPP/Web implementations allowing you to easily speak XMPP from a browser, but the two most popular with web developers seem to be stanza.io and xmpp-ftw - the latter is JSON objects via HTTP/Websocket. - Most server implementations cluster, so a chatroom is highly available across multiple physical servers within the same domain. This latter restriction can be avoided using FMUC. - Hedging your comments with "(without extensions)" is crass, since there are many extensions that have very wide support. - Talking about a minimal baseline is at best ignorant, at worst deliberately misleading. Chatrooms aren't in our baseline, but that doesn't mean we should ignore them. - "Not particularly suited to mobile". We've actively worked on push recently, but on Android it's really optional. As for bandwidth efficiency, you use HTTP, so that's laughable - as noted above, stanzas go across HF radio in their native format just fine.

Yes, you can chuck messages around fast in XMPP. You can also chuck messages through pubsub systems very fast in XMPP.

So Matrix is a brand with no legal entity behind it, where the people operating it and controlling the specification seem to be entirely employed by the same company who owns the domain and sponsors an extensive publicity campaign? I shall never suggest it's merely branding for a commercial outfit again, my deepest apologies for having clearly misunderstood the situation.

As for getting defensive, I admit that too, but it's quite common when being attacked.

Matrix do have a much bigger marketing budget than the XSF, it's true. They fly their employees all over the globe, and appear to have a constant stream of FUD on tap about XMPP, presumably because it's their strongest competitor. Last I saw, though, it's got a thumping great single-point-of-control in the middle, firmly under their corporate grasp. Nothing too critical, though, only your identity. (Looking at the FAQ, the "Can I run my own Identity Server?" question is simply missing, but I can answer that one: No)

XMPP's primary distinction is that all services are fully partitioned by domain. The only shared state - and I use this term very loosely - is in subscription data, which is long term. (By subscription data, I'm intending to mean the roster, pubsub, and also MUC chatroom occupancy). Matrix attempts to share considerably more state, acting much more like a single chat service scattered across multiple servers, but due to the centralized and privileged identity servers, there's a significant loss of autonomy in the system. Important to note, an XMPPservice on a single domain can be spread across multiple physical servers, fully sharing state, in which case you're well into the territory of vector clocking and CAP theorums, but that's an internal matter for your service and doesn't affect the inter-domain federation at all.

In layman's terms, you can run an XMPP service without reference to any external system - although federation would need you to reference DNS. Running a Matrix server leaves you somewhat under the control of Matrix, which despite appearances is just a branding for a commercial outfit.

Well, that, and XMPP is an open standard according to a definition we didn't just make up as part of our marketing. (see OpenStand.org)

So first off, it may be useful to note that when looking for a universal, interoperable standard for text chat, the armed forces around the world went for XMPP. This based on its reliability and handling of low bandwidth situations (much, much, MUCH lower bandwidth than mobile). So when you say XMPP doesn't support your notion of an MVP, it's worth noting it really is just that - your notion. That doesn't make you wrong, but it may be that your focus is different from other people's.

And as a second point, when considering writing something on the web, you (thankfully) don't have to implement JavaScript. Or HTML. Or CSS. Or, thank heavens, HTTP. Do you even know what happens in HTTP? Clearly not, otherwise you'd be a gibbering wreck - it's about as badly designed as a protocol can be and still work. And don't get me started on TCP; nobody gets that right.

So really, if you're implementing something using XMPP, please don't leap in at the low level - just use one of the many libraries for your chosen platforms.

Next: Unique message identifiers are an interesting topic, and you've referenced several XEPs that don't address the problem. One issue is that what's needed varies in different situations - generally, a short-term scoped identifier is good enough, but it's fair to say that's not always the case. The tuple of <from,to,id> is good enough for this, though clients aren't mandated to send an identifier. It's not even clear to me that we need global stable persistent single-valued identifiers, but despite there being no use-case explained to me where this is required, I'm aware it's a popular solution some undefined problem.

Worth noting, too, that XEP-0198 and XEP-0184, though often confused as addressing the same problem, actually address entirely different ones. '198 addresses link reliability, tackling the two generals problem. '184 addresses end to end acks. You end up with acks in both, but 184's can be out of order, and 198 allows session resumption and retransmissions, which addresses the same cases you identify with vector clocks between domains. Your library should handle all this for you, and just note if a message is lost.

Carbons you have entirely wrong, sorry. It does not involve "re-sending your messages back to the server", it just allows non-priority clients to get copies of all messages. The only oddity is that the copies are just that - they're not duplicates. This allows the client being copied to identify that the messages are going to a different device. Which device is usually based on "priority", though different servers can do things slightly differently. Despite this, it doesn't have anything to do with message traffic. And it's not random.

However, clients typically respond to the device which sent the message - but if the sender has moved devices, they'll get a carbon. And yes, there is a massive "if supported" right there, but there are desktop clients that do.

In fact, we don't handle the "every device has the same view of the universe" concept - you're right - we handle the "every device knows the state of every other device". It's functionally identical, but a slightly richer view.

So, let's rewind.

The XSF doesn't write software, though there are many libraries out there. We're working on Push, mostly to cover Apple's platform where long-running sessions are prohibited - this despite the fact that on Android, clients like Yaxim or Conversations don't excessively use power.

The XSF also doesn't do the levels of social media advertising that Matrix does. This is becoming increasingly problematic for us as a community, because Matrix are explicitly targetting XMPP with things that are simply untrue.

Most of the problems you identify are actually solved at the protocol layer. They really are. And it's not as bad a solution as you imply, either - in fact, the solutions to multi-device messaging and archiving are pretty good, and more based on "every device knows about the other devices" rather than "every device is treated identically", which is interesting in and of itself.

Oh, and when I said low bandwidth? XMPP is routinely used over 2400bps SATCOM links with 30s latency, but it'll run lower. Try 9 bits per second, half-duplex, with 2 minutes link turnaround.