JMAP: It’s Like IMAP but Not Really (2019)

https://unencumberedbyfacts.com/2019/01/24/jmap-its-like-imap-but-not-really/

by jbellis • 4 years ago

184 131 4 years ago

bee_rider• 4 years ago

So, it isn't just IMAP but for electrical engineers? (/s)

YPPH• 4 years ago

Is it too late for any significant improvements to email? It seems like, to our lament, we're headed in the direction of proprietary instant chat services and protocols.

Also, you can't overlook that a significant number of people use Gmail/Google Apps and Outlook.com/Office 365 which each use a proprietary communication protocol that is unlikely to change in the future.

So, it does make me wonder, who does JMAP benefit?

mcint2• 4 years ago

JMAP benefits people who want to use broadly accepted, open standards. Especially standards better adapted to present and future needs: mobile use - lower latency & fewer round trips, lower end-user device power usage.

JMAP benefits FastMail (and others). The CEO of FastMail is also a heavy-lifter in standardizing JMAP, Bron Gondwana. It's probably appropriate to mention he's also a lead developer of the Cyrus IMAP free-software project, so it's hard to accuse him of making a private moat for his company.

Self-hosters, and small businesses, who can run software, and possibly contribute, but not write their own email software from scratch—-benefit.

JMAP is targeting email, and soon calendar, contacts, and maybe more general messaging, to run better suited to current network and user conditions.

https://jmap.io/

https://www.ietf.org/blog/jmap/

Who does TCP benefit, JSON, .well-known?

who does vcf/vCard benefit, or .ics (even if it started as a proprietary format)?

mcint2• 4 years ago

The multiplexing of requests has me pretty interested.

If I made myself the spare time, or had a business model, I would love to implement chat bridges (for personal, self-hosted) small-scale use, using JMAP.

I like, I'm hopeful for the potential of, the promise of, REST-ful state transfer but with multiplexing of requests internal to the protocol.

Think of the protocol as sending a git-patch to modify state, instead of sending a diff per file, "resource".

Analogous to how QUIC (which is still adding standardization features) works around the problem of head of line blocking on TCP connections (and on http2's HTTP sequentially multiplex-able response streams), a core theme supporting in existing JMAP for email & attachments is multiple application requests per web-request, on-the-wire payload.

Said another way, I appreciate the multiplexing in: - per web request, - per application request (JMAP adds multiplexing here) - per resource, accessible to the user, on the server, - you can create, read, update, delete, move, etc.

brongondwana3• 4 years ago

Thanks for the callout :) I'm not doing so much development on Cyrus any more - I still attend the weekly team meetings and do code and design review, but trying to keep my focus more on company operations and standardisation work.

JMAP core and email standards haven't changed in the past few years because they're done! In the JMAP working group at the IETF we're currently heavily focused on getting the calendaring and contacts bits finished. That unlocks significantly more benefit for a JMAP client because you don't need multiple different authentications.

As for email clients - yeah, it's been slow going, partly because when COVID hit we all shrank back into our shells a bit, and certainly my travel and networking was cut back! It's good to see how it's perceived from outside our little bubble where JMAP is doing great and we're moving more and more of Fastmail's internal APIs to be JMAP-style because it's so easy to multiplex requests together and route them separately through our middleware into different systems, then combine the results back to together in a single response to the calling system or browser.

throw0101c• 4 years ago

Especially standards better adapted to present and future needs: mobile use - lower latency & fewer round trips, lower end-user device power usage.

IMAP development started in the 1980s (RFC 1064), and was run over POTS modems for many decades: I find it hard to believe that it can't handle the 'low-end' of things.

The fact that many clients assume high-end / high-speed links is not a fault of the protocol.

georgyo• 4 years ago

Also, you can't overlook that a significant number of people use Gmail/Google Apps and Outlook.com/Office 365 which each use a proprietary communication protocol that is unlikely to change in the future.

All of those support IMAP currently, an open standard. Exchange has MAPI as well.

Email is the best federated system we might ever have, and it has thousands of clients that can work with nearly any provider.

JMAP has not taken off, but there is a chicken and egg problem. Most clients don't support it so there is no pressure on services to support it. Though I imagine if Gmail supported it then suddenly all the clients would support it

josephg• 4 years ago

All of those support IMAP currently, an open standard. Exchange has MAPI as well.

Gmail supports IMAP, with custom extensions to support threading. As far as I understand it they use a custom protocol for their own gmail apps because of IMAP's shortcomings.

Its basically impossible to build a modern email app directly on top of gmail's public APIs. You can do it - but only by mirroring and re-indexing all the email.

Gigachad13• 4 years ago

Email has already largely become a newsletter and notification distribution system. Average people don't write or read emails anymore outside of some work settings but even then its usually Slack or MS Teams.

Technology and people move quickly so fossilized systems such as email get left behind. There is too much friction to improve a system which has a large number of different implementations. Where any change is hotly debated and people still resist features added to email over a decade ago. Naturally the proprietary systems win when they can roll out a new feature and have it universally work on day one.

alberth4• 4 years ago

Average people don't write or read emails anymore outside of some work settings but even then its usually Slack or MS Teams.

Do others believe/experience this?

I ask because this sounds like a very Bay Area comment.

trasz• 4 years ago

fossilized systems

What a weird way to call a technologically superior solution.

tengwar2• 4 years ago

This is not the case. One of my jobs is taking funerals, which means that I meet a random selection of the population, of all social classes. This includes people down to about 20 years old, not just the older relatives you might expect. The only people I meet who do not use email are a minority of people over 80 years old (perhaps 3% of my total contacts), and pretty obviously they are not using the more recent proprietary platforms either. Sometimes clients will use messaging for something short like an update on the time of a meeting, and occasionally they might use Teams or Skype for meetings rather than face to face, but the two great constants are voice telephone calls and email.

I think you may have become bubbled.

verisimilitudes4• 4 years ago

Yeah, what can't be fixed by shoveling JSON shit into HTTP instead of a dedicated protocol?

JMAP was stupid when the Fastmail CEO or whoever was here pushing it, and it's still stupid.

ikt• 4 years ago

Why do you call JSON shit? It's incredibly useful and I use it all the time, am I missing something?

shakna4• 4 years ago

JSON has a few rough points that make it frustrating to parse. Like for example, most values requiring open and ending tags, no trailing commas allowed (so you must either be lax and non-compliant or backtrack the parser on last element), and other similar edge cases.

This results in making it extraordinarily easy to break most parsers in use, today. For example, to kill Python:

    n="$(python3 -c 'import math; import sys; sys.stdout.write(str(math.floor(sys.getrecursionlimit() - 4)))')"

    left="$(yes [ | head -n "$n" | tr -d '\n')"

    echo "$left" | python3 -c 'import json; print(json.loads(input()))'

That's a less than 1000-character long object, that can explode. Whilst there are more robust parsers you can get for Python, it doesn't change the fact that generating bad JSON is _easy_.

So when you're handling something like email, on a mobile device which may drop a partial connection, you need a hell of a lot of extra work to show the partial download, which might even be everything except the closing braces, or you have to discard the whole thing - which isn't kind to your user.

verisimilitudes4• 4 years ago

S-expressions are better, and something that sees so much usage should have a dedicated format. In a saner world, this work would already be done so that each protocol needn't define the octets on the line. Regardless, IMAP already exists.

It's a diseased mind that takes a shitty programming language, JavaScript, and turns it into a shitty structured data format.

Gigachad13• 4 years ago

No you aren't missing anything. Most HN users have some weird nostalgia about obsolete technology. Custom protocols and XML are good, JSON is bad because Facebook and Google use it to deliver adverts and it reminds them of Single Page Apps.

yyyk2• 4 years ago

IMAP is an ancient behemoth in strong need of modernization. JMAP is the replacement we have. Could modernization have been done with a dedicated protocol? I guess so. But no one (including you) did the work to get it done, so JMAP it is.

verisimilitudes4• 4 years ago

That's not the way this works. I needn't accept the WWW garbage graphomaniacs attempt to force upon others.

masklinn5• 4 years ago

Could modernization have been done with a dedicated protocol? I guess so.

But probably not because it adds an other layer of friction and complexity to deployment, which doesn’t do much good to your odds.

And it requires an entire additional ecosystem of inspection and protection capabilities to appear.

Not to mention the parsing and serialisation of a bespoke protocol, and the risk of all the usual framing and escaping errors in the protocol.

It also requires going to Iana for a port.

denton-scratch4• 4 years ago

so JMAP it is

Except it isn't. As you say, IMAP could do with modernisation; JMAP isn't that, it's a replacement.

If you want to replace a protocol that's in very widespread use, then your replacement needs to offer decisive benefits to operators and users, otherwise they won't switch. JMAP seems to offer benefits only to developers.

WesolyKubeczek• 4 years ago

(shrug) I could probably say about IMAP that it looks like it has been developed by hardcore Lisp fanboys. Boils down to someone’s yum being someone else’s yuck.

brongondwana3• 4 years ago

Hey, I love IMAP. For all its faults it is pretty powerful and good.

I've spent a lot of time on it, but the reasons that JMAP was developed still exist and are still valid :)

pjscott• 4 years ago

IMAP is a protocol that you can only love if you haven’t had to try to implement it. I say “try to” because there are so many incompatible almost-implementations that you will never actually implement IMAP.

Replacing it with something more sane seems like a good use of time.

verisimilitudes4• 4 years ago

I'd rather implement IMAP than HTTP, TLS, JSON, and whatever other shit is involved here. All of those points apply to the WWW more than anything else.

andris92• 4 years ago

If you want to use JSON and paging but are stuck with regular IMAP servers, you can use https://EmailEngine.app between your app and the mail server.

kyrra• 4 years ago

2019.

layer8• 4 years ago

See here for a list of software supporting JMAP: https://jmap.io/software.html

derekzhouzhen• 4 years ago

There are 3 different kinds of email services that only have limited interoperability:

* Corporate email services, aka Outlook/Exchange

* Surveillance email services, aka Gmail, etc

* Real email services that expose the raw email database, such as Maildir, to the users.

For the first 2 kinds, I'd rather POP and keep my interaction with them to the minimum. For the last kind, IMAP/JMAP is too limiting in capability.

syntaxfree• 4 years ago

Now, to build a better IMAP on top of UMAP.

superkuh3• 4 years ago

JMAP sounds like a crutch for mobile devices that can't keep a TCP connection open. The internet is slowly being gimped to support mobile devices.

aetherspawn• 4 years ago

Pretty much every device is mobile now. For example, people will probably start reading emails on the center console of their autonomous cars on the way to work over 5G.

Nevertheless, a valid current-day example for non-mobile devices requiring a tolerance to dropped connections: I regularly dock and undock my laptop when heading to meeting rooms, and simply switching from docked Ethernet to WiFi causes a network switch that breaks a lot of apps. Engineering tolerance for this kind of thing is only good.

superkuh3• 4 years ago

Yes. That's reality. Most devices people use are terrible, terrible computers and they can't do what computers used to be able to do. This is a bad thing.

Gigachad13• 4 years ago

There is a lot of writings explaining that the reason open email clients largely suck / become unmaintained is that IMAP is extremely difficult to work with. It's not just the connection, you also have to parse and work with this custom protocol as well as deal with massive inefficiencies. Doing bulk actions like moving emails between folders requires massive amounts of requests rather than a single one describing the change.

JSON protocols have completely taken over the market because they are massively easier to deal with and trivially parse in to structs which don't require a 200 page document to detail the protocol.

kitsunesoba• 4 years ago

Most modern languages have a JSON encoder/decoder in their standard library too, making for one fewer dependency to worry about and that's never a bad thing.

e_y_• 4 years ago

I've written multiple IMAP clients, and I agree it's a huge pain to work with. But the real issue is the API design and the fact that more advanced searching, synchronization, etc, is only available via extensions that are not guaranteed to exist on a given server.

You can write a JSON API that's equally bad, doesn't support paging, doesn't let you select just the fields you need, is missing key information in the extracted fields forcing you to parse the entire RFC822 text as a fallback, etc. Dealing with the parser is just one part of the battle.

mlyle3• 4 years ago

Less-stateful and less-connection oriented are good for things like this. Lessons from NFS and HTTP.

nick__m• 4 years ago

  Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”.

So it's an adhoc reimplementation of SOAP using json instead of xml?

masklinn5• 4 years ago

Tunnelling everything through post is the normal way to do e.g. rpc over http. It’s not a soap thing, it’s also how xml-rpc, json-rpc, and graphql work.

More generally it’s how you layer over http when you can’t necessarily guarantee respecting its requirements like purity, idempotence, or cacheability (or don’t want to bother), or when you fear hitting it’s practical implementation limits.

p_l• 4 years ago

And SOAP inherited it from xml-rpc.

Octoth0rpe• 4 years ago

And more importantly, that's _extremely_ un-REST-like, at least in the modern use of the term.

encryptluks23• 4 years ago

I want hashes for email and not just UIDs. Does any protocol do this currently where it hashes the entire message?

andris92• 4 years ago

Email hashes are not very useful as additional headers are added to the email every step on the way. Even if you send an email to yourself, the copy in the Sent Mail folder has a different hash than the one in Inbox.

encryptluks23• 4 years ago

They are useful for verification though that an email is the original, and for detecting changed messages when syncing.

newscracker• 4 years ago

JMAP sounded promising several years ago, and it was great to hear it published as a standard. But it seems like it’ll just remain as what Fastmail (the email provider) uses. Maybe the JMAP team is very small and can’t get more people. Maybe Fastmail doesn’t have enough money and/or enough interest to make it easier to use and to evangelize it with major email providers and email client developers.

I don’t see Outlook365 or Gmail (or even Apple for its iCloud email that has a relatively smaller user base) announcing any implementations of JMAP on their services. I don’t see smaller paid email service providers announce support for it. As it stands now, I don’t see a future for JMAP outside of Fastmail.

A popular client like Mozilla Thunderbird still hasn’t started implementing JMAP [1] and one of the reasons is because…the client and server documentation for JMAP haven’t been updated for a few years. [2]

So yeah, JMAP is like IMAP, but not really. We probably need a successor to JMAP that is liked enough to be implemented by email service providers and by email clients.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1322991

[2] https://github.com/jmapio/jmap/issues/323

brongondwana3• 4 years ago

I have more faith in JMAP's eventual success than you do, but I do agree that it's frustratingly slow. We knew that adoption would be frustrating slow of course, but I do hope that in 3 years' time we'll have a much better story than we have had in the three years since initial standardisation of the core and mail parts.

Certainly once calendars and contacts are done, and hopefully tasks as well, that will make a more complete ecosystem that is even more attractive than just the email component!

Loic• 4 years ago

My personal take is that JMAP would start being widely used if we had some simple Python/Go bindings to easily manipulate the emails.

My first reaction as I have seen this standard popping up was not "my email will be faster/better", but "I will be able to setup a nice support@mycompany.com" email address and have both email/web integration with it.

If developers start to embrace JMAP for a derivative usage, I think it could drive the adoption.

alberth4• 4 years ago

Hi Bron - love Fastmail and the innovation you leading.

Question: what do you think is the rough timing on Calendar/Sharing/Contacts/Tasks to be ratified? Within the next year, 2-years, etc?

x0x0• 4 years ago

Are you all thinking about, or planning to allow plugins?

eg say I wanted to use jmap to implement some of hey's functionality -- for instance the screener that isolates and allows ban of new emails on first contact. Rather than a client, can I have a server that listens to my fastmail inbox and (potentially) moves emails as they're delivered?

Any chance you're thinking about allowing UI space for things like that?

mariusmg• 4 years ago

"to evangelize"

It's sad that superior standards need to be evangelized to even hope to gain traction.

robert_foss2• 4 years ago

I've had it confirmed by on of the top two mobile OS providers, that their email developers have JMAP on the roadmap.

froh• 4 years ago

did they share some ETA?

pjc50• 4 years ago

Outlook365 or Gmail

Why would either of these build a bridge over their lock-in moat?

efdee• 4 years ago

Weird comment considering how both of them already support POP3 and IMAP.

ForHackernews• 4 years ago

Maybe Fastmail doesn’t have enough money and/or enough interest to make it easier to use and to evangelize it with major email providers and email client developers.

All the other major email providers are selling their own proprietary services or only offering webmail clients.

They have no interest in implementing modern open specifications. They only support things like SMTP and IMAP for legacy reasons.

jasonmunro3• 4 years ago

You are likely right about all of these things. I agree commercial E-mail providers don't currently have a reason to offer JMAP support. Thunderbird already has robust IMAP support so why change it. But JMAP provides a new avenue to E-mail client development that is both easier for existing clients to deal with and encouraging for new development.

aorth• 4 years ago

On a related note: the author links to his open source web mail platform Cypht. It looks cool and sounds promising. Anyone tried it? I can't remember if it has been on HN. The git history of the project is very active.

degrees57• 4 years ago

I have the Helm email appliance, and what it does _not_ come with is a webmail interface. (It does have Nextcloud, though). I put Cypht on a different box to access my Helm mail, and it was very useful.

At the moment it is down because I did a migration from one host to another, there is some sort of login problem on the new box with the old database. RSN I'll have time to diagnose and fix.

jasonmunro3• 4 years ago

Hello, author here! Not sure who posted this old blog article but great to see an interesting discussion and new interest in the project. Development is active and we always appreciate new feedback!

qwerty456127• 4 years ago

I would go further and also replace SMTP with the same for clients. As I understand we still need SMTP to pass mail between servers but a client already having access to IMAP/JMAP/whatever could just put the mail into the outbox folder for the server to deliver it.

jeltz• 4 years ago

Agreed, I think the separation between SMTP (as used for submission) and IMAP makes sending emails unnecessarily complicated.

Edit: From a quick glance it seems like JMAP actually supports submission.

asim• 4 years ago

Whoa I love SMTP and email, as a distributed systems nerd I'm a huge fan of the federated protocol and it's simplicity but JMAP I'm sorry is garbage. The http/json part is great, the potential to simplify email access and use is great, even using DNS SRV records is a potential good use of DNS and a compliment to MX records. But the actual JMAP protocol is horrible. This is something built incrementally for fastmail, it's not how you design a protocol. We need to go back to square one and redesign and access and communication protocol as one thing that's ultra simplistic as a building block and starts with very basic primitives. JMAP combines too many things while being very ad-hoc and custom not taking into account a lot of progress made in things like gRPC protos, etc. Tempted to build it from scratch just to prove a point.

lewisl9029• 4 years ago

Does JMAP do anything about the fact that it takes on average multiple whole number seconds, with frequent spikes into 10s of seconds, for email delivered to show up in people's inboxes? Source: https://status.postmarkapp.com/

If not, is anyone doing anything to improve this situation?

Sending packets over the internet across the farthest ends of the earth takes about 300ms roundtrip. What's preventing email from approaching this?

Latency is the biggest UX issue facing email IMHO, and a huge part of why it is continuing to lose ground vs alternative messaging systems, most of which are proprietary, but offer sub-second latency delivery. The world will become a worse place if the trend continues to its logical conclusion and proprietary silos eventually fully supplants email's remaining use cases.

The high average latency and even higher variance leads to people second guessing if an email they sent will ever arrive, and if an email they're expecting has ever been sent, resulting in crude workarounds like resend buttons becoming commonplace and seen as necessary, perpetuating the uncertainty around deliverability, and even further eroding trust in the system.

But most importantly, the latency problem makes email unsuitable for the most compelling use case for messaging platforms: real-time conversation.

Holding a spontaneous real-time conversation over email is painful, as anyone who has tried can probably attest. Both sides are constantly left wondering if the other side is simply taking their sweet time to reply, or if the email is just taking even longer than email usually takes to get delivered. Conversations that could have taken mere seconds end up taking minutes as a result.

This is why email has been consistently losing to messaging apps for fun conversations between friends where email's latency would suck much of the fun out of them, to Slack for mission critical work conversations where wasting time means literally wasting money, to live chat widgets for businesses looking to provide a customer support experience that delights, whereas conversations over email frequently tends to frustrate, the list goes on...

Solving the latency problem for email I believe is the key to slowing and possibly reversing the trend of email's slow but steady decline, and society will benefit massively from the continued prosperity of this rare miracle of an open system that managed to survive for so long despite all its shortcomings.

denton-scratch4• 4 years ago

unsuitable for the most compelling use case for messaging platforms: real-time conversation.

If that's what you need, you use a real-time messaging application. That's not what email is for. I disapprove of the incorporation of email functionality behind a messaging UI; I get fed up of sending a properly-composed email, and getting a one-liner reply, because it's hard to compose a thought-out, formatted email on a phone.

andyjohnson02• 4 years ago

The article explains the technical advantages of JMAP, but I'm not clear how (if at all) it benefits email users. Without clear end-user advantages its hard to see why email providers would want to implement it when they already support just-good-enough IMAP.

danielheath2• 4 years ago

Key things that are hard in IMAP and easy in JMAP are well summarized in this comment: https://lobste.rs/s/er97xr/so_when_did_pop_imap_become_legac...

andyjohnson02• 4 years ago

Global search is clearly important. The other "hard thing" are a little... niche?

AFAIK search in IMAP is folder-based, so you have to iterate over the folders, do multiple searches, and merge the results. Not elegant, and probably not fast, but it doesn't seem like a protocol killer to me.

jgalt212• 4 years ago

JMAP is nice if you sell self-hosting data mining software. One less connector to write.

0x445442• 4 years ago

Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”. A method is an action or query you want to perform like “give me the 20 newest messages in this mailbox”

Can the HTTP spec just add a CALL verb already?

jasonmunro3• 4 years ago

hrm, handful of new github stars in one day? That's odd. Must have gotten picked up by something. Oh, HN repost of a blog post I wrote 3 years ago, cool!

To sum up my thoughts about this thread:

Appreciate the great discussion! Wherever you land on the validity of JMAP, as a long (long) time IMAP client developer I stand by my position that JMAP is a huge improvement over legacy IMAP. Not perfect, but walks a reasonable line between what IMAP client devs understand and the modern world of APIs 30 years later. Is it REST compliant or a SOAP derivative? Most likely no and yes. Does it only benefit client developers? Probably, but that is a big benefit to the E-mail ecosystem. IMO the biggest barrier to new and innovative E-mail clients is the obtuseness of the protocol itself, and JMAP goes a long way to bridge that gap.

JMAP: It’s Like IMAP but Not Really (2019)

Related Stories