Making email more modern with JMAP

https://fastmail.blog/2019/08/16/jmap-new-email-open-standard/

by mfsch • 7 years ago

327 166 7 years ago

fastmail.blog

aquabeagle• 7 years ago

Fastmail should throw some engineering resources towards implementing JMAP support in some mail clients like Mutt.

javert2• 7 years ago

That seems like a pretty entitled attitude.

techntoke• 7 years ago

If they want it to be taken seriously by the open source community, might as well start with mail clients used by the top contributors to things like the Linux kernel and core open source libraries used in most Linux distros. This is how to help ensure adoption and making it a replacement for IMAP.

stonogo2• 7 years ago

The fastest route to MUA support is compatibility with existing MDAs like procmail, maildrop, and the like. At the very least a JMAP client that populates a local Maildir would get you Thunderbird, Evolution, mutt, and more.

namibj• 7 years ago

Could a fuse filesystem do the trick for maildir emulation? Assuming that'd be beneficial to a full maildir, but I guess it'd only be so at large sizes.

inputmice3• 7 years ago

The strong suit of JMAP (among other things) is getting access to specific properties of a server side parsed email. It will do the MIME parsing for you; separate attachments and text bodies from each other and for example allows a client to fetch only Subject and senders instead of the entire email. By brainlessly dumping all that into a Maildir you'd be giving up a lot of the benefits of JMAP. I'm a big fan of JMAP but I don’t see it replacing IMAP for (desktop) clients that will keep a fully synced Maildir locally.

qzygr2• 7 years ago

That's what the ircv3 guys did. I think it's a cancerous attitude: you get your protocol implemented regardless of whether it's good or not, since most managers of open source software can't say no to free code

kuschku• 7 years ago

I don’t think anyone in the IRCv3 team has provided free code to any other project just for implementing IRCv3 functionality. In fact, the opposite is true, most projects have to spend significant effort to support it — but believe that it is worth it.

danpalmer• 7 years ago

Maybe, but mutt seems like a poor choice?

They are providing an open source server implementation, which is a great way to increase adoption. Adding support to widely used open source mail clients is also a great option, but I doubt usage of mutt is anywhere near high enough to provide a good return on investment.

Implementing in the default mail client on Ubuntu might be worth it, or maybe the most popular open source web mail client that exists could be worth it. Ideally though they need adoption in something like Outlook, Gmail, Apple Mail/iCloud, etc, but these are all closed source so a) impossible to do without support from the vendor, and b) of limited use to increasing the community of JMAP implementors.

dbg314152• 7 years ago

So a few years back, everything was super fast on IMAP on my phone. But now it just crawls. On the Gmail App it's still pretty fast. So I just assumed it was slow because Google was being a dick and trying to get people to use their app instead of Apple Mail or Outlook. Is the protocol slow, or is Google throttling it?

javert2• 7 years ago

Well, IMAP works fine, so it's not the protocol.

Felz• 7 years ago

I don't know about throttling, but later Android versions are known to mess with email notifications.

See: https://github.com/k9mail/k-9/issues/3950

jivings7• 7 years ago

Having just this month implemented an IMAP client I can tell you it's blazingly fast for Gmail & Outlook.

It's actually much faster and more efficient than using the Gmail API.

dbg314152• 7 years ago

I just clocked it at 16 seconds to check for new mail in Apple Mail checking against a Gmail Account on an iPhone SE connected to Google Fiber. It feels pretty slow...

Meanwhile, same update vs same account in Outlook took about 6 seconds.

And in the Gmail App, about 4 seconds.

Not super scientific, just me looking at my watch and opening each app and seeing how long it took to update.

So maybe something with Apple Mail?

craftyguy3• 7 years ago

IMAP is fine. It's likely your email provider's fault.

tracker1• 7 years ago

Wondering if it could be your isp prioritization being lower because of the not http(s) port.

y_molodtsov• 7 years ago

Gmail has its own sync engine.

mfsch• 7 years ago

From what I gather it should be possible to use 2FA with JMAP. I‘m looking forward to no longer having to decide between using third-party email clients and properly securing the account that’s probably most worth securing.

u801e3• 7 years ago

What would be nice is if the servers supported using client side certificates as an authentication factor. It's possible to configure postfix and dovecot to support this [1], but I haven't looked into whether it's possible to require both the certificate and the username and password to authenticate.

[1] https://blog.mortis.eu/blog/2017/06/dovecot-and-postfix-with...

Freak_NL2• 7 years ago

Would WebAuthn support be possible with JMAP? That would really move things forward.

tialaramex• 7 years ago

One high level problem that jumps out at me is: When do we authenticate?

For web sites WebAuthn drops nicely into the existing login flow. The user knows what they're trying to do is log in, and so when they're prompted to touch the button or whatever it makes sense in that context.

But today most email apps just silently update all the time. So are we authenticating when the app starts and then re-using some credentials obtained to re-connect as necessary? Is that safe? Even if that's all we do (which might be safe) it means the user gets prompted for WebAuthn (to press the button) when the mail app starts, even if they aren't currently interested in email. If they dismiss it, when do we prompt them again? Because meanwhile they of course don't get any new email.

chrismorgan8• 7 years ago

For IMAP already, there are three main approaches to logging in that get used: use your main password for the service to access IMAP, use a separate token for each client (“app passwords” is the most common term of art, and what we call them at Fastmail), and OAuth, which can then have whatever restrictions you like, 2FA or whatever. IMAP providers each support one or more of these techniques. (At this time, we support the second. Gmail supports the third and, if you have “less secure apps” turned on, the first.)

The JMAP core deliberately doesn’t express opinions about authentication technique, because that’s a divisive topic where there is no clearly right answer. We shall see what happens there; conventions will rise.

For Fastmail’s internal JMAP usage, we have our own authentication flow, which will entail 2FA if you have that set up on your account. I cannot remark at this time about what our plans are around authentication for public JMAP access.

But this is my conclusion here: look, I love JMAP, but I don’t believe it actually changes anything on this front over existing email protocols.

simplecomplex• 7 years ago

Super happy customer of Fastmail of 7 years now.

It’s a joy to use. Fast, responsive, and reliable. No pointless redesigns or 10 second loading screens like gmail.

craftyguy3• 7 years ago

.. what does this comment have to do with the actual article other than keying off of 'fastmail' in the title?

Gaelan• 7 years ago

I mean, JMAP is Fastmail's project, and this is a post on Fastmail's site. Still a little off topic though

jhasse• 7 years ago

Fastmail's web interfaces uses JMAP under the hood.

gumby3• 7 years ago

I believe the filtering is for user-initiated search, right?

Would be nice to have some standardized support for sieve-like server side prefiltering. That's how I sort my mail into mailboxes, send spam (labeled elsewhere) to the spam trap, and the like.

eridius• 7 years ago

What would this support look like, in the context of JMAP? By the time the email is in your inbox it's already gone through server-side prefiltering. So the only integration I can imagine is a way to fetch and store Sieve scripts so you can edit them from your mail client, but that seems like a very provider-specific thing.

For example, Fastmail themselves support Sieve, except they don't give you a single editable script, instead they convert your structured rules and spam settings into 8 labelled sections in a Sieve script and give you 4 free-form edit boxes to insert your own code at specific points (so you can't edit the code that they synthesize for you). I don't know how a protocol like JMAP would expose this in a generic fashion.

gumby3• 7 years ago

Two mechanisms: 1, the ability to write arbitrarily nested if all/any/none of various parameters (any/specific header w/restricted reflex) for server-side searches. The Mac mail app has these for the client side, as does my rss reader. A simpler grammar than needing inMailboxOtherThan yet more powerful. Could be represented as a json tree. 2 - the ability to run that filter n incoming mail basically where sieve is run today, with a similar small grammar for specifying a small set of commands to run like file in mailbox, tag, etc.

Sir_Cmpwn3• 7 years ago

Though I have no complaints about the way they went about implementing it (open standards hooray!), as a mail client author and someone with lots of experience with mail in general - I'm NACK to JMAP. I think that it comes with some questionable design decisions (HTTP? JSON?) which on their face I imagine look fine to the typical HNer, but in many ways is a regression from IMAP. I would prefer to see the limitations of IMAP addressed with a few carefully written extensions to the spec - which is already extensible and has a thriving set of useful extensions in the wild.

By no means is IMAP a particularly good protocol, but it is extremely broadly supported and designed more with email in mind than JMAP appears to be, which seems to focus more on bringing shiny web tech to email regardless of its utility in such a context. It's also one of many closely interlinked mail-related specifications, and JMAP fails to integrate well with many related specs imo.

pjc50• 7 years ago

questionable design decisions (HTTP? JSON?)

Those are the premises for JMAP, from which the rest follows.

ori_b2• 7 years ago

Yes, which is why the above post finds it not particularly desirable.

tjoff5• 7 years ago

JMAP is literally the only positive thing that has happened to email in decades, please don't dismiss it (or do you know of anything else in the works that has any chance of success?).

HTTP and JSON are such an insignificant price to pay for the sliver of hope that email can be saved and not being in complete control of google et al.

asjo• 7 years ago

HTTP and JSON are such an insignificant price to pay for the sliver of hope that email can be saved and not being in complete control of google et al.

How would JMAP accomplish that, in a way that other protocols don't?

teknopaul• 7 years ago

Some interesting work was done on adding prescence and chat to email that im disappointed not to see in jmap. Be great if we could chat to anyone with a name and dns domain instead of being stuck in a silo like facebook.

Tharkun3• 7 years ago

The real positive thing that needs to happen to email, is putting an end to near-monopoly of gmail. Email is great. It's decentralized. You can simply roll your own. You can use whatever client you like. Apparently even using HTTP/JSON.

But you can't send mail to anyone using gmail, which is nearly everyone, unless you play by Google's rules. And even then they'll still put you in their spam folder, or just randomly blackhole you without any explanation or recourse.

JMAP vs IMAP vs POP is moot as far as I'm concerned. As long as my mail client continues to work, I'm good. If this gains any traction, I'm sure dovecot and the likes will add support for it (if they haven't already). But no amount of protocol fiddling can fix the fundamental issues of email centralization.

bad_user5• 7 years ago

I think HTTP and JSON are great.

You might not like it, but every programming language in existence has easy to use HTTP and JSON parsing libraries. It's basically a solved problem.

As a software developer I dread working with protocols that are not via HTTP because most often then not the clients are badly designed, leaky and I can't fix it by changing the client.

And I love JSON in spite of its inefficiencies, because it is easy to debug and there are really good JSON parsing libraries for the statically typed languages I'm using (Scala, Haskell). You basically declare your high level types and the library will derive (de)serialization and validation logic for you.

Also consider that this isn't necessarily about building full fledged email clients. Maybe you want an automated way to read an Inbox and import certain messages in your database. Maybe you build a browser extension that counts the number of unread messages. There are many, many micro tasks that developers would be interested in doing, if they had an easy way to do it.

And do you know what protocol does NOT have good libraries for every language in existence? IMAP.

It's basically not a competition from my perspective, HTTP/JSON wins by default.

TheOtherHobbes• 7 years ago

"We have libraries for these technologies" is a poor substitute for "These technologies don't really solve the problems that need to be solved."

The nice thing about the old internet protocols is that they tried to distinguish between abstraction and implementation layers. (Not always successfully or effectively, but they did at least try.)

Modern protocol designs seem to do the opposite. They start from a default implementation layer - usually REST, HTTP, and JSON - and then try to build the abstraction layer out of that.

This isn't necessarily a good approach. But it is a reflection of the difference in culture - the change from hackers who were comfortable with open-ended bottom-up development, and modern developers who seem to prefer snap-together development using standard libraries.

Tharkun3• 7 years ago

every programming language in existence has easy to use HTTP and JSON parsing libraries

That argument doesn't hold much water. Every programming language in existence also has easy to use socket libraries. Are you implying that HTTP and JSON are less error-prone than whatever-over-TCP? There's nothing magical about either HTTP or JSON that makes them the right tool for every job. But I guess when all you have is a hammer ...

qzygr2• 7 years ago

What's unappealing about HTTP and JSON? What's better about having a completely custom protocol and format?

Sir_Cmpwn3• 7 years ago

HTTP is not designed for persistent connections (WebSockets were hacked on later, but JMAP does not use them anyway). This turns a push protocol into a poll protocol, which is a stark regression. It's also got all sorts of other crap built-in which is really not necessary for this use-case but which an intrepid implementer will have to deal with regardless.

JSON does not cope well with binary data, which is common in emails. It fails to elegantly deal with the various email encodings which exist in the wild. Consider as well that all JSON numbers are floating point numbers, despite the fact that floating point numbers provide absolutely nothing of value to this spec and in fact are more likely to introduce bugs than not. And embedded devices can't deal with floats quickly or elegantly, but still need to implement them if they want to use JSON-based protocols. And for what gain!

In short, JMAP reinvents the wheel but worse for the sake of making it easier for web developers to build web shit around email. This kind of stupid change for change's sake is an epidemic in the webshit scene. IMAP is warty but it's fine. Let it be. When your only hammer is JavaScript, everyone else's thumb looks like a nail.

brongondwana8• 7 years ago

You're very welcome to join the EXTRA working group at IETF and propose the extensions that you think would make IMAP better. Apart from working on JMAP, I have also written an extension to IMAP and may write another one next year bringing some of the push stuff from JMAP back to IMAP as well. But JMAP is based on many years of working on servers, middleware and (at least on the web) clients.

We did actually extent IMAP with some experimental stuff first (cross folder threading and thread collapsing for example) but it never solved the batching and proxying difficulties and the cost of instantiating message numbers per-session that are un-avoidable parts of the IMAP model. By the way, engineers who work on other servers have said similar things to us - IMAP is costly to implement in terms of server resources when you're doing true concurrency, high availability, etc.

_urga6• 7 years ago

IMAP is costly to implement in terms of server resources when you're doing true concurrency, high availability

Yes, IMAP requires expensive coordination among multi-master servers to generate UIDs in such a way that won't mess with client synchronization. In other words, IMAP is not a fully distributed design, it assumes a centralized control plane (think Raft or Paxos), i.e. a single central server with many distributed clients, which is a shame for new systems building on CRDTs.

How does JMAP address this achilles heel of IMAP? Merkle tree sync?

More detail on the problem: https://news.ycombinator.com/item?id=20479011

jivings7• 7 years ago

It will be interesting to see how this develops.

I've spent the last month implementing and testing IMAP support for my app [1].

It has... not been very fun.

Before we were limited to providers with a REST API, so implementing IMAP should in theory allow us to support a much wider range of email servers with a much lower support burden.

Although the protocol spec is tight, it seems like some providers randomly bug out or hang [2], respond with custom error messages, offer bespoke functionality [3], or generally don't comply with the spec in breaking ways [4].

As a result I'm more worried about this going forward than I had hoped.

I'll now also happily support anything that can help modernize email, so I'll apply to the developer program and will be watching JMAP closely.

[1] https://leavemealone.app/

[2]: Yahoo Mail always down (https://downdetector.com/status/yahoo-mail)

[3] https://developers.google.com/gmail/imap/imap-extensions

[4] https://github.com/mscdex/node-imap/issues/775

mattmanser• 7 years ago

Having worked with IMAP before myself, the spec is very vague a lot. It's a bad spec.

We had our own IMAP server implementation and had all sorts of problems even between different versions of outlook.

Each project had an inbox which you could cc in, resulting in huge inboxes (for the 2000s at least). You could also drag and drop an email you'd received into the project folder to share it with colleagues. And we had mysterious disappearing emails, but only at some clients.

One (of the many) problems I remember is that messages could have an ID, but the spec declared that it could change between sessions, so one version of outlook stored that in a short uint (max 65535) because why would you need more for a temporary id. So have fun if you sent anything more than that. And of course the client had different versions of outlook on different sites so it took ages to even figure out what was happening.

Another was that you had to make several back and forths on every request, messages about the messages, to say the payload and number of messages, but the spec was gleefully vague about the order this should happen in. So back then what worked for one email client resulted in an empty inbox for another.

I don't think we ever got it working properly with Firebird.

jivings7• 7 years ago

Validation of UIDs is thankfully something I don't have to worry about. It looks like a total nightmare.

dboreham2• 7 years ago

Outlook is known to have a horrible IMAP implementation.

vinay_ys• 7 years ago

Could you highlight what you consider as vague?

Aloha• 7 years ago

I hope it does go somewhere, as IMAP isn't well suited to mobile, and neither is webmail.

jivings7• 7 years ago

Why is IMAP not well suited to mobile?

octorian2• 7 years ago

Its still far better than POP. Its really nice how you can let the server decode the structure of a message and then selectively download only the parts you actually want to display.

(Yes, I once wrote a mobile IMAP client.)

gumby3• 7 years ago

This is ironic as imap was designed specifically for mobile and remote machines.

The definition of those terms is quite different of course today, so my statement isn’t meant to contradict yours. I doubt Mark ever saw a laptop, at least when working with imap. Imap was a real step forward at the time.

tonyedgecombe• 7 years ago

In the past I've used a 3rd party component [1] for IMAP, it was like outsourcing all the work of dealing with the quirks of each server.

[1] https://sockettools.com

asaddhamani• 7 years ago

Your app looks really cool! I've been looking for something like this for a while now to deal with my crazy inbox. You've made a lot of right choices like opt-in cookies, credit-based purchases, and audit logs, so keep on keeping on! Thanks and best of luck!

jivings7• 7 years ago

Hey thanks! :) Glad to hear you like it!

Nevermindmine• 7 years ago

Although the protocol spec is tight

Are you talking about the IMAP RFC? It's the worst spec I have ever seen and so many parts are open to interpretation. If you think it's tight it must be because you didn't realized all the ambiguities.

jivings7• 7 years ago

My use case is pretty specific, I'm only dealing with a small subset.

I guess I was just lucky and can remain happily ignorant.

verisimilitudes3• 7 years ago

So, how long until everything is JSON-shit over port 80 and how long until port 80 is old news and only port 443 is used for anything and everything? I can't claim to be familiar with IMAP, but it can't be worse than this garbage that requires an HTTP client and a JSON parser.

To process JSON, you must inspect every character, which is reason enough it's an awful ''format'' that only an ignorant could hold dear. Douglas Crockford brags about how smart he is for disallowing comments, but there's whitespace allowed everywhere. Let's not forget this asinine integer nonsense, because JavaScript lacks real integers.

I'm sure most people think HTTP clients and JSON parsers are standard-library affairs, nowadays, though; let's just entirely ignore matters of licensing and whatnot.

It's disgusting to see this as an IETF standard, but people have been complaining about how that moves along for decades; take a look at Google's robots.txt ''standard'' that doesn't do anything useful and effectively makes every existing file standard, rather than do much of anything worthwhile. The WWW and JavaScript are nothing more than a continuation of this worse-is-better filth that UNIX and C previously championed; decades ago, the IETF revised the RFC for the Finger protocol and warned about security, solely because some idiot wrote a shitty Finger server in C; now, we have useless and unnecessary standards such as this that only serve to further this next generation of idiocy.

I wonder when all of this will finally collapse under its own weight.

_fbpt2• 7 years ago

Can you provide some actual arguments as to what's wrong with JSON? It's a text format that can be inspected by humans, unlike binary formats. (Maybe IMAP is a text format too, so this isn't an advantage inherently) And JMAP may have better semantics than IMAP, and there's nothing inherently wrong with JSON (other than edge cases).

verisimilitudes3• 7 years ago

Can you provide some actual arguments as to what's wrong with JSON?

It's a wasteful, textual format derived from a programming language vomited forth in under a week. It has asinine integer characteristics for no good reason; it's not a random-access format; validating it is even more work; hash table keys that appear multiple times, instead of being in error, instead overwrite each other, defeating most any kind of optimization; it is thus wildly inefficient to manipulate, as manipulating it correctly in all cases requires full and comprehensive parsing with checking. It is an asinine ''format''.

It's a text format that can be inspected by humans, unlike binary formats.

Even then, there are much better textual formats that could be used, but I'm of the opinion that numerical (binary) formats are strictly superior, especially as they're used increasingly often. It's better to have good tools that will provide a human-readable representation, rather than pretend a single, inefficient representation for both man and machine is somehow the best option.

(Maybe IMAP is a text format too, so this isn't an advantage inherently) And JMAP may have better semantics than IMAP, and there's nothing inherently wrong with JSON (other than edge cases).

Computer programs written by anyone competent are built around edge cases; that's much of what any reliable program is, accounting for edge cases. To disregarded edge cases, of which there are many in JSON, speaks to me that you have little concern for writing software correctly, but I imagine you'd deny this, as who doesn't want to write correct software? Simply understand that using a format with so many edge cases and other defects as JSON is foolhardy for much of anything.

rapsey• 7 years ago

- Anything outside of int32 must be encoded and decoded as a float. Oh the bugs/crashes this has caused.

- It is way too easy to change the schema and wreak havoc with downstream users.

- Some libraries will encode empty strings as null for instance, some as empty strings. Some will have empty objects, some null. Switch some configuration, change your serializer, ORM or anything that touches your data. Everything will work perfectly in your code, but json output can change and you will again break your downstream users.

If you are designing an API, use protocol buffers. Json maybe as an optional output/input to your service. Keep it as a discouraged dirty optional feature and you will save yourself a whole lot of debugging and time wasting.

IshKebab• 7 years ago

The problems with JSON are pretty obvious:

* It's inefficient compared to schema-based formats like Protobuf - you have to encode key names again and again.

* It's inefficient compared to binary equivalent formats like CBOR - both in space and speed.

* It's less reliable than explicit schema-based formats because it can't be validated (yes I know about JSON-Schema, nobody uses it). You essentially end up with an implicit schema that everyone has to recreate themselves from the documentation - especially if using statically typed languages.

* It's a pain to use from statically typed languages. You have to write the schema in your language, whereas with things like Protobuf it will generate the code for you.

* It doesn't have a proper type for arbitrary binary data. Most people (but not all!) encode it as a base64 string which is not very efficient and not actually a different type to a string, so again you rely on the implicit schema.

* Maps can only have strings as their key type.

* Minor annoyance, but trailing commas would be nice!

* Its number type is not well specified. E.g. does it support uint64?

"Human readable" is overrated. CBOR and similar formats can easily be inspected by humans just by converting them to JSON. The Smile format even starts with a :) smilie as its magic number to make it easily recognisable.

Would you recommend not using compression or encryption on HTTP requests because it stops them being human readable?

octorian2• 7 years ago

(Maybe IMAP is a text format too, so this isn't an advantage inherently)

Actually, IMAP is a binary format that looks like a text format. This distinction is extremely important when you want to make a reliable Email client. There are also multiple ways of encoding text within it, that you need to keep track of.

Personally, I wish they used a structured binary encoding. This becomes especially important when data size efficiency actually matters. With something like protobuf, you can even have the parser for it generated for you. (However, I have worked with binary formats that are more loosely structured.)

Of course I am a fan of well defined and strongly typed formats.

tenderfault• 7 years ago

google defined websocket protocol not long ago. We didn't actually need it. They did. With such a huge browser market share, all they crave for is information. All of it. And most of us (developers) embrace this flow because each and every "fault" they "discover" and use to create a new idiotic standard is an opportunity for us to profit. And we feed it. I don't need jmap, none of us do. They need it, so they wrote 200 pages of stupid, extremely complicated specs with the hope that it will gain traction and we'll start developing. They can't do it alone. They need us to embrace it. I won't and I hope none of you will. I hope this standard will die with a whimper. I am doing my best to keep my e-mails where they belong. Far away from browser. I use SMTP. There's nothing wrong with it. It works. I use MIME. There's nothing wrong with it. It works. I personally don't use IMAP. ODMR is all I actually need. But I configure and maintain IMAP servers. I see nothing wrong with it either. It works. What baffles me is cyrus project accepting this idiocy. I just hope it won't become the new deal and the others (dovecot et all) will stay away from it. Panic and fear can and will escalate the process.

So how long until "everything is JSON"-shit will take over? I'm 40 years old and my bet is I will live to see it happening. This is actually scary. And not because json or hydras.

isodude2• 7 years ago

I tend to agree with you, even so, if not for these moves we wouldn't have any change at all. Email is not a solved problem, we just have a lot of fixes that makes it sort of stable. If not else we get a competing extension to IMAP because someone does not want JMAP.

microcolonel• 7 years ago

To be honest, JSON is pretty reasonable. The parsing speeds are very good, the grammar is a lot simpler than SMTP/HTTP/IMAP, it actually has numeric and boolean types.

It is no surprise to me that the two people here complaining that IMAP is somehow acceptable don't even use it.

JMAP solves real problems inherent with IMAP, particularly on clients that are not suitable for local archiving of messages (hence, ODMR/POP is not an option), and where IMAP sends considerably more data than the client actually uses (making it slow).

I don't like the use of JSON for local IPC, as is showing up in some systems, but for RPC over an unreliable network, JSON is more convenient than not.

As for the use of HTTP, it's not ideal in my view, and I tend to think it's overused. The one argument though, is that it allows JMAP to go over port 443 to a host that has an existing https service on that port (since you likely have a reverse proxy, probably NGINX, as your listener anyway) without having to send a bespoke upgrade request, which defeats the simplicity benefits of a custom protocol, since you have to implement HTTP to do the upgrade.

thelittleone• 7 years ago

"Modern", the darling adjective of startup lingo. We thank you for your service and efficacy. It's time to abdicate the throne. All hail "more modern" the rightful heir, protector of the realm.

Pre-modern -> modern -> more modern -> post modern?

Jokes aside I enjoyed the experience of Fastmail with the exception of the migration tools. Very responsive. Kudos for submitting JMAP as an open standard.

NetOpWibby• 7 years ago

FINALLY

wruza• 7 years ago

I read imap description long time ago, and I know what it does in principle (along with issues itt), but can someone explain what value does it serve today? I have two confusing thoughts.

First, imap seems to be an overkill for a line wider than 64kbps. Texts do not consume much traffic and an entire mailbox can be served easily to everyone. There are some huge mailboxes out there, but messengers have this problem too and they solve it somehow (I suspect trivial &last_date=... from programming tg bots).

Second, and this may be due to bad clients (haven’t seen good however), imap is just sloooow. It doesn’t update instantly, but instead I have to enter every folder and pull down with my finger to update it. It can show outdated messages which I deleted on a pc for ten seconds before it figures out that they were deleted. My guess that it has something to do with an infinite number of roundtrips which this protocol is based on. Occasionally it shows N unread when there is none. Even push notifications fail to work properly most of the time (same server-side issues?) And with all these complications, I still have to download every attachment by finger. How come that heavy sites/forums/boards/messengers always instantly show new data and mail cannot? I know that I can simply use webmail, but wtf?

I’m a developer, but when I see software, I always try to evaluate it from user-only perspective. And from that, mail tech seems simply incompetent. I wish there was a port smtp->tg and tg->smtp, and special per-subj chats, so I could leave all that legacy for someone else to deal with and just talk to them without missing messages or waiting for updates.

(Not even considering mail message formatting, spam mismatches, size restrictions and non-delivery laws)

chrismorgan8• 7 years ago

The slow mail client issue you’re describing is very probably an artefact of how IMAP IDLE is designed; JMAP fixes this design, as explained at https://jmap.io/#push-mechanism.

inputmice3• 7 years ago

Pretty cool protocol. I've been following this for a couple of years. I had already written a library for an earlier version of JMAP (back then references worked differently / didn’t exist); and it was interesting to see how that improved in the IETF process. I wrote a library [1] and a POC email client for Android [2] earlier this year. It takes a moment to fully understand a now fairly complex protocol but when you get the hang of it it becomes very powerful.

Sadly the server support isn’t really there yet. The support for Cyrus hasn’t been released yet (you need git) and some vital functionality like push [3] is still missing. Also no word from Dovecot yet.

[1] https://github.com/iNPUTmice/jmap

[2] https://github.com/iNPUTmice/lttrs-android

[3] https://github.com/cyrusimap/cyrus-imapd/issues/2714

Leace• 7 years ago

Do you use Lttrs daily? I'm wondering if it works in practice (even when using non-standard setup) or if it's just a proof-of-concept (that'd mean it may take some time to mature).

inputmice3• 7 years ago

No I don’t use it daily. It can’t even send emails yet and you have to set your login credentials at compile time. I got frustrated having to deal with HTML emails and now I’m waiting for the servers to mature a bit. And honestly also for dovecot to announce support. I mean I love what the fastmail+cyrus team is doing; but if Dovecot is not going to support it, it is probably not worth it to build a JMAP-only MUA.

brongondwana8• 7 years ago

I've started defining the bits necessary for storing push subscriptions: see

https://github.com/cyrusimap/cyrus-imapd/commit/7144a747267c...

But it's rudimentary - it will need sql_db and dblookup components to make it fast enough to integrate with an mboxevent reading component to do the actual pushes. At least - that's how we'll probably implement it at Fastmail, and there's a plan to extract the key parts of our non-blocking push daemon and lift them up into the Cyrus project.

chiefalchemist3• 7 years ago

I didn't see anything about security. I understand sending is outside the scope. But if storage and downloading is to be portable, doesn't encryption have a place?

bad_user5• 7 years ago

The encryption of the connection is orthogonal. That's what HTTPS is for.

And client-side encryption is managed by solutions like PGP. Doing it on the server-side is incompatible with the email protocol IMO.

chiefalchemist3• 7 years ago

So server side is secured how?

z3t43• 7 years ago

Lets ignore the biggest issue (spam) and focus on the bike shed.

dane-pgp2• 7 years ago

No one's stopping you from coming up with your own anti-spam technology.

In fact, it's possible for individuals and organisations to work towards multiple goals at once. For example, Fastmail and the IETF have concurrently been working on RFC 8617, which allows for stricter spam filtering:

https://dmarc.org/2019/07/arc-protocol-published-as-rfc-8617...

z3t43• 7 years ago

The reason people choose Gmail or Office instead of smaller providers, is 1) They get less spam 2) Less likely get their mail market as spam. This centralization is killing e-mail. But that's just e-mail. What protocols do people use to send messages ? Facebook messenger, Whatsapp, etc. More proprietary centralized solutions. We need a simple decentralized protocol, as simple as SMTP, bit it need to solve the spam issue, and it need not to be over-engineered (like XMPP). And no, it should not be built ontop of SMTP, SMTP has so many issues, like you can send mail pretending to be anyone. There are solutions like SPF, DKIM etc, but 1) they are built ontop of SMTP 2) They are not stopping spam. We need a new protocol that replaces SMTP, and have spam prevention, identify, etc built in.

rounce• 7 years ago

One thing I find weird (probably because I’m new to JMAP and still ignorant) is having position AND anchor+anchorOffset. Why have both? Surely the latter is sufficient and safer (as any position is only guaranteed to be valid when the last response was written)? From a glance of the spec I didn’t see any explanation of why. If anyone could shed more light it’d be much appreciated.

chrismorgan8• 7 years ago

Both are necessary because you may not have the necessary IDs.

anchor + anchorOffset is useful when you use pagination and new entries might appear at the head of the list but you don’t want them to mess with your pagination once you’re inside.

position is useful when you have a total count and wish to jump in at the middle; as a concrete example, when you use lazy loading instead of pagination. For example, if I have a mailbox of 100,000 emails, and jump to the bottom: I don’t want to need to fetch all the IDs (probably well over 1MB to fetch), just so that I can specify an anchor. I want instead to ask “give me the thirty messages from position 99,970”.

(Note that when I speak of lazy loading I’m not talking of infinite scrolling, but rather of when you know that you have N messages, and that each message is allocated M pixels of vertical space, so you can allocate an exact space, and provide a proper scrollbar. This is how Fastmail does it and how desktop email clients have historically done it.)

psim1• 7 years ago

I'm a believer in open protocols, but in recent years, I'm also loving Exchange and ActiveSync. I know that's not going to be a popular opinion here. To be honest, I don't know much about how the magic works, but the configuration is simple, the push is reliable, and everything just works. It doesn't drain my mobile and it doesn't lose its indexing and download for hours like IMAP always seems to. And besides, it's mature -- Exchange and EAS have been around for many years.

So here's to JMAP becoming the open standard that replaces that.

znpy• 7 years ago

Yes, but I can’t tun it on my own low hardware at home for free.

amaccuish3• 7 years ago

Sure you can, I use z-push [0] integrated with Kopano which works great. You can also use z-push with IMAP, CalDAV and CardDAV. The only issue I know of using IMAP is that the z-push server polls your mailbox, so it's not true push, but vastly better than the battery drain of IMAP IDLE.

[0] http://z-push.org

jchook• 7 years ago

I dislike the details of JMAP’s design.

While I appreciate the value it brings, I can see the 5 years of struggle in the sheer size of the multiple specs needed to define some pretty rudimentary mail client/server functionality (retrieve, send, push, pagination, etc).

At some points they use JSON (a data format) to describe procedures... like a programming language written in JSON. It feels a bit like using a screwdriver handle as a hammer.

How did we end up with JMAP instead of something simpler or modular, especially with GraphQL and other similar innovations on the radar?

zeveb• 7 years ago

At some points they use JSON (a data format) to describe procedures... like a programming language written in JSON. It feels a bit like using a screwdriver handle as a hammer.

And folks laugh at me for suggesting S-expressions until I'm blue in the face!

Seriously, this is one of the areas in which they excel (ref.: literally every line of Lisp ever written). They're a pretty nifty data format too (cf. https://sites.google.com/site/steveyegge2/the-emacs-problem).

0x0• 7 years ago

They list the wrong RFC number "8261", but the linked RFC is actually "8621".

brongondwana8• 7 years ago

Thanks! Have fixed the copy. The link was going to the right place, but somebody (possibly me) put the digits in the wrong order.

For your efforts, please enjoy a joke: https://comic.browserling.com/tag/multithreading

(that's my excuse - brain was running multithreaded output to the fingers)

Nasreddin_Hodja2• 7 years ago

https://jmap.io/#push-mechanism JMAP going to use rfc8030 push for mobile (3rd party service for notifications, really?) and EventSource (kind of http long polling) for desktop clients, as I understood. I don't like that idea, it's even worse than IMAP4 IDLE. Why not just WebSockets for mobile and desktop clients?

Or just building it on top of XMPP protocol (or any other chat protocol).

nailer• 7 years ago

HTML5 Server Sent Events are better than long polling, and they're about as old as websockets, but simpler, as they maintain the concept of HTTP resources whereas websockets replace the concept. HTML5 SSE is also easier to implement.

Not hating on websockets, but saying HTML5 SSE is outdated tech is just wrong. If you were making a real time app that works over HTTP either would be fine.

Nasreddin_Hodja2• 7 years ago

HTML5 Server Sent Events are better than long polling

But it works the same way: infinite http connection.

wchest• 7 years ago

The JMAP protocol is transport agnostic and can be easily transported over a WebSocket, for example, as well as HTTP

From section 10. of the jmap.io link you referenced. I haven't used JMAP or looked into the spec in any detail, but wanted to provide that context for anyone coming here and seeing the parent comment first. Sounds like you can use WebSocket or any other transport mechanism.

cotelletta• 7 years ago

Give me the speed of fastmail with the encryption of protonmail, make the two talk to each other out of the box, and actually make something new instead of just the same old thing.

Actually, loan some of your devs to protonmail because both their mobile app and their imap bridge are embarrassing.

akulbe• 7 years ago

Is the protocol itself important to users? or is it mostly important just to developers?

If it makes a practical difference to a user, how so?

Making email more modern with JMAP

Related Stories