How to Replace IMAP

http://www.gaborcselle.com/blog/2010/02/how-to-replace-imap.html

by mk • 16 years ago

118 60 16 years ago

ajross2• 16 years ago

Not to spend too much effort defending IMAP (which does, to be fair, kinda suck) but... come on. It's not that hard. It works just fine, enjoying pervasive support with very high quality clients and servers available freely on essentially all platforms.

A new protocol would be ... prettier, I guess. But to pretend that the reason facebook and IM are replacing email is IMAP (and not, y'know, spam) is borderline delusional.

sandGorgon2• 16 years ago

http://jerakeen.org/notes/2010/01/yay-more-email-clients/

The Android GMail client is a perfect example of what a client looks like in this world. It talks to the (secret / private) GMail API, it does offline mail reading, and queues actions so you can archive / filter / whatever mails while offline and it’ll push changes later. You can read and write mail. It doesn’t try to do anything clever, because anything clever done on one client isn’t reproduced on any other client. And if I don’t have a client on my current computer for GMail, I can use a web browser, and still get all the features of the server. I use the web gmail interface for everything anyway, because it’s better than any GUI client I’ve got.

I dont know - this guy makes for pretty convincing arguments on why email should be done right on the server rather than the client. Which effectively means a new protocol.

tedunangst• 16 years ago

I started using this new "mail on the server" protocol way back in 2000, but back then we called it SSH. :)

_delirium• 16 years ago

If not designed very carefully, though, it can reduce the user's ability to view things as they like them. With the current status quo of mailservers not handling message threading and such, I can choose a client that handles it how I want. If the server is doing it, it had better either do it how I want it, or provide some way to customize what it's doing.

(For example, I don't like gmail's flat "conversation" view; I much prefer a "threaded conversation" view, like classic Usenet readers, or HN discussions.)

coryrc• 16 years ago

Mozilla Seamonkey and Thunderbird do that all with IMAP (except the web client part).

gduffy• 16 years ago

This is not about making it easier to write email clients or servers, it's about making it easier to write email apps. Currently, you either have to write a plugin for an email client or interact with IMAP directly. Opening up email to mainstream developers means ditching IMAP/MAPI/etc as the API.

ajross2• 16 years ago

And again, I don't see why that's such a huge improvement. IMAP is clunky and weird, but it's not rocket surgery. If there was a huge market opportunity for someone to write custom mailoid gadgets using IMAP, clearly it would have been done already.

A better protocol will only be "better", analogous to, say, the Mac (c. 1990) being better than Windows 3.1. While true, compatibility pressures aren't going to allow someone else into the market simply for being "better". You have to be transformative, and this isn't.

And in any case what's killing email isn't the lack of apps, it's the lack of authentication and moderation.

neilc• 16 years ago

You rarely need to use IMAP directly, though: most email apps are fine using a library that manages the details of the IMAP protocol under the covers, and presents a more sane interface.

gaborcselle5• 16 years ago

The combination of IMAP and MIME are hard.

Yes, we could be happy with the status quo. But I do believe that one of the things that's holding up innovation around email is the lack of easy access to data. Proposed exercise: Write a script that downloads and displays your Twitter feed. Then write a script that downloads and displays your email. The email exercise will take you an order of magnitude longer.

Spam filtering isn't the problem you make it out to be. I get more spam in my Facebook inbox (mostly viruses) than in Gmail. If you're hosting you're own email or are using a sucky provider, that's a different story.

Email will be hard to replace long-term anyway. I doubt your bigcorp employer really going to let you send work-related Facebook messages. IM is a fundamentally different protocol with different use cases (think async vs. sync, fax vs. phone).

IgorPartola2• 16 years ago

  $mbox = imap_open( $host, $login, $password );
  foreach( imap_search( $mbox, 'UNSEEN', SE_UID ) as $uid  ) {
    $header = imap_fetch_overview( $mbox, $uid, FT_UID );
    echo $header[0]->subject . "\n";
  }
  imap_close( $mbox );

IgorPartola2• 16 years ago

You might want to revise your challenge.

IMAP and MIME are hard because there is a higher level of information that is available. Can you attach an archive over Twitter (not link to but attach)? Can you send HTML? They are not the same thing. If you want something really simple your code will be simple. If you want more use cases than "i can haz cheeseburger" then you need more expressive power and more complexity.

ryanwaggoner• 16 years ago

But to pretend that the reason facebook and IM are replacing email is IMAP (and not, y'know, spam) is borderline delusional.

As is believing that it's about spam (or probably believing it all, for that matter, but let's stay on topic).

I don't understand people bitching about spam in 2010. I have probably a dozen email addresses that I publish all over the web and I get maybe one spam message a month. I get way more "spam" through Twitter than I do through my email.

sorbits• 16 years ago

I don’t understand people declare the spam problem solved.

I have had false positives with every single spam filter I have tried. This is unacceptable when running a business, and if we manually have to double-check hundreds of spams each day, what is the point of the filter?

Additionally we spend hours each week working with users who do not receive our emails because of spam filters unknown to them (many believe the fault is ours).

It is 2010 and spam is still a major problem.

gibsonf1• 16 years ago

It seems like the key is to work on top of an interface that does all the dirty work of negotiating with IMAP for you, such as the lisp mel-base library. Then you really don't care how messy imap might be, as you can build on top of it at increasing levels of abstraction.

nfnaaron• 16 years ago

I can't say whether his implementation makes sense.

Regardless, I would like to see public key encryption of all email. There's no reason why messages should be plain text anywhere except at the two ends.

I realize key management would be a problem, but it's probably a solvable problem. The solution would have to allow most people to be practically unaware of encryption issues.

webignition• 16 years ago

Encryption of all email would make intermediary virus scanning impossible and spam scanning next to impossible.

conover• 16 years ago

Wouldn't it make spam impractical because you would all of a sudden need much more processing power to encrypt the messages (especially millions of them)?

mrduncan2• 16 years ago

The description for #8 sounds more like long polling to me. For push email, I'd envision something like registering a location to send notifications to (web hooks).

gaborcselle5• 16 years ago

Yeah, it is long polling. I could be misinformed, but isn't the problem with web hooks that you might not have ports open and could be behind a firewall?

mrduncan2• 16 years ago

Yea, I'd say long polling is probably a better solution in this case than web hooks since clients (likely) could be behind firewalls, NAT, and be going on and off line relatively often. Sorry, the title and description on that one just threw me off a bit - I think you chose the best option though.

tdmackey• 16 years ago

First the solution doesn't really fulfill all of imaps use cases (especially push mail) but that is unimportant.

What is important is that the IMAP protocol isn't difficult, and I don't think that is what limits "email innovation." It is more so that it has been around since the 70s and email clients haven't been able to remove themselves from their past. It isn't the protocol limiting them from moving forward, it is the shear number of other clients that wouldn't support any of the "innovations" one would try to bring to email.

gaborcselle5• 16 years ago

Would love to hear your thoughts about how to improve my proposed design. My email's on my HP in case you want to do it offline.

beh• 16 years ago

I'm very hopeful for Letters, a project just taking off under the leadership of some pretty big names: Gruber, Rands, Brent Simmons. They're hoping to collectively create a "lean and programmable IMAP email client, with plugin and automation APIs, designed for developers and power users." While still sparse and very early-stage, their vision document is ambitious: http://pastie.org/785269 If something comes from this, it'll definitely be a step forward to modernize email clients.

Project wiki: http://letters-wiki.heroku.com/

Joe_Bananas• 16 years ago

Ah, yes. A new email scheme that would require us to replace all clients and servers in one fell swoop. I can see that taking off.

gridspy• 16 years ago

All it requires is a major server (i.e gmail) providing it as an option and a major client (say a iPhone email app) supporting that option.

there• 16 years ago

oh, that's all?

it'll require an RFC or some other kind of standard, which will take a long time to finalize. then someone will have to write a reference spec server and client, then "real" versions in "real" languages, then get them both to the stability, security, and scalability of current imap clients and servers. then once big email providers (isps, google, etc.) start supporting it on the server end (gotta factor in all those different operating systems, authentication systems, storage systems, etc.), popular clients can support it. then work out all of the minor implementation details that some servers or clients get wrong (i think almost every smtp and pop3 server has hacks in it to support the stupid things outlook express does wrong), and then once everyone complains long enough, the iphone will finally get support for it (while a small but vocal minority shouts that android already supports it).

there is a reason why smtp, imap, pop3, dns, http, and all of the other core internet protocols are still around after 20 years and it's not because there aren't any faults in any of them.

jwhitlark2• 16 years ago

My responses to your Design Outline.

1. I'm not convinced that http/https is the best protocol. I agree that it's the defacto standard, but if you're going to go through this, there might be a better way. (Push, in particular, is a problem).

2. Stateless is wonderful, agreed.

3. I agree with JSON over XML, but what about binary data? That is the weakest part of JSON, in my opinion.

4. I like where you're going with this, but I'm not sure this is the exact right way to handle it.

5. Agreed.

6. how about SHA-1 hashes of content? Done properly, that would eliminate any duplicate messages, and could make it very easy to include other messages. This might also allow content to be transfered between accounts only once, dropping bandwidth and speeding delivery.

7. Don't know enough about MIME to comment.

8. According to your comments on this page, you're referring to long polling. I agree that this use case needs to be addressed, but I'm not sure that this is the way to do it. Perhaps multiple options here?

9. Good idea, but do you think that stemming/language differences is going to be a problem? What about other types of content? attachments?

The biggest thing missing from this list is some sort of Public Key Encryption. Anyway, it looks like a good start.

pyre3• 16 years ago

> 3. I agree with JSON over XML, but what about binary data? That is the weakest part of JSON, in my opinion.

What binary data? All attachments in emails are Base64 encoded. That's what MIME is all about.

> 6. how about SHA-1 hashes of content? Done properly, that would eliminate any duplicate messages, and could make it very easy to include other messages. This might also allow content to be transfered between accounts only once, dropping bandwidth and speeding delivery.

Developers should start to shy away from SHA-1 since it's already started to get broken.

jwhitlark2• 16 years ago

>> 3. ugh. Hadn't realized that. I like IMAP even less, now. (Told you I didn't know much about MIME ;-)

>> 6. I wasn't intending it as a security mechanism, but as a way to know which bits you have and which you don't. Do you think SHA-1 is still a bad idea in that case?

blasdel• 16 years ago

3) Handling binary data is incredibly easy when you're already using HTTP: just use hyperlinks to other resources!

gaborcselle5• 16 years ago

Yeah, that was my thought about that too. Attachments downloaded separately via HTTP.

bhousel• 16 years ago

> 3. I agree with JSON over XML, but what about binary data? That is the weakest part of JSON, in my opinion.

Maybe BSON?: http://www.mongodb.org/display/DOCS/BSON

JoachimSchipper3• 16 years ago

This is a collection of the author's favourite technologies/gripes, not a useful proposal. A large number of errors can be found by considering that it'll have to talk SMTP on the back end.

I'll only point to some "highlights".

OAuth isn't nearly secure enough (http://hueniverse.com/2009/04/explaining-the-oauth-session-f...) for your e-mail (which, don't forget, gives access to everything via password resets). Let's just ignore the part about third-party access.

Encoding everything as UTF-8 is nice, but breaks the first time some idiot mail client puts Shift-JIS encoded Japanese in the Subject: field. Without any charset marker, obviously. (This goes right back to the "where does the mail come from?" issue.)

IMAP has a THREAD extension (solving #4), a SEARCH extension (solving #9 and parts of others), and all sensible servers support IDLE (http://en.wikipedia.org/wiki/IMAP_IDLE). It also supports downloading MIME parts separately, solving the part of #7 that can actually be solved (again, you'll have to accept mail.)

__david__• 16 years ago

With regards to search, I recently installed Cyrus Squatter for full text IMAP searching and it is blazing fast. To me it seems just as fast as any of the special case local searches I've seen people build (Spotlight/recent Thunderbirds).

qjz2• 16 years ago

Any new mail storage protocol should anticipate the need to store encrypted messages at rest. Sure, it can play a part by storing an encrypted index on the server to deliver to authorized clients, but all decryption and index updates should take place client-side with the user's private key. Stable UID's can help make this a reality. There is no reason to store messages in plaintext on the server, and no new protocol should depend on it.

gaborcselle5• 16 years ago

Thanks for your suggestions Joachim! This is why I wrote that blog entry in the first place: To get some high-quality feedback.

I think IMAP could be replaced independently of SMTP. You could conceivable replace an IMAP server with a reMAP server and not affect the rest of the mail ecosystem. SMTP will likely not die for another hundred years.

1. I know about the OAuth problems. Do you think we'll see a more secure solution in the next few years?

2. Is there an easy way to validate UTF-8ness? Happy to hear suggestions.

3. I know about THREAD and SEARCH and IDLE but find them clunky. Under the assumption that we're replacing IMAP with something RESTful you might as well get those ones right.

Happy to take this offline as well - my email is on my HP

JeffJenkins• 16 years ago

It's easy to validate UTF-8. The danger is if other implementations crop up that are more permissive and your users are complaining that they can't get mail from those people. I think you could probably manage to avoid that, but I don't know if it would really be preferable. I want to claim that in asia and europe there are some widely used non-unicode encodings used.

JoachimSchipper3• 16 years ago

First off, thanks for taking this well. It was a bit of a rant, even if it was toned down from the first version. Sorry.

We agree SMTP will be around forever; but how do you plan on abolishing MIME then? Your system will have to handle it, no matter what. Do you want to push MIME handling to the server? IMAP already does that, there's nothing revolutionary about that.

Due to the special structure, it's usually possible to distinguish UTF-8 from other 8-bit text. The tricky part is what to do with unmarked non-UTF-8 8-bit text - it could be pretty much anything. Mutt, for instance, allows the user to configure this according to what the user expects to see. Quoting the manual,

    For example, Japanese users might prefer this:
    set assumed_charset="iso-2022-jp:euc-jp:shift_jis:utf-8"

There is a lot more information about such things on the Mutt site. Basically, Mutt tries to parse the headers/message according to a specific character set and falls back to the next one if that fails.

I probably wouldn't trust my e-mail to a protocol that was only a few years old. I don't anticipate any new security protocols with significant uptake either; "serious" stuff already uses SSL/TLS plus an authentication method (passwords/certificates/SASL), and the web crowd is unlikely to replace that (OAuth is about weakening security in a controlled way, not replacing SSL.)

And yeah, parts of IMAP are clunky. But that's not enough reason to rip-and-replace it.

Skeuomorph2• 16 years ago

First, I love reMail. Gabor breathes email and search. If you own an iPhone and use Gmail or Google Apps, get reMail--you'll thank him.

That said, it's easy to skim this article and nod, "Yeah! Yeah!", but when we're discussing these, laymen may need to know a few of these seem over-simplified.

> "TCP connections are great, but for transferring large amounts of email securely, HTTP is the way to go."

1. http://wiki.answers.com/Q/What_is_the_difference_between_tcp... - HTTP uses a TCP connection. For that matter so does SMTP and IMAP and the proposed REMAP.

2. I generally have at least 5 devices using IMAP (on Google Apps) at once, and interleave my interactions with these devices arbitrarily, yet expect each to show me the exact same state when I look at it. Keeping them all in the apparently same state is the sort of intractable software problem bedeviled with implementation details, and I agree throwing out IMAP and starting over with lessons learned could be easier than, say, the RFC linked below.

3. http://ajaxian.com/archives/json-vs-xml-the-debate - Seems this point is something like GIF vs JPG vs PNG, with a bit of "Markdown vs HTML" thrown in. Meanwhile, both XML and JSON are missing an important concept given labels/folders and conversations: multiple references to single objects.

4. Conversations or threads exist today as In-Reply-To: GUID and References: GUID headers, among others, but see the JSON problem above. We don't realize these headers are there because clients generally ignore these and try to group on Subject instead. Gmail grouping respecting existing headers is much better, but from usability point of view, Gmail's approach makes it challenging to bulk delete individual matching messages across a broad set of conversations. For example, to bulk move or mark all messages on a particular date results in the full conversations being moved or marked. So I like this idea as long as I can still optionally operate on sets of messages without affecting the rest of the conversation.

5. http://googlesystem.blogspot.com/2007/08/organizing-chaos-fo... - It's not yet clear that people prefer tags to folders. After all, monkeys don't expect a banana to be in two boxes at once. I don't want to label anything with keywords. I want semantic search.

6. Excellent point. Changing GUIDs out from under us is just lame. Having had to resort to http://github.com/rgrove/larch recently to move a decade of email onto Google Apps because Thunderbird was putting new IDs on each moved message, I fully agree this problem is evil.

7. Seems we're stuck with MIME until we get rid of every legacy email server in the world, or control what servers our recipients use. Between the new server and new clients, sure. But the server will have to know it to send it. Granted, IMAP is for receiving, and SMTP is for sending, but something in the chain has to know MIME.

8. "Call" an HTTP endpoint that "returns" when messages arrive? Just nomenclature, since IDLE is a "call" that "returns" when state changes. Either way, the socket is held open, so neither of these is "push". Maybe Gabor meant "posts back" instead of "returns"? In any case, under the hood, iPhone push and Microsoft ActiveSync are exploiting out-of-band channel such as SMS to notify the phone it should poll (or in case of ActiveSync, maybe doing a "long pull"). See http://msexchangeteam.com/archive/2004/04/26/120520.aspx versus http://tools.ietf.org/internet-drafts/draft-maes-lemonade-p-...

9. Will the RFC specify a search results algorithm, or will the same search return different results depending on the implementation of mail server the client is connected to? We manipulate a "client" tool to perform the search and access the email, so our natural mental model is of the client at hand, not of the remote server. This lets the brain wrap itself around Gmail's web search versus Apple Mail's search box vs OS X Spotlight vs Gabor's reMail all returning different results. Practially speaking, search at the server does make far more sense from both a data retention and security standpoint. I have 509 MB of index in Gabor's "reMail" app on my iPhone, and would rather not.

Thanks for brainstorming this, and thanks for a kickass iPhone app.

Nwallins2• 16 years ago

1.HTTP uses a TCP connection. For that matter so does SMTP and IMAP and the proposed REMAP.

Yes, of course. He is not claiming that TCP is bad, but rather that it is too low-level. i.e. One wants an application protocol (e.g. HTTP, SMTP, IMAP) rather than a transmission protocol (e.g. TCP).

The HTTP verbs should be sufficient for email, so why not leverage them?

It's not yet clear that people prefer tags to folders. After all, monkeys don't expect a banana to be in two boxes at once.

Heh, dare I reply? Tags can be used as pigeonholes, and this can be enforced if desired. But why? Tags are so much more useful because they are not pigeonholes.

Here is a classic example:

Does my app config go in a system directory or an app directory or a config directory? the executables? the daemon? the log files?

The classic approach is to put system-level config files under /etc, log files under /var/log, executables and daemons in various locations, etc. Some newer approaches (HomeBrew, GoboLinux, etc) keep everything organized per-application.

Instead, keep every file in the global folder / namespace. Tag the executables as exe, the log files as log, config as cfg, and also tag everything with the app's name. Possibly the version, etc.

A file is uniquely identified by its tags and its name. Any combo can be used, so long as it doesn't already exist.

You don't have to make painful decisions about the unique place in the tree that a piece of information should exist. Just tag it appropriately and move on. The tag system could even create an optimal concrete hierarchy based on tag and tag association metrics if needed.

But the server will have to know it to send it. Granted, IMAP is for receiving, and SMTP is for sending, but something in the chain has to know MIME.

I'm not sure if this is GoF, but this sounds like the classic gateway pattern. You put MIME gateways at the edges of the new email network. MIME-only servers must go through the gateways.

eitally• 16 years ago

Your intended point about pigeonholes can sometimes backfire within the context of email. Consider a Gmail user who sends a query to a number of different recipients, expecting unique replies from each. Gmail, because of the like headers, will group all of the responses into the parent conversation. Let's say the users tags the first response with a label corresponding to that sender, and then the second response comes in. At this point, the user is at an "oh shit" moment, finally comprehending the implications of Google's labeling of conversations rather than individual messages.

You need to be able to specify both parent and child tags to be able to facilitate search and give users something that isn't too confusing re: organization.

graywh• 16 years ago

Having had to resort to http://github.com/rgrove/larch recently to move a decade of email onto Google Apps because Thunderbird was putting new IDs on each moved message, I fully agree this problem is evil.

What version? This doesn't happen to me on 2.0.0.23

dlsspy• 16 years ago

> All data that's ever sent to or received from the server would be in JSON format. JSON is much more human-readable than XML.

I don't need my protocols to be human readable. I need my computers to read them and know that I've got good, bug-free, reusable, efficient, and easy-to understand implementations.

This isn't an argument for XML over JSON -- I've made good use of both, but I don't want to consider how easy it is to type up communication protocols when I'm defining stuff the replacement for something I expect to take over the world.

It's by far easier to implement the memcached binary protocol as a client or a server than it is to implement the text protocol. Easier to get right, more regular, and just generally more pleasant.

regularfry3• 16 years ago

As far as I'm concerned, the argument for JSON over XML is that the parser should be much lighter. No bloody entities to worry about, for a start.

viraptor• 16 years ago

There are some good points, but:

1. HTTP / HTTPS - please don't... There are stupid/broken transparent proxies in many places that will effectively block this protocol. Also custom TLS connection gives some hope for domain-specific certificates instead of ip-specific - there is enough problem with them in HTTPS as it is.

3. People still don't know how to set the encoding (or what the encoding is) and will happily send you Greek in latin1 or something equally silly.

4. Would it really work for mailing lists where conversations naturally fork into many branches?

5. IMAP supports labels... even if it's crazy number-based labelling where every client needs to set its own names for the labels. We'd just need to make it not suck this time ;)

So it's not all that bad in IMAP - there are many areas where we can do worse by accident. But yes - it would be nice to have server-side text searching...

pyre3• 16 years ago

> 4. Would it really work for mailing lists where conversations naturally fork into many branches?

I agree with this. The idea that messages must be accessed by conversation is a bit much. Support for conversation group could be nice, but ignoring that feature would have to be allowed so that clients could implement their own views.

For example, the following structure would be crappy to force on clients:

  Label
  `-> conversation 1
      `-> msg0
      `-> msg1
  `-> conversation 2
      `-> msg0
      `-> msg1

Needing to iterate through every conversation to pull tease out the full list of messages (in order to use a different conversation-grouping method) would be ridiculous. Not to mention the question of whether or not conversations should span labels and how that should work.

cturner• 16 years ago

You could implement this as a service on top of couchdb. The oreilly couchdb book that has just been released shows examples of developing apps using the javascript engine that's inside couchdb. You'd get easy access to some functionality by doing this - tagging, email versioning.

How to Replace IMAP

Related Stories