robn_fastmail

I don't know about bounties, because I'm not personally in favour of vigilantism, but I do take your point.

Honestly, I think the ease in which people can be anonymous is major problem here. Anyone with an internet connection can buy botnet time with Bitcoin and accept a ransom in the same way. It makes it incredibly difficult to follow the path back to the attacker.

At this point pretty much the only thing you can do is collect data and share it with CERT and other relevant law enforcement. I don't have a good sense of how effective they can be, but it makes sense that the more data they have, the better chance they have at identifying specific botnets and follow the path back to the owners.

Generally, we talk about what we're doing because we're all excited about what we do. It's always been that way for us, and our customers really appreciate the honesty and transparency.

On this particular one, we really did learn a lot and we were keen to share some of that. It's really difficult to run an internet service and we feel we have a duty to try and make this easier, or at less better-understood, where we can.

Our business can't exist with a large diverse network that anyone can get involved on, and we couldn't have got to this point without the knowledge of others, whether that's embedded in the open-source software we run or in the blog posts and emails of other people that figured out hard stuff. It wouldn't be right for us to take and not give something back in return.

There's also an element of defiance in this post too. We got punched in the face. We're not going to respond by hiding in a corner. We're going to say "you know what, fuck you" and we're going to help (and have helped) others to do the same in whatever way we can.

Quite. As my dear colleague said this morning, "hubris" is the word of the day. Still, I'm not sorry we posted it.

Datacentre confirmed a DDoS attack, which took a little while to mitigate. We're looking good now, but we're continuing to monitor.

Indeed. As I note in the post, this wasn't something we _needed_ to do at that time, we were just curious.

Replacing a function with a faster implementation is trivial; its just another deployment and we do several of those each week. Changing data format adds operational complexity - two codepaths need to be run and maintained and we increase system load and lower response times for the duration. (The data format is versioned, so that's easy - no architectural problem there).

Its totally worth doing if you need to, and we're not afraid of that, but you don't do it on a whim - it takes proper planning and testing and needs multiple people involved.

The results in the tests are all with optimisations (-O3 -march=sandybridge -mtune=intel), as mentioned in the post.

The exception is the Debian packaged version of zlib, because we don't control that. That's the reason I include stock zlib in the tests - if it had been wildly different from the system zlib, I would have looked into recompiling the Debian package with more optimisations. There were no major differences and indeed, inspecting the package further shows that it is compiled with optimisations.

On the Cyrus side, we used to link to the Debian zlib, so we get those optimisations. For the new code bundled in Cyrus itself, we have to enable optimisations ourselves.

I think that mostly, your benchmarks have to match your workloads. Most of the CRC32 benchmarks I've seen are looking at larger buffers. The xxhash function mentioned elsewhere in this thread was claimed to be "an order of magnitude" faster, but again, large buffers - the gain over CRC32 on the same tests were rather more modest (though not at all insignificant).

In this case, I think (but am curious, will investigate further at some point) our Cyrus servers are doing enough checksumming work to keep any tables hot in the cache. So the tests are hopefully a useful indicator of where improvements can be made.

Same day, in fact: https://git.cyrus.foundation/diffusion/I/browse/master/lib/c...

My quick tests here (using the same methods outlined in the post) suggests its around 30% faster on 64-byte buffers. If we're ever shopping for a new hash function entirely, I'll make sure its considered. Right now its not worth it because as noted in the post, we're not actually under any particular stress and changing hash functions is a big job.

They do work, but only if you upload them via CardDAV - we don't have any UI for them in the web client yet.

Not a chance :)

In what way do you consider the privacy policy cavalier? What would you prefer it said?

I don't believe we made any claims that Australia is a "privacy Eden" or that we're "paragon of privacy". Indeed, we frequently say we're _not_ a privacy service, just an email service that cares about privacy among other things.

More specifics on both these points would help us discuss them properly.

Maybe. They'd have to issue that warrant to our datacentre operators though, not us, because there's nowhere to send the documentation. And then they can compel our datacentre not to talk about it, if they like, but they can't stop us talking about.

Really though, the point of all this isn't to say they can't take our servers - of course they can, via legal and illegal means. The point is more to say that they can't do it _quietly_, which greatly raises the bar, because now you've got a PR shitstorm to deal with.

But really, it's not going to happen, because we have good legal processes in place. There are proper channels from most countries in the world to the appropriate Australian authorities, and from there to us, and once that request comes in we service it and that's that.

If you want reasonably secure and private email, and you're not doing really dodgy shit, we're probably a safer choice that many. But we're not selling a privacy service, just an email service. If privacy is 100% non-negotiable for you, then you'll need to look elsewhere.

Yes, we're an Australian company.

https://www.fastmail.com/about/company.html

Dumb storage is fine if you only have a single actor that can maintain a copy of the state at all times. As soon as you've got multiple actors on the data (another client on your desktop or phone, or even something doing mail delivery), then you need a way to determine what changed on the server so you can update your state.

If the server has no ability to tell you what changed, then you're left with having to download and check everything. On a large data set, that's pretty much impossible to do quickly and without a lot of network traffic.

Of course, no server is actually that dumb - even a file listing with file sizes can get you part of the way there. But if you've got 10000 files in a file store, that list can still get pretty heavy. If you're willing to make the server smarter, eventually you can get to the point where the server can give you only what changed since the last time you checked. JMAP isn't unique in this; IMAP has MODSEQs, *DAV has collection synchronisation, etc.

JMAP specifically doesn't really care much about the actual format of the data it works with. The only thing it really needs is an immutable ID, so you could use the same model to store all sorts of things (and at FastMail we do, with things like client settings).

JMAP server implementations probably will require a good database, not just a dumb FS like in remoteStorage.

To do JMAP well you need to be able to calculate change sets, which does mean a database, though a fairly light one. I think you could do a server without delta updates by always returning a cannotCalculateChanges error in response to getUpdates calls, but it would be very inefficient on both the client and the wire.

However, I would like to collaborate wherever possible.

Sure! Best place to start is probably the jmap-discuss list:

  https://groups.google.com/forum/#!forum/jmap-discuss

No, because it's an actual application. Edit this and submit a PR: https://github.com/jmapio/jmap-perl/blob/master/htdocs/index...

If you want a laundry list of SSRF methods you should protect against, a great place to start is this slide deck from a talk at ONsec a couple of years ago:

http://www.slideshare.net/d0znpp/ssrf-attacks-and-sockets-sm...

Its thrilling and terrifying :)

Done: http://jmap.io/software.html

There's not much around yet. The only things that are in any kind of usable/complete state are the proxy, the demo client and the JS client library, all mentioned in the original post.

Cyrus has partial support (the basic data model and contacts) and is running in production at FastMail. The aim is for full support this year.

On the client side, Roundcube has committed to adding support in the next version (after their recent successful Indiegogo campaign) and Thunderbird is serious investigating it (there's a GSoC student looking at it).

I'm aware of two companies that will be shipping products (client and server) based on JMAP this year. I'm not going to mention names because that's not cool.

Finally, I'm personally building a JMAP data model library in Rust, a standalone (non-IMAP) server on top of it, and I'm part way through adding support to fetchmail and mutt.

I'm not aware of a sync service but it would be pretty straightforward to build. I'll have a look at it sometime, probably taking code from the FastMail app for Android, which has a partial JMAP client in it already.

I need to pull this info together into a "software" page on jmap.io - I'll try to do that soon. In the meantime, if you're working on something that uses JMAP, let me know :)

I don't believe there's anything in it that would prevent that. If you squint, the auth exchange looks a lot like SASL, and that supports pretty much anything.

This is something we'll gain more experience with at FastMail over the coming year. We'll be sure to share that experience.

That's pretty much exactly what it does. Client supplies a HTTP endpoint, that gets called with new state strings. That endpoint arranges for that information to be shipped to the client and then the client issues calls to update methods to get its view of the world up to date.

Its easy to extend to a custom (non-URL) method type if you like, and works well with mobile push channels. This is what we do with our mobile apps at FastMail to push through GCM, APNS, ADM and Pushy.

Yes, via the Authorization: header with a negotiated access token.

The spec includes a discovery and authentication mechanism that a client can use to get an access token and can work fine with passwords, OAuth, etc. We don't actually have a lot of experience with this part of the protocol though. We expect that vendors may want to provide an alternate method to obtain access tokens (maybe using OAuth tokens directly). We're expecting to learn more about these use cases as more implementations appear.

I, for one, am somewhat leery of people reinventing the wheel - especially for a browser-based environment, where everything has a history of being brittle and short-lived.

Yeah, so are we (and we even introduce JMAP with the "now you have X standards" XKCD comic). This is our last resort. We've been pushing the limits of what's possible with IMAP for years, and we've come to the conclusion that there's just some things you cannot do efficiently with it.

JMAP is not not just for browsers, though that has been where our focus is so far. I have partial implementations of JMAP for fetchmail and mutt already done (I'm just trying to find time to finish them and get them out there). Yes, JMAP uses JSON for its data encoding and (currently) HTTP for its transport, but it's not inherently browser-based.

IMAP works amazingly well if you're using dovecot as a server (including search and push e-mail), and CalDAV/CardDAV also work well if your server actually implements things properly, and projects like Horde have shown you can run and deploy web-based services for them at scale (even though I still regret they're using PHP).

The problem is, if you're not using a server like Dovecot or Cyrus that does implement all the extensions to IMAP that you need to get a modern email experience, then you're screwed. Ditto if your client doesn't support it. So we end up in the situation where we have all this stuff that is technically IMAP-compliant (ie implements RFC 3501) but gives a half-assed experience to the user.

Worse, if you wake up one morning and decide you want to build some amazing new experience on top of standards-compliant email, you'll spend the next few months trying to get your mail client bits working correctly before probably giving up. Even if you do get it working, you'll still have to work out what to do against servers that don't support CONDSTORE and QRESYNC. This is why all these pretty new mail-based apps generally target Gmail exclusively - it's a known environment.

So that's what JMAP is about. Use it and you get a straightforward protocol based on tech you probably already know that supports things you expect in 2015, like push, assistance on flaky networks, battery friendly, etc.

Sure, anything related to DAV has more than a few warts, but my main point here is that things like JMAP won't cater to an installed base or deliver benefits across the board until they finish filing all the corners.

Do you see anything missing from the spec?

We recognise that this doesn't work without everyone supporting it, which is part of why we're releasing the spec and all this code and talking to every company with an email product that will listen. This won't work unless everyone can play and its easy for them to do so. We don't want to own this; that doesn't help anyone.

I don't know if JMAP will win. Something has to change though, that I'm convinced of. The web had its revolution (a couple of times). We think its email's turn.

JMAP itself is transport-agnostic, though of course we've only implemented it over regular HTTP so far.

Some methods have to be batched together and executed in order, as it's possible to arrange for the output of one method to be used for an input to another method within the same batch. It is however entirely feasible to have multiple batches containing a single method in-flight at the same time and indeed, the demo client and the FastMail client do just this.

I'd certainly be interested to see if HTTP/2 could help anywhere but I don't think there's many obvious wins - JMAP itself has very little overhead.

We've written about this before:

http://blog.fastmail.com/2013/10/07/fastmails-servers-are-in...

FastMail's already pushing their own JMAP[2] protocol which I imagine they'd be very happy to become a standard. Perhaps everyone should wait for a standards body to come up with a solution, but personally I'd rather they be pragmatic and actually help end-users until that happens, even though I agree with you on principle.

That's pretty much it. We want open standards as much as anyone, and our efforts with JMAP and building solid IMAP support into Cyrus reflect that. That said, the iOS Mail app is our #1 IMAP client by a long away, and a very common complaint is that we haven't been able to do push with it - its even been enough to stop people moving from iCloud. As a business decision its a no-brainer.

I can't wait for CardDAV syncing.

Its actually enabled for all accounts and fully supported. We just haven't advertised it widely yet because its a big, complex feature and we've got enough people travelling at the moment that we didn't want everyone to start using it all at the same time!

Documentation is at: https://www.fastmail.com/help/clients/applist.html

You can see that it is leasted on our pricing pages: https://www.fastmail.com/signup/

We'll announce it officially in the next 2-3 weeks.

Since you work there.. why did you guys choose that approach for 2fa? Reading what's written on the help page about the master password, it sounds like you want the master password to serve as a recovery code. So why not just have recovery codes instead?

History, mostly. Ten years ago 2FA wasn't really a thing for most sites. We built an "alternative login" system, where you could create secondary passwords for your account and optionally set that password to be a "restricted" login. This was when using FastMail on untrusted machines (eg public libraries) was not at all uncommon, and HTTPS and even cookie-based login wasn't widespread and being able to create a throwaway password was very useful!

Over time we added paper-based OTP, Yubikey, SMS and so on but at its core its still just for secondary passwords.

We have started work on a rewriting the whole way this system works, which will work like more "conventional" 2FA systems, and will allow a second-factor on the master password. We're expecting it to be available later this year.

IMAP "push" usually means the IDLE extension, where a client holds a connection open and waits for the server to report that something has happened. This works ok, but isn't great on mobile because holding a TCP connection open is usually difficult on flaky networks and consumes battery.

iOS Mail doesn't implement it, instead doing a poll every 15 minutes. However it also implements a separate push system which allows true push if the server supports it. The details on how to support this aren't public information.

We talked to Apple, and they were kind enough to give us access to this system. We implemented it on our side, and now when an email arrives at FastMail we can immediately signal to Mail.app that there's new mail available.