"mbox" is a family of several mutually incompatible mailbox formats

http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/mail-mbox-formats.html

by gnosis • 15 years ago

77 36 15 years ago

homepage.ntlworld.com.

Interesting URL there. A dot after the tld, and hosted on ntlworld, which was once the UK's most hated ISP (now owned by virgin and hated very slightly less.)

tententen• 15 years ago

A dot after the tld

That's actually how it's supposed to be done:

http://www.dns-sd.org/TrailingDotsInDomainNames.html

"It's a little-known fact, but fully-qualified (unambiguous) DNS domain names have a dot at the end. People running DNS servers usually know this (if you miss the trailing dots out, your DNS configuration is unlikely to work) but the general public usually doesn't. A domain name that doesn't have a dot at the end is not fully-qualified and is potentially ambiguous. This was documented in the DNS specification, RFC 1034, way back in 1987 ..."

but the site works just fine without the trailing dot as well.

djcapelis2• 15 years ago

> but the site works just fine without the trailing dot as well.

In your current environment! And probably in all the environments connected to the Internet. But potentially without the dot it would be perfect valid for this to route elsewhere in someone's environment. :)

julian37• 15 years ago

The dot after the tld is optional (if you don't need to specify a fully-qualified DNS name). Try for example: http://news.ycombinator.com./item?id=3438428

EDIT: I learned something new today, adding the trailing dot can be a good idea:

http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/we...

mike-cardwell5• 15 years ago

https://news.ycombinator.com./

Chrome doesn't accept the certificate when the trailing dot is in the request.

I bet if you configured your site to redirect to a URL with a trailing dot, it would break a lot of badly written robots. I'd be interested to hear about how it affects the major search engines before I'd want to implement it on any important sites.

slaven3• 15 years ago

Another wrinkle: some mailers stuff full 8-bit chars into mbox which will choke some parsers and end parsing prematurely (most expect 7-bit ASCII).

But it beats mdir handily in one way: if you're dealing with mailboxes with 100s of thousands of messages it is way more efficient.

ZenPsycho• 15 years ago

way more efficient along which dimension? Space, or processing speed? By roughly how much? Enough to be worth the costs in potential data loss?

sliverstorm• 15 years ago

Efficient use of inodes, perhaps?

slaven3• 15 years ago

Each mdir email is a file, so depending on your OS it consumes disk fragments very quickly and relies on your OS file handlers to allocate and load messages. mbox as a mapped file and a good index blows it away in speed and space usage.

kijin2• 15 years ago

One way to get around the different escaping rules between mboxo and mboxrd is to insert a space (0x20) at the beginning of any line that could be misinterpreted. This eliminates the need for any further escaping. Leading whitespace is almost always ignored by mail clients when displaying the message, especially when the text is flowed.

I've received received e-mails that were formatted this way, though I'm not sure exactly which software, whether client or server, is responsible for doing it.

dsr_• 15 years ago

Now you now have n+1 standards to keep track of.

As a side effect, this breaks every reader which is unaware of it and handling binary content in mail (many, many messages) or languages or encodings which are space sensitive.

kijin2• 15 years ago

Binary content and non-ASCII characters are almost always base64-encoded, and base64 never produces a From_ sequence (because of the space).

Anyaway, I'm not endorsing the leading-space technique. I'm just observing that it's out there in the wild.

riobard2• 15 years ago

A side question: what's the best way to build an email message store which stores millions of messages (and adding tens of thousands on average each day) on disk so that it can support:

1) Fast looking up messages sent/received by a particular email address, time, tag, priority, and a handful other properties

and

2) Appending thousands of new messages per second at peak?

gwillen• 15 years ago

If you only ever want to append messages, and never want to mutate or remove them, I would do this scheme in two parts: 1) an append-only message store; 2) a set of indices stored alongside (1). To append thousands of messages per second, I would probably want a system with multiple writers appending to multiple files (mbox) or directories (maildir), and cycling/archiving the mailboxes as they get large.

As for (2), I know a lot less about indexing messages than I do about storing them, but I would try out an off the shelf SQL or noSQL system and measure the performance, before trying to build anything fancy.

JoshTriplett4• 15 years ago

Maildir (or a very similar one-message-per-file store) plus an index of some kind.

If not for the continuous addition of messages, I'd suggest notmuch (http://notmuchmail.org/), which does a great job of indexing in any way you could want. Unfortunately, as far as I know notmuch does not have an "online" indexing mechanism; someone wrote a notmuch-deliver that adds a message and indexes it, but I doubt it would meet your performance requirements. You could probably make notmuch do what you want, but you might want a new indexing mechanism instead.

I still think you want something very similar to Maildir underneath though.

riobard2• 15 years ago

I was using the one-message-per-file approach, but keeping millions of small files in a filesystem is quite a challenge: I have to worry about available inodes, and listing/manipulating these files takes minutes!

djcb• 15 years ago

There are some systems based on the Xapian full-text indexer, including my own 'mu'[1] which might be able to do this. It expects the messages in a Maildir, and it can handle > 150 messages per sec, on fairly slow hardware - so after a peak of thousands of new messages, it would take a few seconds to catch up. Tens-of-thousands per day would be no problem. There have been people using [1] for a few million of messages without any problem, but I'm not sure how well it scales beyond that.

[1] http://www.djcbsoftware.nl/code/mu, or 'maildir-utils' in debian/ubuntu

beza1e1• 15 years ago

You could look at http://notmuchmail.org/ which builds on top of maildir.

Karellen• 15 years ago

See also JWZ's article on the "Content-Length" header (mboxcl and mboxcl2 formats from the original article)

http://www.jwz.org/doc/content-length.html

"mbox" is a family of several mutually incompatible mailbox formats

Related Stories