DIY Mail

https://computer.rip/2021-12-26-diy-mail.html

by zdw • 5 years ago

173 75 5 years ago

YPPH2• 5 years ago

Pretty much reflects my view.

Mail is hard to set up properly, but it's not ridiculously hard to set up. Anyone with average Unix administration skills can do it. I might add that OpenSMTPd is a lot easier than postfix and makes a dangerous open relay misconfiguration hard to do, so it's my recommendation to the uninitiated.

It requires maintenance attention, but not a ridiculous level like some suggest.

What is ridiculous is the effort you will go to to stop your connections being bounced, or (worse) your emails going to spam folders. It's this third reason why I use Fastmail to run my email. Technical accumen is of some but limited assistance here. In essence you will you will be struggling to build an IP reputation with low traffic flow. This article has some interesting insights on this topic.

akerl_• 5 years ago

Saying it’s not ridiculously hard to set up your own mail server, unless you want recipients to actually get your emails when you send them, feels a bit irrelevant?

Most things are easy if you’re willing to accept an end result that doesn’t do one of the two (3? Maybe? Send, receive, search?) primary functions.

walrus014• 5 years ago

The frustrating part is that you can have the most impeccable Postfix configuration for your smtpd, and your system has never once emitted a single spam outbound to anyone, but your mail to office365 or other places might disappear into a black hole.

I know probably 4 or 5 people who are far more than capable of running their own mail server and have given up on it these days, bowing to the inexorable market pressure of gsuite/google workspace/office365 for their MX, because they just can't be bothered by having outbound mail deliver-ability issues. These are people who were running smtpd for regional ISPs in 1997. It almost seems like the google and microsoft spam filters are a bit of a protection racket, forcing people to buy their services.

The author of the article covers this:

If you self-host email, you will run into an elevated number of delivery problems. That is a guarantee. Fully implementing trust and authentication measures will help, but it will not eliminate the problem because providers weight their IP reputation information more than your ability to configure DKIM correctly. Whether or not it becomes a noticeable problem for you depends on a few factors, and it's hard to say in advance without just trying it.

Federated systems like email tend to rely on a high degree of informal social infrastructure. Unfortunately, as email has become centralized into a small number of major providers, that infrastructure has mostly decayed. It was not that long ago that you could often resolve a deliverability problem with a politely worded note to postmaster @ the problematic destination server. Today, many email providers have some kind of method of contacting them, but I have never once received a response or even evidence of action due to one of these messages... both for complaints of abuse on their end and deliverability problems [3].

YPPH2• 5 years ago

Perhaps I should have been upfront with the third point. I was trying to convey that, in my view, getting condemned to spam is the most compelling reason to not self host mail - and it's reason enough. The other common criticisms, by comparison, aren't that strong. If you're relying on your email to work, apply for jobs, or even just send things to friends using consumer services, you either have to sacrifice self hosting, or go through a lot of effort to obtain and build IP reputation.

I think it's a shame you can't reliably self host mail without jumping through hoops around IP reputation. But it's just the reality. You can be an idealist or you can be pragmatic.

In a similar vein, I now only ever use transactional email services (SES, Sendgrid) for automated emails. The sacrifice you make with self hosting is also significant in this context.

brazzy• 5 years ago

I think what GP means with "not ridiculously hard" is purely the technical effort to set up everything correctly.

The IP reputation issues that will prevent your emails from being delivered are not really a technical matter.

Hard_Space• 5 years ago

I realized the truth of this this year, when I launched my own self-promotional website (just a portfolio site) and assiduously configured its email, which passes every single test on the range of spam-rating testers available.

Almost none of the mail gets through.

Now I am researching outsourcing the site's mail to one of the usual suspects: mxroute, FastMail, ProtonMail, Exchange, etc. Perfectly configured mail that no-one gets to read is worthless.

sliken• 5 years ago

How old is your domain? I have noticed that email providers are biased against new domains.

aidenn0• 5 years ago

The worst for me has been Office 365; I have an e-mail with an organization where I'm not allowed to configure the spam settings, and my spam folder is about 90% ham typically. I finally found out that you can whitelist wildcards (though it won't let you whitelist *@*.tld) so every time I get ham in my spam folder, I just whitelist the entire domain, and I run the spam filtering client-side.

I'm now down to the point where I only get one or two ham messages in my spam folder a month. I don't get very much actual spam to that address at all; presumably they have an even higher level of filtering where it doesn't get delivered at all.

walrus014• 5 years ago

I actually clicked this thread after reading the first part of the article, and saw that the author has already made 3 very important points (in ASCII bullet points) that I came here to say basically the same thing. All of this is extremely important in terms of real world outbound mail deliverability.

Do not cheap out on providers. Run your mailserver in the most reputable IP space you can find. Most likely you will use some type of cloud instance or VPS (probably in such a way that the line between the two is blurry). Unfortunately, per-month cost tends to correlate directly with IP reputation quality, as does name recognition. But neither are any guarantee. AWS has plenty of IP reputation problems. Sometimes a little-known, brand-new VPS provider can be a surprisingly good option because they got their IP space off of some staid corporation that maintained an impeccable security program. Do some research to try and find out what kind of results people get from different providers.
From an IP reputation perspective, consistency is key. Mailservers and IP reputation are like credit cards and your credit report: once you open one, you want to keep it as long as possible so that it will build a good IP reputation. Do not move your mailserver around between providers to save money, it will result in headache.
Do not run a mailserver off of IP space that is not intended for commercial services. For example, running a mailserver on a residential ISP is a terrible idea (if it even works, many residential ISPs drop outbound on port 25). You will be guaranteed to have IP reputation problems and, moreover, IP space allocated to end-users usually gets reported to a "policy block list" such as the one Spamhaus maintains... meaning your outbound email will be blocked by recipients simply because it's coming from an IP that should not have mailservers, so it's most likely to be from a compromised end-user device.

===================

Now to add my own 2 cents:

If you are serious about doing this, find a medium sized long-established ISP where it is still small enough that you can get in contact with the people who run the routers. And that ISP should be one that really cares about its IP space reputation.

If you can afford it, anywhere with long-held IP space belonging to its ASN that has never had $5/month VPS customers on it, or any form of shared hosting environment, is much more likely to have "clean" IP space.

geocrasher2• 5 years ago

Great. Can you suggest such a provider, or 3?

walrus014• 5 years ago

What region or city?

jcrawfordor6• 5 years ago

I'd recommend looking around locally in your city/region. Lots of cities have small, well-established companies that do a grab-bag of MSP, IT, Microsoft hosting, and offer some VPS or colo. They're usually on the expensive end but they come with the benefit that they've usually had their IP space for a long time and manage it carefully (unlike basically any "modern" provider), and since they're in town it's usually easier to get someone on the phone if there's a problem. Make friends with the people who run these kinds of businesses anyway, in a lot of smaller markets especially there's a bit of an informal club of IT service providers and if you chat with them regularly you'll end up getting good advice and business leads. I've had a colo inquiry to a local company turn into them sending me one of their clients before just by being chatty with the network guy.

rsync• 5 years ago

"Of course requiring a validated telephone number as part of identity is a substantial compromise on privacy and effectively eliminates identity compartmentalization ..."

That's what the "2FA mule" is for.

It's a stock android phone with no google account and no apps installed except for "SMS Forwarder"[1].

It is configured to forward all SMS to an email address via encrypted SMTP. This means that I can receive these 2FA codes anywhere I have Internet access - such as an airplane or newly arrived in a foreign country where my SIM card does not work.

The "2FA Mule" itself is plugged in at my office in a corner.

I'm not employing this for anything sensitive but it's interesting to consider that I can use SMS based 2FA while divorcing it from my day to day SIM identity ...

[1] https://play.google.com/store/apps/details?id=com.frzinapps....

commoner• 5 years ago

A free and open source, ad-free alternative to SMS Forwarder is SMS Gate:

- F-Droid: https://f-droid.org/en/packages/com.github.axet.smsgate/

- GitLab: https://gitlab.com/axet/android-sms-gate

g5095• 5 years ago

One of the biggest hurdles I've experienced is that a number of large players in the industry (think G..argantuan) systematically flag private mail servers as spam almost as a matter of course and it's up to you to prove your innocence. There is a clear financial benefit for this sort of behavior as it pushes people back onto their services.

sbuk• 5 years ago

The bulk of this post discusses IP reputation which is the biggest hurdle to self hosting. It is the first line defence for spam/phishing/fraudulent mail. SPF and DKIM can align for DMARC, but if the IP has a bad, or no reputation, chances are the message will be bounced. Think of it like having a low or no credit score. As an example, if a “cold” IP address suddenly starts sending mail, mail security will take the (arguably cynical) view that chances are it’s some sort of spam. It even more cynical to assume that this is done as a money grab, shepherding users to big mail services, when in fact, it’s the best way to prevent bulk mail.

jcrawfordor6• 5 years ago

The irony too is that, if you run a mailserver of substantial size, you will become very sympathetic to the other side on this. Dropping email from new/suspect IPs will tremendously reduce your user complaints about spam. It's not really a winnable situation.

upofadown• 5 years ago

Monopolies gotta monopoly. In the end though they can't lose too much of their incoming email before people will move to email services that are reliable.

jmconfuzeus• 5 years ago

Scrolling on this web page is really annoying.

meling• 5 years ago

This was a great read and goes to show how broken email has become… I often wonder if we will ever migrate to something better collectively, or we end up on a variety of different communication platforms.

vbezhenar• 5 years ago

I very rarely use e-mail for anything. Most interactions are via whatsapp, telegram. Not sure if it counts as a better alternative.

jiggunjer3• 5 years ago

Yes, if you think about it it's weird how govt. application forms have an email field. Perhaps email should be nationalized.

alex_duf2• 5 years ago

nationalized by which government?

tommek40773• 5 years ago

To whoever always ends up in spam: stop using cloud providers. Spammers use cloud providers, so you get banned, too. I am having my own mail server for over 15 years, no it is not a lot of maintenance. I had to create some spf records at some point in time and thats it.

jiggunjer3• 5 years ago

I read some ISP will block port 25, while other mail providers block residential IPs. It seems there is no rule that always applies.

tommek40773• 5 years ago

The rule is: no residential ip and no cloud provider.

amelius• 5 years ago

One problem is still if you're sending email on behalf of other people. One of those people might do something (maybe even something innocent) that will put your entire service on a blacklist.

NetOpWibby• 5 years ago

I think this has happened to me.

normaler• 5 years ago

i never had an issue with the big mail servers. All the smaller ones broke me. T-online.de etc.

brongondwana2• 5 years ago

Agree - the really big providers are pretty good because you can get in contact with somebody who gives a shit at the other end and can fix things up. They do so much volume that they can afford somebody who cares about it.

The really painful things are either places that subscribe to dodgy blocklists and never check their configuration, or... yeah, places like t-online.de and free.fr.

messo• 5 years ago

How is IPv6 coverage in the email provider landscape these days? I guess it would be easier to build up a decent IP reputation if it was on a "virgin" IPv6 address used only for this purpose, and not tainted by prior use etc.

IPv6-only for email is probably still a pipe dream, I guess...

brongondwana2• 5 years ago

It's pretty bad - nobody is really doing it.

On the plus side, since it's all pretty fresh - you can do things like "MUST be DKIM signed from an aligned reverse DNS" on IPv6 and have a chance of it working.

LeonM2• 5 years ago

It's pretty bad - nobody is really doing it.

I'm curious how you came to that conclusion.

Messo's question above made me curious, so I just did a quick query on the database of Mailhardener. >99% of email coming from Gmail is delivered over IPv6 (from ~10k samples).

This appears to be provider specific though. For example: Yahoo seems to mail exclusively over IPv4, Zoho also IPv4. Most exchange based services appear to prefer IPv6. Microsoft 365 also uses IPv6.

So your conclusion doesn't really reflect what I am seeing in our database. Obviously, we process mostly DMARC reports, so maybe regular email is more IPv4 biased to support mail services that are not IPv6 capable.

LeonM2• 5 years ago

I guess it would be easier to build up a decent IP reputation if it was on a "virgin" IPv6 address used only for this purpose, and not tainted by prior use etc.

IP addresses (especially IPv6) are ephemeral, email providers know that. Just about every email service has a TTL on IP reputation.

Usually, if an email is DKIM signed (and aligned) then domain reputation is used as the key indicator, the sender IP becomes less relevant in that situation.

Obviously, each implementation is different. There are no silver bullets here.

But what I'm trying to say here is: do not worry about inheriting a 'tainted' IP address. With the limited number of IPv4 addresses available, and the huge volume of spam, just about every publicly available IP would have been 'tainted' by now.

sam_goody• 5 years ago

I run a mail server [Ubuntu + Postfix + Dovecot + Sieve], and then use a service , (for me - Postmark[1]) to actually send the mail.

The integration is extremely easy to setup [2], and solves the deliver-ability issue. Most of these services also take care of the DKIM and DMARC setup, so it is much easier to get things right. And they all offer debugging tools if something isn't sending or returning correctly.

On the other hand, they are not in privacy abusing position that Gmail is, and I also don't have to worry that they will close my account with no recourse, as Gmail once did to me.

The biggest downside is no email forwarding. There is no way to do email forwarding using a service, since they only send mail from approved senders. (There are Bad Idea solutions, such as rewriting headers with SRS. Don't.)

And of course, I need to pay for sending. Mailgun used to have a generous free tier, and Amazon SES is decent at $1/10K mails, but things add up. I chose Postmark for many reasons, but the pricing of different companies is worth looking into if you are sending enough mail. (Sendgrid and Mailchimp Mandril, for example, have a unlimited per deliveree charge, that could conceivably be cheaper for some use cases.)

OT, the only reason that I support PoP3 is for Gmail: The only way to "check an external account" to write and send mails in Gmail is PoP3.

[1]: postmarkapp.com [2]: https://postmarkapp.com/support/article/832-can-i-configure-...

narag• 5 years ago

So in case you don't want to send email, just receive for you, there's not much problem... ???

z0f• 5 years ago

Correct. You can receive emails on any IP, sending is the only time reputation comes into play.

WhyNotHugo• 5 years ago

Another big issue is silently rejected emails.

A client of mine had corporate clients who relied on MS's hosted email solution. Often emails would be accepted from our MTA onto theirs, but never show up in inboxes or spamboxes. They were just silently discarded, even though the IP had pretty good reputation (and had been used to send email for years).

Eventually, the problem got so bad they just had to switch to a hosted email solution, since there was no way to work around it. I would have insisted the other company deal with it, but they wouldn't pay my client if the invoices weren't emailed, so we didn't really have a choice.

southerntofu2• 5 years ago

The problem in this case is Microsoft. They've been known to do this for years in order to reinforce their email oligopoly. Unfortunately instances where your business lost money is the only thing Nation State regulators care about, but in this specific instance, you could have sued Microsoft and made a ton of money and maybe paved the way for their stopping to do so entirely from fear of further fines.

Maybe it's not too late. You'd be doing all of us a major service.

tiahura• 5 years ago

“in order to reinforce their email oligopoly.”

Cite?

syntheticnature• 5 years ago

I have seen this with all the major email providers; least often with Google, but it has still happened.

pydry• 5 years ago

These stories have hammered home two points for me:

* You probably shouldnt use email for sensitive communications ever again.

* While it's not worth self hosting a mail server to send and receive email there's still a huge privacy value in hosting email storage and using a service that only passes on emails via POP3/SMTP (ideally with minimal logs).

akshayrajp2• 5 years ago

Does anyone know how to go about building a blog like that? What SSG/theme does the author use? Looks really good.

jcrawfordor6• 5 years ago

It's a short and pretty ugly Python script that reads a directory of text files and plugs them into Jinja2 templates (virtually all of the actual logic is in the templates). I find myself using this pattern pretty often because I am very much not a web developer and find most SSGs to be insufferably complicated for my use cases.

When I write new posts, I use a shell script that runs them through aspell, then runs the site generator, does a simple concatenation to build the email version, and calls pandoc to build the secret hidden pdf version.

akshayrajp2• 5 years ago

Thank you for describing how you build/develop your blog. Maybe I'll implement some of that for myself.

emptybottle• 5 years ago

Running mail is complex, and maybe it’s not the best use of your time personally. But please stop dissuading engineers from attempting to run their own mail infrastructure if desired.

More people getting familiar with how reliable email systems work is a good thing.

With regard to sending mail, a few things go a long way. Check your IPs on dnsbls before sending email from them. Don't expect to send mail successfully from a public cloud address. have at least one backup mail system that isn't listed in any dnsbl. Use separate IP addresses for transactional email, user email and bulk email. Deploy SPF, DKIM and DMARC for your domains.

deltarholamda• 5 years ago

Totally agree. I think nearly everybody of the HN stripe should run an email server, if only on a vanity domain. There has to be some kind of pushback against centralizing email among Google, Microsoft, et al. Email is fundamental to the open Internet. It's the "killer app," as they used to say in ye olden tymes. It does not belong to 3-4 international corporations, it belongs to us.

The article pooh-poohs iRedmail and some others, and while he's not wrong, I can say that iRedmail is about as close as you can get to learning all about the inner workings of a mail delivery system and still be reasonably one-click install. It is opinionated to some degree, but it doesn't lock you in. I've used it for years and have been happy. Unfortunately, delivery problems (as the author noted) meant I have to use Mailgun for their relay service.

I think this is a fight we should continue to have. It's worth it.

nickdothutton• 5 years ago

I feel the same way. I wonder if users of hackernews would be interested in a community (semi)Pubnix and email service. Traditional multiuser system with thousands of concurrent users catered for and all the “old” services plus some new ones.

lowwave• 5 years ago

Don't forget rDNS.

0xbadcafebee2• 5 years ago

But please stop dissuading engineers from attempting to run their own mail infrastructure if desired.

I don't think we're trying to dissuade them. I think we're fighting the insidious memes of "Look, it's easy! I did it in 5 minutes!" which perpetuates a potentially hazardous misunderstanding. If you don't do e-mail right, you can end up sending or receiving an important piece of official mail that nobody will ever read. Job offers gone unseen, billing reminders turned invisible, customer service tickets lost, warranty redemption missed, urgent family notices not gotten in time, etc. We're just saying it's much more nuanced and unintuitive than just turning on the software.

It's like the blind leading the blind: somebody's going to end up in a pit. We're putting caution tape around the pit. They can still choose to go under it and explore.

kkjjkgjjgg• 5 years ago

Reading that article, I had the thought that maybe it is time to abandon mail for all communication and only use it for the required signup processes with web sites.

I had actually given up on mail months ago, in the sense of not considering it a reliable channel anymore. But I haven't been strict about enforcing it, and instead live with the risk of missing important messages.

lazyweb• 5 years ago

A lot of good points. I've been running my own private mail server for almost seven years [1], it's been a very valuable learning experience.

[1] https://jschumacher.info/2021/05/running-a-private-mail-serv...

simonebrunozzi• 5 years ago

Should be Email, not Mail.

softwarebeware• 5 years ago

This pretty much convinced me not to run my own mail server. Thanks!

Anyone using Protonmail with a custom domain?

ipodopt• 5 years ago

I have used both pm and migadu with custom domains. Miguda ends being much better on that front feature, pricing, and ease of use wise.

https://www.migadu.com/pricing/

You do lose the layer of encryption protonmail provides. But it is essentially boils down Encryption at Rest and e2e when between two protonmail users. It can be mimicked by pulling down the mail from the servers to your machine and using pgp.

3np2• 5 years ago

It needs to be said that you can get most of the benefits of your own mail-server and still route outgoing SMTP through a provider like mailroute/sendgrid/mailgun/AWS/even google.

It’s not an all- or nothing question.

Similarly, if you’re worried about the risk of potentially losing incoming email if you have issues, you can keep your existing mail provider choice as fallback in the DNS and fetch+delete mail going there into your primary mailbox every 5 min or so using mbsync.

andris9• 5 years ago

When I started building https://wildduck.email (a completely independent email server implementation) then my goal was to build sonething that is deeply integrated, there would be only a single way of doing things, no file based storage, and management would be done over a REST API. I don’t regret a single decision from back then, the non-unix way has proven to have a lot of upside.

PS. It’s not mentioned often but when you send out emails you need to send at least 100 emails per IP per large destination _every day_ (100 to Gmail, 100 to O365 etc). Otherwise you’re not even on the radar and can’t build any reputation. You need to have a constant mail flow going out for the big guys to take you even remotely seriously.

DIY Mail

Related Stories