Tweaking Your eMail System

This page is updated from time-to-time; most of the updates will be at the bottom.

Setting up your own eMail system is no easy task. It's likely that your Drupal system will need to use the eMail functionality on your server even if you don't think you will email out with your system. A wrinkle in this effort for you is that many ISP's block normal incoming email from personal servers; e.g. outbound on port 25 coming from a low rent DHCP connection. You aren't paying for a "Business" connection so you look like a potential scammer/spammer if you are trying to send out email. That means you may set everything up right but still never get a single eMail to leave your server and get to the rest of the world. You will only get messages out via the SMTP server of another provider some how.

The first thing you should do is go to the Ubuntu Server Guide for Email Services of Postfix. Start doing what it says from the Basic Configuration section, because you should already have the Postfix on your system by now. Look in your Servers section of Webmin to see it. You probably have Dovecot running too. As I understand things, Postfix is a complete email system, and Dovecot is a server that lets your users collect their email via IMAP or POP.

Once you get to the step 2 part of the SMTP Authentication section of the instructions above, it will get tricky. That's why I have the video below. Once I followed these instructions and lost all Webmin access because Ubuntu didn't assume you might have used a competing management system, and Webmin sets itself up as the CA on your server. I followed the link in the Step 2 instructions and turned on a new CA without telling Webmin and didn't know how to fix that, so gave up and rebuilt the server.

Before you watch the video and try it at home, you should have executed commands from

• sudo dpkg-reconfigure postfix

to

• sudo postconf -e 'smtpd_recipient_restrictions = \ permit_sasl_authenticated,permit_mynetworks,reject_unauth_destination'

Because I'm going to send eMail from my OutsourcedMath server through the Google G-Suite system...

• sudo apt-get install mailutils mailutils-mh mailutils-doc nmap fetchmail

...they weren't on my system yet, but Postfix and Dovecot was. The nmap program will help you diagnose port errors and other domain settings.

And because I'm trying to work out a problem of sending the email to the Google Suite eMail servers, I don't believe I'll be able to generically tell anybody how they should set up their particular situation. It seems like their's more ways to do eMail than their are countries on the globe. An extensive reading of the mail logs showed me this gem...

• Jun 22 14:54:06 OutsourcedMath postfix/smtp[29811]: CA8D214814AE: to=<me@gmail.com>, relay=smtp-relay.gmail.com[173.194.206.28]:587, delay=0.56, delays=0.09/0.02/0.42/0.04, dsn=5.7.1, status=bounced (host smtp-relay.gmail.com[173.194.206.28] said: 550-5.7.1 Invalid credentials for relay [68.133.1.126]. The IP address you've 550-5.7.1 registered in your G Suite SMTP Relay service doesn't match domain of 550-5.7.1 the account this email is being sent from. If you are trying to relay 550-5.7.1 mail from a domain that isn't registered under your G Suite account 550-5.7.1 or has empty envelope-from, you must configure your mail server 550-5.7.1 either to use SMTP AUTH to identify the sending domain or to present 550-5.7.1 one of your domain names in the HELO or EHLO command. For more 550-5.7.1 information, please visit 550 5.7.1 https://support.google.com/a/answer/6140680#invalidcred i37sm368317qkh.2 - gsmtp (in reply to MAIL FROM command))

...witch led me to this page for the how to fix it. The moral of the story is, learn to look at your log files and search for the errors on Google.

After struggling with one error after another, I decided to loosen the requirements in Google Suite to the "Not Recommended" level, "Allowed Senders: Any Address". This temporary move allowed me to guess and test my way through difficult decisions about what was the root cause of my troubles. It turned out to be me.

Their were two decisions I made to change things without full knowledge of why they were default settings. One was changing the Postfix server's General Options setting for "Local internet domain name" from "com" to nothing and radio button for "Default (provided by system)". The other was to change the Address Rewriting and Masquerading settings for Rewrite to all "Yes" radio buttons. The real problem was getting rid of the "com".

Experimenting with the Rewriting settings allowed me to see what they were doing to my sent from, and I discovered that sometimes I was sending from user@sample.sample.com, or from user@sample. Eventually I got wise to that thing I did and put "outsourcedmath.com" in the "Local internet domain name" field, which was also wrong, then I reverted it back to just "com" and all was well. You can have all rewrites turned on when this is the case, and I think sending from Virtual Domains will be easier with the "com" value. If I hadn't loosened the settings in G-Suite, I wouldn't have seen the malformed sender ID. After making things good again, I could tighten-up the security at G-Suite to only allow from authenticated users and only from my IP address.

At long last, I'm sending encrypted and signed eMail messages from my own server.

10/21/2017 - Okay, Jesus-balls-Christ, email servers are freakin' complicated!

This as an attempt to self-organize my current efforts to get email to work right from my server instead of bouncing off of another relay like Google. Don't read it yet as I haven't had success yet, just anguish.

The following linked notes are almost irrelevant if you use MailGun as described in the next section.

• Testing your existing system https://intodns.com/

• SPF Record creation in Domain Custom Resource Records

• Add DomainKeys Identified Mail (DKIM) signature to outgoing mail

  • then finish up the DKIM with this posted solution https://unix.stackexchange.com/a/74491

• And something about encryption related to DANE but via LetsEncrypt.

• Maybe the best instructions I've seen from the redhat Linux people

12/16/2017 Update Finally got everything eMail working the right way behind DHCP

Most of the default settings for Postfix were good, but if your domain is happysales.com and you want to email from and to somebody@happysales.com from behind a DHCP connection, you need a Relayhost Service. I'm not sure how many their are, but MailGun is the one I was suggested by a trusted acquaintance. I love it so far.

If you are only doing one FQDN with your server, just put the settings in the way their help tells you to after you finish "getting started" with them. MailGun will gently walk you through the process of putting your SPF, DKIM & DMARC records in your custom resource records wherever you have your domain listed.

If you are hosting many sites, you'll need to employ the Sender Dependent Authentication portion of Postfix. Your /etc/postfix/main.cf file will look like this somewhere after you are done:

# See http://www.postfix.org/SASL_README.html#client_sasl_sender for details

smtp_sender_dependent_authentication = yes

sender_dependent_relayhost_maps = hash:/etc/postfix/sender_dependent_relayhost_map

smtp_sasl_auth_enable = yes

smtp_sasl_password_maps = hash:/etc/postfix/smtp_sasl_password_map

relayhost = [smtp.mailgun.org]:587

smtp_sasl_security_options = noanonymous

# TLS support

smtpd_tls_security_level = may

smtp_tls_note_starttls_offer = yes

And you'll need to create /etc/postfix/sender_dependent_relayhost_map, then put this in it,

# Per-sender provider; see also /etc/postfix/smtp_sasl_password_map.

user1name@domain1name.com [smtp.mailgun.org]:587

user2name@domain1name.com [smtp.mailgun.org]:587

user3name@domain2name.com [smtp.mailgun.org]:587

And you'll need to create /etc/postfix/smtp_sasl_password_map, then put this in it,

user1name@domain1name.com MailGunUserDomain1:MailGunPassword4Domain1

user2name@domain1name.com MailGunUserDomain1:MailGunPassword4Domain1

user3name@domain2name.com MailGunUserDomain2:MailGunPassword4Domain2

Then to "activate" the two newly created files (or to refresh them when they get edited), run:

• sudo postmap /etc/postfix/sender_dependent_relayhost_map

• sudo postmap /etc/postfix/smtp_sasl_password_map

So, Ubuntu's LAMP set-up doesn't have Postfix using TLS, and installing Webmin doesn't activate the TLS either. Virtualmin does, but it's only via the snake-oil key & cert. This part of the Postfix documentation shows how to DIY the TLS if you aren't paying for those keys somewhere else. Here's the command's I put in the SSH Terminal Window:

• sudo postconf compatibility_level = 2 (only if you see the "Postfix is running with backwards-compatible default settings" error in your mail logs)

• sudo postconf smtp_tls_security_level = may (unless you already have "smtp_tls_security_level = may" in your /etc/postfix/main.cf file)

• sudo postconf smtp_tls_loglevel = 1 (unless you already have "smtp_tls_loglevel = 1" in your /etc/postfix/main.cf file)

• sudo postconf smtpd_tls_security_level = may (unless you already have "smtpd_tls_security_level = may" in your /etc/postfix/main.cf file)

• sudo postconf smtpd_tls_loglevel = 1 (unless you already have "smtpd_tls_loglevel = 1" in your /etc/postfix/main.cf file)

• sudo postfix tls new-server-key (the on screen messages will give you the following two commands when you do it)

• sudo postfix tls deploy-server-cert /etc/postfix/cert-20171216-062044.pem /etc/postfix/key-20171220-062044.pem

• sudo postfix tls output-server-tlsa -h <your hosted FQDN> /etc/postfix/key-20171216-062044.pem

  • The output of that statement gives you the signature for your server that you put in your domain hosting's Custom Resource Records section. If you are using the Let's Encrypt for your server key and cert, it will change every 90 days, and so will the TLSA signature. You'll re-run the above command and then ADD the new signature to your list of Custom Resource Records. For me on Google Domains, I clicked edit on the existing record, then a "+" icon, and put the new 3 1 1 segment into the field. The suggestions I see is to leave that old one on for a few Time To Live cycles before removing it such that your TLSA record only has the new one.

  • Edited on April 8, 2019: I erroneously wrote in the last bullet point that it would change as Let's Encrypt updated. That's wrong. It's only when you do a "sudo postfix tls new-server-key" that you need to do a "sudo postfix tls output-server-tlsa -h example.com /etc/postfix/key-20171216-062044.pem" command to find the new info for your DNS. Also noteworthy, that "<your hosted FQDN>" section seems irrelevant because its the same output for all my domains and even "sudo postfix tls output-server-tlsa /etc/postfix/key-20180123-173720.pem" gives the same results. I don't know anything about having different keys for different domains, but that's probably where it matters.