I like StackExchange newsletters, which send you weekly emails for each site you sign up for containing top rated discussions. I subscribe to Electronics, Physics and, for fun, Aviation.
In February, I changed the email address associated with my StackExchange account to use my own server. After that, I noticed I no longer received any StackExchange newsletters! For a while, I figured there was something wrong with my email server that caused StackExchange not to deliver the mail. I eventually found time to debug the server last week, and discovered, incidentally, that the PTR record for my server (I run my own DNS server), which is used to map IP addresses to hostnames, pointed to attackllama.com and not alpaca.attackllama.com as it should*. After fixing that, I thought I would start to receive emails from StackExchange again. After waiting for a week, though, it was clear that this was not the problem.
I finally decided to do something about it, and forced StackExchange to send me an email by clicking the "I forgot my password" button. I then looked in my mail server's logs:
1 2 3 4 5 6 7 8 9 10 11 12 | Jun 16 09:30:41 alpaca postfix/smtpd[9251]: connect from mx-out.stackexchange.com[198.252.206.125] Jun 16 09:30:42 alpaca postfix/smtpd[9251]: Anonymous TLS connection established from mx-out.stackexchange.com[198.252.206.125]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) Jun 16 09:30:42 alpaca postfix/smtpd[9251]: D943F18400CE: client=mx-out.stackexchange.com[198.252.206.125] Jun 16 09:30:42 alpaca postfix/cleanup[9255]: D943F18400CE: message-id=<2.a00e7ff480d1dd548444@NY-WEB02> Jun 16 09:30:43 alpaca opendkim[1413]: D943F18400CE: mx-out.stackexchange.com [198.252.206.125] not internal Jun 16 09:30:43 alpaca opendkim[1413]: D943F18400CE: not authenticated Jun 16 09:30:43 alpaca opendkim[1413]: D943F18400CE: DKIM verification successful Jun 16 09:30:43 alpaca opendkim[1413]: D943F18400CE: s=dk d=stackoverflow.com SSL Jun 16 09:30:43 alpaca postfix/qmgr[25776]: D943F18400CE: from=<do[email protected]>, size=14102, nrcpt=1 (queue active) Jun 16 09:30:43 alpaca dovecot: lda([email protected]): sieve: msgid=<2.a00e7ff480d1dd548444@NY-WEB02>: marked message to be discarded if not explicitly delivered (discard action) Jun 16 09:30:43 alpaca postfix/qmgr[25776]: D943F18400CE: removed Jun 16 09:30:43 alpaca postfix/smtpd[9251]: disconnect from mx-out.stackexchange.com[198.252.206.125] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7 |
The first 3 lines show the successful, secure connection to my mail server from StackExchange - good. The next line shows the message ID received, before any filtering. The next four check that StackExchange's email server has DKIM records to prevent some forms of email spoofing. The next line shows the email entering into the processing queue, then the next shows that a sieve (filter) is discarding it because it was flagged by a filter. The next line shows the message being removed, and the last line shows the connection with StackExchange being closed.
So, it turns out one of my sieves was discarding the messages. I opened up my sieve file and found the following likely candidate:
# Trash messages with improperly formed message IDs if not header :regex "message-id" ".*@.*\\." { discard; stop; }
I took this from Ars Technica's guide on setting up a mail server. It's supposed to fight spam, by silently discarding badly formed message IDs that presumably only spammers would create. It turns out that simply checking the message ID with a regular expression like .*@.*\\. is insufficient. RFC guidelines on message ID validity are complex, leading to the need for horrendous regular expressions like
((([a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+(\.[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)*)|("(([\x01-\x08\x0B\x0C\x0E-\x1F\x7F]|[\x21\x23-\x5B\x5D-\x7E])|(\\[\x01-\x09\x0B\x0C\x0E-\x7F]))*"))@(([a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+(\.[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)*)|(\[(([\x01-\x08\x0B\x0C\x0E-\x1F\x7F]|[\x21-\x5A\x5E-\x7E])|(\\[\x01-\x09\x0B\x0C\x0E-\x7F]))*\])))
to be more correct, and even then, apparently not strictly correct.
Checking old emails from StackExchange, I found the message IDs lacked a trailing full stop (.) which caused the regular expression in the filter from Ars Technica to fail. However, as far as I can tell, having no full stop at the end of the filter is perfectly reasonable. Perhaps the filter should have had an asterisk after the full stop to ensure that there is a full stop somewhere after the @. I don't know.
It seems to me that trying to validate the message ID in a sieve to detect spam is not easy, and should not be attempted with a regular expression, so I just disabled the sieve by commenting it out. After that, boom! I was able to receive emails from StackExchange again. I guess I'll have to wait and see if the amount of spam I receive goes up (from zero, currently!) to check if this filter was having a positive effect before.
Isn't running your own mail server fun!?
*For DNS nerds: servers should not be given a hostname that is the same as the full domain name that points to it. I initially set up my server with a hostname `attackllama.com`, which essentially meant that the server should be the provider of all services on `*.attackllama.com`. While that is in fact the case for me, that's not how domains should be used: the mail server providing `alpaca.attackllama.com` can in principle be hosted on a different server, without the need to ever talk to the main `attackllama.com` server - that's the polymorphic magic of DNS. So, I fixed this by simply changing the hostname of my server to `alpaca.attackllama.com`, and correctly pointing the IP address via the DNS `PTR` record to that. That means the IP address points to a specific server on the `attackllama.com` domain, as it should. And, since some email providers check the reverse `PTR` record to make sure the IP address points to the mail server, this fixes some issues with mail not being sent by strict sending servers.