Fixing discarded messages in Dovecot sieve filters due to incorrect message IDs

I like StackExchange newsletters, which send you weekly emails for each site you sign up for containing top rated discussions. I subscribe to Electronics, Physics and, for fun, Aviation.

In February, I changed the email address associated with my StackExchange account to use my own server. After that, I noticed I no longer received any StackExchange newsletters! For a while, I figured there was something wrong with my email server that caused StackExchange not to deliver the mail. I eventually found time to debug the server last week, and discovered, incidentally, that the PTR record for my server (I run my own DNS server), which is used to map IP addresses to hostnames, pointed to and not as it should*. After fixing that, I thought I would start to receive emails from StackExchange again. After waiting for a week, though, it was clear that this was not the problem.

I finally decided to do something about it, and forced StackExchange to send me an email by clicking the “I forgot my password” button. I then looked in my mail server’s logs:

Jun 16 09:30:41 alpaca postfix/smtpd[9251]: connect from[]
Jun 16 09:30:42 alpaca postfix/smtpd[9251]: Anonymous TLS connection established from[]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Jun 16 09:30:42 alpaca postfix/smtpd[9251]: D943F18400CE:[]
Jun 16 09:30:42 alpaca postfix/cleanup[9255]: D943F18400CE: message-id=<2.a00e7ff480d1dd548444@NY-WEB02>
Jun 16 09:30:43 alpaca opendkim[1413]: D943F18400CE: [] not internal
Jun 16 09:30:43 alpaca opendkim[1413]: D943F18400CE: not authenticated
Jun 16 09:30:43 alpaca opendkim[1413]: D943F18400CE: DKIM verification successful
Jun 16 09:30:43 alpaca opendkim[1413]: D943F18400CE: s=dk SSL
Jun 16 09:30:43 alpaca postfix/qmgr[25776]: D943F18400CE: from=<>, size=14102, nrcpt=1 (queue active)
Jun 16 09:30:43 alpaca dovecot: lda( sieve: msgid=<2.a00e7ff480d1dd548444@NY-WEB02>: marked message to be discarded if not explicitly delivered (discard action)
Jun 16 09:30:43 alpaca postfix/qmgr[25776]: D943F18400CE: removed
Jun 16 09:30:43 alpaca postfix/smtpd[9251]: disconnect from[] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7

The first 3 lines show the successful, secure connection to my mail server from StackExchange – good. The next line shows the message ID received, before any filtering. The next four check that StackExchange’s email server has DKIM records to prevent some forms of email spoofing. The next line shows the email entering into the processing queue, then the next shows that a sieve (filter) is discarding it because it was flagged by a filter. The next line shows the message being removed, and the last line shows the connection with StackExchange being closed.

So, it turns out one of my sieves was discarding the messages. I opened up my sieve file and found the following likely candidate:

# Trash messages with improperly formed message IDs
if not header :regex "message-id" ".*@.*\\." {

I took this from Ars Technica’s guide on setting up a mail server. It’s supposed to fight spam, by silently discarding badly formed message IDs that presumably only spammers would create. It turns out that simply checking the message ID with a regular expression like .*@.*\\. is insufficient. RFC guidelines on message ID validity are complex, leading to the need for horrendous regular expressions like ((([a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+(\.[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)*)|("(([\x01-\x08\x0B\x0C\x0E-\x1F\x7F]|[\x21\x23-\x5B\x5D-\x7E])|(\\[\x01-\x09\x0B\x0C\x0E-\x7F]))*"))@(([a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+(\.[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)*)|(\[(([\x01-\x08\x0B\x0C\x0E-\x1F\x7F]|[\x21-\x5A\x5E-\x7E])|(\\[\x01-\x09\x0B\x0C\x0E-\x7F]))*\]))) to be more correct, and even then, apparently not strictly correct.

Checking old emails from StackExchange, I found the message IDs lacked a trailing full stop (.) which caused the regular expression in the filter from Ars Technica to fail. However, as far as I can tell, having no full stop at the end of the filter is perfectly reasonable. Perhaps the filter should have had an asterisk after the full stop to ensure that there is a full stop somewhere after the @. I don’t know.

It seems to me that trying to validate the message ID in a sieve to detect spam is not easy, and should not be attempted with a regular expression, so I just disabled the sieve by commenting it out. After that, boom! I was able to receive emails from StackExchange again. I guess I’ll have to wait and see if the amount of spam I receive goes up (from zero, currently!) to check if this filter was having a positive effect before.

Isn’t running your own mail server fun!?

*For DNS nerds: servers should not be given a hostname that is the same as the full domain name that points to it. I initially set up my server with a hostname, which essentially meant that the server should be the provider of all services on * While that is in fact the case for me, that’s not how domains should be used: the mail server providing can in principle be hosted on a different server, without the need to ever talk to the main server – that’s the polymorphic magic of DNS. So, I fixed this by simply changing the hostname of my server to, and correctly pointing the IP address via the DNS PTR record to that. That means the IP address points to a specific server on the domain, as it should. And, since some email providers check the reverse PTR record to make sure the IP address points to the mail server, this fixes some issues with mail not being sent by strict sending servers.

Spectra Microvision Plus install guide for modern operating systems

The Spectra Microvision Plus is a nice residual gas analyser (RGA), allowing one to examine the elements and molecules present within a vacuum system. In the speed meter group at the Institute for Gravitational Research, we use it to test our vacuum system for cleanliness, leaks and the presence of unwanted elements before we place our delicate optical surfaces inside and shine high power lasers upon them.

Spectra Microvision Plus. This box is able to analyse the residual gas profile of a vacuum system, but it’s tricky to install the aged software on modern machines.

Despite being around 15-20 years old, this hardware is solid and still works nicely. The problem is that the software used to control the box is a similar age, and has well and truly suffered from bit rot. In fact, it came with the unit in the form of three floppy disks. These were old even when I was growing up, and CDs and later DVDs became the standard distribution channel for software. Adding insult to injury, the software is only designed to work on Windows 3.1 (!), 95, 98 or XP, meaning any modern computer is going to struggle to run it. Finally, it demands a physical RS-232 connection to the machine, since it harks from the days when USB was but a teenager with braces and oily hair, and Apple and IBM were still preventing one standard from ruling them all by pushing FireWire.

Incidentally, if you’re looking to download the Spectra RGA For Windows software that is used with the Spectra Microvision Plus, it’s unfortunately not available directly from the manufacturer. They apparently insist on users paying $2k for newer hardware. If you want the software, though, just send me an email – I’ve made images of the floppy disks for safe keeping.

Spectra RGA For Windows software running on VirtualBox on a Linux host

I managed to get the software to work by using VirtualBox to emulate Windows XP on a Linux host. I run Ubuntu on my machine, so VirtualBox is easily installable via the package manager. Fortunately, my university still has XP campus licences kicking around, so I fired up an ISO and installed XP as a virtual machine within VirtualBox. If you need to know how to install XP on VirtualBox, just google it. If you run Windows or Mac OSX, don’t dismay – VirtualBox is also available on those platforms, so this will still work provided you can obtain a Windows XP disk or image.

Interface to RS-232

Now comes the slightly more tricky part. As stated earlier, the RGA uses RS-232 for communication between the hardware and the RGA For Windows software, and modern computers don’t come with RS-232 ports. However, given that USB is, eponymously, universal, you can buy an adapter which emulates an RS-232 serial interface over USB for a few pounds:

RS-232 to USB adapter. Could it be any simpler?

I got mine from, but you’ll find them anywhere with a quick search. Ubuntu 16.04, but probably all recent Linux kernels, supports this adapter out of the box, giving you a virtual RS-232 port that you can hand over to the virtual machine.

If you use Ubuntu or a Debian derivative, make sure your user account is part of the “dialout” group (run usermod -a -G dialout [your-username] as root, then log out and back in again), otherwise you won’t be able to read from and write to the serial adapter.

Once you’ve got an RS-232 USB adapter, plug it in and fire up VirtualBox after having installed Windows XP. Before starting the operating system, however, you need to tell VirtualBox to pipe the RS-232 connection through to the virtual Windows XP system. In the settings for the Windows XP virtual machine, go to “Serial Ports” and enable Port 1. Then choose “COM1” for the port number and “Host Device” for the port mode, assuming that your kernel has successfully identified the adapter (run dmesg in a terminal and look at the kernel messages to check). Put /dev/ttyUSB0 as the Path/Address, but check that this is actually the path to the serial adapter you plugged in (again, use dmesg or similar to find out). Once this is done, you should be able to fire up Windows XP and look in the device manager to see a new COM port present.

Installing, configuring and running the RGA For Windows software

Given that the software was distributed on floppy disks, I first had to get a USB floppy disk drive and create images for each of the three disks (using dd in the terminal – again, Google is your friend). VirtualBox can mount these images in turn, allowing your virtual Windows XP machine to think it actually has a real floppy disk drive attached. I inserted the first “disk”, opened up “A:” in Windows and started “INSTALL.EXE”. It is very simple: after clicking next a few times and inserting the second and third virtual floppy disks when asked, it was installed.

To get the software to run, the first thing to do is to run the “Configure” program from the start menu. For me, it found the only COM port configured on the computer (COM1) and was happy. After that, fire up the “RGA” software with the RS-232 connector plugged in to the Spectra Microvision Plus unit. You should be presented with the default screen – hurrah! – and after that, you can take some measurements. Here is a screenshot of the RGA For Windows software running on a Windows XP virtual machine, itself running on Kubuntu 16.04:

The Spectra RGA For Windows software running on a virtual Windows XP machine, running on an Ubuntu host.

There we go! A virtual machine that is able to talk to the Spectra Microvision Plus RGA, future proof and fit for modern operating systems. Hopefully this allows this old but solid hardware to continue to serve a purpose in modern research labs. Let me know if you run into any issues when getting the software to work – I’d rather not let Spectra hustle thousands of dollars out of customers just by refusing to update or release 20 year old software…