How to Host a Personal Email Server on Google Cloud (for Free!): Part VI

Filtering Spam with Rspamd & Sieve

Articles in this series

1. Introduction & GCP Setup
2. Configuring Postfix, Mailgun, & DNS Records
3. Configuring Dovecot & Encryption
4. Managing Virtual Mailboxes with MariaDB & Postfixadmin
5. Hosting Webmail with Roundcube
6. Filtering Spam with Rspamd & Sieve

If you have not read the previous articles in this series, please follow the links above to catch up. We now have our server running on GCP configured to send and receive email via the SMTP protocol and allow IMAP connections, while encrypting traffic with TLS. Our DNS records are configured properly to ensure email is delivered to & from us, and we are using virtual mailboxes so that there is no need for a local unix user for each mailbox. These are stored in a MariaDB database and managed using Postfixadmin via a web browser. We can access our mailbox from email clients or in the browser via Roundcube webmail.

The only step left is to handle spam mail. We will install Rspamd to detect spam. Rspamd scans messages for patterns to determine a spam score. The higher the score, the more likely it is to be spam. Mail with a high spam score will be outright rejected, but messages with a moderate spam score will still be delivered because we want to be sure we’re not rejecting legitimate mail due to false positives. In these cases, an X-Spam header will be added to the message. Therefore, we can use Dovecot’s Sieve plugin to filter messages with this header into the Junk folder. Awesome, right? Let’s do it!

Installing Rspamd

Rspamd uses Redis to store data, so we have to install both.

sudo apt install rspamd redis-server -y

Configuring Redis

As we are on a tight RAM budget, let’s limit the memory that Redis can use. We will also tell Redis to remove the key closest to expiration when memory needs to be freed. Add these lines to /etc/redis/redis.conf.

maxmemory 256mb
maxmemory-policy volatile-ttl

Save and restart Redis.

sudo service redis restart

Configuring Rspamd

Postfix communicates with Rspamd over the milter (mail filter) protocol. Since we are running Postfix & Rspamd on the same machine, we can use unix sockets to communicate between the two more efficiently, so add the following line to /etc/rspamd/worker-proxy.inc.

bind_socket = "/var/spool/postfix/var/run/rspamd/milter.sock mode=0666 owner=_rspamd";

We can also set the Proxy worker to self_scan mode which allows us to disable the Normal worker, freeing up our precious resources. Edit the upstream section of the same file to enable self_scan.

upstream "local" {
  default = yes;
  hosts = "localhost";
  self_scan = yes;
}

Then, add the the following line to /etc/rspamd/worker-normal.inc.

enabled = false;

We need to create the socket directory and give ownership to the _rspamd user & postfix group.

sudo mkdir -p /var/spool/postfix/var/run/rspamd/ && sudo chown -R _rspamd:postfix /var/spool/postfix/var/run/rspamd/

Next, we need to tell our Rspamd bayes classifier to use Redis. We will go ahead and turn on auto learning, which is just one of MANY ways that Rspamd will do its job. To see a list of all the ways it filters spam, read this. Create /etc/rspamd/local.d/classifier-bayes.conf and add the following lines:

servers = "127.0.0.1";
backend = "redis";
autolearn = true;

We should also tell Rspamd to not mark emails in a thread as spam once a user has replied so create /etc/rspamd/local.d/replies.conf and add:

action = "no action";

Then restart Rspamd.

sudo service rspamd restart

Rspamd is now ready to detect spam and improve through learning. If you want a boost out of the gate, you can find collections of email to train with. As this is not necessary, I will not be covering it here. Perhaps it merits a follow up article of its own, if there’s interest. Now we need to connect our email software to Rspamd.

Connecting Postfix to Rspamd

Add the following lines to /etc/postfix/main.cf to connect the two.

smtpd_milters = unix:/var/run/rspamd/milter.sock
milter_default_action = accept

milter_default_action tells Postfix what to do in case there is an issue preventing Rspamd from being reached. Restart Postfix to apply the changes.

sudo service postfix restart

Incoming messages will now be scanned by Rspamd. High scoring spam will be rejected, and moderately scoring spam will be delivered with the X-Spam header. Now we need to configure Dovecot to sort any mail with this header into the Junk folder.

Using Sieve to Handle Spam

Sieve is a scripting language used to filter email. To use it, we first need to install the dovecot-sieve plugin. Let’s also create a directory to store our Sieve scripts.

sudo apt install dovecot-sieve
sudo mkdir /etc/dovecot/sieve/

Moving Spam to Junk Folder

We must tell Dovecot to use the plugin by editing the protocol lmtp section section at the bottom of /etc/dovecot/conf.d/20-lmtp.conf.

protocol lmtp {
  mail_plugins = $mail_plugins sieve
}

Next, we need to write a short script that tells Dovecot to move emails containing the X-Spam header into the Junk folder. Create /etc/dovecot/sieve/move-spam-to-junk.sieve and copy the following script into the file.

require ["fileinto"];
if header :is "X-Spam" "Yes" {
        fileinto "Junk";
}

Then we have to compile this script using the sievec utility.

sudo sievec /etc/dovecot/sieve/move-spam-to-junk.sieve

Lastly, open /etc/dovecot/conf.d/90-sieve.conf and edit the following line:

sieve_after = /etc/dovecot/sieve/move-spam-to-junk.sieve

This tells Dovecot to run the move-spam-to-junk script after running any user-defined scripts.

Learning from User Actions

We can train Rspamd when a user moves a message to or from the Junk folder. To do so, we must activate the Sieve plugin for IMAP. Open /etc/dovecot/conf.d/20-imap.conf and edit the protocol imap section at the end of the file.

protocol imap {
  mail_plugins = $mail_plugins imap_sieve
}

Now, let’s write our Sieve scripts for reporting spam & ham.

Create /etc/dovecot/sieve/report-spam.sieve and copy the following script:

require ["vnd.dovecot.pipe", "copy", "imapsieve"];
pipe :copy "learn-spam.sh";

Then create /etc/dovecot/sieve/report-ham.sieve with the following script:

require ["vnd.dovecot.pipe", "copy", "imapsieve"];
pipe :copy "learn-ham.sh";

These scripts call shell scripts that we are about to write. The shell scripts will in turn call Rspamd’s learn_spam or learn_ham functions. Let’s go ahead and compile them.

sudo sievec /etc/dovecot/sieve/report-spam.sieve && sudo sievec /etc/dovecot/sieve/report-ham.sieve

Next, create /etc/dovecot/sieve/learn-spam.sh containing:

#!/bin/sh
exec /usr/bin/rspamc learn_spam

Then, create /etc/dovecot/sieve/learn-ham.sh with the following lines:

#!/bin/sh
exec /usr/bin/rspamc learn_ham

We also need to make our shell scripts executable.

sudo chmod u=rwx,go= /etc/dovecot/sieve/learn-{spam,ham}.sh

Now open /etc/dovecot/conf.d/90-sieve.conf once more and edit the following line to enable the Sieve plugin for IMAP.

sieve_plugins = sieve_imapsieve sieve_extprograms

Then, add the following lines inside of the plugin section of the same file.

# Move to Junk folder
imapsieve_mailbox1_name = Junk
imapsieve_mailbox1_causes = COPY FLAG
imapsieve_mailbox1_before = file:/etc/dovecot/sieve/report-spam.sieve
# Move from Junk folder
imapsieve_mailbox2_name = *
imapsieve_mailbox2_from = Junk
imapsieve_mailbox2_causes = COPY
imapsieve_mailbox2_before = file:/etc/dovecot/sieve/report-ham.sieve
sieve_pipe_bin_dir = /etc/dovecot/sieve
sieve_global_extensions = +vnd.dovecot.pipe

These rules tell Dovecot which Sieve scripts to run when a user moves mail to or from the Junk folder. sieve_pipe_bin_dir tells Dovecot where to find our shell scripts and our sieve_global_extension setting enables the pipe plugin, enabling us to pass email to external commands.

Finally, restart Dovecot to apply all of our changes.

sudo service dovecot restart

That’s it! Our server is all set to handle spam and learn from user interaction!

FYI, Rspamd has a lot of configuration options that are beyond the scope of this article. You can adjust the score parameters, configure training, etc. You can also further configure mail filtering by writing Sieve scripts or using Sieve manager clients, such as the plugin for Roundcube. To learn more, start here.

Now let’s test it out.

Testing Spam Filtering

Testing Rejection

We can use the Generic Test for Unsolicited Bulk Email (GTUBE) pattern to test our spam filtering. To do so, simply send yourself an email (from another account) containing the following string:

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

You should get the following delivery status notification that the email was blocked. Nice!

We can check that Rspamd is scanning & scoring all incoming messages that make it to our mailbox as well by enabling additional headers and checking them. To do so, create the file /etc/rspamd/local.d/headers.conf with the following line:

enable_test_patterns = true;

Then restart Rspamd.

sudo service rspamd restart

NOTE: I choose to leave this off on my deployed server, leaving only the X-Spam header for classification. However, if you choose to leave it on, there is no negative impact.

Now send yourself an email. In Roundcube, we can easily check the headers.

If we click on Headers, a popup will show us all of the message headers. We can scroll down and find the headers added by Rspamd. A small snippet of the available information is shown below.

You can find the headers using any email client as well, if you prefer. The method will vary, so you may want to search for instructions for your preferred client online. If the message had been determined to be spam, the X-Spam: Yes header would have been added and our Sieve script would have moved the message to our Junk folder.

Conclusion

We made it! Both literally and figuratively. It’s been a long, tedious, and hopefully, fun journey! We went from nothing but a dream and maybe a domain name to hosting our own private email server in the cloud always having reliable access to our mailbox. To make our accomplishment even sweeter, we can keep it running at no cost!

Feel free to brag to your family & friends about what you achieved. You earned it! You don’t want them to be jealous though, so hop onto your admin portal and make them a mailbox @ your domain. It’s easy!

Let’s quickly recap how we wrapped up our personal cloud email journey:

• We installed Rspamd, configuring it to use Redis to store data & learn.
• We configured both Rspamd & Redis to limit resource usage.
• We connected Postfix to Rspamd to scan & classify incoming messages.
• We configured Dovecot using Sieve scripts to move spam to our Junk folder and to train Rspamd when we move messages to & from Junk.

Congrats again! I hope you learned something cool on this journey, and I really hope you enjoy your new personal email!

As always, thank you for reading! If you have found this series or article helpful, please clap and follow. Comments are always welcome too!