NGINX as Mail Proxy Server: A Beginner’s Guide

Photo by Mikaela Wiedenhoff on Unsplash

Let’s first look at the definition of an SMTP proxy server.

SMTP proxies are specialized mail transfer agents (MTAs) that, similar to other types of proxy servers, pass SMTP sessions through to other MTAs without using the store-and-forward approach of a typical MTA. When an SMTP proxy receives a connection, it initiates another SMTP session to a destination MTA. Any errors or status information from the destination MTA will be passed back to the sending MTA through the proxy

In this blog, we will be configuring and using NGINX as a Mail Proxy Server. NGINX mail proxy supports SMTP, IMAP and POP3 protocols and can proxy email client traffic to one of the upstream mail servers. It brings several advantages like easy scaling of mail servers and distributing load among the mail servers based on some rules (like the recipient to address).

Follow this guide to install NGINX

Configuring SMTP Mail Proxy Server

Let's start with NGINX configuration file.

mail {
    server_name mail.test.com;
    auth_http   http://127.0.0.1:8000;
    # disabling xclient command
    xclient off;
    server {
        listen     3333;
        protocol   smtp;
        smtp_auth  none;
    }
}

server_name: name of the mail server

auth_http: HTTP authentication server for redirecting the request based on some rules (We will discuss this in detail in the next section).

server: proxy mail server running on 127.0.0.1:3333. We are not using any authentication for our server (SMTP auth is none).

Let’s verify the proxy mail server by telnet to 127.0.0.1 on port 3333.

Connect to Mail Proxy

Setting Authentication Server

The proxy server on receiving a request from a client, asks the HTTP server for authentication (in our example there is no authentication). On successful authentication, it returns the mail server’s IP address and port based on rules.

HTTP/1.0 200 OK
Auth-Status: OK
Auth-Server: <host> # the server name or IP address of the upstream server that will used for mail processing
Auth-Port: <port> # the port of the upstream server

Below is a simple python authentication server using the recipient address rule for choosing the mail server. The recipient address is sent in the header Auth-SMTP-To. Proxy also sends other important authentication-related headers which are out of scope for this example. You can read the details here.

Note: We are running two local mail servers on ports 2525 and 2526 for accepting the redirected traffic by the mail proxy. I am using go-guerrilla, any mail server will do the job.

In this example, we are redirecting traffic to 127.0.0.1:2525 if the recipient is eng.abc.com and to 127.0.0.1:2526 if the recipient is hr.abc.com

Before moving forward, let’s verify the authentication server.

curl -vvv -H “Auth-SMTP-To: RCPT TO: <foo@eng.abc.com>” -i http://localhost:8000

Authentication server response for recipient eng.abc.com

Working Flow Example

NGINX Mail Proxy Flow

1. Mail client sends an SMTP request to NGINX Mail Proxy server.
2. Mail Proxy sends the client request to the Authentication server along with a header like Auth-SMTP-To.
3. The authentication server decides the upstream mail server to connect to based on the recipient address rule and returns the mail server IP and port in the response header.
4. Mail Proxy redirects the client’s request to an appropriate mail server.

Let’s look at an end-to-end example to understand the complete flow.

In the first example, RCPT TO is user@eng.abc.com, so the mail proxy redirects the traffic to the 127.0.0.1:2525 mail server.

When RCPT To is eng.abc.com, traffic is redirected to 127.0.0.1:2525

In the second example, RCPT TO is user@hr.abc.com, so the mail proxy redirects the traffic to the 127.0.0.1:2526 mail server.

When RCPT To is hr.abc.com, traffic is redirected to 127.0.0.1:2526

I have attempted to show a basic end-to-end flow to get started. Hope this was helpful. Thanks for your time:-)