Lemon Email Infrastructure, pt. 1

Our goal was to build a fault-tolerant, secure and auto-scalable system that could, potentially, handle a huge amount of users. As you will see, we used a lot of different, mainly new, technologies while working on this project. Below, we will talk about these technologies and what best suited our needs. Also, the main focus of this post is about how we built our regular email system and how it fits in the overall Lemon Email picture and what our plans are in terms of open-sourcing this project.

Why create a regular email system when there is a private one?

If you’re familiar with Lemon Email, you probably already know that we have two email systems working together — regular and private email. We won’t be talking about our private system in this post, but it is important for you to know that these are two different systems and that they share just a part of the overall infrastructure. Let’s say someone wants to send an email from our private system to another external email service(e.g. Gmail) and vice versa. If that would be the case, a relay between those two “worlds” would need to exist. That’s where our regular email comes in, as we use its SMTP server as a relay. Also we figured users would want to send a regular email from time to time that doesn’t need a good secret question and answer when sending informal messages to their friends.

If you’re interested in learning more about our private system, we wrote a little bit about it in our previous blog.

What technologies did we use and how do they work together?

As it turns out, building an email service can be quite complex. With so many pieces working together, things are bound to break, no surprise there. We made some good decisions early on, when we started working on this project, regarding which pieces of technology we were going to use to create our email system. Some might say we have kind of an obsession with open-source products, and they would admittedly be right. Throughout our project we’ve only used open-source solutions and we’re quite proud of that. Following the same open-source logic, for our SMTP and IMAP solutions, we decided to go with Postfix and Dovecot.

Postfix is powerful enough on it’s own to handle hundreds of thousands of incoming and outgoing emails being sent every second. We chose Postfix because it is in constant development, improving its performance and security with every new release, appeared to be the best solution. Dovecot serves as a Mail Delivery Agent, delivering emails to their respective email folders. To avoid overloading the IMAP container with email storage, we decided to use NFS to store all emails and mount the volume on the IMAP container. This works perfectly with our auto-scaling infrastructure in a way that all containers, especially those added through auto-scaling, would have access to all emails saved on NFS.

Nowadays, every system needs a proxy server standing in front of other servers providing an extra layer of security and traffic management. For that, we decided to go with NGINX. Nginx comes with an authentication module that works as a separate container dedicated for authentication only. When there is a login request, it’s forwarded from the proxy server to the authentication container which authorizes the user and sends the response back to the proxy server.

An email system cannot be complete without a spam filtering service. Whether emails are incoming or outgoing, they are sent to our spam filtering service which is done by Apache’s SpamAssassin. Headers and message content are scanned and awarded positive or negative points, depending on what score the email gets determines if the email will be flagged as spam or not.

Now let’s see how it all works in action with some visual representation and a step by step walk-through:

Incoming email example:

1. An email from an external email service arrives to our system.
2. The email is forwarded from our proxy server to our SMTP server.
3. SMTP server checks if the recipient exists in our MySQL database. If the recipient exists, the email is sent back to the SMTP server.
4. Next, the email is sent to the spam filter. Message content and headers are scanned for keywords that could indicate a spam email. If the filter determines that the email is genuine, it is sent back to the SMTP server.
5. The SMTP server forwards the email to our IMAP server.
6. IMAP server saves the email to our NFS storage.
7. Email is sent to your inbox.

Outgoing email example:

1. An email is being sent by a Lemon Email user.
2. IMAP server saves the email to our NFS storage and forwards it to the SMTP server.
3. SMTP server checks if the recipient is a Lemon Email user in our MySQL database.
4. Email is forwarded to the spam filter and scanned for potential spam content, if the system determines that the email is genuine, it’s sent back to the SMTP server.
5. If the recipient is a Lemon Email user, SMTP forwards the email to the IMAP server that saves it to NFS. Otherwise, the email is sent to an external email service.

So far, we’ve covered the underlying technology we use and how it all works, now for the part where it all comes together.

How is everything managed?

As we’ve said, we wanted to make a fully scalable and containerized system that could potentially support millions of users sending hundreds of thousands of emails per second. Even though we don’t have millions of users (yet!), we feel like we’ve done a pretty good job at achieving goals we’ve set for this project. We decided to use Docker to create images of our SMTP, IMAP and other services and store them in AWS’s EC2 repositories. This means that they can easily be used for auto-scaling and, in case of system failure, restarted with speed and ease.

To help us with cluster management, we decided to use Kubernetes to create a cluster of containers that use the Docker images we have built. That way, when one container in our cluster fails, another container is started in its place, guaranteeing we have no downtime of our service. Also, if there is a lot of traffic on our for e.g. SMTP container, the system creates another container and alleviates the load between all active SMTP containers.

There are probably things we’ve left out in this post regarding the overall infrastructure, however, these are the key components that are running our systems.

Open-sourcing in progress

Recently, we’ve open-sourced an email service that you can run on your local machine or server. It allows you to send and receive emails locally, as well as send emails externally. For more details and if you’re interested in testing it out, visit our GitHub page. Our next step will be to open-source the decentralized part of our infrastructure and, after that, our production infrastructure, along with our entire AWS infrastructure, which requires a little bit more work and tweaks. You can visit our website for more information about the project and the people behind it.