Diving deep into emails: SMTP, envelopes, and headers
How email works is a lot more complicated than you might expect
I’m working on an exciting new open-source project that is essentially a self-hosted/managed email aliasing service (more on this at the end!). In a nutshell, it works by generating disposable aliases (e.g. uniquealias@mydomain.com) and forwarding any received emails on that address to your personal email address.
In the simplest use case (single recipient), all you have to do is to receive the original email, change the “to” email address to your personal email address, and send it out.
Things get significantly more complicated, however, when there are multiple recipients. Consider the following example:
From: sender@company.com
To: uniquealias@mydomain.com
Cc: boss@company.com...
If we were to simply receive, modify, and send the email, Mr. Boss will receive 2 copies of the email (one from Mr Sender, and one when Mr Unique Alias re-sends it).
If we clear out the “cc” header, we would receive an inaccurate version of the email (because the email has been destructively modified).
What we want is to preserve email metadata without actually re-sending the email during the forwarding process. We want Mr Unique Alias to send an email to my personal email address with Cc to include Mr Boss, but without actually Cc-ing Mr Boss in this re-sent email.
How email works
To figure this out, I dived into a rabbit hole of technical details surrounding this almost 50 years old technology. This is what I’ve learned.
Email (or electronic mail) works like… well, the electronic version of regular mail. Regular mail is delivered as a letter contained in an envelope. The envelope contains information that instructs the postal service where to deliver the mail to, and after delivery, the recipient reads the letter contained within. (Let’s assume that the recipient does not actually bother with reading the information on the envelope.) As you can imagine, some information, such as delivery address, is replicated twice — at the top of the letter and on the outside of the envelope. (Let’s also assume that fancy see-through envelopes are not a thing.)
Similarly, an email message consists of a few parts: headers, body, and an envelope (read more about them here). The body of an email is straightforward, so we’ll skip that and go straight to headers and the envelope.
Headers
Headers contain information such as “from”, “to”, “cc”, “subject”, etc. When you open an email in an email client such as Gmail, you’re reading from the email headers.
Email headers are the equivalent of the information at the top of postal letters. No, that’s not a typo. Email headers do not determine where your emails get sent to.They are read by email clients to display information to the user. Most of the time, they match the information of the email envelope, because why would it not?
To understand how emails actually get routed and delivered, we need to go a level deeper to the world of SMTP envelopes.
SMTP envelopes
SMTP (or Simple Mail Transfer Protocol) is the communication protocol for email transmission. SMTP envelopes tell email servers where to send the emails to, where email clients read the email headers and body to display to the user. For the purposes of email delivery, the servers are only concerned with envelope information — not what the headers say. (You wouldn’t expect the postal service to open up your envelopes to read the addresses at the top of your letters, right?)
For more information, check out this Superuser answer.
Solution
Therefore, the solution to our original problem is theoretically simple enough. If we want to forward an email from Mr Unique Alias to our personal email address without re-sending it to Mr Boss, we only have to ensure that:
- The SMTP envelope’s recipient information (contained in
RCPT TO) contains only our personal email address. - The email’s Cc header contains Mr Boss’s email address.
Implementing this idea, however, is a lot trickier. My project is written in Javascript/Typescript and runs on AWS’s Simple Email Service, so naturally, I started looking at npm packages and AWS’s SDK.
Firstly, AWS’s SDK is too high-level to implement this solution. Both its sendEmail and sendRawEmail methods are too high-level, working at the level of email headers only. Fortunately, after reading through its documentation, I discovered that it supports sending emails through SMTP.
There aren’t a lot of libraries that can work with emails and SMTP at such a low-level. Some SMTP client libraries that I found worked well enough for connecting to a SMTP server to send a message, but had no way of customizing the SMTP envelope and headers separately.
Eventually, I found nodemailer. By default, it accepts a message object, from which it automatically generates an SMTP envelope.
// Message object copied from nodemailer's documentation
{
from: '"Fred Foo 👻" <foo@example.com>', // sender address
to: "bar@example.com, baz@example.com", // list of receivers
subject: "Hello ✔", // Subject line
text: "Hello world?", // plain text body
html: "<b>Hello world?</b>" // html body
}However, it is possible to specify the SMTP envelope by passing it as a property on the message object.
// Modified from the snippet above
{
from: '"Fred Foo 👻" <foo@example.com>', // sender address
to: "bar@example.com, baz@example.com", // list of receivers
subject: "Hello ✔", // Subject line
text: "Hello world?", // plain text body
html: "<b>Hello world?</b>", // html body
envelope: {
from: "fakesender@example.com",
to: "someoneelse@somewhereelse.com"
}
}In this example, the email would be sent to Mr Someone Else, where he would see it as being sent from Mr Fred to Mr Bar and Mr Baz. (That’s right, someoneelse@somewhereelse.com will not appear anywhere in the email headers.)
I can’t confirm this, but I suspect it is how Gmail implements its email forwarding feature :).
You can check out my work-in-progress project here! This has been an incredibly exciting project, and I’ll be writing about that in an upcoming post.
