A/B Testing An Email — First Impressions

George Abraham
Jul 13, 2019 · 6 min read

This is the first post in a 3 part series about A/B testing an email.

Email is an invaluable channel in a comprehensive messaging program for any business. Depending on your audience there are a few key levers within an email that lends itself well to testing. There are multiple areas of testing outside of the actual email itself such as time, day, frequency, personalization signals, etc. that are also valuable to test but those areas are not the focus of this particular series of stories.

First impressions

You need to make a compelling case to get your customer’s attention in a crowded inbox. There are a few levers you can experiment with to make a good first impression and persuade your customer to see the value in your message.

From Name

The From Name is how you identify yourself to your customer. A few examples follow. Can you recognize the different tactics and guess what those may be trying to do?

AllModern is using the From Name as an area to push their July 4th clearance sale like what you would expect in a Subject Line. DoorDash and The New York Times are leaning into their brand by setting the field to simply reference the company name. bobdylan.com is maybe using the field as a way to draw the user back to the website even if they don’t open the email. Domino’s is differentiating their transactional order confirmation message in the name. Domino’s likely has a different From Name for their offers email. Using long names could result in some email clients truncating the field as is the case in the emails from PG&E and AllModern.

Subject Line

The subject line is probably the most widely tested area in email marketing. However, it is also one of the most abused areas since a lot of email marketing effectiveness is measured in email opens rather than core business metrics. Sometimes all the information a customer needs is right in the subject line, that the customer doesn’t even need to open the email. A notification that an item has shipped is a basic customer expectation and typically has all the information a customer needs in the subject line. So an email open doesn’t necessarily translate to the effectiveness of the message. My colleague Chris Beaumont has a post on this publication that illustrates the folly in relying on opens as a core metric of an email campaign

Let’s look at some examples of Subject Lines in action.

Emojis tend to be used a lot in retail emails. The same tactics used in brick and mortar stores to grab your attention are also effective in email. Flashy signs and $xx.99 numbers tug on customers FOMO strings in stores. Similar tactics are used in the subject line to draw customers into opening an email. The following subject lines seem to have a sense of urgency. Even when there is no time dimension in the copy — you feel as though you are missing out on these “deals” if you don’t check them out right away.

Account Updates are probably the least opened emails but these emails are expected as a standard product experience. Unless you want to use this opportunity to engage customers, a simple subject line can get the job done.

Below are some examples of well crafted subject lines intended to engage the customer as part of a messaging program that creates value for the customer using the product. Subject lines in emails from subscription services to their members tend to be less clickbaity as they are trying to increase their customers’ engagement and help the customer extract maximum value from their subscription.

What do you think of the following email from the Bernie Sanders 2020 campaign? Do you think the missing subject line is a mistake?

Image for post
Image for post

Think again! Here’s an example email I sent to myself without a subject line. The iOS mail app defaults to a gray colored text with “No Subject” as the fallback copy. Compare that with the subject line above viewed in the same email client.

Image for post
Image for post

Yes — subject lines are really a great lever to use in implementing an effective email program.

Preview Text

Preview text is the text that is displayed below the email subject line in email clients to give the user of the client an idea of what is in the email. It is typically displayed in lighter font compared to the subject line by mail clients. Email marketers are leaning into preview text copy as a way to distinguish their emails in a customer’s crowded inbox. If you don’t explicitly set the preview text, various email clients will render the copy from a fallback option such as the plain text version of the email or by scraping the initial part of the HTML email. Take a look at the following two examples of the preview text and the actual email below it.

Notice the preview text is set explicitly and not left to chance. Both campaigns are hammering home the fundraising deadline message with concise copy. This is likely going to differentiate these emails from others in an inbox because of the whitespace surrounding the email in the inbox list. Here’s a view of the Bernie campaign’s email among a few other emails. Notice the whitespace drawing all the attention in a crowded inbox.

Image for post
Image for post


Your customer’s inbox is crowded. To implement an effective messaging program for your business, you need to understand what your customers want. A/B Testing is a powerful tool to get at that understanding. Making a good first impression and maintaining that trust is a vital step in building a valuable messaging program. Why do customer-obsessed businesses invest time, money, and effort in seemingly simple levers such as an email first impression? When executed at scale, a well-crafted email first impression in a messaging program can bring in millions of dollars per year in incremental revenue for the business. No joke.

In the next post in this series, we will delve into the details of the body of an email.

The Notifications Blog

All about Notifications

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store