A/B Testing An Email — First Impressions

George Abraham
Jul 13 · 6 min read

This is the first post in a 3 part series about A/B testing an email.

Email is an invaluable channel in a comprehensive messaging program for any business. Depending on your audience there are a few key levers within an email that lends itself well to testing. There are multiple areas of testing outside of the actual email itself such as time, day, frequency, personalization signals, etc. that are also valuable to test but those areas are not the focus of this particular series of stories.

First impressions

You need to make a compelling case to get your customer’s attention in a crowded inbox. There are a few levers you can experiment with to make a good first impression and persuade your customer to see the value in your message.


From Name

The From Name is how you identify yourself to your customer. A few examples follow. Can you recognize the different tactics and guess what those may be trying to do?

AllModern is using the From Name as an area to push their July 4th clearance sale like what you would expect in a Subject Line. DoorDash and The New York Times are leaning into their brand by setting the field to simply reference the company name. bobdylan.com is maybe using the field as a way to draw the user back to the website even if they don’t open the email. Domino’s is differentiating their transactional order confirmation message in the name. Domino’s likely has a different From Name for their offers email. Using long names could result in some email clients truncating the field as is the case in the emails from PG&E and AllModern.


Subject Line

The subject line is probably the most widely tested area in email marketing. However, it is also one of the most abused areas since a lot of email marketing effectiveness is measured in email opens rather than core business metrics. Sometimes all the information a customer needs is right in the subject line, that the customer doesn’t even need to open the email. A notification that an item has shipped is a basic customer expectation and typically has all the information a customer needs in the subject line. So an email open doesn’t necessarily translate to the effectiveness of the message. My colleague Chris Beaumont has a post on this publication that illustrates the folly in relying on opens as a core metric of an email campaign


Let’s look at some examples of Subject Lines in action.


Emojis tend to be used a lot in retail emails. The same tactics used in brick and mortar stores to grab your attention are also effective in email. Flashy signs and $xx.99 numbers tug on customers FOMO strings in stores. Similar tactics are used in the subject line to draw customers into opening an email. The following subject lines seem to have a sense of urgency. Even when there is no time dimension in the copy — you feel as though you are missing out on these “deals” if you don’t check them out right away.


Account Updates are probably the least opened emails but these emails are expected as a standard product experience. Unless you want to use this opportunity to engage customers, a simple subject line can get the job done.


Below are some examples of well crafted subject lines intended to engage the customer as part of a messaging program that creates value for the customer using the product. Subject lines in emails from subscription services to their members tend to be less clickbaity as they are trying to increase their customers’ engagement and help the customer extract maximum value from their subscription.


What do you think of the following email from the Bernie Sanders 2020 campaign? Do you think the missing subject line is a mistake?

Think again! Here’s an example email I sent to myself without a subject line. The iOS mail app defaults to a gray colored text with “No Subject” as the fallback copy. Compare that with the subject line above viewed in the same email client.

Yes — subject lines are really a great lever to use in implementing an effective email program.


Preview Text

Preview text is the text that is displayed below the email subject line in email clients to give the user of the client an idea of what is in the email. It is typically displayed in lighter font compared to the subject line by mail clients. Email marketers are leaning into preview text copy as a way to distinguish their emails in a customer’s crowded inbox. If you don’t explicitly set the preview text, various email clients will render the copy from a fallback option such as the plain text version of the email or by scraping the initial part of the HTML email. Take a look at the following two examples of the preview text and the actual email below it.

Notice the preview text is set explicitly and not left to chance. Both campaigns are hammering home the fundraising deadline message with concise copy. This is likely going to differentiate these emails from others in an inbox because of the whitespace surrounding the email in the inbox list. Here’s a view of the Bernie campaign’s email among a few other emails. Notice the whitespace drawing all the attention in a crowded inbox.


Conclusion

Your customer’s inbox is crowded. To implement an effective messaging program for your business, you need to understand what your customers want. A/B Testing is a powerful tool to get at that understanding. Making a good first impression and maintaining that trust is a vital step in building a valuable messaging program. Why do customer-obsessed businesses invest time, money, and effort in seemingly simple levers such as an email first impression? When executed at scale, a well-crafted email first impression in a messaging program can bring in millions of dollars per year in incremental revenue for the business. No joke.

In the next post in this series, we will delve into the details of the body of an email.

The Notifications Blog

All about Notifications

George Abraham

Written by

imgeorgeabraham.com

The Notifications Blog

All about Notifications

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade