Testing Emails on Action Network
A Step-by-Step Guide to A/B Testing
This post is Part Two of a series on optimizing your email program on Action Network. To learn how to make sure your emails arrive in your supporters’ inboxes (and not the spam filter), check out our Deliverability Guide.
What is email testing? Why is it important?
A/B email testing is a great way to figure out what content resonates with your activists and what (despite your best efforts) does not. It’s the ‘scientific’ way of finding out what kind of content your activists will engage with because it prioritizes data over your instincts (your instincts are probably very good, but they aren’t as accurate as the data). Your activists may react unexpectedly to your content, and you’ll never know for sure until you test and measure your results.
A/B email testing allows you to send different variations of an email to a subset of recipients in order to determine which variation gives you the best results. Then, you choose your highest performing, ‘winning’ email and send it to the rest of your email list. You can test email subject lines, ‘from’ lines, and email bodies on Action Network to find which content your list likes best.
Here’s how:
Step 1: Determine what metric you’re going to test
Your first step to A/B testing is to decide what metric you’re trying to improve. There are three main metrics that A/B testing is used for:
- Open rate (the percentage of recipients who opened your email): The best way to improve your open rate is to test your subject lines and from lines.
- Click/open ratio (the percentage of email openers who clicked on a link in your email): The best way to improve your click ratio is to test your email body. Why use click/open ratio instead of click rate? Your click rate will increase as more people open your email, whereas the click/open ratio will stay more constant and allow you to make faster decisions about what’s working well and what isn’t.
- Action/click ratio (the percentage of email clickers who took action): The best way to improve your action/click ratio is to test your action landing pages via links in the email body. Why use action/click ratio instead of action rate? As with click rates, your action rate will increase as more people open your email. Your action/click ratio will remain more constant and produce useful results sooner than action rate.
Step 2: Determine what variable you’re going to test
Once you’ve decided what metric you’re going to test, you can decide what variable you’re testing in your email. There are three variables you can test on Action Network: subject line, ‘from’ line, and email body.
- Subject line: Email subject lines are the first message an activist receives from you in an email. It needs to interest your activists enough to motivate them to open your email.
- ‘From’ line: The ‘From’ line can be your name, your organization, or a combination of both. It allows you to build a relationship with your activists, so it’s important to decide who that relationship should be with. Whose name at your organization should your activists recognize? After you’ve determined which from line performs the best, it’s important to stay consistent to build rapport with your activists.
- Email body: This is the content that your activist sees once they open your email, and it should encourage them to take action. The email body lets you test a multitude of different variables, including button color, text copy, images, links, and more.
Note: You can test multiple variables on Action Network. We will display your test results for every combination of variables you create in order for you to select the best email. For example, if you’re testing two subject lines and two email bodies, we will send four variations of the email. You will be able to view the results of each variation.
Step 3: Make a hypothesis
Next, make your hypothesis (think back to 10th grade science class). This is what you think will happen when you test your email. Some examples of hypotheses include:
- Emojis in my subject line will increase open rates.
- Emails from my executive director will increase open rates.
- An orange call-to-action button will increase my click/open ratio.
- An image in my email will increase my click/open ratio.
- An action landing page in my organization’s colors will increase my action/click ratio.
You should already know your baseline result (what your metrics currently are) so you can gauge what to expect when you send your test email. You want to test option A and B, but you also need to know if neither option A nor B are performing as well as you’d like — it’s possible that neither option A nor option B will perform better than your baseline.
Step 4: Create variations of that email
Now that you’ve decided which metrics and variables you’re going to test and made a hypothesis, it’s time to test that hypothesis! Here’s how to test subject lines, ‘from’ lines, and email bodies within Action Network.
Testing Subject Lines
To set up a subject test, use the alternate subject field for a second subject, and click the green plus icon to add as many additional alternate subjects as you want.
If you want to remove a subject line, delete the content and leave the alternate subject field blank.
Note: Don’t make your subject too long! Long subjects tend to perform worse, and subjects over the byte limit will be truncated and will break statistic reporting on opens and clicks. Less than 200 characters should be your goal.
Testing ‘From’ Lines
You can enter more than one ‘from’ line if you’d like to set up an automatic ‘from’ line test. If you enter more than one ‘from’ line, we will randomly distribute each ‘from’ line you enter among your email’s targeted universe and allow you to track the statistics of each ‘from’ line individually to see which one performs best.
If you want to remove a ‘from’ line, delete the content and leave the alternate ‘from’ field blank.
To set up a ‘from’ line test, use the alternate ‘from’ field for a second ‘from’ line, and click the green plus icon to add as many additional alternate ‘from’ lines as you want.
Testing Email Bodies
You can test multiple email bodies, which lets you test your creative content like email text, buttons, images, and more. Underneath where you write your original email body, click the green plus button labeled ‘add alternate body’. This will copy your original body over to the alternate, where you can fully edit the body. Give your alternate body a name, which will appear on the ‘test results’ tab of your email manage page so you can track how your email is doing.
On the visual editor/HTML, if you want to remove an alternate body, delete the content and leave the alternate body blank. On the drag & drop email builder, click the ‘delete body’ button in the lower right.
Step 5: Determine who to test your email to
Now comes the important part: deciding who will receive your email tests. The answer is different depending on the size of your email list. An important thing to remember in email testing is that this is a ~ scientific ~ process, so an ample sample size is essential to getting useful results. The larger the sample size is, the more accurate your results will be.
The first step to figuring this out is to calculate the number targeted in your email. For example, if you’re targeting people in Texas who are donors, calculate the number of Texan donors you have. If you’re sending to your full list, you can go by the number of subscribers on your group manage page. However you’re targeting your email, you can find out the number of activists by pulling a report OR by using the email targeting feature and letting it calculate.
Now that you know how many people are targeted, you can decide which method you want to use to test:
Method 1: For large lists (>10,000 targeted), send to a test batch
If you have a larger list (more than 10,000 activists who meet your targeting criteria), then you can send a test batch of your email. This will let you evaluate which email variation is performing better, then send that email to the remainder of your list. For example, if you have 100,000 activists subscribed to your list, you may test your email by sending to 10,000 activists. Then, you would send whichever email does best to the remaining 90,000 activists.
To determine a good sample size, we recommend using a tool like Optimizely or EvanMiller.org.
Whatever sample size you choose, scroll down to the bottom of your email targeting page. Input the amount you calculated into the ‘random limit’ box. This will take the total number of activists targeted with your ‘includes’ and ‘excludes’ and cut it down to the number you entered by choosing random activists in your targeted universe.
Method 2: For small lists (0–10,000 targeted), send a 50/50 split
If you have a smaller number of activists you’re sending this email to (less than 10,000), your sample size is not large enough to send a test batch. Instead, you should send the email to your full list and employ long-term list testing. The data you gather on the best performing email should inform the strategies you employ in the future. What type of subject line works best, what type of ‘from’ line works best, and what type of content works best? Over time, you’ll see these answers start to emerge.
Step 6: Watch the results come in
The hard part is done! On your email manage page, navigate to the ‘test results’ tab, and watch your results come in! We know you’re excited to watch those email statistics jump up when you refresh the page.
Step 7: Choose the winner and duplicate
Wait an hour or two to see which test does best — was your hypothesis correct? Remember to make sure whatever metric you’re measuring is above your baseline.
Once you’ve chosen your winner, click the “duplicate with exclude” button to the right of your highest performing email. You’ll be taken to an email writing page that will automatically duplicate that email version you chose. Then, click the button to save and go to the next step, and you’ll see everyone automatically excluded who received any version of the email you previously sent. From there, send your winning email to the rest of your email list.
And you’re done! Remember that your email list changes over time, so be sure to test your emails frequently.
We hope you found this guide helpful! Keep an eye on our Upcoming Trainings page to stay up to date on our latest 101, 102, and 103 level trainings. Thanks for reading!
