To Gain or To Give: What Motivates User Action?
Analysis of an Email Marketing A/B Experiment
Email…Marketing…those damn things I get all the time? Hmmm, I hear you say, they’re damn, damn annoying. Well, they often are, but those are the ones that don’t inspire an emotional reaction, don’t try to engage or motivate you to do something effectively, or from some mailing list you forgot to unsubscribe from. However, Email is such a fantastic tool when done properly, and as you’re reading this, you probably know that already.
This is an analysis of an A/B experiment we did with the first email campaign for our startup emble.io (it helps you to plan fun with your friends).
Right now, we’re not quite at beta (so we don’t have any site activity to track with the aim to personalise etc) and we really wanted to test something, to start as we intend to go forward.
We were keen to gather some data from the lovely people who have signed-up so far, in the form of the exciting and irresistible survey. Of course, email was the most appropriate way to ask if our prospective emblers would answer a few questions for us.
With this in mind, we decided to test a big idea on the emails themselves, to see what insight we could glean. We decided to test what subject line and CTA would drive the greatest response — in opens, clicks and completes — whether a gain for them or a give to us would show significant differences in the success rate.
Here’s what happened:
We sent out two emails with differing subject lines and primary CTAs with N=498:
A — Gain, aka “Jump”
B — Give, aka “Help”
The carrot for each remained the same: if they completed our super short survey they’d be given earlier access to the emble beta. However, both the subject line and CTA were different. We designed the email so that the CTA would be very prominent and likely to be the first thing they’d see when opened.
NB: We did want to take away the carrot entirely in our B email, however, that would have been counterproductive as we wanted survey completes.
The Race
The email was sent. Our fingers were poised on the stat refresh button. At the beginning “Jump” opened up a huge 7.3% CTR lead and pulled away on opens right out of the gate:
And continued to extend up to nearly 9%:
It looked like “Help” would be trounced…but then it asserted itself, closing the CTR gap and overtaking on opens:
Whilst early results in any experiment should be taken with a large pinch of salt, the wide initial variance and subsequent catch-up was still interesting to note.
The Final Result
With the initial excitement dissipating we hunkered down to other work and left the our two horses to duke it out for supremacy, secure in the knowledge that both emails had fought bravely to demonstrate that — whether gain or give — people will act readily to both types of request (…this gave us a warm fuzzy feeling).
Whilst “Jump” won in terms of creating a statistically significant larger CTR, it was great to see that “Help” illicited more opens, even though this wasn’t significant enough a difference to be something to learn from.
“Jump” therefore inspired more action, but not by a huge amount. We thought it fantastic and inspiring to see that “Help” held it’s own in the end — what an awesome bunch we must have on our list so far!
Survey Completes
As a final measure, we wanted to see how the email click activity fed through to the completion of the survey itself (as we consider that a conversion in this scenario), which was our main reason for the reach out.
As you may recall, “Jump” had a +4.8% CTR over “Help”, however it garnered a survey completion rate of +8%, disproportionately high compared to the CTR from the email. We know, on the surface it seems like a “well, duh” moment, however, it does give nice insight into a specific group of people.
Some other stuff we learnt
- General Response Rate — we are hoping that we’ll continue to get this kind of response rate when the beta goes live. We are delighted by this first endeavour and the engagement of the awesome people signed-up:
- We probably distracted up to 24% of people with secondary links, even though these weren’t obvious. There was a good reason why we had to in this campaign, but we’ll be more conscious of this in future.
- Great data from the survey that will really help us in refining going forward and serving our members to the best of our ability. We can’t share it though, it’s private.
Have a good one!
The emble Team
Written by @ACharles_writer with the @embleapp team for the Vero Email Marketing Blog. Also posted on Growthhackers.com
If you liked this article please share it…!
More by me:
Should You Start Being A Skill Hacker Today?
72hrs in the Life of Content on Twitter (& other Social Media)
C’mon, People, Be Social: A Personal Twitter Experiment
