Telling stories about shark jumping & A/B testing
I think I first remembered hearing the expression ‘jumping the shark’ around the same time that I started A/B testing — which is probably a classic combination of too late on one & too early on the other trend.
These concepts combined when the Manchester Umbraco meetup jumped the shark. It was that fateful day when I presented on adding A/B testing to everyday development, which still seems to me an area that many developers remain ignorant about. Perhaps developers see A/B testing as a bit woolly & not for them as it is often the domain of those dread marketing types or at best front-enders.
Since the Umbraco meetup which didn’t mention Umbraco, Pete Donaldson waxed lyrical on React development, the meetups have dried up & Sam Flanagan has taken over the job of reviving the sessions. Hopefully in a few short years the sessions can recover from this meta A/B testing — because obviously not covering the actual tech was a test too far.
Back in 2015 we were using Optimizely for A/B testing, which is expensive at around £20k; but is easy to use, full featured & had good tie ins with other marketing tools we use like Sessioncam, GA and so on. So if you have the budget you can record really rich detail about journeys.
To save some money I moved us onto Site Gainer — which a bit rough round the edges but about 10% of the cost of Optimizely & still does a good basic A/B testing job.
Yesterday I was at a ‘masterclass’ about content superbly organized by John McCambley with an excellent keynote from Sam Jones (who has a more than passing look of Gary Barlow about him) where once again the phrase ‘content is king’ seemed to be making the kind of return usually reserved for Lazarus.
Coincidentally we’ve been testing this week whether on one of our ecommerce websites adding additional content in the form of a story page does anything for sales. Without preconceptions we set up an A/B test with 50% of traffic travelling from a top-level-navigation link going into a story page that lovingly describes a product range; & 50% dropping straight into a product lister.
After a week’s data we are approaching 10,000 tests & a statistically relevant result & it is that by delaying the customer journey to explain the story behind a product the likelihood that the customer would eventually add that product to the basket improved by over 70%.
Telling stories isn’t just for A/B testing marketing charlatans.
