12 Basics to remember when A/B testing videos in ecommerce

A/B testing can be exciting. You get to check how different elements on your site affect shopper behavior. You get to create a hypothesis and see how it plays out with real visitors.

Yes, working on conversion optimization is both fun and empowering.

Every ecommerce store owner probably has a list of things they’re itching to test on their site. It could include testing the copy, testing the placement of free delivery message or perhaps product videos.

If running A/B tests with videos is on your list, then this post is for you.

Before you start placing your bets on the video variation most likely to win, remember these basic tips.

1. Have a clear idea of where you’re standing. Know what your current conversion rate is before you start testing. Outline the most important KPIs for your ecommerce site. That way you’ll understand better not only what sort of engagement results the winning video variation brings, but also whether it creates a tangible difference in the conversion rate on your site.

2. Make a plan. Your list of ideas to A/B test is handy, but you need to lay out an action plan of how you’ll go about working on them. Separate priorities from wish-to-do tests. Isolate website elements that require immediate attention and outline which sub-elements can be tested. For example, if you’d like to run A/B tests with videos, make a plan to test video position, size, call-to-action etc. Formulate your hypotheses clearly. Set out the processes and then schedule your tests. Having a solid plan will help you advance your conversion optimization strategy faster and help you from getting distracted.

3. Define what a conversion is. If you’re testing videos, it might be clicking on the “add-to-cart” button on the page with video, clicking “play” on the video or reaching a “thank-you” page after watching a video (completing the purchase). Define micro-conversion goals as well as macro-conversion goals before you start testing so that you can have an objective view of how the variants perform.

4. Run A and B versions simultaneously. You want to test both versions of your page at the same time. That way both versions have equal chances, conditions and time frame to generate results. Testing sequentially introduces timing bias, so you wouldn’t know if a test worked because a variable changed or because the timing played a role.

5. Test one thing at a time. You might think that testing just the color of the video play button might be too small of a change to make any real impact on your conversion rate. But if you introduce multiple variables to test simultaneously (e.g. color of the button, the position of the video, AND video thumbnail), you won’t be able to single out exactly which element influenced visitors to convert the most. (Testing how multiple elements work together in different combinations would be a job for multivariate testing, so we’ll talk about it in the future posts). Keep control over what is tested each time.

6. Focus on incremental changes. You might be tempted to give your site a major overhaul and then test how the new version performs against the old one. But don’t just pour all your money into a complete revamping project before you know which elements work in converting visitors. Instead, test one element and one variant of that element at a time. For example, before you decide that you need a whole lot of new product videos, test how changes in your current video positioning or play button work on your site. Small, incremental changes will help you isolate the problems and focus on what really makes a difference.

7. Test the changes on new visitors. Your existing customers already have pre-set expectations about your site, so their behavior is conditioned. They might fall into habitual paths and ignore the changes you introduced to your store. Test on new visitors who haven’t interacted with your site before. That way you can isolate the effects of the variables on an unbiased audience.

8. Test on a sufficient sample of visitors. Your A/B tests need to report results that are statistically significant. Testing video elements on a dozen of visitors, when your website runs with a six-figure traffic won’t help you draw correct conclusions. You won’t have any assurance that the winning variation will produce the same results once applied at large. To ensure your results are reliable, make sure you’re running tests on a visitor sample large enough to render meaningful data. This is a handy tool to help you calculate your required sample size.

9. Take your time. Although you might see immediate changes in the visitor behavior as soon as you introduce your A/B tests, give it time. Allow each test to run for a few weeks before you draw conclusions. You want to be sure the changes in conversion rate are persistent. You might get a notable conversion lift when a video is first introduced to the site, but as the novelty wears off, the conversion rate might drop back to “normal” again. Collect enough data to be sure the results of your experiments are sustainable over time.

10. Avoid testing in volatile seasons. Picture this. You’ve decided to add videos to your most promising product pages and run “video vs. no video” A/B test. Your conversion rate goes up in the weeks that follow. You conclude that videos motivated the shoppers to place orders, only to realize that the test actually took place the week before Christmas when all your metrics were running above average. Bear in mind these kinds of seasonal fluctuations. Try to schedule your tests in periods with stable traffic not affected by seasonal fluctuations in traffic and conversions.

11. Be ready to iterate continuously. A/B testing is not a one-off exercise. Your conversion rate could always be better, so keep on testing new variables. Your visitors will appreciate the continuous effort towards improvement.

12. Trust best practices, but trust your visitors more. Not all ecommerce stores are created equal, and not all visitors behave the same. While some sites achieve drastic lift in the conversion rate just by switching a video play button, it might not bring as significant results in your store. Perhaps you need to test other elements, such as length of videos or video thumbnails. Be in tune with what your visitor data tells you. Create A/B tests that make sense for your store and your customers.

Finally, act on your results! Don’t perform A/B tests just for the sake of saying you’ve been busy with a bit of conversion optimization activities. Use your learnings to improve your ecommerce site. Your visitors will appreciate it and reward you by becoming loyal customers who return over and over again.

If you’d like to save time on adding videos to your ecommerce store and need help A/B testing videos on your pages, Goodvidio is the good place to start.

Originally published on Goodvidio Blog