How A / B Testing Improves Your Product and Service Launch

It doesn’t matter how much you love your product, service, or app. There’s always room for improvements — meaning checking your tools, apps, and touchpoints about what we can do better. Improving the world, business, or flow of my workers. When we talk about eCommerce it is about how I can make browsing, finding, and purchasing easier, more effective, and just more fun. That means asking yourself and your users, and not any leader or decision-maker. It means asking the real users, the real consumer, ideally — if possible — not in an artificial milieu and environment.

written by Holger Maassen

A while ago I wrote about the value of research being a key factor for success in design and business. This is where testing can make life easier by providing a focus “UX Research: A Key Factor for Success in the Design Process”.

I’ll get into how you can begin testing in a minute, but to kick things off, deciding which research or testing method you should or might use has to come first — first think about what are the goals and reasons for testing. If we talk about eCommerce and customer experience the goal and reason are often the conversion rate.

Conversion testing allows you to see if the change and its impact is significant or negligible or just a kind of random outcome.
When you begin testing to optimize your store’s conversion, you need to define which parameters define your conversion as to be done or fulfilled and that you’re tracking and measuring your parameters. This can be achieved using various analytic tools e.g. CDS (Context-Driven Service) and ensuring you’re measuring macro-conversions and micro-conversion.

Macro-conversions are high-value interactions

The most common types of macro-conversions are leads (people providing their details) and transactions if you’re selling online.

Micro-conversions can be seen as secondary objectives — they’re like an ancillary but welcome side effect, for example, because consumers are engaging with you on a deeper level like signing up or logging into a member area, commenting, and giving feedback, or downloading a PDF, watching a video.
It’s crucial that you know and define your macro and micro parameters because they’re what you’re going to be testing against.
I also want to flag that your objective for improvement might not be financially driven. You might be looking to improve usability or improve engagement with your content — this is just as valid as testing to improve transactions or leads.

In general, there are two types of tests.

A/B testings, also known as split testings, are most common. It is a method of comparing, for instance, two websites, two pages, UIs and you split your pool or a segment of users in two. One half of the user segment will see for example the original UI (variant A) and the other half will see the new UI test variation (variant B). This means you have ‘A’ and ‘B’ and you then look at your data to understand whether it’s A or B that is driving more people through to the conversion action.
You also have the option of adding additional variations, so instead of testing only one new interface variation against your existing one, you could actually run multiple variations against your original or your preferred one. However, the more variations you add, the longer the test will take to run.

And here you see four variations of POV Product Overview pages

Why do I like A/B testing? Well because of a few reasons:

• It is a cheap and easy method that can even perform in paper prototyping
• It is a great way to figure out the value of interaction, feature, labeling, or content and what works best for all or particular users.
• It is 100% user-centered.
• It is oriented around user experience and expectations. After the run, the real data and results help us to make an UI or service even more user-friendly.
  Since it is usually focused on a few certain metrics, it’s not that hard to determine which version performed better.

Multivariate testing is the second one I like to mention. It allows you to test multiple UI elements on the same page. For example, you might test different buttons and different content all within the one page. In short, it is a technique for testing a hypothesis in which multiple variables of UI elements are modified.

Below you can see one PDP Product Detail Page — two versions with multiple UI elements and Text elements and variables

The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations.

How to run A/B testing

A/B testing is not a difficult method. It can be accomplished quite easily if you take the following steps.

1. Gather data
First of all, A/B testing is about optimization. Optimization of usage, conversion, and clicks which is gathered through the rate of the flow or sequence of pages (e.g. a one-page vs. three-page checkout steps). Therefore, the first step before you run your test should be the collection of data, the what to test for which purpose. This sounds self-evident but is sometimes forgotten.

2. Set the goals
To make the improvements work correctly, the designers or the marketer, if we are talking about an e-commerce store, need to determine what they expect to gain from the testing. For example, it can be time spent on a website, or less time spent in checkout, or more clicks on a carousel of products.

3. Shape / Create an assumption
Which alternative or selection would you like to test? What areas can be improved in the current version? Based on those questions, you can create your A and B versions.

4. Define who should see and test your A and B versions
When you have decided what changes you want to test, it’s time to bring them to life and define which users e.g. a certain segment of your users (e.g. role, rights, or buying behavior, location, …) should execute the testing. You can test all users / all visitors but in almost all cases we are in a world where and when we have to show and offer customized elements, interactions and interfaces — that leads you to the need to define target groups or segment — in order to test for specific usage and expectation. And that might mean if you test something in a running environment that it might make a lot of sense that you define a time frame for your test phase.

5. Testing
Now, it’s time for users to test the variants. People use an application or visit a website, where everything they do is measured and transformed into useful data.

6. Analyze results
After the testing, it’s time to work on the results again. All the data and metrics from the two versions are gathered and compared. Based on the results, marketers or designers decide which variant performed better and the goals that were set at the beginning of the project can be accomplished.

Wrap-up

Testing allows you to make your data actionable. You can test just about anything, and as mentioned you can test early and when ever you like or see the need — but here are my top picks for things you might want to consider testing:

• Forms and UI flows
• Navigation (including checkout paths
• Headings, text content and language (way of talking)
• Calls-to-action and buttons
• Symbols, images, graphics, and video content
• Page layout and hierarchy

And at this point, I like to make you aware or remind you of CDS the context-driven service by SAP which gives you all at hand to run tests.

This article originally appeared on SAP User Experience Community.