Conquering A/B Testing (4): How should I set my user identifiers for an A/B test?
This article is the fourth post of Hackle’s ‘Conquering A/B Test’ series. This series deals with the common questions many people have with regard to the entire process of A/B test design, preparation, result interpretation, and final decision-making. To check out the first post in the series: “When should I conduct A/B testing?”, click here.
Hackle’s Conquering A/B Testing covers the following topics:
1. When should I conduct A/B testing?
2. How should I set my metrics for an A/B test?
3. How long should an A/B test run for? What should my sample size be?
4. How should I set my user identifiers for an A/B test?
5. Can we deduce a causational relationship from an A/B test?
6. When is the right time to stop an A/B test?
7. How can I reach my conclusions with ambiguous A/B testing results?
8. What should I do if I want to restart an A/B test?
9. How should I conduct review processes within my organization?
***
Today we’re covering user identifiers.
Indicative.com provides the following definition for user identifiers:
“A user identification or user ID is an entity used to identify a user on a website, software, system or within a generic IT environment. It is the most common authentication mechanism used within computing systems.”
What Is A User Identification (User ID)?, Indicative.com
User Identification and A/B testing
When put into the context of A/B testing, user identification or user identifier is basically the entity that defines each unique user participating in your A/B test.
This concept is especially important in online experiments because it is hard to keep tabs on all users to make sure that each user is considered as a single, unique person online. Preventing duplication (where a single user is mistakenly counted as two or more users) is crucial to ensure the accuracy of your A/B test results and metric calculations.
In the online world, there are many ways or criteria to define and identify users. For example, some of the most common subsets of different types of user IDs include usernames, email addresses, or account numbers. Especially with services like Instagram and Snapchat that require users to log in with a specific username or email address in order to perform many of the actions online, simply logging in will allow companies to associate a user with a specific, unique identifier.
This brings us to our next question, How about for services that do not require users to log in for the majority of web browsing? How do we set user IDs for such services?
This is why a “golden standard” is needed to accurately define each unique user or subject of the online experiment. Using this standard as unique identifiers will ensure the reliability of A/B testing results. Through this post, you will be able to understand the “why” and “how” of user identifiers. 1) Why is setting the user identifier so important and 2) How do we set up the right user identifiers with the ideal criteria.
Why is setting user identifiers important in A/B testing?
A/B testing is an online experimental methodology that exposes either version A or B of a webpage to users to find out how different versions affect the final action taken by users. When there is a clear user identifier or criteria to identify users, we can ensure that 1) each user is only exposed to the same version throughout the whole experiment, and 2) different actions taken by each specific user such as the history of purchase, search, and product inquiry can be accurately collected to calculate accurate results.
For example, say that your company is conducting an A/B test on the header content of your website (e.g. banner image). You want to see how the text in banner A as compared to the text in banner B affects the click-through rate.
Once a user is assigned to a specific test group (either A or B) of an A/B test, he/she should keep seeing the same banner until the test is finished. If a unique user is somehow assigned multiple identifiers (e.g. a user is given different identifiers whenever he/she revisits the website), it becomes possible that he/she is exposed to both versions of the banner before making the very first click, making it impossible to distinguish which version of the text has affected the decision to click.
However, when there is a standardized user identifier for distinguishing each user, behavioral data such as purchase, search, and product inquiry can be accurately collected. This means that A/B test metrics such as purchase amount per user, search conversion rate, and product page views can also be accurately calculated. Especially in cases where the A/B test results are further analyzed in comparison to other datasets, it is very important to synchronize a standardized user identifier criteria amongst the whole company (product, development, data teams).
Now that we learned about the importance of setting up unique user identifiers to track the users and behavior of each unique user, let’s move on to the actual “how”. What are the different criteria that can be used for user identifiers?
How do I set up user identifiers for an A/B test?
First of all, the user identifier must be a unique value without duplicates. One of the common user identifiers you can think of is the “User ID” used for logging in. User IDs can be used as a user identifier when performing A/B testing on a page that can only be accessed after users log in.
However, there are many cases where users can use a service without logging in. In such cases, you can collect behavior data by setting different values as the user identifier. Device IDs or device information are generally used as the user identifier value. In the case of a mobile app, the device ID is the UUID or ADID value, and in the case of mobile web/PC, the device ID is the PCID based on the cookie value.
- UUID: A universally unique identifier/value established by the Open Software Foundation and generated by the app itself. It consists of 32 characters and 4 hyphens.
- ADID: Advertising ID created by the mobile operating system. In the case of Google, it is called GAID, in the case of Apple, it is called IDFA.
- PCID: ID created using Persistent Cookies.
When you are collecting advertising IDs, you need to follow the policies of Google and Apple. (Note: Users can also reset their own advertising IDs by themselves)
There are many different user identifiers that can be used in this way. Generally, it is recommended to use the device information (Device ID) as the user identifier when most functions of the service can be used even without logging in, but other identifiers may be used depending on different situations.
How Hackle’s SDK defines and handles user identifiers for A/B testing
Hackle’s A/B testing tools allow you to use automatically defined user identifiers or customize your own user identifiers.
Sample code for setting up user identifiers on the Hackle platform through client SDKs such as JavaScript, Android, iOS, React, and server SDKs such as Java, Python, Node.js, PHP, Ruby, etc. can be found within the Hackle documentation.
Check out Hackle at www.hackle.io in order to start creating your own A/B tests to release the right features to maximize your customers’ experience.
