A/B Testing — Remote Configuration

Published in

Huawei Developers

8 min readJun 25, 2020

Hi everyone, I will talk about what the A /B Testing, one of the Huawei AppGallery tools. Let’s start by getting to know the A / B Testing tool :)

What is A / B Test?

A / B Testing offers a range of study tools to optimize the application you are developing and to improve key conversion and growth indicators. The service can be used for purposes such as application user interface design, copywriting, product functionality, or comparing solutions for marketing activities for performance metrics.

You can use it to create one or more A / B tests that meet user needs and deal with different groups of users to find the best one. This helps you make the right decisions after the test. For example; Let’s develop an e-commerce application where products are exhibited and sold, and let’s test the effect of the UI change on the display where the products are displayed on the sales and choose the right UI according to this test result. When we want to do such a test, A / B testing does it for us!

When we want to test on an example like the one above, basically A / B testing provides us to identify different user groups and direct differently designed UI views to the groups and measure the difference in performance between the groups. It allows you to find ways to display products that most appeal to users, to write text and to marketing through comparison tests to ensure the effectiveness and retention of users. What we mean by the performance here can be any criteria you choose when defining A / B testing. At the end of the identification, the testing process begins and after the test period is completed, A / B testing shows you the results on the performance monitoring screen.

A / B testing can also be used in marketing transactions. For example, when a new product is released, A / B testing can be used to measure how users will act against this update or innovation. It can be observed, for example, that only 5% of users show this innovation to cover only some of the users and how users act against this innovation in the results screen. Thus, a more accurate decision is made for the release of the product or the update to be made.

What is the path to follow for A / B testing?

First of all, Push Kit and Analytics Kit services that A / B testing is dependent on should be integrated into your application (Detailed information on this subject is available later in the article).

Identify one or more treatment groups you want to evaluate. For example, 50% -50% of users are denominated in two groups, or only 10% is selected to be tested.

Specify the trial user group, for example, the user group of a particular application version. Testing can be done separately for different versions of the application. Different content for 1.0, like different content for 2.1

Define the experiment targets to measure the achievement of the experiment targets (such as user retention rate) by the test group in the experiment.

Select other indicators (such as user interaction time and clickthrough rate) to better understand experimental results and their impact on your app’s user experience.

Once an experiment is created, you can manage it as follows:

-Define the experiment.

-Start the experiment.

-View the experiment report.

-Increase the percentage of participants.

- End the experiment.

Perform other test management procedures.

How is A / B testing defined?

After logging into your Huawei developer account, we go to the AppGallery >> apps screen by switching to the console screen. After selecting your application from this screen, choose to develop from the left part of the screen, and then select Growing >> A / B Testing among the develop options on the next screen, and select Enable now at the first use. must be activated.

If you did not optionally select a storage location later, you can set the storage location by selecting the country and region from the Project location field.

After completing the activation process, we will see the final A / B testing screen.

In this step, necessary integrations must be made for the use of A / B testing. Necessary actions; It is the integration of Analytics and Push Kit into the project and completing the Remote Config process.

You can get detailed information about Push kit integration link.

You can get detailed information about Analytics kit integration link.

For remote config operation, I will talk about the A / B testing definition steps in Step 2 in detail.

There are two different test types in A / B testing, Remote Config and Notifications.

Remote Config: When you want to determine whether your application updates can serve users better, you can use the A / B Test to create a remote configuration experiment. By comparing multiple treatment groups of one or more Remote Configuration items, you can determine whether your users are like a new function or a new user interface.

Notifications: When you want to send notifications to existing users or send new marketing notifications but are unsure of its effect, you can use the A / B Test to create a notification experiment and test treatment groups in the selected user group. Find the most appropriate notification copywriting and viewing mode. In my test, I developed two different interest screens and measured which user was directed to which interest screen was more active in the application.

Since I did Remote Configuration A / B testing on the project I developed, I will continue my narration on this.

1. Step: First you need to define remote configuration on AppGallery and within the application.

Remote Config Identification in AppGallery

App Gallery Content >> My Apps >> We choose our application for A / B testing and then switch to the Develop screen. After selecting Growing >> Remote Configuration from the menu on the left, we select Add Parameter and we define our parameters as follows.

After saving, we will create our parameter as follows.

The important thing here is that for Remote Config, the data that will return to User groups must be equal in the code and AppGallery definitions. For example, when I define the Remote Configuration parameter in AppGallery, I defined the parameter name as “Interests” values a and b, and the default parameter as “a”. Therefore, while defining Remote Config in the code, I defined the configuration name as “Interests” and the value that will return from AppGallery as a or b. I showed it in the “text_abtesting” text in the function you see below.

The equivalent of the default parameter we define in AppGallery in the code should be taken as an XML. The content of the XML file we have defined is as follows.

Then it should be pulled within the applyDefault function of the AGConnectConfig class. (You can also see it in the code block below.)

Then I defined activity and defined a variable of type AGConnectConfig in it and pulled the values sent to the groups over it. The code block required for Remote Config is as follows.

2. Step: Since we have to define user attributes during A / B Testing;

Create User attributes

We select Analytics over AppGallery Content and then select the project we will test. We move from the screen to the Advanced Analysis screen and select User attributes from the left menu. Here we define the users we will use during A / B testing definition.

After selecting Create, the user definition is made as follows.

Since I will divide it into two groups, I made two user definitions. You can define as many users as you want according to your purpose.

Since our user definition is finished, I continue my narration with Remote Configuration A / B testing.

3. Step: A / B Testing Identification

We switch to My apps on the App Gallery and after selecting the project we will define the test, we choose Develop >> Growing >> A / B Testing. Select Create remote configuration experiment from the A / B testing screen;

Step 1: Define the basic information step according to your test content as fallows.

Step 2: Determine the test conditions in the target user step. Which conditions are used here and why are shown in detail in the table in this link.

Here you can run a single-user test of your choice, covering only half the users. Here, according to the rate given to the users, the system randomly allocates users to each trial.

Step 3: Treatment & control groups

In this step, we select the Remote Config parameter we defined in Step 1.

Step 4: Track indicators

In the last step, the indicator step, we select the indicators that we want to see as a result of the test. The contents of all the indicators on this screen are explained in a table in the link below.

From this table, you can choose any indicator up to 5 max.

As you can see above, I chose the User A / B Test Participation indicator since I wanted to get test results for the user groups we defined in this step. We have completed the A / B Testing identification steps. When we save and exit, we will see the following screen.