A/B Testing — What are the Solution Design and Engineering Processes for Your Startup?

In Design Thinking, testing your solution by giving it to real users is the only way to evaluate your idea. If your system doesn’t support the ability to serve different users with different content, your system is not A/B Testing readiness.

From Wikipedia — “A/B testing is a way to compare two versions of a single variable, typically by testing a subject’s response to variant A against variant B, and determining which of the two variants is more effective.”

In a Startup, you are running an e-commerce website that offers clothes for women. After the first sales year, your system has recorded numbers saying customers likely to visit than paying through your platform. Given the purpose of this screen will lead two user groups A and B, who has a similar audience characteristic, to “pay” for your product:

Figure — 1. A/B test payment method on the same audience characteristics.

Which one has a better sale? We need to run a user-test on both possibilities for the same audience set but divided into two different groups. This test will answer just one question:

Does the audience who ages from 28–32 years old, living in a urban city, using iPhone likely to pay by PayPal, by ApplePay, by Credit Card or doesn’t care at all?

You are leading the Software Engineering team, to work with this test, your system must adopt the following requirements:

• Develop two new features: “Pay with ApplePay” and “Pay with PayPal”.
• Display the feature “Pay with ApplePay” for only user-set A,
• Display the feature “Pay with PayPal” for only user-set B.
• Display none of them for the other audience sets (not in the A/B test).

The engineering process for A/B Testing

A “User Story” statement format is always a good method to describe what user needs and what system will provide to that user. Let’s link the business idea to the engineering process:

Figure — 2. The user requirement for A/B Testing.

Your team will have three statements as User Stories to implement:

• As a user in A, I want to see the “Pay with PayPal” button, so that I can measure how the effectiveness of using PayPal.
• As a user in B, I want to see the “Pay with ApplePay” button, so that I can measure how the effectiveness of using ApplePay.
• As a user is NOT in A or B, I don’t want to see any of them, so that there will be no impact on the sales.

On a pretty big team scale, there are three different engineering teams for those who work on a specific stage. The principal of the A/B Test engineering process is the same as in three-stage: Data engineering, Software engineering, and System Engineering stage.

Figure — 3. The engineering process for A/B Testing.

Data engineering: refactoring the data source to determine the audience of interest, then divided it into the A-set and the B-set. The result of this process can be one of those two:

• Mark a selection for A/B and NOT(A/B), visible to the client-side.
• A list of A/B or NOT(A/B) users to be referred by a routing service.

Software engineering: develop new features for A-set and B-set. The previous version will become the feature NOT(A/B-set). The result of this process will be:

• 3 different software features in three different service endpoints.
• 3 different displayed features at the client-side for user interfacing.

System engineering: network wiring the audience set of A/B or NOT(A/B) to the corresponding feature A/B or NOT(A/B). The result of this process will be:

• A link between client-side requests and the corresponding features.
• Traffic routing among audience set and feature set on request.

The responsibility of A/B Testing solution design is clearly on the System Engineering team’s shoulder. What should be a good strategy to design a resilience system that can seemly switch user on-request from one to the other service? The overall system architecture and analyzing business impact are important to make any decision of choice.

The solution design for A/B Testing

An end-to-end solution design must cover the differential software traversal among different users and devices. Depending on web apps or mobile apps, the deployment of new features will impact the traversal of usage differently.

With web technology, your team can push the content from the server-side with minimal, controllable time-to-update. But with mobile apps, the choice to upgrade a new version that contains your new A/B testing feature is put on the customer’s hands. For mobile apps situation, you must ensure your audience-set that you want to test having the newest version in hands.

You don’t want your customers are using an out-of-date version of your mobile apps. A new upgrade with breaking change may block your customer’s access to your apps that is the unwanted-ever situation. “Forcing” upgrade is one of the solutions that can work in this case or some time it works very well in an urgent case that entirely blocking your user access.

Given all customers have up-to-date software in hands, what is the best scenario for A/B testing solution design?

Figure — 4. Service routing rule base on user properties.

Your data engineering team has already delivered an A/B audience set that allows determining a specific customer belong to A, B, or neither A nor B. At step 1, the client-side request to the A/B audience service to know what is the specific routing customer property? This property will be used for specifying the feature to interact in the following steps.

One of the methods to write the customer space to application space is using the “Reverse-Proxy/Rewrite URL” technique by mapping from “user-property” to “service-endpoint”. This is an example of a service mapping table:

Figure — 5. Service endpoint mapping by using the resource URLs.

The other method that I prefer to apply for a small software development scale or micro-teams organization whereas you own what you did:

Figure — 6. Service endpoint mapping by using an HTTP header.

Next step, we will experience an example of using NGINX reverse-proxy to setup a dynamic routing rule bases on HTTP headers. This method is practical for a monolithic application architecture or a docker based microservices architecture.

Using NGINX reverse-proxy for A/B Testing

Your system has an endpoint configured as a gateway. Every request will hit to the https://yourdomain.com. In the payment method selection example above, you want the new payment method button will call to the same API but work either for PayPal or AppPay depends on a particular customer to be controllable from the server-side:

One endpoint https://yourdomain.com/purchase works:
— for Credit Card with the old version and NOT(A/B-set).
— for PayPal with the new version and A-set.
— for ApplePay with the new version and B-set.

Nginx configuration supports converting HTTP headers to variables that you can use in your configuration to redirect endpoints. Let’s define an “x-service” header to specify the service URI “/v2a” or “/v2b” corresponding to PayPal or ApplePay feature.

server {
   listen       80;
   server_name  localhost;
   server_name_in_redirect off;
location /purchase {
      proxy_pass https://yourdomain.com$http_x_service$request_uri;
   }
}

This is the mapping table explaining how it works:

Figure — 7. Using an HTTP header with NGINX to rewrite the service URL.

NGINX is one of many methods that you can utilize to setup a dynamic routing table base on HTTP headers. Mapping Template of Amazon API Gateway is another choice if your solution using AWS Services. Istio in Kubernetes also supports a header-based routing method that works in the same way (Istio is developed base on NGINX).

Using Lambda’s Layer Version for A/B Testing

Amazon API Gateway Stage Deployment supports canary, tagging with Lambda Alias version to a different stage of applications. By doing so, your applications will access the different stages of one service through the different endpoints:

• https://xxxxx.execute-api.us-east-1.amazonaws.com/v2a/purchase
• https://xxxxx.execute-api.us-east-1.amazonaws.com/v2b/purchase
• https://xxxxx.execute-api.us-east-1.amazonaws.com/v1/purchase

Those endpoints are configured to one Lambda function but in the different stages of deployments. Whenever you want to upgrade a small patch of v2a or v2b, you must deploy another new tag: v2a-patch-1 or v2b-patch-2. Bad news comes with the need to upgrade your client-side code. It is not a good way to change client-side apps every-time a small change has just released.

So, what is the best solution to use a single endpoint with a different version of the code? Changing inside the code is the responsibility of developers who own that feature.

By forwarding HTTP headers to the code inside a Lambda function, developers have full control of what to behave with the incoming request. But modifying the last qualified code (v1 feature) is not alike for the modern software development process.

Fortunately, Lambda has Layers, an architecture that allows a function can utilize the code that deployed differently after the function was deployed. This way can allow overriding the existing file structure in it.

Figure — 8. Using Lambda layered architecture to switch among features.

An example of using NodeJS runtime in Lambda function, we deploy 3 layers for v1, v2a, and v2b feature in three different node modules. In the Lambda entry point index.handler(), it lazy loading the layered code as a module:

Figure — 9. Lambda main code to switch between layered codes.

Every time you need an update from a feature (v2a or v2b), there is the only thing that’s need to update is the layered code of it. Even if you extend your A/B testing scenario to another payment method, eg: “Pay with GooglePay”, you use another value for x-service = v2c and developing the “v2c-code”.

Warning: the example code above is not recommended to use in production due to it’s lacking security prevention. You must sanitize the header’s value before it used to not exposing the vunerable of remote execution ability throught the require(“…”) method.

Conclusion

Choosing the right solution at the right time needs your engineering team’s experiences and their awareness about how the system works. There is a no-one-fit-all solution always, the best one just works for a particular situation.

(Subscribe the Problem Solving Blog to watch out…)