Performance optimised TUI Musement’s A/B testing solution on top of AWS infrastructure.

przemkow
przemkow
Jan 22 · 11 min read

Agenda:

Introduction:

  1. What is A/B testing?
  2. Initial design
  3. Frontend AWS infrastructure

A/B testing with AWS Lambda@edge :

  1. Architecture
  2. Implementation
  3. Performance Impact
  4. Costs

What is A/B testing?

A/B testing is a UX research methodology focused on comparing users’ engagement of two different versions of a website. The main purpose of A/B testing is to select the version with a greater return on KPIs (ex. the number of user interactions per visit or conversion rate).
Instead of releasing a completely new design for all users, with A/B testing you can show the new version of a component to a select group of visitors ex. 20% of all users. In doing so, you can compare a brand new UI with the old version, to determine which of the two variants is most effective and will have the greatest impact on your business.

Example of the component redesign.
{
"experimentName": "ActivityCardRedesign",
"variantPercentage": 20,
"control": "OldActivityCard",
"variant": "NewActivityCard"
}

Initial design

What do we need?

Let’s try to summarize everything and prepare a list of requirements. To implement an A/B testing system, we require the following:

{
"experiments": [
{
"experimentName": "ExperimentA",
"variantPercentage": 20,
"control": "ControllNameForExperimentA",
"variant": "VariantNameForExperimentA"
},
{
"experimentName": "ExperimentB",
"variantPercentage": 20,
"control": "ControllNameForExperimentB",
"variant": "VariantNameForExperimentB"
},
]
}
  • Tracking system to compare A/B testing results.
    This depends on the tracking system used by your company. You can use the custom dimension of Google Analytics as well as an internal tracking system implemented by your BI team.

Client-side or server-side?

In the previous point, we wrote that some applications might need to support server-side rendering. The same situation applies to A/B testing solutions, we can divide them into two groups: client-side and server-side.

Client-side A/B testing
Server-side A/B testing

AWS infrastructure

  • If CloudFront used cached response: x-cache: Hit from cloudfront
  • Before CloudFront forwards the request to the origin (origin request)
  • After CloudFront receives the response from the origin (origin response)
  • Before CloudFront forwards the response to the viewer (viewer response)

A/B testing with AWS Lambda@edge :

Architecture

To fully support server-side rendering and not to impact cache performance, our solution needs to assign a variant for every user before request hits the CloudFront instance. AWS gives you a possibility to do so by using Lambda@edge. Lambda@edge is a feature of AWS CloudFront which lets you attach Lambda function to your CloudFront instance and run custom code which modifies request or response (dependent on which stage Lambda@edege is attached).

The architecture of A/B testing solution using Lambda@edge
  1. The request will arrive in the CloudFront instance. Based on various parameters (ex. Request URL, cookies or other Request Headers selected by you) and cache expiration time, CloudFront will then decide if the user can receive one of the previously cached server responses.
    If a valid response is present in CloudFront cache, CDN will immediately use that value without forwarding it to Node.js Server.
    In another case, CloudFront will forward the request to Node.js Server. When your server returns a rendered HTML it will be cached and used for the following requests.
  2. Assuming that CloudFront did not have a valid request within its cache. The request will arrive in the Origin — in our case — Node.js Server.
    On the server, we can assume that every request includes a valid cookie with a selected variant which needs to be presented (as we said before it’s the job of Viewer Request Lambda@edge to add a correct cookie to all requests). Knowing this, we can read that value and display the correct version of the currently tested element.
  3. Origin Response Lambda@edge — Before a response arrives in the CloudFront instance and it’s cached for the specified period of time, we need to add one more thing. In the first point, I wrote that Viewer Request Lambda@edge checks cookie saved in the user’s browser, however, until that point there is no mechanism which saves the cookie added to the request in the first step. To cover this, we need an additional Lambda@edge function added attached to Origin Response which will save Set-Cookie header to the response generated by the server. Doing so will save the cookie in the user browser so that on the next visit we can provide a consistent user interface. This part can also be implemented directly on Node.js server, in this case, additional lambda is not needed. The important part is to have the correct Set-Cookie Response Header before Response arrives in CloudFront instance.

Implementation

The first item which needs to be implemented is A/B test config. In our case, we decided to keep it inside a simple abtestConfig.json stored on S3 bucket. The key is used to describe a version of the config file.

  1. If the user has a/b test config with a timestamp equal to the one from testConfig.json it should pass the request without any modifications
  2. If the user does not have assigned a/b test configuration it should roll the dice for every experiment present in testConfig.json
  3. If the user has a/b test config with a timestamp older than the one from testConfig.json it should roll the dice for all new experiments, delete the finished experiments (not present in testConfig.json) and keep previously assigned experiments which are still present in the config file.

Performance Impact

Request Latency — As additional nodes were added to our infrastructure we expected an increase of time required to fetch musement.com from CloudFront equal to the duration of Lambda@edge. Within the main regions , we sell activities (Europe and USA), the average duration of Viewer Request Lambda@edge was ~2–10ms. Which is unnoticeable for the end customer. However, there are some situations when the duration is much longer. The first situation occurs when internal config cache expires (once every 30 minutes), at this point fetching the config increases lambda duration to ~250–400ms.
The second situation is the so-called “cold-start”. This is when your lambda is invoked for the first time. In this case, we noticed increase of lambda@edge duration up to 1.2s.

Costs

Infrastructure cost of implementing an A/B testing solution on top of Lambda@edge is strictly related to the traffic on your website. The current cost can be found in AWS docs. However, assuming that you have ~10 Million visits per month and the average duration of Lambda@edge is below 50ms you will pay approximately $10 per month.

TUI MM Engineering Center

News, knowledge and technology at TUI Musement

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store