IKEA.com — the problems with static content
In my previous blog post, I wrote about the history of IKEA.com going from a monolithic e-commerce platform to a modular web architecture delivered using static files.
In this post, we’ll go through the problems related to A/B testing and personalisation in this static files based architecture.
These problems then become the drivers for us to change the web architecture towards server-side rendering using Edge Computing: “If we want to increase A/B testing and personalisation, and we can’t do that without also increasing the problems they create in this web architecture, we need to change the web architecture”.
Table of Contents
- Static content
- A/B testing and Personalisation
- As-is problem statement
- A/B testing sub-problems
- Personalisation sub-problems
- From Static content to Dynamic content
Static content
The majority of the HTML page responses for IKEA.com is served from static content, using pre-rendered HTML files uploaded to a file store. The rest of the page responses are Single-Page Applications. Static pages and SPAs can contain static fragments and JavaScript components.
Static content has been a great pattern for us, since it easily enables Web Performance, Availability, Security, and SEO. We currently have 40 million “active” static files, i.e. being updated and serving traffic.
However, we have encountered problems when increasing the use of A/B and personalisation using the static pages approach.
A/B testing and Personalisation
Before going through the problems, let’s have a very short introduction to the concepts.
A/B testing
A/B testing (or A/B/n testing) allows us to get fast and reliable feedback that the changes we do to IKEA.com increase the customer value in reality. And inversely, that the changes we do have no clear unintended negative side-effects.
Verifying changes using A/B testing does not replace other important practices like user research, but instead amplifies them.
The last twelve months, we have conducted 1100 A/B tests on IKEA.com and the goal for the next twelve months is to grow this number by at least 30%. My own ambition is that all changes to IKEA.com should be possible to A/B test (with some exceptions, i.e. changes to DNS).
Personalisation
From a high-level perspective, personalisation is the practice of presenting different experiences to different users. Some important factors are:
- The granularity of the personalisation, i.e. purely individual, user segments, or geo location
- How the functionality of the site is changing due to the user properties, i.e. displaying the user’s name, picking more relevant content for the user’s segment, or changing the order of content/components to be as relevant as possible
- Where the data is coming from, i.e. user settings, logged in profile data, click-streams, and geo location APIs
- Related to user settings, how to handle users logging in and out in different devices, and how to resolve possible data conflicts
- When to update the user UI, i.e. near real-time, hourly, daily, weekly. For example changing the layout of a page within a session would likely confuse the user
It’s important to stress that Compliance (including GDPR), Privacy, and Ethics are also crucial factors here.
Two sides of the same coin
A/B testing and Personalisation in their essence both present different things to different users. The difference is that in A/B testing we want users to be assigned to test groups randomly and the assignment between tests are fully independent, but in Personalisation we want to change the experience based on the user’s behaviour over time and their context.
The consequence of this is that A/B testing bucketing would almost always be a faster operation than Personalisation, because A/B testing only needs access to a seed, a configuration, and a hashing function. Personalisation needs access to user data, which then needs to be available in a performant way globally.
As-is problem statement
Without any other tools than static files and client-side JavaScript, the only option for static content teams to implement A/B tests and Personalisation is to use client-side JavaScript because static files can contain no logic in themselves. Client-side JavaScript has a clear performance impact, impacting our Core Web Vitals, which in turn is a ranking factor for SEO.
At IKEA.com we have already seen an increase of Cumulative Layout Shift (CLS) happening in production due to A/B testing and Personalisation of certain features. Unless we do something different, when we increase our use of these practices over time, we will also increase the problems associated with these practices.
CLS is not the only problem associated with these practices. I’ll try to cover the other problems below. However, some of these problems only happen in some cases, which makes it hard to get a sense of the total size of the problem. Therefore, I’ll try to decompose the problem into different sub-problems, when they occur, and what the impact is.
A/B testing sub-problems
In this scenario, we assume that “A” is already statically rendered, and we alter the experience with client-side JavaScript to B (if the user is bucketed into B).
Cumulative Layout Shifting (CLS)
Measuring CLS in the large is not trivial. Also, it’s important to point out that the Google Core Web Vitals measurements are done in real user experiences measured in Chrome and visible in the CRuX reports.
Most web navigations are “forward”, which makes it very important that above-the-fold content has low CLS. But we should also not forget users navigating backwards, which will navigate to their last scroll position on the previous page.
If we assume that above-the-fold content is the most valuable information on a page (why is it otherwise there?), we can also assume to have a higher ratio of A/B tests above-the-fold.
Flickering (Flash of Original Content)
Flickering is similar to CLS in the sense that the user experiences that elements are replaced in the browser. The difference is that these replacements don’t cause a layout shift.
According to my knowledge, there exists no web performance metric for flickering in web browsers. But it’s still a web performance problem to solve (in my opinion), even if we need to rely on visual tests in order to detect the problems.
Biased/uncertain experimentation process
CLS and flickering are performance problems, which makes the user experience worse. However, in an A/B testing context we can sometimes also get a positive effect, since UI elements that move/blink can increase the attention from the user.
An experimentation process should be as unbiased as possible, so that we can trust the results. A/B testing is a method to create a “all other things being equal” environment, where we can measure the differences between A and B in isolation.
But if we introduce both positive and negative factors of unknown magnitudes into the system, can we then trust the results? For example, a bad scenario would be that we measure that B is better than A, but when we properly release B into production we see that it’s not better, because we had noise in our experimentation process. Or the opposite: in another test we measure that B is not better than A, when in reality the negative effects of CLS and flickering is cancelling out the true value increase caused by B.
CLS can be measured, so if we have zero CLS increase in the test we can rule out that factor. However, flickering can’t be measured, so we wouldn’t know (not at scale at least) if flickering is biasing our tests.
Poor developer experience during experimentation
If we want A/B testing to be adopted at larger scale, we need developers to be able to A/B test in their current web development paradigm. But for the static content teams, they need to implement their tests using client-side JavaScript. Then, if B wins, they need to take that code and adapt that to their static content generation web paradigm.
This approach might be ok for A/B testing in smaller scale, but not when it’s becoming an integrated part of a team’s workflow.
Personalisation sub-problems
Cumulative Layout Shifting (CLS)
If we don’t reserve screen real estate for personalised content or if the personalised content is not fixed height (width is often less of a problem), CLS will occur when the personalised content is rendered.
If the order of components is changed in order to increase relevance, CLS is almost guaranteed to happen.
Again, CLS (and flickering) is mostly applicable for above-the-fold content, even if we shouldn’t neglect backward navigation.
Search Engine Optimisation risks
Search Engine Optimisation (SEO) is very important for IKEA.com. Search engines is also part of the user journey for our users: if a user enters a search term, we want to help search engines to display as relevant results as possible. A fundamental piece of this is to allow search engines to crawl effectively, in order for them to index and rank both new content and updated content.
The faster our pages load and render, the more Google is able to crawl for the limited crawl budget we have.
From Static content to Dynamic content
Static content has been instrumental to help us build, scale, and operate IKEA.com for our globally distributed customers. As long as the files contain the correct information, the operational work from the teams is minimal. And since the files already contain the HTML needed to render the pages, static content enables great web performance and effective SEO crawling.
However, as described above, increased A/B testing and Personalisation introduces great challenges that are not possible to overcome in the static content web development paradigm. Therefore, we much change the web architecture from Static content to Dynamic content, i.e. server-side rendering.
Server-side rendering enables us to respond with different responses depending on the properties of the customer, i.e. A/B testing seed and customer data. The challenge here is to implement server-side rendering in a way that still enable the product teams to easily build web solutions with great global web performance, effective SEO crawling, and with great developer experience. In other words, we want to keep the great properties of the current web architecture, while adding dynamic capabilities. And here’s where Edge Computing is an important technology going forward.
The next blog post in the series will cover more about Edge Computing from a web perspective.
