Fingerprinting and Cloud Storage, or how we became Asset Delivery Champions

At idealo.de, we dramatically improved the caching efficiency of the ‘Offers of Product’ page by exploiting cache busting techniques and Akamai cloud services.

This is an enhanced and lengthened version of a previous post published on the Kreuzwerker blog.

The beating heart of idealo is the ‘Offers of Product’ page, which aggregates the offers from over 50.000 online shops to enable quick comparisons and purchases. The web-application that supports these functionalities is updated almost every day, in the quest for the best user experience. Many of these updates require changes to static assets like images, CSS, and JavaScript files, which are cached in the browser to improve the frontend performance. But, as any web developer will tell you, delivering fresh assets while still taking advantage of the browser cache is not always a piece of cake. It certainly wasn’t for the legacy application behind the ‘Offers of Product’ page, whose assets management performance was far from being satisfactory. We had to find a better solution. And we did.

In this article, we share some experiences and lessons learnt during the redesign of an effective process for delivering the static assets of the ‘Offers of Product’ page.

Motivation

The backend of the ‘Offers of Product’ page is a Spring Boot-based application. In the original architecture, the static assets were packed into a single bundled artefact (a Debian package), deployed on several application servers, and finally distributed via the Akamai Content Delivery Network (CDN) using the application servers as the origin. The application servers sit behind a load balancer to achieve high availability. This architecture is illustrated in the diagram below.

The continuous improvements to the application led to daily deployments and frequent updates of the assets. To tell the CDN which images, stylesheets, and scripts were required for a given application release, each asset was served under a URL specific to that release. The automatic generation of the URLs occurred at compile time, and included the version identifier of the application in the URL path. The version identifier was the short Git version hash. For example, if the internal location of an asset was img/logo-idealo.svg and the application version was ab1234, then the resulting URL looked like cdn.idealo.com/offerpage/ab1234/img/logo-idealo.svg. The same asset had a different URL in the next application release even if the asset did not change, due to the new Git hash prefix.

The rationale behind this versioning strategy was to ensure that the end users always got the freshest assets. However, this strategy is not caching-friendly at all. The browser must download all assets after each application deployment, because the new release-dependent URLs prevent browsers from using their cached copies of the unchanged assets. That is not very efficient, especially for mobile devices or with limited bandwidth.

The original architecture and deployment pattern contributed to another issue with the legacy asset delivery, generating a significant number of 404 Errors at every deployment. Recall that each end user’s request reaches a load balancer running in front of several application servers, with each server compiling the asset URLs based on its release version. It is then paramount that all servers are running the same application version, otherwise, the assets of the outdated versions won’t be reachable on the latest release, and vice versa (see the diagram below). Unfortunately, that’s what happens at every deployment, when — even if only for a short time — some application servers are running the old release because they have not been updated yet.

One might think that configuring the load balancer with sticky sessions might avoid 404 Errors. However, when the application server eventually restarts to run the last release, all its clients will lose their sessions and the issue will appear again, unless a costly failover strategy is in place.

The two issues described above, besides their direct impact on the performance and availability of the ‘Offers of Product’ page, also represent a major impediment toward continuous deployment of the web application. In fact, the more we deploy, the less we benefit from the browser cache and the more 404s we will get.

But so much for the problems. Let’s talk about solutions.

Redesigning the Asset Delivery: Key Ideas

We designed the new asset delivery process around two technologies: file fingerprinting and cloud storage.

File fingerprinting is a versioning technique that binds the name of a file to its actual content, usually by adding the file hash to the name (e.g. logo-idealo-kr3u2vv3k0r.svg). The idea is that a fingerprinted asset will keep the same name — and URL — across different application releases, as long as its content doesn’t change. Hence, the browser can cache fingerprinted assets for the longest possible period, as it will only be forced to download the updated ones as indicated by their new URL. This practice is also known as cache busting, and will overcome the caching inefficiencies of the legacy application.

With fingerprinting, the occurrence of 404 Errors during deployments can decrease significantly, because a fewer number of assets will change their URL between releases. This doesn’t mean that the issue is solved. Consider, for example, that the latest application release has only one updated asset. If the old version of that asset was never requested, and therefore never cached by the CDN, a new request for the old version could not be resolved by any server running the latest release. Then we have a 404 again.

To solve the problem once and for all, we store the recent versions of each asset on a cloud storage service and configure it as the new origin server for the CDN — see the diagram below. This way, previous versions of an asset will be available long enough to prevent any 404 Errors during deployments.

Are we all set? Let’s dive into the technical details!

The right Tool for Fingerprinting

Since version 4.1, the Spring Framework provides a set of managed components to create and resolve fingerprinted assets in an automated and easy-to-configure fashion. The framework calculates the version of an asset at runtime on every first request. Then, it caches in-memory the mapping between original and versioned names for the next requests. Although promising, serving the assets through the application will still lead to 404 Errors during deployments, making the Spring way unsuitable for our needs.

In our solution, we rely on the popular build tool and module bundler: Webpack. Using Webpack, we create a fingerprinted copy of each static asset required by the ‘Offers of Product’ page along with a manifest file that maps the original asset path to the versioned one. An example of the manifest file is as follows.

We chose Webpack for a number of reasons:

• Relevance: the tool is designed to deal with web-assets.
• Reusability: another module of the application already used the tool.
• Simplicity: the tool comes with plugins that greatly simplify fingerprinting, such as MD5 hash calculation, manifest generation, and processing of versioned JS/CSS files to replace each reference to an unversioned asset path with the corresponding fingerprinted path.

The fingerprinted files are packed into the deployable artefact together with the manifest file. The application resolves asset requests at runtime by (i) querying the manifest, (ii) generating the versioned URL using the query result, and (iii) rendering that URL into the DOM. If the runtime cannot find the requested asset in the manifest file, the fallback strategy is to generate a URL using the unversioned path. We implemented these functionalities by extending existing Spring components as well as developing new ones.

Up in the Clouds

Our redesigned asset delivery avoids 404 Errors during deployments by using cloud storage as the origin server for the CDN. Among the available cloud storage services, we selected Akamai NetStorage due to its perfect integration with the CDN of the same vendor currently in use at idealo.

We integrated the process to upload the assets to NetStorage into the deployment pipeline right before deploying the application. Note that if this process fails, then the application will not be deployed. The process consists of the following steps:

1. Extract manifest and fingerprinted assets from the latest application artefact.
2. Upload to NetStorage the assets listed in the manifest that are not already in the cloud.
3. Remove obsolete assets from NetStorage. The clean-up policy is to keep a configurable number of latest versions of an asset and remove the older ones.
4. Check that each asset in the manifest is available via the CDN. As a positive side effect, these checks will issue a request for every asset, forcing the CDN to cache them.

For the implementation of the process described above, we could rely on the NetStorage HTTP APIs to manage resources on the cloud storage. Akamai provides these APIs in a variety of programming languages, including Python, Java, GoLang, and Javascript/NodeJs. We used the Python implementation, mainly because is the most updated and maintained on GitHub.

The deployment pipeline is managed by the Jenkins build server, which coordinates the different stages of the application deployment, including the execution of the Python program that implements the upload process. We recommend to execute the program in an isolated environment to keep the program installation clean as well as to avoid permission issues on the Jenkins server (for example, when installing Python dependencies via PiP). Virtualenv and Docker are both valid tools for serving this purpose.

Results and Conclusions

In this article, we discussed solutions and lessons learnt in redesigning the asset delivery process for the ‘Offers of Product’ page while aiming to improve its efficiency and reliability. We based the new design on two key ideas: (i) assets fingerprinting, to enable cache busting, and (ii) storing the assets on Akamai NetStorage, to increase their availability.

We evaluated the performance of the new asset delivery in the last 80 deployments — a time span of almost three months — tracking number and size of the assets uploaded to NetStorage, as well as the number of 404 Errors. As you can see from the two charts below, the results are stunning. On average, we went from 590 uploaded assets to only five per-deployment, which translates into almost 96% of traffic saving on the end user’s side — from 8.8MB to 365KB of data to download. As for the 404 Errors, relying on the cloud storage solved the issue for good.

We expect this performance to be robust enough to withstand the continuous improvements on the ‘Offers of Product’ page, while enabling the team to take the logical next step toward continuous deployment.

If you found this article useful, give me a high five 👏🏻 so others can find it too. Follow me here on Medium, Linkedin, GitHub, or StackOverflow to stay up-to-date with my work. Thanks for reading!