How we served brotli compressed files through Amazon Cloudfront — Ramakrishna
What is brotli?
Brotli is an open-source, lossless compression algorithm that compresses content with higher density than gzip.
Why brotli?
According to the compression benchmarks, brotli provides better compression(around 15% improvement over gzip). This means that all the static assets sizes would be reduced by 15% if we serve brotli compressed files to our users.(https://www.opencpu.org/posts/brotli-benchmarks/)
The only caveat is that the compressing time for brotli is significantly greater than that of gzip. This problem can be solved by compressing the files beforehand during build time. This would increase the build time but it can be further optimised by caching the built files.
The Problem
We are using Amazon Cloudfront as the caching server for serving our static assets. Amazon Cloudfront currently does not support brotli compression.
Cloudfront(or any standard caching server) reads the Vary header sent by the backend to create its cache keys. For example, if the Vary header contains User-Agent, Accept-Encoding then the cache keys would be maintained with a combination of these 2 headers.
For static assets, User-Agent header is not significant. But Accept-Encoding is extremely important. It is through this header, that the browser tells the server , which type of encoding it can accept. But different browsers have a different way of sending the content in this header. For Example, Chrome’s header may consist of gzip, br, deflate and Safari’s header may be deflate, gzip, br. From the POV of the caching server, these 2 requests are accepting different encodings, hence different keys would be maintained by the server, although content served would be same.
To optimise this, Cloudfront normalises the Accept-Encoding header. It strips the header from all values leaving only gzip or none. Sadly in this process, br encoding gets lost and the backend never finds out whether the request could have been served with brotli compressed content or not.
The Solution
We used lambda edge servers to preserve the Accept-Encoding header values sent by the browser just before the request was processed by Cloudfront. Lambda Edge provides 4 hooks for a intercepting a request. More can be read about Lambda edge here: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html
We used the Viewer request hook to intercept the request and check whether the request is Accepting the brotli compressed files. If it is accepting them, then a new header(say X-Brotli) is added to the request. The backend checks whether the request contains that header. If the request contains it, then brotli content is served, else normal behaviour is maintained.
Catches:
1. The Vary header set by the backend should contain the value of that custom header set by the lambda server.
2. Cloudfront will have to be configured to forward the header set by the lambda server. This can be easily done through the cloudfront panel.
3. There is cost factor associated with lambda servers but it comes out to be negligible. The actual cost will depend on the traffic. In our case it was less than ~$10 a month.
The Results
Brotli reduced the sizes of all our assets by ~20%. The page load time went down by ~0.2s
