Behind the scene of ETag caching
Recently I had a task to implement custom ETag caching mechanism into our core part of the iOS app. My rough estimation of the task was 2 days. Unfortunately, 2 days only took the investigation work, please let me tell you why.
What is Etag?
First of all, what the ETag is? ETag is a response header in HTTP that allows you to check whether a requested resource has changed or not. Thanks to ETag, caches can work more effectively. As instead of the requested resources returning the data it returns 304(NOT MODIFIED) and the data are taken out of local caches. That saves us a bandwidth in our applications. On the first request, the client saves the ETag and then attach it in If-None-Match
header for the next request.
ETag caching behaviour in clients
Along the way of implementing my custom ETag caching, I learnt the hard way how it works behind the scene on client sides (Browsers, iOS, Postman, curl, …).
By default, high-level clients like Postman, Chrome, iOS… will add to the request an HTTP header Accept-Encoding, which makes absolute sense as you don’t want to add that header to it by yourself all the time and you want to have the transfer to be as effective as possible. Thanks to my dear friend Charles I tracked down that different client adds different accept-encoding header values.
Postman: Accept-Encoding: gzip, deflate
Chrome: Accept-Encoding: gzip, deflate, brFirefox: Accept-Encoding: gzip, deflate, brSafari: Accept-Encoding: br, gzip, deflateiOS: Accept-Encoding: br, gzip, deflate
curl: no accept-encoding is presented
(order matters)
ETag on AWS CDN
Our CDN whose the custom ETag caching was meant to be implemented for is hosted on AWS. By default, if you request a CDN resource on AWS in any client besides curl it returns in headers Transfer-Encoding: chunked. That makes total sense. CDN expects that it will be sending large files to clients, therefore, it serves them in chunks.
Now we are getting to the bottom of the problem. Since clients always send the Accept-Encoding header as shown above this led to a mismatch of encodings and the ETag was never calculated and therefore, never returned in the response.
Interestingly, since curl has no Accept-Encoding header present the ETag is returned every-time as the server in my case CDN on AWS decides which encoding to choose and serves it to the client.
Solving the problem
So after roughly two days of depression, applying for a new job in McDonald’s, giving up on SW engineering, I had the solution and my sun shined again.
The solution is quite straightforward, to make it all work you just have to overwrite the default Accept-Encoding header so that the client does not attach the default one by itself. This worked for me on every platform.
Conclusion
Sometimes it takes more time to figure it out the nowadays convenience of clients than to implement a related specific feature in the app. I hope this article saves someone the time that I had to put into the investigation.
