Broken ‘Hyper-lynx’: How a 30-year-old browser caused an issue on our Android app in 2023
What do a 30-year-old web browser, a secret codename and the ASOS Android app all have in common?
This might sound like a trick question, but recently a small handful of our ASOS Android users experienced a bug that linked all three of these seemingly unconnected things together 🤯
At ASOS, we are always keen to learn from any mistakes and share what the experience taught us in the hope others won’t fall foul of similar problems in the future. In this blog post, we’ll explore the issue we encountered, what caused it and our learnings from this rather unique situation.
So, what actually happened?
In a typical fashion and as with all good software issues, this particular bug was brought to the attention of our team over the course of a weekend 😅
We were first made aware that something wasn’t quite right when a small number of similar user reports started to come through our support channels to inform us that their Android ASOS app was not displaying product image content correctly.
This is the type of issue the Apps team takes extremely seriously, as it should go without saying that we want users to be able to actually see the products we have available within the app. Once we received our first screenshot, it was obvious from the user’s messages something was certainly amiss!
So without delay, we kicked off our incident support processes and began to investigate the issue at hand. However, much to our surprise not a single member of the team could reproduce the issue! 🤔 This was most strange and certainly warranted a deeper dive, especially given our user reports indicated this issue was not limited to a single version of our app.
We must go deeper… 🤿
When investigating bug reports from users, it is often helpful to find any common patterns to help narrow down the steps to reproduce and ultimately work towards the fix for the issue. This might include finding out the particular actions the user was taking when they experienced the problem, the screens they opened or the generic profile of the mobile device they were using at the time.
Thankfully when modern apps collect bug reports some of this information, namely the device profile, is available to developers automatically along with the user’s own explanation of the issue they encountered. Having this information available allowed the Android team to analyse the reports in search of commonality and fortunately for us, a pattern quickly emerged.
Device: Pixel 7a , Device: Pixel 7a , Device: Pixel 7a …
All of the reports were coming from users with a Pixel 7a, Google’s newly released flagship “Pixel A-Series” phone launched at May’s Google I/O 23 conference ¹ 😵💫
This was a shock, particularly as the new Pixel 7a was running on supported hardware/software and a similar device profile to others which our team runs thousands of automated tests over (without issue) on a regular basis. There should be no reason for the bug to exist on this single device type alone unless something is uniquely different…
The start of a strange coincidence… 🧐
One line of enquiry as part of our investigation was looking into how the app requests images. On the backend, we utilise a content delivery network (CDN), meaning static content such as images is cached and served via CDN edge servers to all our digital platforms, including our mobile apps.
Our CDN provider is also a tool we use within our cyber-security suite and has the capability to block requests that are deemed suspicious based on common and custom rulesets. This is a standard approach in large scalable solutions to ensure only legitimate users can access specific services or resources and to mitigate any potential denial-of-service (DoS) attacks. Our team reached out to Cyber Security and asked, Could it be plausible that this is blocking these legitimate requests for some reason?
Their answer was somewhat unexpected.
However, before we tie everything together and reveal the answer to the question we posed in the introduction, let me first introduce Lynx. Lynx is a text-based web browser that, according to Wikipedia, is the oldest continuously maintained browser having been started in 1992 by a team of University of Kansas students and still actively supported as of 2023.
Its interface is incredibly rudimentary by today’s standards, but given the information available on the web was primarily text 30 years ago, this is quite understandable.
An entirely text-based web browser might be somewhat of an alien concept in 2023, but in the early days of the internet (and perhaps today for some modern-day purists) Lynx stood out as a popular choice, particularly with visually impaired internet users who found the interface to be helpfully compatible with most text-to-speech software.
How we made the app ‘pixel perfect’ (again) 👾
Now, while Lynx is great at what it does, it doesn’t give users the best experience the modern web has to offer and as such, is commonly blocked by tools such as the security services in our CDN. 😱 Wait, isn’t that what was happening to our Android app?
To our surprise, Cyber Security confirmed that it was treating these requests for images to be coming from an ‘Unsupported Browser’ — i.e. Lynx and these requests were being blocked.
So why would the Pixel 7a be falsely reported as being a Lynx browser? We inspected the blocked request’s User-Agent header, which is commonly used to identify the application, operating system, device, etc. and contained within it was our suspectlynx🤯 After some quick Google-fu, it transpired the codename and therefore the output of Build.DEVICE for the Pixel 7a was “lynx”… we were being blocked simply due to the device’s name!
The lessons learned 👩🏫
After a quick rule change on our CDN’s security dashboard, our Pixel 7a users were able to continue to browse the Android app as normal. In total, we estimate less than 0.3% of our Android daily active users were affected, but nonetheless, this issue taught us an awful lot and is something we will actively monitor going forwards.
In software development, sometimes issues can arise with no prior warning and seemingly no logical explanation. Thanks to the thorough support processes we have in place at ASOS, bugs such as these are dealt with logically and quickly (even at weekends). We do our best to be prepared to ‘expect the unexpected’ and thankfully due to our methodical approach, we usually get to the bottom of things in good time.
Nevertheless, I am sad to announce ASOS’ support for Lynx will remain on the backlog for a little while longer. Until then, I guess we will need to keep an eye out for next year’s flagship “Google Pixel Netscape-Navigator” ² 😅
Thanks for reading — and happy browsing!
Footnotes
[1]: At the time of writing, Google I/O had occurred only a couple of weeks prior. Therefore it was highly unusual to have a brand-new flagship device causing issues
[2]: Netscape Navigator was the most popular web browser in the late 90s, eventually losing its dominance of the market to Microsoft’s Internet Explorer extremely rapidly in ‘the first browser war’.
