How Android updates work: A peek behind the curtains from an insider

Updates on Android were always a topic of discussion in the tech communities. Even though they are a crazily complicated topic and very few people understand it, everybody seems to have opinions about them. From Apple fans criticizing Google for lack of effort to Android users condemning some manufacturers for delayed updates, everybody shows a different way of understanding them and forms their own conclusions that stem from what they think is true. However, maybe it’s time we learn about reality? This article contains everything I’ve learned about Android updates over the years from various people on the “inside”, various sources, documents, code.

This article wouldn’t be possible without the help of Husson Pierre-Hugues (@phhusson), Mishaal Rahman (@MishaalRahman), Kacper Skrzypek (@kacskrz), yshalsager (@yshalsager) and many others. Thanks!

Please note: this article is lacking in sources, because that’s the only way it could’ve been created. Many of the topics mentioned here are locked behind NDAs and other agreements. It’s either no sources or no article.

Background

We can’t go into updates without knowing about some fundamentals of Android. Feel free to skip this section if you have knowledge on the mentioned topics.

Basic Android architecture — Treble

For a more beginner friendly overview of Treble, refer to this article

Android 8 was the most important release in the history of Android. It didn’t offer anything revolutionary to the user, but it made a fundamental change to how Android is created as a platform. Before 8, the integration between system-side components and hardware-specific abstractions was inseparable. Everything was just hard connected, interfaces between various layers were loose and often changed from version to version. System-side was permanently bonded to the hardware abstraction. It was a mess and made developing Android for hardware very difficult.

In Android 8 Google decided that they had enough, and they’d make a fundamental change to how Android functions as an OS. Instead of having a hard connection using a loose interface, they decided to do the opposite. System-side components would expose a stable set of versioned interfaces that are necessary for functionality to function. A new image called “vendor” would house all the hardware-specific code and implement all the necessary interfaces for the system to function. This way, vendor development would no longer be so interconnected to the system. Android would even keep older interfaces supported for versions to come, meaning that the Android system could be updated without a change to the vendor image!

As an extra effect of that, a compliant vendor image that exposed standard interfaces could function under ANY system image exposing the same interfaces. That’s how Generic System Images (GSIs) were born.

Even though GSIs supporting older vendor versions existed even back then, Google still required OEMs to update their vendor images to match Android system versions until recently. Even with that limitation in place, Treble still revolutionized Android development and set a new direction for the whole platform.

Security patches

Security patches can be divided into 2 categories — HLOS (High Level Operating System) and NON-HLOS. The first category is for patches concerning the Android system itself and the Linux kernel, and the other is about code running at much lower levels. Generally, NON-HLOS issues are rarer and might cause way more harm, but it strongly depends, as vulnerabilities are not comparable.

Android X isn’t always Android X — QPRs

Quarterly Platform Releases (or QPRs for short) are major updates to particular Android versions. They bring bug fixes, stability improvements and minor feature additions. There are usually 3 QPRs per Android version, with the first one releasing in December. The aforementioned “QPR1” is often the one with the most important changes, fixing bugs in famously buggy initial Android version releases by basically cleanly redoing all the changes in that version.

Recently, Google started including QPRs as a part of their “Pixel Feature Drops” (although they mainly focus on new features in their services and not Android itself).

QPRs are not mandatory for OEMs to implement, they can use the release versions as long as they integrate the required security patches. This means that OEMs rarely release QPR updates, which is a shame considering the terrible state of initial Android releases.

Android 12L (internally “Sv2”) is actually just the “QPR2” for Android 12 (yes, that means it’s releasing in March). It brings an unprecedented amount of changes for a QPR update, including many bug fixes, huge improvements to large-screen devices, new features and even a new SDK level.

Your Android might not be… Android

Even though your Android device might appear to be running a “stock” or “AOSP” feeling system, it might be using something entirely different. All major chipset vendors maintain their own modifications (or “forks”) of the Google-provided Android Open Source Project (AOSP). How that usually works is that the vendor applies their changes over AOSP and then Google helps them integrate new Android updates (security, QPRs and major platform releases) over their modified version via something called “Keystone program”.

One of the most prominent forks is the one maintained by Qualcomm — commonly referred to as “Codeaurora” or ”CAF”. The current “branch” has been started somewhere around Android 9 and since then all new Android updates have been updated on top of it. This sometimes leads to bugs and messy/incompatible system code, although it does also occasionally bring some improvements like better carrier support or performance uplifts in some scenarios.

While most OEMs use chipset vendor-provided modified versions of Android, it is not a requirement, and they can instead opt for directly using AOSP.

The “N+x” notation

Throughout this article, I’ll be using the “N+x” notation. It isn’t anything formal, but it’s used amongst (at least some) enthusiasts when talking about updates. So what does it mean? It signals how many platform updates were shipped on a particular device.

For example, if a device launched with Android 9 and the last update it got was Android 11 we can say it was updated to N+2 (since it got updated to A10 and A11). You can also substitute “N” for the launch Android version to find out the last version, e.g., 9+2=11.

The process of updating

Source availability

The first topic that’s widely misunderstood about updates is the OEMs’ access to pre-release source code. This topic is a little complicated, so I’ve divided it into 3 sections.

Security patches

As you might know, Android’s public security bulletins are usually released on the first Monday of the month, but when do OEMs get access to preview and apply them? To answer that, first you need to know that there are 2 “variants” of security bulletins — YYYY-MM-01 and YYYY-MM-05 — the first one is just the AOSP-side patches where the second also includes patches from partners like Qualcomm or MediaTek. OEMs can release whichever they prefer, however if they pick -01, they still need to comply with the changes from -05 from the previous month (example: OEM releases 2021–12–01 and then 2022–01–01 next month — the second update needs to also comply with changes from 2021–12–05).

With all that in mind, let’s finally get to the answer: AOSP patches are released as partner bulletin previews. The release timeframe seems to vary greatly between different months, but in general the first preview of the patches is released in the first or the second week of the preceding month. That’s not the final release, though, it can still change over time if new issues of high importance are found.

As for the partner patches, there is no hard schedule. They can release the patches whenever they want and with some partners — *looks at Qualcomm* — where there are a lot of platforms sharing a lot of code (and vulnerabilities) OEMs might not even get the patches before the embargo lifts!

As for the embargo lift, it’s usually 11AM PST on the chosen day. That’s the date when the OEMs can start talking about the patches and release their builds. Sometimes the embargoes are broken by OEMs — with a common offender being Samsung. This doesn’t mean that they’re faster than Google, though. This means that they don’t respect the embargo date.

Google backports Android security patches for 3 years on mobile platforms (as mentioned in this Android Automotive OS document).

Letter updates (aka Android version updates)

Version updates are ultimately what most people care about. They are also much harder to adapt and port to devices than the regular security updates. Google keeps them more secret and only lets some OEMs in. Let’s take a look at an example timeline of the source release to OEMs:

• ~December — big companies working closely with Google get direct access to the source. The details of this type of release are largely unknown, but it’s most likely limited to the biggest players. At this point, the next Android release is still in its early stages — most features are only starting to be developed.
• ~March — OEMs can start applying for source access with their silicon vendors. This kind of agreement is called “keystone” and it’s signed between the OEM, Google and the silicon vendor (e.g., Qualcomm). Seemingly, it’s the most common way to get the source. There are a few minor prerequisites (mostly regarding update timelines) to be able to join, but it’s nothing impossible. This type of agreement grants access to silicon vendor’s Android fork (e.g., “CAF” for Qualcomm, “MocorDroid” for Unisoc) but also unmodified AOSP code for the release and other vendor software necessary to run the specific android version (commonly referred to as “BSPs”). It’s also possible to get source directly from Google via “Early Access Preview”, however most OEMs opt for the first option.
• ~July — OEMs get access to preview versions of test suites to prepare their software for Google’s certification process.
• ~Around the public release date — Final test suites are released

OEMs can ask Google for permission to use preview test suites for the final build so that they can release on day-1.

As you can see, OEMs have plenty of time to prepare their updates in advance if they get in on the action early enough.

QPRs

As mentioned before, QPRs (Quarterly Platform Releases) are major updates to a particular Android version. They usually bring in bug fixes, code quality improvements and minor features. This year QPR2 for Android 12 has actually been promoted with its own marketing name — 12L, because it brings so many improvements and even a new SDK version. Access to these releases is provided directly by Google. Interested OEMs can get access to the in-development branches well before the release. Finalized versions are often frozen about a month before release. There are typically 3 QPRs per release, with the first one bringing the most changes. Updating devices with these patches is not a requirement (12L might be an exception, as it bumped the API level).

Certification process

To be able to release a software build with Google services, OEMs must certify each build.

The tests

There are several tests that need to be fully passed in order to certify a build:

Publicly available ones:

• CTS (Compatibility Test Suite) — automatically checks the compatibility with standard Android APIs to ensure full app compatibility
• CTSV (Compatibility Test Suite Verifier) — supplements CTS, by providing tests that require manual input. This only needs to be run once on launch and major platform updates
• VTS (Vendor Test Suite) — tests the vendor-system interfaces implemented in the OEM’s vendor image and kernel (note: no prebuilt binaries are publicly available, but the source code is public)
• CTS-on-GSI and VTS-on-GSI — the same tests as above but running on generic system images to ensure full vendor-side compliance. On 5.4+ devices, it is necessary to run these with Google-provided boot.img’s containing GKI kernel builds
• ITS (Image Test Suite, technically a part of CTS) — verifies various camera-related requirements that cannot be verified without a physical test. Requires a special box for testing
• MTS (Mainline Test Suite) — various Project Mainline-related tests

Partner-only ones:

• GTS (Google Test Suite) — verifies compliance with GMS related agreements like MADA (Mobile Application Distribution Agreement)
• BTS (Build Test Suite) — scans system images for potentially malicious apps using similar tech to the Play Store. Unlike the other tests, it’s ran offline, without a physical device. OEMs need to send their firmware to Google for analysis.
• STS (Security Test Suite) — checks if vulnerabilities fixed in the declared security patch level are actually patched

This whole set of tests is often referred to as “xTS”.

The process

OEMs usually start from a base provided by their chipset’s vendor. It’s often fairly well-prepared to pass all the tests and requires little effort for 100% compliance. However, due to the extensiveness of the tests, even simple changes can cause failures. The process of certification goes something like this:

1. Prepare a new software build
2. Optionally run the tests locally
3. In case of smaller companies send the build to the 3PL that’s reporting directly to Google, in cases of larger companies officially run the test themselves
4. After 4–5 days, get the results back

The worst thing about this process is how time-consuming it is. Five days might seem like a short time, but remember that in a perfect world OEMs would release builds near the beginning of every month, which means they really only have 2 weeks to integrate and internally test the security patches before they need to submit it for formal testing.

In case a test fails where it shouldn’t (for example due to improper implementation of a test for a particular requirement by Google) you must submit a waiver request to be able to “skip” the test until a new release of the suite is published. Timing of obtaining a waiver differs on a case-to-case basis.

OEM testing

Outside normal internal testing, OEMs might want to perform their own internal testing outside the standardized tests. For example Google equips many of their employees with Pixel phones preloaded with “dogfood” builds of their software and created another, internal test suite called PTS (“Pixel Test Suite”) designed, among other things, to test performance, thermals, security (SELinux) and accessibility options.

Other OEMs take a completely different approach and allow users to test the prerelease software in a semi-private manner. Examples of such companies include Xiaomi with private daily and weekly builds of MIUI China (please note that this kind of testing cycle wouldn’t be possible on a ROM with GMS, as certification usually takes 5 days).

Carriers are tough

For some reason, US people keep buying devices from their carriers, so getting them to sell your phones is basically the only way to get any good sales there.

To get a phone into a carrier’s roaster, you need to do a few things. First, one of them is preparing a full briefing on the device specs and target price well before the proposed launch (typically about half a year in advance). Then you need to comply with a ton of requirements. Some of them might involve modem-related things like support for featured tech (e.g., VoNR, VoLTE, certain 5G bands, RCS, region-specific emergency calling tech etc.) and others might be stuff like preloading carrier apps, implementing SIM lock or complying with misc security requirements.

Carriers usually require each build to be certified before it’s released. This typically involves rechecking the aforementioned requirements and doing other carrier-specific tasks. For example, a major US carrier requires OEMs to list every file, explain its permissions, describe the access of each preloaded app and provide a description for each of the patches in the security patch. Others might require listing all the parameters of various things, like all the supported hardware features, supported codecs and radio tech support. These things are not only basically pointless, but very time-consuming. Catering to all these requirements is not simple, so even OEMs that want clean firmware like Google often release multiple firmware variants to comply with all the requirements.

In general, while not trivial, it’s not impossible to manage to comply with it all. Carrier certification length strongly varies, but it’s generally not very lengthy.

Google’s update incentives

Over the years Google has offered many programs to improve experience on 3rd party devices — Google Play Experience, Android One and most recently Premier Device program. Some of them had update requirements.

Android One required OEMs to update devices with new major versions of Android within 90 days from the release date for 2 years (N+2). It also required monthly security updates for 3 years, with each one of them being within 45 days from their private bulletin release (so generally in the first half of the month).

In 2019 Google (privately) announced their Premier Device program. It is basically a less restrictive version of Android One, with an added benefit of 50% extra Google revenue share (12% vs standard 8%). It requires platform updates within 120 days from major releases. Examples of devices in this program include select Realme, OnePlus, Nokia, Xiaomi and Motorola devices.

Qualcomm

This section will go into Qualcomm’s behavior with updates, as it’s still the most prominent chipset vendor in the industry.

Qualcomm is (was) sorta good with updates

One thing you hear a lot in this community is how Qualcomm is the source of all evil, and they are the reason OEMs can’t update their devices, but that’s not really true. Qualcomm’s update timelines are (were) sorta good. Let’s take a typical mainstream chip for example. Snapdragon 730 is a midrange chip from early 2019. It was released with Android Pie (9). It’s still supported right now with Android 12 (N+3) and it’s likely it will receive Android 13 support too. This means that OEMs could totally provide at least the (currently) standard 3 major platform updates. Guess what — this is the same as Google’s Pixel 6 update scheme.

Let’s take another chip as an example, this time an extraordinary one. Snapdragon 660 was released in 2017 with Android 7.x. It is still supported today on Android 12 (although it’s extended support; more on that later). That’s N → N+6!

While not over the top, Qualcomm’s update policies are (were) always at least decent.

You might be wondering about the past forms in parentheses. Don’t worry, we will talk about that later.

What do you actually get?

Qualcomm’s licensees get a few main things for the price of a general software license (which ranges from around $500k to a few million $, sources say) — access to Qualcomm’s BSP for vendor-side software on a particular Android version and a BSP for system-side software (QSSI). This particular combination is supposed to pass Google’s xTS tests out of the box (and it often does). However, OEMs can also take a “dangerous” route and discard QSSI, instead opting for an AOSP-based system. This takes away the benefit of having a xTS pretested combo and forces the OEM to port parts of Qualcomm’s QSSI BSP to AOSP. This is what OEMs like Google do.

What to do if you lose official support?

Even if you run out of official support and still want to support, the way to go is to forward port the vendor BSP. While not as “plug and play” as official Qualcomm software, it’s certainly possible. Even without Qualcomm’s support in passing xTS it’s very much a possibility. One example of this approach would be Fairphone 2, where the company ported Qualcomm’s Android 7-based BSP to up to Android 10.

PAID SUPPORT 🤑

Qualcomm provides a few types of extra paid software support:

• “Standard” extended support — Qualcomm provides you with security patches for the last platform release of your chip of choice for longer, possibly spanning years. This variant might sometimes only include patches for the firmware part of the BSP.
• “Customized” extended support — Fully customizable. One of the clients of this type of support is Google, who gets “.c9” branches that are not only supported for longer but also tested for building against AOSP (as Google’s vendor and system are both AOSP).
• “Platform” extended support — Qualcomm provides extra years of platform updates, sometimes also including major kernel version bumps. A prime example of this is the aforementioned Snapdragon 660 after Android 10 (currently sitting at 6 major updates when counting extended support). As far as we know, this kind of support often costs upwards of $1 million/platform/OEM (multiple devices can use the same extended support plan if they are on the same platform). It’s only available for select chips.

Summary — updates from a perspective of an OEM

The process

For security updates:

1. Receive the patches from Google around the first week of the preceding month
2. Integrate the patches, create a build
3. Test the build for regressions
4. Prepare and submit the build for certification with the carriers you’re working with (length of this heavily varies)
5. Send the build for certification with Google (usually takes 5 days)
6. Finally, release the build

Major platform release:

1. Sign an agreement granting you access to the upcoming Android version ahead of the release
2. Integrate your changes from the previous Android version on top of the developing AOSP
3. Integrate new patches from AOSP as they come
4. Begin testing the builds internally and/or externally
5. Start testing xTS when Google releases the finalized tests
6. Certify a final build
7. Release

xSSI or what could’ve been

Somewhere around the Android 9 era, Google had an idea to finally begin taking advantage of Android’s split vendor-system architecture. They proposed a solution to chipmakers — split system build targets from the existing vendor ones. They called it “Single System Images”. This could potentially allow whole lineups of devices to run the same exact system image, reducing costs and unnecessary troubles of updating. Both Qualcomm and MediaTek implemented the idea (as “Qualcomm Single System Image”/”QSSI” and “MediaTek Single System Image”/”MSSI”) and… nothing. Basically, no OEM utilized the scheme, instead creating device-specific system builds.

How hard is it to update a device?

Not hard. Very easy, actually. Generally the more obligations you have (from carriers, partners etc.) the harder it is, but generally it’s all pretty simple. The true difficulty might come from justifying it all economically. OEMs might be tempted to save a bit by cutting on engineers working on older projects. And they do it. That’s the sad truth about updates from big corporations like Moto/Lenovo.

So what’s wrong right now?

The Pixel-AOSP-Google love triangle

“Pixel is just another OEM”

…or so Google wants you to think. They like to think that. They want everyone to believe that the Pixel skin atop AOSP is just a “reference implementation” of what OEMs can do. So is that the truth?

AOSP is made for Pixels. The OSS aspect is of secondary importance to making a product they can sell on their devices. Example? The whole Monet/Material You thing. The entire Android UI was redesigned around the extended theming ability. It was touted as one of the flagship features of Android 12, while Google was the only one that had it because they didn’t release the code.

Another one? The whole A12 release fiasco. A12 was super delayed due to the massive changes it brought and long-lasting effects of the pandemic. Realistically, it should’ve been delayed and delivered in the form that 12L is shaping up to be. The only reason Google released it in its state in October was the Pixel 6 launch. Pixel 6 only works with Android 12, so they had to release something to launch it with.

Virtually all features in AOSP are developed for Pixels, A12 was literally designed for the Pixel 6. Pixel isn’t just another OEM — it’s an OEM with an infinite head start and input into what we call “Android”.

GRF

Introduction

GRF is “Google Requirements Freeze” but you’ve probably never heard that name. The only public source really going into what it entails is this blog post from Google.

Previously to ship Android version N+1 on an android device that originally shipped with version N you had to release:

a) N+1 system image able to pass the xTS tests and

b) N+1 vendor image compliant with some of the N+1 vendor requirements with a few N launch device requirements.

This basically meant that to go from N to N+1 you had to update the whole software stack. Google decided to change that.

The idea is simple: a single vendor image built against version N will now be certifiable for up to version N+3. It will only need to pass vendor-side tests for the version it was built against (called VSR-N — Vendor Software Requirements for version N). Chipset vendors will keep updating the vendor-side BSP with security updates and providing OEMs with system-side BSPs for up to version N+3. Please note that a chipset vendor CAN update the vendor later to potentially get more than 3 platform version bumps, but it’s unlikely it will actually happen. OEMs are also not forced to embrace GRF and can instead port a vendor for version N+x themselves, but chipset vendors don’t support this scenario, so OEMs are on their own.

The first GRF devices are already out — all Qualcomm devices except the ones based on the newly launched Snapdragon 8 gen 1 and Pixels are using Android 11 vendor when updating to Android 12.

The problems (plural)

First of all, there’s a very simple timing issue: devices that are released later with a chip frozen on Android version N but themselves have Android version M=N+1 will only be able to go from M to M+2, because the chipset will die on N+3=M+2. This issue is only amplified by the fact that chipset vendors like Qualcomm release “refreshes” of chips that are basically the same things but with slightly higher frequencies — the support window will be the same as that of the original chip!

While it was also the case before, more or less, GRF makes it much harder for OEMs to extend the support themselves. In the “pre-GRF” era it would be simple — port the BSP from N+3 to N+4, work towards passing all the new tests and be done, nothing too hard. Now, however, it’s much harder as they’d have to port BSP from N to N+4. That’s 3 versions of requirements and compatibility changes to worry about. While before this wasn’t a big problem now it really is.

Second, remember the subchapter on Qualcomm where I mentioned that Qualcomm was already providing N+3 updates for their chips? Right… So what’s the point of this? Google says it’s to reduce the costs that the chipset vendor has to take, however I think it’s stupid. OEMs still need to do stuff to update, and chipset vendors were doing just fine before (Qualcomm earns $5 billion a year and they can’t afford to update 6 platforms?). Sounds like Google decided that all devices shall live for 4 years (or less in most cases) and that the profit margins of already greedy hardware vendors should increase.

Last (and the most important) problem is the feature stagnation. Imagine a scenario — there’s a device with vendor and system on version N. Google decides to update the camera HAL in N+1 to include a new feature that allows control of the intensity of the flash. It’s trivial to implement, as current flash implementation is basically just telling the kernel to set the LED to its max brightness. So the device should get the feature when updating to N+1, right? WRONG. The updated HAL definition only appears in N+1 so the N vendor CANNOT implement it.

There’s actually a very good example of a similar feature right now — the 2G toggle. It requires a minor update to the radio HAL, it’s fully software. No current devices got it because they are stuck on the previous revision of the radio HAL since the new one wasn’t implemented on the Android 11 vendor they’re running. GRF has already prevented many devices from getting a new security-related feature, and it’s only the beginning.

Impact on OEMs

If the OEMs want to follow GRF, it’s a fairly small change to their responsibilities when updating. They still have to update the system all the time, merge BSP releases and work towards passing xTS tests. The only difference is with bringing up new platform releases, where it’s a bit easier with GRF.

On the other hand, if they want to ignore GRF and go their own way it’s much harder, as mentioned above.

OEMs pretending to be better at updates than they really are

While this isn’t strictly related to Android updates as a whole, it still seems appropriate to mention.

Fairphone (3 & 4)

Fairphone once really cared about Android updates. They actually bothered to make their own software and then update it. Fairphone 2 went from N to N+6!

However, since FP3 things started to change. The software running on that device was made by an ODM (the same company that was commissioned to make the hardware). FP3 launched with Android 9 and was updated to 10. There is an 11 update planned for later in 2022. It’s not getting Android 12 because they picked an ancient SoC. That’s right, N → N+2 from a company that prides itself on long updates. They did also sorta blame Qualcomm for their own bizarre choices.

Fairphone launched FP4 right before Android 12’s launch. It sported a Snapdragon 750G (launched with Android 10). Fairphone was already behind on updates around the time they launched! They initially only promised an A12 update, later correcting it to A12+A13 and blamed Qualcomm for the lack of A14+ while, again, it was mostly because of their own choices. That means N → N+2 again (effectively N → N+1 compared to other phones launched around that time, though, as A12 should’ve shipped right from the factory).

Google (Pixel 6)

Pixel 6 is the first generation of Google’s phones to sport “their own” CPU — Tensor. Between the time when Google decided to take control of the leaks by just showing off their phone and the actual release there was a lot of speculation regarding the designated lifespan of the devices. Some Qualcomm haters inferred that since they no longer used a chip from the Source of All Evil itself, Qualcomm, they must provide more support than they did before (which for N+3). And guess what… yeah. N+3 with 2 extra years of zero effort security updates. So is Qualcomm the one to blame for everything after all?

Summary — what’s going on?

Android updates are undergoing a revolution as we speak. Is it a positive thing, though? We will see. Google’s focus on N+3 for everyone might make it more common for OEMs to provide 2 or 3 platform updates, but on the other hand, in the cases where they want to provide more updates than it’s expected of them it gets way more complicated than before. It might also cause a feature stagnation, stripping users of useful improvements.Where is Google leading us? I don’t know, but I think we will find out very soon.