Google and Qualcomm: Pixel-Perfect?

Looking at Android graphics driver reliability for Google’s Pixel phones

The Pixel 2’s newer Qualcomm Adreno 540 driver introduces new issues, and trades issues of one kind for another, when compared with the Galaxy S8’s older driver.

See also: posts evaluating:

• ARM S8 vs. Qualcomm S8
• NVIDIA shield

The Google Pixel 2 and Samsung Galaxy S8 are widely considered to be two of the best Android handsets currently on the market. The Pixel 2 and S8 US model both have Qualcomm Adreno 540 GPUs, so we might expect their graphics to be equally reliable.

Try out our web app — the GraphicsFuzz benchmark — on your Android device and let us know what issues it finds!

Actually, not quite: querying for GPU driver details shows that the Pixel 2 ships with a fairly recent Qualcomm driver (October 2017), while the S8 ships with an older driver (March 2017).

We saw in a previous post that the Qualcomm S8 driver is unreliable compared with the driver for its ARM S8 counterpart. Does the more recent driver in the Pixel 2 fix these issues?

It seems not: if anything, the update makes matters worse. Here is a summary of the issues found by the subset of ShaderTest GLES tests that we have been using to evaluate devices:

Click the table image to go to an interactive version where you rank the devices according to issue type.

The Pixel 2, and its Pixel 1 predecessor (with a Qualcomm Adreno 530 GPU) rank bottom if we sort by the total number of issues flagged by the tests. Our interactive table lets you sort by other criteria.

As we discuss in “the details” below, the driver update introduces some new compile and link errors, trades some rendering issues for crashes and vice-versa, and introduces more new failures than it fixes.

Of course, our tests are just one lens through which to view reliability. It is likely that the newer Pixel 2 driver also fixes some important issues that our tests do not flag up. But our results underline the fact that GPU driver regressions are common! Our hope is that ShaderTest can help by quickly flagging regressions before a driver update ships.

The Details

Check out our full results tables.

Pixel 1 vs. Pixel 2

The drivers for the two Pixel phones behave very similarly on our tests. There are a few differences, though:

• Two crashes on the Pixel 2 that render fine on the Pixel 1 — variant 073 of shader family 006 and variant 122 of shader family 009
• A Pixel 1 crash that instead gives a compilation error on the Pixel 2 — variant 065 of shader family 008. The compiler error string is empty, however.
• A compilation timeout on the Pixel 1 for variant 046 of shader family 005; this renders fine on the Pixel 2.
• A bad image on the Pixel 1 that shows up as a crash on the Pixel 2 — variant 051 of shader family 008; however, we have observed this variant to fluctuate nondeterministically between crashing and misrendering on Qualcomm drivers:

For variant 051 of shader family 008, the Pixel 1 renders the unexpected image on the right; the expected image from the reference shader is shown on the left. However, we have also found that Qualcomm drivers fluctuate between crashing and rendering a bad image for this test.

• A fix: a shader that gives bad images on both the Pixel 1 and the Galaxy S8, but gives the right image on the Pixel 2; variant 050 of shader family 009:

The left image is the expected image for variant 050 of shader family 009, and the newer Qualcomm drivers on the Pixel 2 yield this image. The unexpected image on the right is rendered on the Pixel 1, as well as on the Galaxy S8.

Pixel 2 vs. S8

Comparing the Pixel 2 and S8 results in more detail, there are a number of apparent fixes in the more recent Pixel 2 Qualcomm driver, including variant 050 of shader family 009 (see above), and variant 228 of shader family 007, which gives a link error on the S8 but renders fine on the Pixel 2.

However, the newer Qualcomm driver exhibits several additional compilation and linking failures on the Pixel 2 compared with the older S8 driver. In some cases, the relevant variant shaders render just fine with the S8 driver —variant 065 of shader family 008, once again, and also variant 127 of shader family 007.

An instance of one bug type being traded for another is flagged up by variant 092 of shader family 009: the variant crashes with the S8 driver, and renders without crashing on the Pixel 2 driver. However, the resulting image is unexpected:

The reference image (left) is rendered by the S8 drivers for variant 092 of shader family 009, but the unexpected image (right) is rendered with the more recent Qualcomm drivers in the Pixel 2.

What is less bad, an explicit crash, or a silent miscompilation? The former is more directly disruptive, but the latter could pose a greater debugging challenge for developers, who may believe their code is wrong when really an issue is due to a miscompliation.

Check out variant 199 of shader family 010: in our posts so far we have yet to see a device that successfully renders the expected image. But, once again, fear not: we shall in due course!

Next time…

…we’ll look at a Nexus device with a PowerVR GPU from Imagination Technologies.

See also

Our original series of Medium stories describing the underlying technology.

Our OOPSLA 2017 research paper with full technical details.