Not all Galaxies are made equal

A summary of Samsung Galaxy S6, S7 and S8 Android graphics reliability while we await the S9

Summary of results for 150 tests on Samsung Galaxy S6-S8 models. Results for the older ARM GPUs are still markedly better than the newest Qualcomm GPU.

With the Samsung Galaxy S9 hitting the stores as we go to press, let’s look again at the reliability of OpenGL ES drivers for current Galaxy S8 models using the GraphicsFuzz ShaderTest GLES tests, and also take a look back at how these compare with older S6 and S7 models.

For background, check out our previous story comparing Samsung Galaxy S8 models, as well as our “How it works” page.

Spoiler

• All Samsung Galaxy models suffer from graphics driver reliability issues: at least 10% of tests failed on all devices;
• In the worst case — for the Galaxy S7 with a Qualcomm GPU — 42% of tests failed;
• ARM’s GPU drivers appear to be more reliable than those from Qualcomm, with fewer tests failing on the ARM S6 and S7 drivers compared with the Qualcomm S8 driver;
• Reliability does not always get better: the ARM S7 drivers fail more tests than the older ARM S6 drivers.

We are excited to see how the S9 performs on our tests once we get a hold of it. But, clearly, GPU vendors seem to struggle on shader compiler quality: even though Samsung continues to support all Galaxy models for years, these bugs continue to plague their entire range of devices, which holds back the Android platform as a whole.

Galaxy Graphics: Qualcomm and ARM

The S7 and S8 phones come with either ARM or Qualcomm GPUs, depending on territory, while the S6 ships exclusively with ARM GPUs. Here are details of the GPUs in the models we tested:

• S6 (ARM): Mali-T760
• S7 (ARM): Mali-T880
• S7 (Qualcomm): Adreno 530
• S8 (ARM): Mali G71
• S8 (Qualcomm): Adreno 540

Reliability: Qualcomm vs. ARM

Our results table, shown at the top of this post and available here in an interactive form, suggests that the ARM GPU drivers embedded in Galaxy phones are more reliable than their Qualcomm counterparts.

In particular among the 150 tests for which we report results, 64 tests failed on the Qualcomm S7 model — a 42% failure rate. In contrast, the worst failure rate for ARM — also for the S7 — is 20% with 30 tests failing: still not good, but significantly less bad.

To get some sense of this visually, here is a snapshot of results for the ARM (first row) vs. Qualcomm (second row) S7 models on one of our shader families — shader family 001:

On the Galaxy S7, all rendered images are correct with ARM GPU drivers, though some link errors occur. With Qualcomm drivers, we see three distinct bad images, as well as a crash.

Switching attention to the S8 models, the ARM results improve on the drivers used in the S7, with no link errors and one crash:

However, the Qualcomm results are different but no better: the crash remains, and there are just as many “wrong image” results — just for different variants in the shader family.

Things don’t always get better!

The ARM S8 is the most reliable of the devices we tested, at least according to ShaderTest GLES.

But it’s worth noting that there is a marked increase in issues between the ARM drivers for the S6 and the S7. Our tests exposed 8 rendering issues and 5 crashes in the S6 drivers, increasing to 13 rendering issues and 9 crashes in the drivers for the S7.

This is illustrated by the results for shader family 007:

The ARM S6 drivers (top row) lead to accurate rendering of the reference image for every variant shader, while four bad images and one crash are observed with the S7 drivers (bottom row).

Keeping up shader compiler quality in the face of a changing architecture and surrounding device components is a challenge, and without intensive testing it is inevitable that new bugs will creep in and old bugs will creep back. It is the goal of ShaderTest to put a lid on this!

Bugs can be beautiful

It’s kind of fun to see a few cases where a rather dull reference image is transformed into a much more interesting variant image by a shader compiler bug! For example, shader family 005 does not have the most exciting associated image:

The expected image for shader family 005.

However, on our Qualcomm S7, we instead (but wrongly!) get this pretty cool alternative image for variant 052:

Beautiful but incorrect image rendered for variant 052 of shader family 005 with the Qualcomm S7.

It’s a shame this is a bug. As we observed once before, bugs can be beautiful.

Try for yourself!

Got a Galaxy phone or another Android device? Then see whether the GraphicsFuzz demo exposes any issues, and let us know what you find!