Playing with Compose Screenshot testing and Github Action

5 min readJul 12, 2024

Introduction

So recently, Compose Preview Screenshot Testing has been available and is now in alpha-02 as I’m writing this.

This article will present quickly how we configure it and how we can start making it run with a simple Github Action.
If you are not familiar with the concept of screenshot testing, I would recommend this great article by Domen Lanišnik, who presents the concept and the already existing alternatives.

The long term advantage of using this one over an already existing tool is that we are more sure it’s going to be maintained over the years and it’s going to follow “immediately” all the evolution of compose.

The tool and the documentation are at their very early days so this article will be very short and won’t go in deep configuration as it’s still very early and lots of stuff might change in the upcoming months.

Setup

You are going to need at least to use Android Studio Koala.

Then, you can follow the steps from the original doc.

Here are the differences when adding the tool to a new empty project (commit here):

Adding the plugin to the version catalog

There is nothing crazy here; just follow the documentation setup, and for once, it works like a charm.

Writing our first test screenshot

I have this very basic composable:

@Composable
fun Greeting(name: String, modifier: Modifier = Modifier) {
    Text(
        text = "Hello $name!",
        modifier = modifier.padding(16.dp)
    )
}

Let’s generate a screenshot test of it:

Compose screenshot testing seems to be relying on placing a test class in a screenshotTest folder under your src folder:

class PreviewScreenShots {
    @Preview(showBackground = true)
    @Composable
    fun GreetingPreview() {
        ComposeScreenshotTestTheme {
            Greeting("Android!")
        }
    }
}

Now, we have to generate the image that is going to be a reference for our test with the following command:

./gradlew updateDebugScreenshotTest
// or
./gradlew {:module:}update{Variant}ScreenshotTest

It’s going to output a PNG of our composable:

Now, we can run our test and generate a report with the following:

./gradlew validateDebugScreenshotTest
// or
./gradlew {:module:}validate{Variant}ScreenshotTest

It’s going to re-run our composable, compare the new PNG output with the existing one, and generate a report like this:

Let’s make the report fail.

I’m updating my original composable with a background color:

@Composable
fun Greeting(name: String, modifier: Modifier = Modifier) {
    Text(
        text = "Hello $name!",
        modifier = modifier.padding(16.dp).background(Color.Blue),
    )
}

Note that we can run the test from the Gradle panel:

It’s going to trigger the Android Studio test failed panel, which is completely useless at the moment as the outputted error is not helping at all:

In comparison, the new test report is very complete and helps you figure out what went wrong:

So, with this output, we can clearly identify what has changed and what is broken.

Let’s link this to the Github Action.

Screenshot testing is the kind of stuff you might want to link to your Continous Integration, for example when a pr gets opened.

So, let’s see how to do this with Github Action. So we create a folder: .github/workflows where we add the following file:

name: Build Gradle project

on:
  push:

jobs:
  build-gradle-project:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout project sources
      uses: actions/checkout@v4
    - name: Set up JDK 18
      uses: actions/setup-java@v3
      with:
        java-version: '18'
        distribution: 'temurin'
    - name: Setup Gradle
      uses: gradle/actions/setup-gradle@v3
      with:
        build-scan-publish: true
        build-scan-terms-of-use-url: "https://gradle.com/terms-of-service"
        build-scan-terms-of-use-agree: "yes"

    - name: Run build
      run: ./gradlew validateDebugScreenshotTest
    - name: Upload build reports
      uses: actions/upload-artifact@v4
      if: always()
      with:
        name: my-artifact
        path: app/build/reports

So, for testing purposes I’ve sticked to

on:
  push:

But we could replace it with pull-request to make it as we want.

This Github Action script is pretty simple; the “interesting” part is:

- name: Run build
      run: ./gradlew validateDebugScreenshotTest
    - name: Upload build reports
      uses: actions/upload-artifact@v4
      if: always()
      with:
        name: my-artifact
        path: app/build/reports

It runs the validate Gradle command and will upload its outputted artifact to make it available in your Github Action page in the artifact section:

So with this the test is correctly running but it’s very likely it will never pass with the following error report:

The diff is very small but still annoying (I want my test to pass).
I’ve opened an issue in the Google Issue Tracker and it looks like it’s coming from differences in anti aliasing between local host and Github Action runner. The dev team is having the same kind of issue and it’s very likely they are going to fix it by allowing some small pixel variation in tests due to different platform implementations.

Conclusion

So for an alpha it is very pleasant to use locally, it’s simple, easy to setup and it leverages very well the Android Jetpack Compose preview feature. But the CI issue is kind of blocking its use, but let’s be honest in alpha release this is the kind of issue we are supposed to have, I think. So just stay tuned for next version.

I’ll try to write other articles to follow up on the upcoming changes.

Let me know what you would like to see or if you have any feedback.