Golden tests in Flutter: A comprehensive guide

Ensure UI consistency in Flutter with golden tests

TL;DR: Golden tests in Flutter ensure visual consistency by comparing UI outputs against reference images, known as “golden” images. These tests help catch unintentional UI changes, especially in complex projects.

• How It Works: Golden tests render a widget, capture a bitmap image, and compare it pixel-by-pixel to the golden image. Any difference triggers a test failure.

Using golden tests: Add the golden_toolkit package, write tests using pumpWidgetBuilder(), and run flutter test — update-goldens to create/update golden images.

What are Golden tests?

In mobile app development, visual consistency is critical to delivering a polished user experience. In Flutter, golden tests are a unique type of testing that allows developers to verify that the UI has not unintentionally changed. Essentially, golden tests compare the current state of the UI against a reference image, often referred to as a “golden” image, to ensure that the visual output of a widget or screen remains as expected.

Why should I use them?

When changes occur in the UI codebase, it’s easy to miss subtle visual discrepancies that may affect the overall design. Golden tests help developers avoid these issues by running comparisons between the current and expected state of the UI. If the widget or screen deviates from the baseline image, the test will fail, indicating that the change might be unintended. This is especially useful in large projects with complex UIs, where manually inspecting each visual change is impractical.

How do the golden tests work?

Golden tests work by capturing an image of a widget in a specific state and comparing it pixel by pixel to a previously approved “golden” image. If all pixels match, the test passes; if there are differences, the test fails, prompting developers to review the changes.

Getting started

First and foremost, we’ll need to create a new flutter app, so:

flutter create goldens
cd goldens
flutter run

This will leave us with the famous Flutter Demo Home Page:

Flutter demo home page

For the sake of simplicity, let’s change the homepage to a StatelessWidget (we won’t need states) and remove the unused elements from our main page and make it cleaner:

Flutter demo home page, without counter

Now, we need to create a widget that will be our subject of the golden tests. For this, we’ll create a new box_with_icon.dart file and put the following code in it:

import 'package:flutter/material.dart';
class BoxWithIcon extends StatelessWidget {
  const BoxWithIcon({super.key});
  
  @override
  Widget build(BuildContext context) {
    return Container(
      height: 200,
      width: 200,
      color: Colors.cyanAccent,
      child: const Icon(Icons.abc),
    );
  }
}

Now, we can use this widget in the main.dart file:

class MyHomePage extends StatelessWidget {
  const MyHomePage({
    super.key, 
    required this.title,
  });
  
  final String title;
  
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        backgroundColor: Theme.of(context).colorScheme.inversePrimary,
        title: Text(title),
      ),
      body: const Center(
        child: BoxWithIcon(),
      ),
    );
  }
}

This will leave us with this result:

Home page with new widget

With our current app, we are ready to start working on our golden tests!

Using golden tests

First, we’ll need to create a test file for our BoxWithIcon widget, so let’s add a new file in /test/widgets, called box_with_icon_test.dart. Now, our folder structure should look something just like this:

Project’s folder structure

Since our widget is very very simple and minimal, there’s not much to test visually, but we can still assert that the color and icon are right. Without further ado, let’s write our first golden test:

Initially, we’re gonna need to render our widget for the golden tests be able to see it:

testWidgets('BoxWithIcon default appearance', (tester) async {
  await tester.pumpWidget(
    const MaterialApp(
      home: Scaffold(
        body: Center(
          child: BoxWithIcon(),
        ),
      ),
    ),
  );
});

Note that the pumpWidget function is responsible for rendering the widget in the test scope.

Now that we have our widget rendered, let’s add our assertion:

testWidgets('BoxWithIcon default appearance', (tester) async {
  await tester.pumpWidget(
    const MaterialApp(
      home: Scaffold(
        body: Center(
        child: BoxWithIcon(),
        ),
      ),
    ),
  );
  await expectLater(
    find.byType(BoxWithIcon),
    matchesGoldenFile('goldens/box_with_icon_default.png'),
  );
});

What’s left for us now? To run the tests, right?! WRONG, if we try to run them now, we’ll be prompted an error like the following one:

Flutter test framework error

What’s happened? Simply enough, we don’t have our golden images (the images that the flutter test framework is going to compare our tests to) because it is our first time running this test suite. To fix this, we need to run the flutter test command with a new parameter, the — update-goldens, this will make Flutter generate these images for us to have something to compare against.

The flutter test — update-goldens command will leave us with the following result:

Tests folder structure

Golden image generated by the tests

Congrats, you have generated your first golden image and coded your first golden test in Flutter! Cool, isn’t it? But we need to understand better what is happening with our image (the little square on the center of the cyan box) and how to ensure visual quality with these results.

The first strange behavior is the middle square inside our box. This is simply a placeholder for our icon because the flutter test framework doesn’t load any images/icons unless they’re pre-cached before running the tests. Since we only need to assure the icon is inside the box, not what icon is inside the box, we can proceed.

To ensure our visual consistency, running the — update-goldens line isn’t enough, because we’re not effectively comparing any image, we’re only generating them. To make this assertion, we need to run flutter test again, with the generated images on its respective folders:

flutter test result

Now, if we change the color of our box, without updating the golden file, an error will be prompted:

import 'package:flutter/material.dart';

class BoxWithIcon extends StatelessWidget {
  const BoxWithIcon({super.key});
  
  @override
  Widget build(BuildContext context) {
    return Container(
      height: 200,
      width: 200,
      color: Colors.red,
      child: const Icon(Icons.abc),
    );
  }
}

flutter test result, with errors in golden tests

The best part about golden test errors is that we don’t need to search or debug hard any part of the code, because the failed tests generate a failures folder for us, containing some files:

Tests folder structure

The isolatedDiff file, like it’s name give us a clue, makes sure only the difference is in the image:

box_with_icon_default_isolatedDiff.png

The maskedDiff is a mask of the differences found during the test:

box_with_icon_default_maskedDiff.png

The masterImage is basically a copy of our golden image:

box_with_icon_default_masterImage.png

And finally, the testImage is the widget rendered during the tests:

box_with_icon_default_testImage.png

These four images are here to help us find the error in our visuals. In this example, it is very simple to see that the error is the color of the box, but when some harder errors are found, the maskedDiff and isolatedDiff files help us a lot.

Conclusion

In conclusion, golden tests are a powerful tool in Flutter for maintaining visual consistency across an app’s UI. By comparing the actual rendered UI against predefined “golden” images, these tests allow developers to catch unintentional design changes early, improving both design accuracy and user experience. With Flutter’s golden_toolkit package, creating and updating golden images is straightforward, and the test result visuals make it easy to pinpoint UI differences when tests fail.

Golden tests aren’t unique to Flutter; snapshot testing is also widely used in other development environments. In JavaScript, libraries like Jest for React and Jasmine for Angular offer similar capabilities. These tools allow developers to verify UI integrity across various frontend frameworks by comparing DOM snapshots or component images. Leveraging snapshot testing across platforms can greatly enhance app quality by ensuring that any visual changes are intentional, ultimately leading to a more polished and consistent user experience.

References

golden_toolkit — Dart API docs

matchesGoldenFile function — flutter_test library — Dart API

Golden File Testing | VGV Engineering