How to run Flutter Golden (Snapshot) Tests with Codemagic CI/CD

Katarina Sheremet
May 7, 2020 · 4 min read
Photo by Louis Hansel @shotsoflouis on Unsplash

Testing is a very important aspect of development, it helps to make sure that your app works and looks properly for the users. It is great that there are CI/CD tools that automate some of the work for you in checking the app quality, notifying the team if something goes wrong, and releasing the app to the end-user.
In this article, I’m going to explain how to run Golden tests with Codemagic CI/CD.

Step 1: Create the Golden (Snapshot) Test

In one of my previous articles, I’ve covered how to create Flutter Snapshot tests, therefore I won’t focus on this topic here. Instead, I’ll just copy over some code snippets so that you don’t have to go back and forth between articles to follow.

Create in the folder of the counter app with the next code:

void main() {  testWidgets(‘Golden test’, (WidgetTester tester) async {    await tester.pumpWidget(MyApp());    await expectLater(find.byType(MyApp),                      matchesGoldenFile(‘goldens/main.png’));   });
}

Generate snapshots:

flutter test — update-goldens

I’ve created a counter app with a Golden test and pushed it to GitHub. I was able to see my new app in Codemagic Applications.

Step 2: Run Golden (Snapshot) Test with Codemagic

When you generate Golden (Snapshot) Tests, the different OS platforms generate different files. There is an issue filed on GitHub, but it’s closed because this is working as intended, even if not very convenient. It means if you generate your Golden files on Linux and, afterward, run your tests on MacOS, your Golden Tests will most likely fail. However,

Thus, people who use MacOS have an advantage. All passing Golden Tests will succeed on Codemagic CI/CD without any extra work.

If you are a MacOS user you need to press “Start new build” and you are all set, just do not forget to locally generate (see above) and commit golden files before.

Start new build with Codemagic
Start new build with Codemagic
Start new build with Codemagic CI/CD

If everyone in your team uses MacOS, you are done 🥳. No need to continue reading.

Step 3: Run Golden Tests if you are not a MacOS user

Codemagic supports free remote connection to MacOS infrastructure and also has variable to upload any files from builder machines when a build has finished.

You don’t need to generate Golden files on your machine, instead, you can do it using Codemagic. I’ve pushed my code without generated snapshots to GitHub.

After that, you need to create a new workflow just for the generation of snapshots, since you don’t want to lose the build minutes and run the release workflow with tests that will fail anyway.

A new workflow with the name “Generate Goldens”

In the Test section, you need to enable Flutter test and set Flutter test arguments to:

test --update-goldens
Enable Flutter test and set test arguments

The command above will generate Snapshots for you. Then you should copy generated Snapshots to with the following command in section:

cp -r test/goldens $FCI_EXPORT_DIR
Add Post-test script

Don't forget to disable Build and Publish section as well!

Run tests only in the Build section

Save settings and start a new build. After finishing the build you should see the generated golden files in the Artifacts section on the left.

Artifact section

What you need now is to just download this zip and extract generated golden files into your project.

If you have Golden tests in your project you should add the parameter if the platform is not MacOS. Then if you run locally, the tests won’t fail because of golden differences with Mac.

testWidgets('Golden test', (WidgetTester tester) async {    await tester.pumpWidget(MyApp());    await expectLater(      find.byType(MyApp), matchesGoldenFile('goldens/main.png'),    );}, );

Flutter Community

Articles and Stories from the Flutter Community