Producing a CircleCI Test Summary with Fastlane

Published in

Big Nerd Ranch

8 min readJun 27, 2018

by: Jeremy Sherman

The heart of Continuous Integration is running tests. Whenever a test fails, you want to know why ASAP so you can correct it. Whenever a CI build fails, you want to see that failing test and how it failed.

CircleCI’s Test Summary feature puts this info front-and-center so you can respond directly to the test failure without anything getting in your way. The trick is to feed CircleCI your test info the way it expects.

Why set up Test Summary when you already have a build log?

The build log might be fine to start. You expand the failing step, scroll to the end of the page, and then scroll up till you hit the test failure.

This is not too bad. At first.

But with a big enough project, the build and test logs grow too long to view in-place on the web page. Then you find yourself downloading the log file first.

Sometimes the failing test isn’t really that near the end of the file. Then you’re fumbling around trying to find it.

Across a lot of developers on a long project, this time and friction adds up.

Don’t CircleCI’s docs cover this already?

If you’re building an iOS app, and you copy-paste the Example Configuration for Using Fastlane on CircleCI, you should luck into something that works.

But you’ll want to better understand what the Test Summary feature is looking for if:

Your Test Summary omits info you want in there, like linter output.
Test Summary isn’t working for you, and you want to fix it.
You’re not building an iOS app using Fastlane, and one of the other example configs doesn’t meet your needs.

What does CircleCI need for a Test Summary?

CircleCI’s Collecting Test Metadata doc calls out one big thing:

Report tests using JUnit’s XML format.

The store_test_results step reference calls out another:

Your test report should be in a subdirectory of another “all the tests” directory.

This subdirectory name is used to identify the test suite.

There’s one more requirement that I haven’t seen documented anywhere, though:

The JUnit XML test report file must literally have the xml extension.

The rest of the filename doesn’t seem matter for the test summary, but if you have the wrong path extension, you won’t see any test summary.

What does that look like on disk?

You’ll wind up with a directory layout like:

/Users/distiller/project/
└── fastlane
    └── test_output
        └── xctest
            └── junit.xml3 directories, 1 file

This ticks all the boxes:

XML file: junit.xml
“Test Suite” directory: xctest
“All Test Suites” directory: test_output

(Fastlane only produces a single test report, so the nesting of report-in-folder-in-folder admittedly looks a little silly.)

How do you get Fastlane Scan to write JUnit XML to that path?

Scan provides a lot of config knobs. You can view a table of the full list and their default values by running fastlane action scan.

We need to arrange three things:

The report format: JUnit
The report filename: junit.xml
The directory that report file should be written to: fastlane/test_output/xctest

Conveniently enough, Scan has three config settings, one for each of those needs.

Scan also happens to have three different ways to set those three options:

In your Fastfile: Using keyword arguments to scan()
In your shell: Using option flags
Anywhere (but probably in your shell): Using environment variables

Keyword Arguments to scan()

In your Fastfile, you can set them using keyword arguments to the scan method call:

scan(
  # … other arguments …
  output_types: 'junit',
  output_files: 'junit.xml',
  output_directory: './fastlane/test_output/xctest')

Option Flags to fastlane scan

If you’re invoking fastlane directly, you can set them using CLI options:

fastlane scan \
  --output_directory="./fastlane/test_output/xctest" \
  --output_types="junit" \
  --output_files="junit.xml"

Environment Variables

Because Ruby is a true descendant of Perl, TMTOWTDI, so you could also configure Scan using environment variables:

env \
  SCAN_OUTPUT_DIRECTORY=./fastlane/test_output/xctest \
  SCAN_OUTPUT_TYPES=junit \
  SCAN_OUTPUT_FILES=junit.xml \
  fastlane scan

(You could also set those environment variables in the environment stanza in your CircleCI config. Six one way, half-dozen the other.)

How do you get CircleCI to process the JUnit XML?

Now you have Fastlane Scan writing its test report using the JUnit format into a *.xml file under a suggestively-named subdirectory.

To get CircleCI to actually process this carefully arranged data, you’ll need tell the store_test_results step to snarf everything at and under fastlane/test_output.

That’s right: not just the xctest subdirectory that holds the test report XML, but the directory.

Add this step to the pipeline that runs scan:

- store_test_results:
    path: "./fastlane/test_output"

What about the rest of Scan’s output?

At some point, you’ll probably want to be able to look at the test report yourself, as well as the overall build logs.

You can send both of those on up to CircleCI as build artifacts using a couple store_artifacts steps:

- store_artifacts:
    path: "./fastlane/test_output"
    destination: scan-test-output
- store_artifacts:
    path: ~/Library/Logs/scan
    destination: scan-logs

What about more than just Scan?

You’re not limited to just one artifact or just one test output. In fact, handling multiple kinds of test output is precisely why there’s the folder-in-folder nesting.

Say you wanted to have CircleCI call out SwiftLint nits. You could drop this snippet into your jobs list:

lint:
  docker:
    - image: dantoml/swiftlint:lateststeps:
    - checkout- run:
        name: Run SwiftLint
        command: |
          mkdir -p ./test_output/swiftlint
          swiftlint lint --strict --reporter junit | tee ./test_output/swiftlint/junit.xml- store_test_results:
        path: "./test_output"
    - store_artifacts:
        path: "./test_output"

The key links in the chain here are:

Create an “ALL the tests” directory: ./test_output/
Create a “test suite” directory: ./test_output/swiftlint/
Write JUnit output into a .xml file: ./test_output/swiftlint/junit.xml
Aim store_test_results at that "ALL the tests" directory: path: "./test_output/"

Any output you can massage into meeting those requirements, you can cadge CircleCI into calling out in your Test Summary.

Conclusion

There you have it:

Quickly debug CI build failures
By putting relevant test details in prime real estate
By feeding a folder two steps up from a carefully located, precisely named test report file
To CircleCI’s store_test_results build step.The heart of Continuous Integration is running tests. Whenever a test fails, you want to know why ASAP so you can correct it. Whenever a CI build fails, you want to see that failing test and how it failed.
CircleCI’s Test Summary feature puts this info front-and-center so you can respond directly to the test failure without anything getting in your way. The trick is to feed CircleCI your test info the way it expects.

Why set up Test Summary when you already have a build log?

The build log might be fine to start. You expand the failing step, scroll to the end of the page, and then scroll up till you hit the test failure.

This is not too bad. At first.

But with a big enough project, the build and test logs grow too long to view in-place on the web page. Then you find yourself downloading the log file first.

Sometimes the failing test isn’t really that near the end of the file. Then you’re fumbling around trying to find it.

Across a lot of developers on a long project, this time and friction adds up.

Don’t CircleCI’s docs cover this already?

If you’re building an iOS app, and you copy-paste the Example Configuration for Using Fastlane on CircleCI, you should luck into something that works.

But you’ll want to better understand what the Test Summary feature is looking for if:

Your Test Summary omits info you want in there, like linter output.
Test Summary isn’t working for you, and you want to fix it.
You’re not building an iOS app using Fastlane, and one of the other example configs doesn’t meet your needs.

What does CircleCI need for a Test Summary?

CircleCI’s Collecting Test Metadata doc calls out one big thing:

Report tests using JUnit’s XML format.

The store_test_results step reference calls out another:

Your test report should be in a subdirectory of another “all the tests” directory.

This subdirectory name is used to identify the test suite.

There’s one more requirement that I haven’t seen documented anywhere, though:

The JUnit XML test report file must literally have the xml extension.

The rest of the filename doesn’t seem matter for the test summary, but if you have the wrong path extension, you won’t see any test summary.

What does that look like on disk?

You’ll wind up with a directory layout like:

/Users/distiller/project/
└── fastlane
    └── test_output
        └── xctest
            └── junit.xml3 directories, 1 file

This ticks all the boxes:

XML file: junit.xml
“Test Suite” directory: xctest
“All Test Suites” directory: test_output

(Fastlane only produces a single test report, so the nesting of report-in-folder-in-folder admittedly looks a little silly.)

How do you get Fastlane Scan to write JUnit XML to that path?

Scan provides a lot of config knobs. You can view a table of the full list and their default values by running fastlane action scan.

We need to arrange three things:

The report format: JUnit
The report filename: junit.xml
The directory that report file should be written to: fastlane/test_output/xctest

Conveniently enough, Scan has three config settings, one for each of those needs.

Scan also happens to have three different ways to set those three options:

In your Fastfile: Using keyword arguments to scan()
In your shell: Using option flags
Anywhere (but probably in your shell): Using environment variables

Keyword Arguments to scan()

In your Fastfile, you can set them using keyword arguments to the scan method call:

scan(
  # … other arguments …
  output_types: 'junit',
  output_files: 'junit.xml',
  output_directory: './fastlane/test_output/xctest')

Option Flags to fastlane scan

If you’re invoking fastlane directly, you can set them using CLI options:

fastlane scan \
  --output_directory="./fastlane/test_output/xctest" \
  --output_types="junit" \
  --output_files="junit.xml"

Environment Variables

Because Ruby is a true descendant of Perl, TMTOWTDI, so you could also configure Scan using environment variables:

env \
  SCAN_OUTPUT_DIRECTORY=./fastlane/test_output/xctest \
  SCAN_OUTPUT_TYPES=junit \
  SCAN_OUTPUT_FILES=junit.xml \
  fastlane scan

(You could also set those environment variables in the environment stanza in your CircleCI config. Six one way, half-dozen the other.)

How do you get CircleCI to process the JUnit XML?

Now you have Fastlane Scan writing its test report using the JUnit format into a *.xml file under a suggestively-named subdirectory.

To get CircleCI to actually process this carefully arranged data, you’ll need tell the store_test_results step to snarf everything at and under fastlane/test_output.

That’s right: not just the xctest subdirectory that holds the test report XML, but the directory.

Add this step to the pipeline that runs scan:

- store_test_results:
    path: "./fastlane/test_output"

What about the rest of Scan’s output?

At some point, you’ll probably want to be able to look at the test report yourself, as well as the overall build logs.

You can send both of those on up to CircleCI as build artifacts using a couple store_artifacts steps:

- store_artifacts:
    path: "./fastlane/test_output"
    destination: scan-test-output
- store_artifacts:
    path: ~/Library/Logs/scan
    destination: scan-logs

What about more than just Scan?

You’re not limited to just one artifact or just one test output. In fact, handling multiple kinds of test output is precisely why there’s the folder-in-folder nesting.

Say you wanted to have CircleCI call out SwiftLint nits. You could drop this snippet into your jobs list:

lint:
  docker:
    - image: dantoml/swiftlint:lateststeps:
    - checkout- run:
        name: Run SwiftLint
        command: |
          mkdir -p ./test_output/swiftlint
          swiftlint lint --strict --reporter junit | tee ./test_output/swiftlint/junit.xml- store_test_results:
        path: "./test_output"
    - store_artifacts:
        path: "./test_output"

The key links in the chain here are:

Create an “ALL the tests” directory: ./test_output/
Create a “test suite” directory: ./test_output/swiftlint/
Write JUnit output into a .xml file: ./test_output/swiftlint/junit.xml
Aim store_test_results at that "ALL the tests" directory: path: "./test_output/"

Any output you can massage into meeting those requirements, you can cadge CircleCI into calling out in your Test Summary.

Conclusion

There you have it:

Quickly debug CI build failures
By putting relevant test details in prime real estate
By feeding a folder two steps up from a carefully located, precisely named test report file
To CircleCI’s store_test_results build step.

If you like what you just read, please hit the green applause button below so that others might stumble upon this. For more blogs like this visit Big Nerd Ranch

Producing a CircleCI Test Summary with Fastlane

Why set up Test Summary when you already have a build log?

Don’t CircleCI’s docs cover this already?

What does CircleCI need for a Test Summary?

What does that look like on disk?

How do you get Fastlane Scan to write JUnit XML to that path?

Keyword Arguments to scan()

Option Flags to fastlane scan

Environment Variables

How do you get CircleCI to process the JUnit XML?

What about the rest of Scan’s output?

What about more than just Scan?

Conclusion

Why set up Test Summary when you already have a build log?

Don’t CircleCI’s docs cover this already?

What does CircleCI need for a Test Summary?

What does that look like on disk?

How do you get Fastlane Scan to write JUnit XML to that path?

Keyword Arguments to scan()

Option Flags to fastlane scan

Environment Variables

How do you get CircleCI to process the JUnit XML?

What about the rest of Scan’s output?

What about more than just Scan?

Conclusion

Written by Big Nerd Ranch