iOS build, test and deliver using Bitrise — Part 2[TEST]

In the previous story, we have set up a simple build workflow with unit test in Bitrise. In this story, we will continue enhance the workflow with automated testing. At the end of the story, we will have a production-ready main branch for our release.

About Testing

In NE Digital, we are practising Agile Testing, with the principle Tests performed by the whole team, we don’t have a tester in a team instead we have a SET (Software Engineer in Test) as a test specialist champion diffuse inside the team.

Chart created in Canva

Developer will work together with SET to design and set up the most suitable pipeline for the project.

Also, by following Testing in continuous principle, we are encouraged to set up unit test, snapshot testing, end to end testing and static code analysis before our feature release to production.

There are two automated tests will cover in this story:

1. Snapshot testing
2. End to end testing

I have added two new screens in the Moviz project, I will create both tests in the new screens to ensure they are working correctly before the release.

Snapshot Testing

The snapshot testing library I’m using is SnapshotTesting by Point-Free. It is a modern snapshot testing library written in Swift.

If you are interested to learn more about SnapshotTesting library, feel free to check their blog post.

Install SnapshotTesting with Carthage

We can simply add the following line into Cartfile.private:

github “pointfreeco/swift-snapshot-testing” ~> 1.8.0

You will only need to add SnapshotTesting.framework into test targets, you don’t need to add it into your non-test target. For details, please check on Carthage for adding framework to unit tests

SnapshotTesting Example

Snapshot testing with SnapshotTesting is very straightforward as the above example. We can have the first run on the snapshot testing to gather our snapshot now.

On the first run, you will see the above error on your snapshot testing test cases. Repeat the test with isRecording set to false and our test will work as expected.

Integrate Snapshot Testing on Bitrise

Let’s include the snapshot testing into our Bitrise workflow.

1. Install fastlane and AWS CLI

We will need to use fastlane and AWS CLI to automate our snapshot testing.

1.1 Install fastlane
brew install fastlane

1.2 Install fastlane unzip plugin
fastlane add_plugin unzip

1.3 Install AWS CLI
brew install awscli

2. Upload and Download Snapshot Reference Images

Why do we need to upload and download the reference images if we have committed them into our repository?

The more snapshot and varieties of device we have tested, the more we will consume for the repository space. The avoid a bulky repository, we will use S3 (any cloud storage you prefer) to store the reference image which will upload from our machines and download when we running the tests on Bitrise. The download step also can be used by a developer who has set up the repository first time.

Ok, let’s have a look the Fastfile I have:

1. Start of the upload snapshot function.
2. A fastlane action to zip our snapshots folder.
3. We will prompt the credentials for AWS S3 if they are not set as an environment variable in our system.
   We will set the credentials in Bitrise Secret Environment Variable, so that it will not be stuck on the prompting.
4. Upload the zip file to our S3 bucket.
5. Start of the download snapshot function.
6. Check for the AWS S3 credentials before download.
7. Download the zip file from S3 bucket.
8. Unzip the zip file with the plugin we have installed to the test folder.

3. Integrate with Bitrise

We have a complete workflow for the snapshot testing, it is time to integrate it into our build workflow.

We just need to add two steps before our Xcode Test.

1. Brew install — set Formula name to awscli to install AWS CLI in our CI.
2. fastlane — set fastlane lane to download_snapshot to download our snapshot reference images to our CI.

I have added these two steps after Bitrise Cache: Pull and before

That’s all we need. On the next trigger from PR, we will run the snapshot testing to ensure our UI is in the correct layout and prevent unexpected change.

Example build from our demo repo

Example PR: https://github.com/lihao6485/moviz/pull/5

Example Bitrise build: https://app.bitrise.io/build/47688b09dc47068b

End to End Testing

In NE Digital, we are using Cucumberish as our testing library. You can set up the library by the following Install with Carthage for Swift.

After the preparation works, we can start to create the feature file and begin our end to end testing journey.

Here is an example feature file for the demo app, the library will parse the feature file based on Gherkin style.

Next, we will need to implement the steps. Take note that same step will just need to implement once so you should implement them without context. We will put the following implementation in the CucumberishSwiftInit

That’s it, now you can press ⌘+u and make sure the app has gone through the step as you expected.

Integrate End to End Testing on Bitrise

There are 2 ways for us to integrate our end to end testing on Bitrise:

1. Physical device testing (based on Firebase Test Labs)
2. Bitrise’s simulator

For physical device testing, Bitrise’s has provided a native step for us to test on Firebase Test Labs directly and we can find the resulting logs, videos and screenshots on Bitrise at the end of test also. For more details about the physical device testing, you can have a look at Device testing for iOS on Bitrise.

For Bitrise’s simulator, it is pretty straightforward. We will use Xcode Test for iOS with the end to end testing scheme’s name. This will be how it looks like in our workflow:

We have added the end to end testing to run after our unit test, in the configuration, we will set the Scheme name into the end to end testing’s scheme. We are also able to configure the device type and OS version we would like to use.

After that, you will able to run end to end testing in your workflow!

Example PR: https://github.com/lihao6485/moviz/pull/6

Example Bitrise build: https://app.bitrise.io/build/47688b09dc47068b

Physical device testing or Simulator testing

Either physical device or simulator testing has its own pros and cons. Physical device testing based on Firebase Test Labs will automatically provide logs, video and screenshots. Simulator testing is faster than the physical device (because it doesn’t need to connect to third-party service) and able to use the latest iOS version based on Xcode.

We can select either of them based on our project and workflow.

Let’s wrap it up. Both snapshot testing and end to end testing not only help us to ensure the app is working as expected, and also increase our confidence when rolling out the new feature on top of some existing feature. They can make sure the previous flow still look the same and working as expected.

Some tips for end to end testing:

As we know, end to end testing is an expensive test in the workflow and the time consumption will keep growing when more and more tests are added. After discussing with our QE, we have decided to only run crucial flow and newly added test in our PR build workflow, once the PR has merged we will trigger another workflow to run a complete set of end to end testing. By doing this, we have a balance between testing and PR build time.

So, how we do this?

1. You will need to accept an environment variable in your test schemes:

2. Replace the Cucumberish.executeFeatures in your CucumberishSwiftInit() to the following:

3. In your Bitrise’s Xcode Test for iOS under Debug, you will need to add our new environment variable in the Additional options. So I have decide to use the name PR to distinguish the test I would like to run in the PR build workflow.

Add E2E_TYPE to run specific test

4. Lastly we just need to add the name we use for E2E_TYPE as our tag for feature:

Feature with tag

That’s it. We will run those feature with both PR and New tags in our feature when we running PR build and when we don’t provide any environment variable, we will run all the tests as default.

Example bitrise.yml: