Decreasing CI Duration of iOS via Fastlane in Udemy
The Udemy iOS app gets thousands of users every day. It is so important that we have a testing mechanism and make sure everything works as expected before merging changes to the repository or releasing products to the end-users.
We have Unit tests, E2E tests, and Snapshot tests in CI for the Udemy iOS app to make sure that it works as expected and does not break any existing features while publishing new changes.
In CI
We use Jenkins for continuous integration and Fastlane for automation — i.e. building, testing, and reporting results back to GitHub. Fastlane is a great tool that has lots of lanes implemented and ready for use. We are using scan lane to take build and run tests in CI. We can give various parameters to scan to configure it according to our needs. For example, we can specify which project to build, the path of the built product, which tests to execute, or on which device these apps or tests should run.
What We Had Before
We used to have three schemes and targets for Unit tests, E2E tests, and Snapshot tests in addition to the Udemy iOS app target and scheme. Xcode target specifies files, build settings, and build phases to create products. Xcode scheme defines a collection of targets, build settings, and a collection of tests that will be executed. The test schemes (Unit, E2E, and Snapshot) have an explicit dependency on the Udemy iOS app which means that before the test target is compiled, the Udemy iOS app needs to be compiled to get the product to be tested. In the picture below, we can summarize the flow for an old configuration of one of the test schemes.
Unit tests, E2E tests, and Snapshot test schemes have the same dependency on the Udemy iOS app. Each scan command compiled the Udemy app target along with its test target since the scheme included both targets and the application was compiled three times from scratch for different test schemes in CI. We can see the summarized way of the usage of the scan lane as below.
scan(workspace: "udemy.xcworkspace",device: "iPhone 8",scheme: "unit-tests")
scan(workspace: "udemy.xcworkspace",device: "iPhone 8",scheme: "snapshot-tests")
scan(workspace: "udemy.xcworkspace",device: "iPhone 8",scheme: "e2e-tests")
What Was the Problem?
The job execution was sequential; each test job had to wait for the other jobs to finish. At first, when the number of developers on the team and test counts were low, this wasn’t considered as a blockage in CI. As the team grew, we added more tests which resulted in increased execution time for our jobs and as a result, we ended up blocking our developers for a longer time period.
New Configuration
We reconfigured the Udemy iOS app scheme and included all of the test targets including Unit tests, E2E tests, and Snapshot test targets which will prevent redundant rebuilds of the app. The new configuration flow becomes something like this:
With this new configuration, we build the iOS app only once and the tests are run on this ipa. Scan lane provides awesome parameter options for us to achieve this! Let’s take a look at the parameters that can be used with this new configuration on CI.
- build_for_testing: Builds for testing only, does not run tests. By setting this option to true we say that only compile all of the schemes including app and test targets but do not run tests under this scheme.
- derived_data_path: Derived data location is where all the intermediate and final products of the build process are stored. By default, the derived data path is specified by Xcode uniquely for each project. However, with this option, we can set the derived data path as we want, and all of the compiled files will be stored in this path that we specified.
- test_without_building: Test without building, requires a derived data path. By setting this option to true we can use the derived data path that we specify with the build_for_testing option.
- only_testing: Array of strings matching test bundle/ test-suite/ test cases to run. With this option, we can run tests under specific test targets, for example only unit tests under Unit Test target, at a time.
Let’s take a look at the new configuration of the scan. Firstly we need to build the Udemy iOS app without running any of the tests. Since the Udemy scheme includes Unit test target, E2E test target, and Snapshot test target all of these targets will be compiled in addition to the Udemy iOS app.
scan(workspace: "udemy.xcworkspace",device: "iPhone 8",scheme: "udemy",build_for_testing: true,derived_data_path: "/desired-path-for-building/udemy" )
Now we can start to use a compiled app and test targets and execute each test one by one.
To run Unit tests run:
scan(derived_data_path: "/desired-path-for-building/udemy",device: "iPhone 8",test_without_building: true,only_testing: "unit-tests")
To run E2E tests run:
scan(derived_data_path: "/desired-path-for-building/udemy",device: "iPhone 8",test_without_building: true,only_testing: "e2e-tests")
To run Snapshot tests run:
scan(derived_data_path: "/desired-path-for-building/udemy",device: "iPhone 8",test_without_building: true,only_testing: "snapshot-tests")
What We Gained?
That’s all! With this new configuration:
- We no longer make redundant builds of the app for the various test schemes.
- It saves us two build times of the app.
- With this approach, we can save approximately 20 minutes for each test cycle.
That’s an awesome improvement! With Fastlane and scan, you can improve your mobile CI execution, too!
Thanks for reading, happy testing!
