Running tests on multiple iOS versions on GitHub Actions

This Note originally published on my Personal Blog here. Read original note so that you won’t miss any content.

As with everything, there are at least two ways to run tests in parallel on multiple iOS versions:

• Parallelization at the CI level
• Parallelization at the test framework level

In our case, the CI will be Github Actions and the test framework will be Fastlane — the wrapper on top of XCTest. Just to be clear, we are going to use Github Actions as well as Fastlane in both scenarios, the only difference is where exactly we will parallelize the tests.

Parallelization at the CI level

Let’s create a Github Actions workflow, that will:

• be executed by cron
• use a matrix with different iOS versions, devices and macOS versions if needed
• install the required iOS Simulator
• pass a device name and an iOS version to Fastlane

name: Cron
on:
    schedule:
        # Running the pipeline "At 03:00 every night"
        - cron: '0 3 * * *'
jobs:
    test:
        name: Run tests on multiple iOS versions
        strategy:
        matrix:
            ios: [12.4, 13.7, 14.5, latest]
            device: ["iPhone 8"]
            macos: ["macos-12"]
            fail-fast: false
        runs-on: ${{ matrix.macos }}
        steps:
        - uses: actions/checkout@v2
        # Grabbing the full name of a macOS image
        - run: echo "IMAGE=${ImageOS}-${ImageVersion}" >> $GITHUB_ENV
        # Caching ruby gems
        - uses: actions/cache@v2
          with:
              path: vendor/bundle
              key: ${{ env.IMAGE }}-gem-${{ hashFiles('**/Gemfile.lock') }}
              restore-keys: ${{ env.IMAGE }}-gem-
        # Installing ruby gems
        - run: bundle install
        # Installing iOS Simulator if required
        - name: Install iOS Simulator ${{ matrix.ios }}
          if: ${{ matrix.ios != 'latest' }}
          run: xcversion simulators --install='iOS ${{ matrix.ios }}' || true
        # Here we go
        - name: Run Tests
          run: bundle exec fastlane cron_checks device:"${{ matrix.device }}" ios:"${{ matrix.ios }}"

Then let’s create a Fastlane lane, that will:

• take in a device name and an iOS version
• run tests on the provided device

lane :cron_checks do |options|
    device_name = options[:ios] == 'latest' ? options[:device] : "#{options[:device]} (#{ios})"
    scan(
        project: 'SampleApp.xcodeproj',
        scheme: 'SampleApp',
        testplan: 'SampleAppTestPlan',
        configuration: 'Debug',
        devices: [device_name]
    )
end

This way, the tests are run on multiple iOS versions in parallel in different matrix jobs on Github Actions.

Parallelization at the test framework level

Let’s create a Github Actions workflow, that will:

• be executed by cron
• pass a device name to Fastlane

name: Cron
on:
    schedule:
        # Running the pipeline "At 03:00 every night"
        - cron: '0 3 * * *'
jobs:
    test:
        name: Run tests on multiple iOS versions
        runs-on: macos-12
        steps:
        - uses: actions/checkout@v2
        # Grabbing the full name of a macOS image
        - run: echo "IMAGE=${ImageOS}-${ImageVersion}" >> $GITHUB_ENV
        # Caching ruby gems
        - uses: actions/cache@v2
          with:
              path: vendor/bundle
              key: ${{ env.IMAGE }}-gem-${{ hashFiles('**/Gemfile.lock') }}
              restore-keys: ${{ env.IMAGE }}-gem-
        # Installing ruby gems
        - run: bundle install
        # Here we go
        - name: Run Tests
          run: bundle exec fastlane cron_checks device:"iPhone 8"

Then let’s create a Fastlane lane, that will:

• take in a device name
• install the required iOS Simulators
• run tests on multiple iOS versions

lane :cron_checks do |options|
    ios = ['12.4', '13.7', '14.5', 'latest']
    # Installing iOS Simulators if required
    ios.each { |version| sh("xcversion simulators --install='iOS #{version}' || true") if version != ios.last }
    # Setting up device names
    devices = ios.reverse.drop(1).reverse.map { |v| "#{options[:device]} (#{v})" }.append(options[:device])
    scan(
        project: 'SampleApp.xcodeproj',
        scheme: 'SampleApp',
        testplan: 'SampleAppTestPlan',
        configuration: 'Debug',
        devices: devices
    )
end

Thus, the tests are run on multiple iOS versions in parallel in the same job on Github Actions.