Mobile Automation Testing
Test Automation as Part of CI/CD
Continuous integration (CI) and continuous delivery (CD) are an important, if not standard, aspect of most mobile application development teams. We have adopted these methodologies at Clover as well. Having a CI/CD system allows our Clover Go iOS team to integrate new features continuously, track the progress of fixes on issues individually, and have a (theoretically) working master build at any given time that QA can use to verify tickets.
However, realistically, with having a new build every few days, it is nearly impossible to verify the state of all features on the app for each build. There were times when a bug fix in what was thought to be an isolated part of the app affected the function of another. Other times, the build that worked locally did not actually build and run properly when our QA team tried to test it. When that happens, it is a waste of time for our QA team to wait for a new build to be released, and then test the build again. With these uncertainties on the quality of our builds and the time it takes to manually test features on our app, our team needed to add mobile UI automation to our CI/CD pipeline. Mobile UI automation allows for automated testing of main features so that we can be confident the basic features that need to be working are in fact working for each build. It also helps catch bugs early if a build has issues.
Our Flow
Our system can be best understood by looking at the diagram above. The parts highlighted in blue show where the tests are run. Developers check in locally tested code changes which are reviewed by peers and committed to the master branch. This branch is built on a Jenkins server, and the entire set of UI tests are run again at this stage. Successful builds are evaluated by QA, and new bugs are reported when tests fail.
In order to prevent the UI tests from being implemented once and forgotten once they start failing, we needed to take the same CI/CD approach to our automated UI tests. This means that all of our testing code is in the same project as our app and therefore part of the continuous integration pipeline. When there are new features added to the app, new UI tests can be simultaneously added to the tests. When there are UI test failures, the bug can be fixed, integrated into the main branch, and then we can ensure that the tests pass. Therefore, the flow shown above is the same flow used for normal code changes (for features, bug fixes, etc) as well as adding and fixing test code. Having both the project code and the testing code in the same CI system allows for a productive feedback system so that bugs are caught early and builds are reliably verified.
Testing Framework for iOS
For the UI tests, we use Earlgrey 2.0. Earlgrey is a testing framework developed by Google to test Google apps for iOS, and runs with XCTest, Apple’s official testing framework that is integrated in XCode. After experimenting with writing tests in Earlgrey and XCTest alone, we decided to use Earlgrey. The main reason for this choice is that Earlgrey was substantially more stable due to built-in functionality such as waiting for views to completely load before interacting with elements and checking whether the element is actually visible, not just in the UI stack.
Page Object Model Design Pattern
For our UI tests, we decided to follow the Page Object Model design pattern.
This pattern separates the classes that deal with identifying UI elements from the test classes. Each “page,” or a screen that a user sees, gets its own page object. Each page object has common methods such as verifyUIElements(), which is all you need to call to verify that all elements that should be on the screen are visible. It would also have methods such as tapDone() or inputEmail() that represent user actions. The test classes can then call these methods to go through a test flow such as navigating to the login page, verifying the elements, typing in an email address, and then tapping “Login.” This approach decreases duplication of code that specifies UI elements. Furthermore, if a UI element gets updated, updating the tests is a one-line change on the page object, and the test classes are unaffected. This separation also means that the test classes are more readable.
See the example below for UI testing on the login page.
Above is the login screen to be tested. First, we made the screen object that represents the UI elements on the screen and methods to interact with the elements.
Then, we wrote the test cases that use the screen to test the login flow. The first test case is if they tap “Forgot Passcode” button, and the second is when they input a valid passcode.
Test Suites
We have our test suites separated into sanity tests, smoke tests, and regression tests. The sanity tests check a minimal number of features to ensure that the build is in fact building and running. If these tests fail, we are in big trouble. The smoke tests check more of the basic features of our app, and regression tests cover more granular and less-used features of our app. The sanity and smoke tests together take about 7 minutes to run. Therefore, we’d like the developers to run those two suites locally to ensure that the build is working for the basic functionalities before landing changes. If any of the tests in smoke or sanity test suites are failing, they should be looked at by a developer before the build gets to QA, since that could indicate a significant issue. Since the regression test suite takes about 30 minutes, our Jenkins server runs those tests every night and results are checked the next day. Regression test results are also important, but not as critical as sanity and smoke tests to have every test pass since testing of granular details is expected to be flakier.
Modular Test Design
We strive to design the tests such that each test is as modular as possible. This means that a failure in one test does not cause other tests to fail. In order to ensure this, each test scenario starts with logging in and ends with logging out. A test scenario might be “Take Discover credit card payment with 15% tip,” and another might be “Take a MasterCard payment with 15% tip.” Even if the Discover test fails, we use XCTest’s teardown() function to go back to a screen with a logout button, and then logout before starting the next test. Modularity helped us when writing tests because you can run tests individually and debugging is faster. When looking at test results, this modularity also helps in pinpointing the issue. When a test fails, the test framework takes a screenshot of when it failed, the stack trace before the failure, as well as the entire UI hierarchy of the screen. The last call to the testing framework that caused the exception comes from the screen object that it is trying to test, which makes it easier to see what element on what screen was the problem.
The Impact So Far on the Team and Product
So far, building and running the tests allowed us to find a lot of bugs, especially with iPhone models that aren’t used frequently. By running the tests in the simulator, we can simulate different phone screens that could cause UI elements to be moved out of view and catch those display bugs. As a side effect of writing the tests, we were also able to find inconsistencies and bugs in the existing code, which increased the quality of the app experience.
After the tests were implemented, we found many bugs before the builds ever got to QA to be tested. With our tests, a bug can be found and fixed in a few hours. On the other hand, if we hadn’t caught the bug with local testing, we would have pushed out a build (30 minutes or more), and then our QA would have tested the build (anywhere from hours to days), hopefully found and reported the bug, assigned it to a dev, and then waited it for to be fixed. This whole process would have taken days as opposed to hours, and that is only if the bug was caught. If it was not one of the fixes to be verified by QA, they might have missed the bug entirely. The amount of time saved by our automated tests is significant and has allowed for an overall increase in the quality of our product.
However, maintaining UI automation takes work. A few things we’ve learned so far is that tests shouldn’t be too granular to the point that they fail too easily, and that documenting the common test failures and the usual source of error helps speed up debugging tests in the future (for example, “Could not find app bundle” just means the app code needs to be rebuilt). After the initial setup is done and becomes part of the CI/CD pipeline, it becomes easier to maintain and is worth the effort for the time that the team saves in the long run.
