Seriously into Quality: Testing the ImmobilienScout24 iOS App
written by Shyam Bhat
Writing tests for iOS apps has always been a much debated and often avoided topic. The tools for writing tests in the past were half-baked, apps too simple to benefit from the rewards of testing, and most developers (me included) blissfully ignorant.
The design patterns promoted by Apple in the early years of the platform — the infamous Cocoa MVC — made writing unit tests for your classes almost impossible. Besides, unit tests do not really cover all of the behavioral functionality of your app’s UI.
Automated UI testing had been in a dismal state of it’s own. In most cases, the time and effort invested in writing and maintaining fragile UI tests never really paid off.
All this, however, has been rapidly changing in recent times as the platform has matured. Apps have become increasingly complex and codebases are required to be maintainable and extensible. The iOS community has come up with more robust design patterns to serve the complexity in modern apps. Good architecture and testability are now one of the top areas of concern for developers. There are plenty of open source tools and frameworks mushrooming to ease writing of automated tests. You can now test your Apps in ways you could never before.
Testing in the ImmobilienScout24 App
At ImmobilienScout24, writing tests has always been an important part of our development workflow. We’ve been continually moving away from manual testing and relying increasingly on automated tests. Not having the luxury of a dedicated QA team makes us more diligent as developers.
The iOS codebase is roughly 70,000 lines. We have a healthy test coverage and over 3500 developer-written tests. These tests run in parallel on a pool of CI build servers each time there’s a pull request raised. A reviewer of the pull request, apart from reviewing code, also manually tests the feature before he approves it. This way, we try to ensure the master branch always has stable, tested, production-ready code at any given point.
This, along with a well modularized codebase, continuous integration and delivery, allows us to work independently in multiple teams on different features in the App, often remotely, and yet effortlessly ship to millions of users with minimal manual testing.
Oh, and despite having over a million unique visitors a month, we manage to maintain a phenomenally low crash rate of less than 0.08%!
The Test Suite
Our test suite consists of unit tests, snapshot tests and automated UI tests. We have a number of shared modules as private Pods and all of these have tests of their own. Besides Apple’s good old XCTest framework for unit tests, we make use of a number of open source testing frameworks. We’ve tried and tested different frameworks at different times and given up on some. These are the ones that worked for us and sticked on.
OCMock
Despite Apple’s effort to improve Xcode and the tooling for unit testing, a mocking framework was still one of the missing pieces in the puzzle. OCMock is a great open source alternative which helps you create mocks and stubs in Objective C with relative ease. It’s a mature and well maintained code base, and there’s plenty of documentation and community help out there if you’re looking to get started.
A caveat, however, is that it relies on Objective C’s runtime and there’s no swift implementation for it. They do however point you to this great blog post in their documentation which outlines how Swift does great with Protocols, without having to use a Mocking framework at all. We’ve been actively using this approach for unit testing all new code we write in Swift and we’re doing fine indeed without one.
GitHub Repo — https://github.com/erikdoe/ocmock
Documentation — http://ocmock.org/reference
FBSnapshotTestCase for UI Testing
You’ve most certainly already heard about this incredible tool from Facebook.
Snapshot testing is a way to automatically and accurately test what finally appears on the UI. It consists of creating a view from your app and comparing it against a reference image of the same view. Any new code in your App that introduces a change in the UI from the reference image will cause these tests to fail.
FBSnapshotTestCase offers a simple way to record these reference images and compare them in tests such that even a pixel of unintended change in the UI will make the tests fail and complain.
The image comparison is surprisingly fast and we’ve never had an issue scaling to having thousands of these tests and reference images in the project.
The true magic of snapshot testing is only realized in a codebase as large as ours. Our project uses a private pod which consists of all the custom UI elements our App uses. It also builds into standalone catalog App that acts as a reference for our designers. We’ve recorded snapshots for each of these UI elements in all it’s different variations. Additionally, we record snapshots with each UI related task to ensure there aren’t any unintended modifications.
I encourage you to give it a shot, if you haven’t already.
GitHub Repo — https://github.com/facebook/ios-snapshot-test-case
Other references -
devmonologue.com/ios/ios/snapshot-testing-with-fbsnapshottestcase
Earl Grey for UI Testing
Earl Grey is a rather new entrant to automated UI Test frameworks for iOS. Google initially built it for internally testing all of their own Apps- YouTube, Google Photos, Google Translate etc. They recently open sourced the tool and made it available publicly.
Earl Grey’s biggest feature is synchronization with the UI. You don’t need to add waiting delays in your tests while your App performs navigation animations or network requests. It keeps track of the main thread and executes each line in the test only when the main thread is freed up.
In that sense, Earl Grey fixes a huge pain point that plague most other available UI Automation frameworks like KIF and Apple’s own UI Testing framework. This essentially results in tests which are far more reliable, produce accurately identical results when run on CI, as with your local machine and ends up saving us hours fixing false negatives.
Apart from that, since EarlGrey runs in the same process as the app under test, it has access to the same memory as the app. So you can mock parts of your code and do things like, say, injecting user login credentials during tests. It also has a robust API and some other nice features which you can find in their sufficiently vast documentation.
We’ve just started to use Earl Grey for as a tool for integration testing and testing complete functionalities of features. Another library we can happily recommend!
GitHub Repo and documentation — https://github.com/facebook/ios-snapshot-test-case
Google’s developer blog — https://developers.googleblog.com/2016/02/earlgrey-ios-functional-ui-testing.html
I also gave a talk on this topic at swift.berlin. You can find the slides here — http://www.slideshare.net/bhatshyam/ui-testing-with-earl-grey
These are tools we’re using to ease the whole process of writing and maintaining tests. Each of these come with a certain learning curve involved. Adopting these in the project, starting to write tests and seeing real benefits of these does take time. But if you’re maintaining a fairly large codebase such as ours, there’s a high chance it will pay off well.
Not completely there yet
There are of course still some gaps and we have to manually test a lot of features. We do have manual testing sessions before release with all stakeholders involved. iOS and mobile in general has still not reached a point where testing can be completely automated like on web or server side platforms. But we’re surely making good progress in that direction. And the tools are no longer to blame.
If you’re seriously into quality as well with your iOS App, we’d love to hear your experiences with these tools or recommendations for more.
