Testing Mobile Application Views at Workday
By Scott Bishop, Sr. Software Development Engineer, Workday
Workday’s architecture uses metadata-driven development, adding a clear separation of business logic from its user interface design. This approach has enabled Workday to radically change its UI over the years while delivering a consistent user experience across all of our applications, without having to rewrite application logic. Let’s use our Android application as an example. The client supports thousands of views, each of which have many states, that are all configurable by our server. It is possible for both Android engineers and server engineers to introduce visual regressions. Testing every state of every view quickly became a problem we needed to solve.
Like many other large software companies, Workday practices modularization and code reuse to save time and resources. However, this also makes applications very prone to regressions, as code changes are rarely localized to a single place in the application.
Despite all of these challenges, we wanted to be able to deploy our application at any point in time. We set up continuous integration, which automates the building and testing of code every time a team member commits changes to version control. We ran unit tests as well as instrumentation tests on every changeset, but still had visual regressions slipping through the cracks. How could we catch these visual regressions before they made it into the application?
To see what code changed, we look at the set of files that changed, also known as a changeset. Changesets are created by taking a diff which shows how or whether files differ from one another. This is precisely how developers do code reviews. It allows you to see exactly what changed in an easy to interpret way. Here is an example using Git that clearly shows that the variable “holidays” was changed to an empty list instead of getting the full list of holidays.
But how does that code affect the UI? What if we could diff application views in a similar way? We can, and it’s what we call a visual diff, or visual difference. Here is an example using a tool called ImageMagick.
To set up visual automation, we created a client testing framework that enables developers to easily write tests that can access the application’s views, capture images of them, and save those images to a directory. For Android we did this with Google’s Espresso testing library. Here is the Espresso test for the time off holiday example.
After running all the tests, we have a large directory of images. We needed a way to not only calculate diffs, but manage the images with a large team of developers. We use Github as our version control system. So naturally our first idea was that we should store the images in the Github repository. This way, we could version our images with the code base. If the changes were intentional, we “approved” the new image by replacing the base image in the repository. This led to a full list of issues: process was manual, repository size skyrocketed, it was hard to maintain with a team of developers, merge conflicts were not handled, and there was no validation to confirm if a developer checked in their images correctly. We needed a tool that could alleviate these problems.
We built and recently open-sourced Vizzy, a Ruby on Rails web server that facilitates visual automation and integrates into the developer workflow. It handles storing the images, the image diff process, and image approval.
The process Vizzy goes through is as follows:
- First a test suite is run against the master branch. This generates a set of images which becomes the baseline to test against. These base images are uploaded and managed by Vizzy.
- Then, for every subsequent pull request changeset, the same test suite runs and generates a new set of images called est images. Checksums are calculated and sent to Vizzy so only the images that are different from their base image get uploaded. Vizzy goes through each matching test, which contains two images, one base image and one test image, and performs a pixel by pixel diff.
- After all diffs are calculated, Vizzy updates a Github status with the link to the visual overview. It displays: diffs, new tests, missing tests, test history, comments, and the ability to approve test images. The developer is then presented with the option to approve the differences if the changes were intentional, or to fix, commit, and rerun visual automation if the changes were unintentional or indicated a bug. Once the pull request is merged and subsequent master build finishes, the approved images replace the base images for those tests, and subsequent visual tests are compared against these new images. Large development teams can run multiple builds simultaneously because the base image set always matches the master branch.
Testing views with instrumentation tests have a high cost associated with them. They can be difficult to write and when a test fails, it is not trivial to find the root cause. However this becomes easy with the visual representation that Vizzy provides. Here is the corresponding Vizzy overview.
Conclusion
Visual diffs test both views and data simultaneously. If a developer changes the font size of a title, a diff will be created. If a function changes and now produces a different output on the screen, a diff will also be created. The tool we built, Vizzy, is device agnostic, meaning it will work to test any github client application regardless of screen size. It is written with a plugin architecture so it can be integrated with any tool. Slack, Jira, Bamboo, and Jenkins plugins are already written, but contributors are encouraged to add more. Join the developer community and get started today!
