Dragoon: an automated UI testing tool

Introduction

End-to-end tests for UI are notoriously work-intensive and brittle. But my team has developed a way to tackle this problem: a tool that automatically visits all our UI pages and looks for errors on them.

The Big Idea

Historically, my team’s automated tests have mostly consisted of function-level unit and integration tests. These give us great code coverage, but have a few drawbacks. The biggest is that each test has limited scope and requires using mock data. In addition, creating these tests is time-consuming as each needs to be written by hand.

We decided to augment these small-scale tests with automated tools that:

1. test whole systems rather than just individual functions or components
2. automatically cover most of our code with a single generic function, and automatically test new code without having to update the tool
3. test using a variety of accounts (which each have different permissions and feature flags)
4. record how long the code takes to execute

The first tool we created to achieve these goals was called Jeffbot. Jeffbot iterates through a set of accounts, then for each one makes a GET call to each of our API endpoints, recording errors and response time. These calls test the whole backend, from API endpoint to database. As new APIs are added, they’re automatically picked up and called by Jeffbot.

Jeffbot was a great start, but it doesn’t test the interface between the frontend and backend. We had tests for our APIs, and tests for our frontend code using mock API data, but we needed another tool to make sure that the data that our APIs provided matched what the frontend expected.

The Solution

Enter Dragoon. Historically, dragoons were mounted soldiers who sometimes dismounted and walked. We named our tool after them because it “walks” our pages and “mounts” each one. Dragoon is a lot like Jeffbot, but instead of calling the APIs directly, it loads pages in our UI. Like Jeffbot, it uses several different accounts, and for each one iterates through our site’s pages using that account’s permissions and feature flags. The data for the pages comes from APIs running on a test environment, and is very similar to our production data. Dragoon records how long each page takes to load and any errors or warnings it encounters.

First Implementation (Enzyme)

Our initial implementation of Dragoon used much of the same tooling we already had in place for our frontend unit tests. Karma connected the tests to an environment, ran them, and collected output. Enzyme rendered the frontend code locally in headless chrome and connected it to the test environment APIs. Jasmine was used for the test cases and assertions.

This mostly worked, but we ran into issues with the test environment denying requests due to CORS on certain versions of chromium. When we initially built Dragoon, the current version of chromium denied requests, but a new version fixed that. Later, an even newer version broke it again.

Second Implementation (Cypress)

Fortunately, around the second time that Dragoon broke we discovered a new technology called Cypress. Crucially, Cypress can bypass CORS, so we decided to scrap our initial implementation and rewrite Dragoon to run on Cypress instead.

Switching to Cypress required a significant change in tooling. Our old implementation ran the frontend code in the same process that tested it. However, Cypress needs a separate process to run the frontend code. So, we decided to use webpack-dev-server to run it since we were already using Webpack to bundle our code. We also added start-server-and-test, which let us write a one-line command to start webpack-dev-server locally, wait until it’s running, and then run Dragoon.

Of course, there were some obstacles that we had to work through:

1. Even though Cypress can bypass CORS, we don’t get that benefit unless Cypress is actually making the request. In our case it was the frontend code making the request, not Cypress, so we still ran into CORS issues for POST requests. We solved this by intercepting all POST requests from our frontend code and making those requests from Cypress instead.
2. Dragoon needs to know the list of routes to iterate through, but that list comes from a helper function in our code that Cypress doesn’t have access to. We fixed this by bundling our code in with Cypress using webpack.
3. We want console errors to count as test failures, but, by default, Cypress ignores them. We resolved this by having Cypress intercept calls to ‘console.error’ and fail the test if there are any.

Even after adding solutions for these issues we still found that using Cypress to drive Dragoon drastically reduced the amount of code required. In the initial implementation we hand-built functions to visit a page, wait until it loads, and check for errors. Cypress is made for running tests like Dragoon, so it has built-in commands to handle all of those tasks. Using Cypress as our driver also provides significant opportunity for future enhancements using its rich feature suite.

Conclusion

Despite some challenges along the way, Dragoon is now consistent and stable. It automatically provides broad coverage of our UI, acting as a smoke test that gives us increased confidence in the code we ship. It also provides consistent timing metrics so that we can track the performance of our pages, either across all users or just for a specific account. We’re excited to take advantage of these features and explore how we can use Cypress to continue improving Dragoon.