How we are improving our Test Automation Strategy

Insider Inc. is on a mission to greatly enhance how we conduct our automated testing. Our engineers are constantly looking for new and innovative automated solutions to ensure we have adequate testing coverage across our product lines.

Test engineers at Insider Inc. realized that having an automated testing framework that was flaky, complex, and executed late in the development cycle was not very useful. We had failing tests that were difficult to debug based on an outdated custom framework (written in Java with Webdriver and Cucumber). Writing and maintaining tests took a lot of work.

These issues and a lack of Java specialists led the team to start exploring alternatives to the existing framework.

Before we started rewriting all of our tests in a new framework, we wanted to have something in place that was reliable and provided a fair level of coverage. We came up with a solution that is simple, fast, and easy to implement and maintain: we coupled Cheerio.js, Chai Assertion Library, Superagent, and Mocha to create a simple lightweight framework to test some of the most business-critical parts of our sites. We get a page’s response code and body using Superagent. Cheerio is used to parse the body text, and we use Chai to make assertions on the response code and what is parsed from the text.

Here is some example code that shows just how straightforward these tests look:

We recognize that this is not a perfect solution. One limitation of this framework is that we can only test a page’s response code and what appears in the source code. We are not able to test functionality or anything user facing (CSS or JS).

While a limitation, we have been able to cover many business-critical areas. For example, we’re making sure that all of our templates actually load and that our pages include the expected meta data and ad targeting information. We see this not as the ultimate solution for all our testing needs but as a stepping stone towards a better automated testing ecosystem for the company.

The best part of this new framework is that it’s easy to get up and running. Once you’ve installed the dependencies, you can begin to run tests — in a way that’s blazingly fast. Our current suite of tests run in less than 15 seconds against our production environment.

Going forward, we will start having Cheerio tests run as part of our pull request process. The goal will be to begin catching more issues upstream when they are isolated to feature branches. As we’ve said, this framework is not meant to replace a robust end-to-end testing framework. However we do find it useful as a supplement in cases where reliability — and speed — are foremost.

If you‘re interested in the cool stuff we‘re working on, we‘re hiring!