It’s been about a year since I wrote the blog post “How we are improving our Test Automation Strategy”. Since then, we as an organization, have continued to make strides in the area of test automation. As mentioned in my previous post, we abandoned our failed Java-based automated testing framework and replaced it with a lightweight framework that gave us some level of reliable coverage for our main codebase, but we still needed a solution for our browser-based tests.
In choosing our next framework, the team made the decision that we should stick with a Javascript-based framework due to our level of familiarity with Javascript and also that of the wider team’s. We objectively evaluated some frameworks and tools such as Puppeteer, Nightwatch.js, Cypress.io, and Webdriver.io. Each member of the testing team chose one of the frameworks/tools under evaluation and was tasked with setting it up and writing a predefined test. We presented our findings with the conclusion being that we settled on WebdriverIO. WebdriverIO has a wonderful support community and is continuously being worked on and upgraded. It was also easy to set up and everyone felt comfortable working with it. The choice was easy.
In the span of a couple of months, we were able to develop a test suite which provided an extra layer of coverage on top of the existing Mocha/Cheerio tests we currently have. Due to the limitations of the codebase we were working with, the closest we could come to CI was having our tests run on deployment to our testing and production environments against Chromedriver. Despite this, we saw success! For the first time we had a complete suite of tests running reliably. As part of our process, we always deployed release candidates to a staging environment, and at this point our tests would run, giving us the opportunity to catch bugs that we may have missed otherwise. As time went on we uncovered different capabilities of the framework (most notably making use of the DevTools Service). The DevTools Service allowed us to integrate performance testing into our test suite and also allowed us to test that certain network requests were taking place. This capability is very important in the context of testing ads, analytics, and other third-party integrations.
Evolution of Our Testing
As our test automation framework was maturing we realized that the way we were running our tests wasn’t sustainable. Running the tests in Jenkins required periodic troubleshooting and maintenance and there was a significant cost associated with how we used it. We started exploring cloud-based solutions and evaluating their pros and cons. For our vendor, we were very close on settling on Sauce Labs but then we got news that our organization got accepted into the Github Actions Beta program which gave us the opportunity to explore an alternative to Jenkins in the context of continuous integration. At about the same time, we were close to completing work on a Node.js powered platform, which gave us an alternative option, since it would be possible to build the app in an isolated environment in a matter of seconds.
I started tinkering with Github Actions and looking into ways we could utilize it for testing. With its Virtual Machine selection and the ability to easily run concurrent jobs, it looked like a viable alternative to Sauce Labs and Browserstack, even though it required an extra layer of configuration. Its tight integration with Github also made it an attractive option, because it gave us a chance to streamline and optimize our workflow, by connecting directly to source control. With some tinkering and trial and error we were able to get the app to build and start on a GH Actions VM. At this point, it became my mission to get our WDIO tests to run on PR against our app. This was something that was never done before in our organization in the context of e2e tests. A setup like this would give us the ability to have our tests run much earlier in the development lifecycle, which is an absolute advantage in any well engineered project.
The journey was difficult and after almost giving up utterly, I was able to get the tests to run successfully! I found that the most reliable way to run the tests was against a Selenium Grid using Docker. Building and starting the app together with running the whole suite of tests took less than 4 minutes. Not long after, we had our tests officially running on PR and we saw a lot of success in catching bugs earlier. Developers were approaching the testing team often mentioning how beneficial it was to be able to catch bugs so early. One of the testing team’s goals has been to help the organization embrace the concept of Shift Left. This was a great example of how the organization continued its journey towards shifting left in testing!
Looking for a new job opportunity? Become a part of our team! We are always looking for new Insiders.
