Shortening the test feedback loop with Jenkins, Saucelabs, Nightwatch.js

The chief purpose of test automation in an agile setting is to reduce the time taken for regression testing. The tests should provide a feedback to the devs and other stakeholders during various phases of development.

see mind map — Nightwatch test automation — MindMeister Mind Map

Our mobile web team used to have very good set of automation tests written in C# and have an internal runner that can configured to run them against different environments and schedules. This system worked out pretty well for our team when there are SDETs maintaining the code base actively. Our big problems with maintaining these tests are

1. .NET environment: Most of the team has now moved over to Mac eco system and having a huge parallels instance running on the machines. Opening the VM, Visual studio, making changes and pushing to TFS took up a lot of effort to little yield
2. Test flakiness: This happened to us for a numeber of reasons including original SDETs moving on, product changing a lot and tests not being able to keep up and test design that is very customized and broken at places.
3. Time to diagnose a failure:A hallmark of a good automation suite — being able to tell within 30 secs if the failure is a fluke or a genuine product failure. With this suite it took about 15 mins to triage a test failure
4. Adding tests is difficult and agains for the same reasons time consuming. Changing a simple locator could mean opening of 3–4 visual studio projects, making changes in multiple files and tests.
5. Test infrastructure: Though we had a scalable selenium grid for running test automation, it frequently had scalability problems and sometimes had bottlenecks.

Following is how we solved this problems

1. Popularity: Choosing a framework that is actively being used for new projects — whole company is moving away from .NET to JS and along with that test automation. The weapon of choice for web automation is nightwatchjs.org. This enabled collaboration across teams and faster turn around.
2. Descriptive error messages Just as the C# framework nightwatchjs is too based on selenium but what it got going for it is failure diagnosability. Without writing custom code just looking at the logs will tell us the failure 8/10 times. If not see 4) below.
3. Simple architecture: This is the power of open-source, since we embraced a well known framework and it has conventions on where to keep Page objects, data, tests, helpers etc. It is easy to follow and even if the current SDETs move on new folks should be able to bank on open source resources
4. Cloud based Test Runners: The new automation projects embrace saucelabs.com which gives us great scalability in browsers, OSes, parallel running, videos/screenshots, logs with virtually no complexity in the automation project. We define the desired configuration in acapabilities hash and we get it. Also we don’t have worry about maintaining servers, file sizes etc. There is some else doing it for us.
5. Jenkins: Running tests through Jenkins gave us a lot of flexibility in terms of how we can trigger these tests. Now tests can be triggered on a Pull Request, on a schedule and even at will — just by typing a URL.
6. Customizations: Another advantage of choosing a popular framework like Jenkins is that we get to choose from a number of plugins and integrations. For e.g: I configured a slack channel that tells the team when a scheduled run is failing along with the URL for the logs for further exploration.

Overall this framework enables everyone in the team to invoke tests, see the results and makes it easy to write tests and debug failures.