Nightwatch.js after 12,000 tests and 3000 hours

What is nightwatch.js?
Nightwatch.js is an easy to use Node.js based End-to-End (E2E) testing solution for browser based apps and websites. It uses the powerful W3C WebDriver API to perform commands and assertions on DOM elements.

How are we using it?
We are using Nightwatch.js on a project where we are running simple end to end tests to verify a complex micro service architecture is functioning as expected. We are checking that each service is running and that they can communicate with their neighbours. We have recently hit the landmark of running over 12,000 test run iterations which resulted in over 3000 hours of end to end tests!

What did we like about Nightwatch.js
Nightwatch.js is a framework so we should review it as such:

What is the role of a framework? The role of a framework is to provide a way of performing actions in an easy, quick and consistent way.

In this way, Nightwatch.js is a good framework.
Nightwatch.js gives you the following ‘out of the box’:

• Nightwatch.js allows a developer to set up page objects (following the page object pattern) in a quick, easy and consistent way — I cannot fault it here. The framework discovers page objects that have been declared within a directory and makes them available to the tests in an easy to use way. There is a standard way of adding extra directories for Nightwatch.js to trawl — again, very good.
• Within page object declarations, Nightwatch.js has a layer of abstraction separating the HTML selectors from the page objects — enabling reuse and promoting general tidiness. This is a nice layer of magic the author has created.
• Nightwatch.js has a series of commands that are available when writing tests. These commands are documented well and are pretty comprehensive. Using these commands and using page object commands have a similar feel but have slight differences in return types. This was a hurdle for our team to overcome and caused us to write a lot of console.log!
• Nightwatch.js allows you to set pre and post test handlers should you want to clean up or perform some more complex actions. Working with this was a little complex. Using fat arrow syntax gave us the scope needed to access the settings we needed but this was not very well documented.
• Nightwatch.js handles commands and page object commands differently. There is a complex queue that you build up as you write a test and drain as you run a test. Commands can be inserted into the queue at test time. This queue has its own wiki page to help you understand it. This again caused some issues on boarding our team and led to more console logs.

As a framework the page objects are good and the magic is great but there is still some work needed to reduce the complexity of the queue and to improve the API around commands and page object commands. In general the framework feels a little unfinished in parts; for example you cannot run tags and skip tags at the same time.

There are some other bigger gaps; Nightwatch.js does not support gherkin based tests. There is a library called nightwatch-cucumber that we have used to combine the running of gherkin based cucumberjs tests with nightwatch.js. The library patches in the life-cycles of cucumberjs and nightwatch.js as well as performing the obvious gherkin passing and step definition calling.

What didn’t we like about Nightwatch.js

From our experience the Nightwatch.js commands have not had as much success as we would have wanted whilst executing. We have found when clicks (and other actions) are executed there are times where Nightwatch.js cannot contact our selenium grid. We have also seen examples where we found an element and requested a click only to be told the wait for element command can see the element but the click command cannot. These stability issues led us to write a set of durable commands that we have rolled out within the client site. Since implementing durable versions of commands with built in fault tolerance we have found an uplift of around 3x — 4x successful test executions.

Within Nightwatch.js it is possible to chain page objects commands. This makes it very natural to build up composable sequences within our tests and make the tests feel very JavaScripty. These follow a promise pattern and we have found that in many instances failures in asserts do not terminate tests. The use of promises with poor JavaScript error handling makes it a difficult choice to justify. You can use async/await if you like and there is also a callback approach. We have tried both of these approaches but think there must be a promised based architecture under the nightwatch.js bonnet.

Would we use it again?
If we knew then what we know now we would have pushed harder to not use nightwatch.js. The library makes it very easy to write tests but they are not reliable enough. We have had to invest a lot of time into creating durable commands and the integration with cucumberjs still feels a little wonky. Also, there is no support for the screenplay pattern which would have been a nice addition for test writers. We have managed to separate our page objects so that we have accessors, mutators and elements so we have achieved a nice separation of concerns and have been able to come up with a good workflow but it has been a lot of work. We are happy with our results but it feels like it was too much work. We now have a lot of code that is tied to the framework and so are subject to keeping it up to date with the framework to ensure smooth upgrades. At some point there could be an architectural change that would mean a significant rewrite of our custom commands.