Tests Being the Soundcheck for Code

Test, test, one, two, one, two… — a proper soundcheck before going on stage ensures a successful gig and a happy audience. Same is true for an application and the users being the concert goers!

Tuning the instruments are the unit tests in coding, checking the ensemble playing equals the purpose of integration tests and corresponding to the final soundcheck, the end2end-tests validate the audience experience: Do they like what they hear or see? Do they get what they expect? Do they find their way?

As a Junior Frontend Developer, the following field report is based on my first experiences with a frontend end2end-testing, comparing two tools and sharing my learnings.

Testing Tools and Instruments

You can pick a testing tool from a variety of options: Cypress, Nightwatch.js, WebdriverIO or Selenide just to name a few. In the following I will compare the testing of Cypress, being the most recommended, and Nightwatch.js to take the option of page objects to its fullest.

Especially from a junior position with a limited background of experience, the most valuable criteria are learnability, error logging and documentation to solve the issues and be able to work with the tools independently.

So, we test to test and evaluate the differences focusing on the following aspects:

• Learnability/easy-to-understand
• Documentation/Problem Solving Support
• Logging/Recording of the Testing Process
• Syntax/Structure

Let’s start with…

CYPRESS

Facts & Numbers

The latest version 4.9.0 was released on June 20th in 2020. With 1.220.421 downloads, Cypress has an enormous weekly download count on Npm and

21,300 Stars on github. The community seems to be quite active and is keen on improving the version on a daily basis to fix bugs and solve issues for everyone.

On May 30th, 2020 Nightwatch.js latest version 1.3.6 was released.

With 194.638 downloads it covers only about 15% of the cypress downloads. I guess this also leads to a noticeably lower engagement on its github commits and stars.

Learnability

One can find a number of videos and articles on Cypress testing cases, examples and tutorials. The syntax is quite easy to understand and follow. Tests can be implemented with a basic understanding of the code structure, as the UI is very intuitive. With enough time and passion, one can dig deeper in the page object options to simplify the tests via assertion libraries for common assertions or using a DomainSpecificLanguage (e.g.
Cucumber).

There are some videos and essays on Nightwatch.js but one will not be drowned by its range. Following a tutorial will help to install the packages and was way easier than following the documentation on the Nightwatch.js-website. Nevertheless, the structure of the website is very helpful to find one’s way through the different pillars of the tool. A detailed explanation on the purpose and advantage of using page objects made it clear that this could be the main advantage of Nightwatch.js.

Documentation / Problem Solving Support

The Cypress documentation is quite extensive. Especially because it is one of the most popular tools at the moment. For almost every problem, there is a Q&A, stackoverflow chat or a github repo. In addition to that, the documentation carries a proper explanation and syntax-examples for each method.

Having the full documentation on one page helps to easy search for and find what you need — if there is a method for it. Unfortunately, the syntax is often not presented in an example code context which makes it harder to understand which parameters can and must be used to apply the method successfully. Additionally, some cases, such as comparing two input values, are not covered and it is tricky to work around the missing methods.

Logging /Debugging

With starting Cypress to run the test, one can follow the process in the testing browser and a Cypress console which logs each step, issue and command. In case of a failing test, the full test cases will be executed, and the error messages displayed in the log. This is a very helpful approach, as one failure doesn’t block the execution of the following tests. And by hovering over the logs, the progress will be highlighted in the test-browser.

Testing the usability by watching the test run in the browser helps to follow each step, but you have to focus hard to see the fast process of the test. Therefore, one can get more insights via the error logging in the terminal.

As soon as there is a failure in the test, the process will be stopped and following tests skipped which is one big disadvantage of Nightwatch.js. Furthermore, one cannot get any details on the backend calls and related timing issues. Working process is literally a try and error process.

Let’s move to…

NIGHTWATCH.JS

Facts & Numbers

On May 30th, 2020 Nightwatch.js latest version 1.3.6 was released.

With 194.638 downloads it covers only about 15% of the cypress downloads. I guess this also leads to a noticeably lower engagement on its github commits and stars.

Learnability

There are some videos and essays on Nightwatch.js but one will not be drowned by its range. Following a tutorial will help to install the packages and was way easier than following the documentation on the Nightwatch.js-website. Nevertheless, the structure of the website is very helpful to find one’s way through the different pillars of the tool. A detailed explanation on the purpose and advantage of using page objects made it clear that this could be the main advantage of Nightwatch.js.

Documentation / Problem Solving Support

Having the full documentation on one page helps to easy search for and find what you need — if there is a method for it. Unfortunately, the syntax is often not presented in an example code context which makes it harder to understand which parameters can and must be used to apply the method successfully. Additionally, some cases, such as comparing two input values, are not covered and it is tricky to work around the missing methods.

Logging / Debugging

Testing the usability by watching the test run in the browser helps to follow each step, but you have to focus hard to see the fast process of the test. Therefore, one can get more insights via the error logging in the terminal.

As soon as there is a failure in the test, the process will be stopped and following tests skipped which is one big disadvantage of Nightwatch.js. Furthermore, one cannot get any details on the backend calls and related timing issues. Working process is literally a try and error process.

The Syntax-Beat

The syntax of the two tools varies slightly. To get an idea of the individual structure, I refer to the test of a generate password functionality (highlighted with a red square below) which tests the creation of a password on a button click and compares the value of the firstly generated password to the value after clicking another time. We assume the values are not identical to prove that there is a new random password created on each button click.

Cypress syntax

Tests are written in spec.js-files, where one can define reusable selectors and list the tests following the given — when — then-approach. User-generated commands or helpers can be used to create functions that simplify and shorten the test code and make it easier to read and follow.

Below you can see an example of a reusable method in a helpers.js to be able to test each page without having to log in each time in advance:

The following example is the code of the spec.js where the tests are written. The userIsLoggedIn-method is imported from the file above and the tests for the password generator are running afterwards.

This will lead to a successful test of the create-password-generation, as you can see below. Not only can you see the commands that Cypress is running, but also the backend-calls and actual validation values.

Nightwatch.io syntax

Tests are split into the main test.js and a supporting testCommand.js operating as the page object. The page object will cover the given/when-steps of the tests, while the test.js itself will integrate the functions of its page object and add the assertions to evaluate the success of the test. This pattern of wrapping pages or page fragments makes it possible to write and run the tests from a user perspective without seeing the underlying html actions.

The very well introduced page objects-methodology of nightwatch.js (see the following example) helps to structure the code and keep the tests easy to follow.

The Test.js-files are therefore quite short and if you follow a proper command-naming, you can easily understand the single steps of the test.

Unfortunately, the logging of the test is only traceable in the terminal as the running test in the browser is faster than you might think. This means that you have to deal with the messages in the terminal and the fact that an error stops the execution of the following tests. This forces you to decide on a testing sequence because as far as I know, you cannot work around it. Lucky you if you see something similar to the log below:

End of Concert

Having to choose between Cypress and Nightwatch.js, I would go with Cypress. The documentation is a big plus and it seems that I can benefit from the experiences and learnings of many other developers and their documentation on the web. In addition to that, the configuration is quite simple and there is no need to install an extra browser or driver for running the tests.

To make the tests even simpler and easier to understand, I would go deeper into the page object options within Cypress and explore a similar structure to separate the test into page objects and HTML wrapper.