E2E testing Services with WebdriverIO

End-to-end tests give you an overview of the quality of your application. Photo by Benjamin Voros on Unsplash.

Our products as a service require constant evolution and refactoring. One of the web front-ends I’m working on is based on jQuery and Vue.js. To make it most valuable, we wanted to be able to end-to-end test it automatically.

Goals

Our goals were:

• To innovate faster, by not having to re-test everything manually after introducing new features
• To keep maintenance cost low, by reducing risk and cost of refactorings, thus making many more immediately affordable, creating a virtuous circle
• To support multiple browsers, by running basically the same tests on all of them, and knowing immediately what browser requires a specific solution

With these goals in mind, we went for WebdriverIO, as at the time it was the second most popular Node.js testing solution after Protractor, which is Angular-oriented. After all, 200.000 weekly downloads and counting Could Not Be Wrong.

Challenges

By using WebdriverIO we faced a couple of challenges I’d like to share with you.

Installation

One does not simply install WebdriverIO. First you run:

npm install webdriverio --save-dev

Then at http://webdriver.io/guide/getstarted/install.html you are given several alternatives to run Selenium.

npm install selenium-standalone --save-dev
./node_modules/.bin/selenium-standalone install && ./node_modules/.bin/selenium-standalone start

And this:

npm install wdio-selenium-standalone-service --save-dev
// Now set `services: ['selenium-standalone']` in wdio.conf.js

And this:

// Manually download the latest version from http://selenium-release.storage.googleapis.com/index.html and then:
java -jar /your/download/directory/selenium-server-standalone-3.5.3.jar

And then you are offered to download Chromedriver to a folder of your system that is included in the PATH environment variable.

Now chances are you have three instances of Selenium and two of Chromedriver in your system, because no combination was clearly sufficient and you were getting errors as you tried. Finally, something works, but you don’t know what, and you wonder about a reproducible process other than removing and installing everything every time.

Selecting more than one element at once

Such a complex installation procedure should give good testing horse power, or at least peace of mind. Well.

The $$ command (http://webdriver.io/api/utility/$$.html), which allows to select multiple elements at once, is basically broken. Executing it, will sometimes give you an obscure error `UnhandledPromiseRejectionWarning: Error: java.net.ConnectException: Connection refused`. Other times will work just fine.

In fact, the $ command suffers from the same error. You can’t use it with a selector that matches more than one element, even if you want to access only the first one.

This will keep you busy for hours or days thinking that some buffer is getting full and you have to restart some of what you installed before, or that your tests have race conditions and you have to add more wait instructions. None of that. Googling enough will teach you that is a bug (https://github.com/webdriverio/webdriverio/issues/2406) that exists for over a year.

Now you know and it’s time to find an alternative to all your $$ calls and some of the $ ones too.

Selecting elements created or modified asynchronously

WebdriverIO does not automatically wait for DOM elements to be added or displayed asynchronously, nor for CSS transitions to finish. You have to manually add wait instructions before accessing elements that depend on the completion of a non-blocking call.

For the average modern web application (with plenty of asynchronous transformations of the DOM) this means finding out where to put a dozen of these function calls per CRUD test. See:

describe('Some CRUD', function () {
  it('Loads the list', function () {
    browser.url('some-crud');
    // wait instruction here because of Ajax
    ...
});
  it('Adds a record', function () {
    $('#addRecordButton').click();
    // wait instruction here because of Ajax or animation
    ...
});
  it('Checks the added record', function () {
$('#sortByDateDesc').click();
    // wait instruction here because of Ajax
    $('#editRecordButton').click();
    // wait instruction here because of Ajax or animation
    ...
});
it('Edits the added record', function () {
    $('#sortByDateDesc').click();
    // wait instruction here because of Ajax
    $('#editRecordButton').click();
    // wait instruction here because of Ajax or animation
    ...
});
  ... // and so forth
});

If you approach this by unconditionally waiting N milliseconds until resuming the execution, will make your tests as slow as the slowest test execution you want to support on your laptop, or worse, on your CI server.

Internet Explorer 11

Good old Selenium and good old Internet Explorer should be allowing a solid, battle-tested, straightforward testing experience.

So you find http://webdriver.io/guide/services/iedriver.html and follow those steps and you just get this error:

Error in process: Error: Command failed: [...]\my-app\node_modules\iedriver\lib\iedriver64\IEDriverServer.exe
ERROR: connect ECONNREFUSED 127.0.0.1:5555

You can google for “IEDriverServer.exe” “ECONNREFUSED” without success.

You can also try to omit the suggested path and port properties, which does launch Internet Explorer, but fills the textboxes at one character every 4 seconds.

Then you hack `node_modules\iedriver\lib\iedriver.js` to use the 32-bit version instead as suggested by some YouTube video. No speed-up.

Resolution

Installation

The happy path of the above was this:

npm install webdriverio --save-dev
npm install wdio-selenium-standalone-service --save-dev
./node_modules/.bin/wdio // And follow the config assistant

Now set services: ['selenium-standalone'] in wdio.conf.js. Next time you run ./node_modules/.bin/wdio, your tests will be launched.

Selecting more than one element at once

Well, now you can’t leverage on CSS selectors that return more than one element anymore. Instead, do something equivalent using CSS selectors that match only one element when they are run.

Here some examples:

// Trying to get the number of .attachment items in a list:
return $$('#coverList .attachment').length;
// Same, but reliably:
return browser.elements('#coverList .attachment').value.length;
// Trying to get the second keyword
expect($$('[name="keyword"]')[1].getValue()).equal('lorem');
// Same, but reliably:
expect($('#templateDialog fieldset:nth-of-type(2) [name="keyword"]').getValue()).equal('lorem');
// Just caring about one of many being visible
$('a*=View record').waitForVisible();
// Same, but reliably:
$('tbody tr:last-child').waitForVisible('a*=View record');

So yeah, now they will work. However, the price to pay is not only a thicker syntax, but also coupling with DOM structure that is non-essential to your test. Altogether making your tests slower to understand and more fragile.

Alternatively, you can add id or highly specific data- attributes to the nodes that should be tested. This will allow you to write robust test selectors, in exchange for some additional source design effort.

Selecting elements created or modified asynchronously

As said, we don’t want to statically wait for a worst-case period of time to pass, because this would multiply the overall test execution time.

The most appropriate solution is to use some of the wait instructions that WebdriverIO provides. Which one is best will depend on the object of dynamic behavior. Often though, as follows:

• For DOM structure: waitForExist
• For visibility: waitForVisible
• For other properties: waitUntil, provided custom evaluation logic

Internet Explorer 11

What to do with IE11 then? No tests at all?

We have only proved that filling text boxes is too slow. Which means, we can still run tests that don’t write text fields. So, let’s have those tests in different files, so that:

• For IE: We can launch only the ones that don’t write text
• For other browsers: We can launch them all

Conclusion

Testing is hard, but one test is always better than no tests.

WebdriverIO allows you to do plenty. Not, however, everything you’d want. We found the following limitations:

• Install documentation requires clarification — can get inexperienced developers lost.
• Testing browser-side dynamic applications requires at least an extra 10 % of wait instructions. For reactive UIs, probably more than 20 %.
• Avoiding unnecessarily fragile test selectors requires coming up with attributes or classes that are specific enough.
• You can’t test text input on IE. Write such tests aside, so you can reuse the others for all browsers.