Courtesy Pixabay: https://pixabay.com/en/cog-wheels-gear-wheel-machine-2125169/

Primer on Front End Automation

Automation Testing is a specialized job that is done with QE engineers who are familiar with the frameworks involved. In my team this year we have an ambitious goal of having developers write these automation tests as part of the development process thereby making test coverage (both unit and automation) a metric of the code quality.

As developer on one of the FrontEnd teams in Walmart and a noob to the automation setup and development process, I am sharing my experience on how I went about setting up the automation code base and executing it. My team works on Walmart’s online Returns experience , our front end responsive web framework uses Electrode a framework open-sourced by Walmart for large scale front end development and the Mobile App experience using React Native. This post is centric around the frameworks that WalmartLabs uses and is primarily intended for a developer getting into automation test development.

First the Frameworks

Automation has been around for a while in Walmart Labs and we have a great team dedicated to working on the tools (TestArmada). So I did not really have to go far to figure out which tools I needed to use. For those who are absolutely new to the automation world, it’s useful to know that Selenium is the most ubiquitous test automation framework out there (started out in ThoughtWorks and later enhanced by Google) and many of the tools in various languages are developed on top Selenium’s web driver apis.

Selenium — its a Java based server implements the WebDriver protocol (which is a Rest based protocol that receives commands to execute on a browser). The Server spawns out separate processes which launches a browser that has remote debugging enabled over which it issues commands to the browser.

Magellan — this is the test runner which runs the tests in parallel . Its similar to KarmaJS in that running tests is a separate concern than actually defining the test. Magellan works by finding files or tags and spawning up new worker processes that work on these tests. Each file is a new test case that will be executed within a worker. The worker environment is a sandbox for each test. It can work on these in parallel or in serial mode depending on the settings provided.

NightwatchJS — is the BDD framework that abstracts out Seleniums webdriver apis and provides a framework for defining the test. It supports a design pattern called a page object model through which an HTML page or Mobile page can be abstracted out as page-fragments with commands , sections elements and pages .

Appium — is the Selenium equivalent for mobile app testing. Nightwatch uses Appium to interact with the Mobile device. Appium implements Web Driver specifications making it compatible with any client that uses Web driver api.

Shifu — is a mocking framework developed as plugin with HapiJS. It essentially provides an easy way of detecting and configuring mocks for particular URI route. For local or dev end to end testing, you would want to start by mocking all of the webapp’s dependent downstream services. So when you start your automation setup, you also start a mock server which will handle all your downstream api calls and return a mock response. The cool thing about Shifu is that it has a concept of a session, which allows sending different responses for a route per session. You do this by invoking a variant for a session. Variants are different responses for the same route, tagged differently. The shifu-magellan-nightwatch npm module has out of the box integration with Shifu so that this can be easily plugged into your automation setup.

The Page Object Model

• Elements — these are logical placeholders for the elements you will query on your page using CSS or XPath query selectors

elements

• Commands — these are abstractions that will act on your elements

commands

• Pages — a page is composition of the elements and commands all of which are available to the page object.

page

The Actors in our Setup

Entities in an automation

Note on the actors

1. The Mocked services are optional — you can just as well do automation testing on a live or pre-prod setup
2. Each Automation tests runs in its own sandbox which starts a Webapp in that process space.
3. The entire setup can be run in one local machine or can be distributed over the network. You could have tests executing from your local machine talking to remote selenium servers that load up webapps from yet another domain.

Setting it Up

So essentially if you are automating a full front end application, you need to be able to load the page, click elements, set values to inputs, click Buttons, assert on element values etc. From the onset, its a good idea to provide a data-automation-id or some such identifier to your elements so as to not have to depend on CSS and element ids. Also where there are dynamically generated elements (such as a grid or list) , selecting elements using CSS parent child hierarchy is next to impossible if all the elements on the grid have the same class attributes. Under the hood, Selenium uses document.querySelectorAll for getting the elements and asserting on these. If you do not have unique id or css selector, this returns a list if there is more than one match for the query. In which case, clicking on the element will not work and Nightwatch Extra will give a warning that looks like

21:39:35 [WARN] [Nightwatch Extra] getEl saw selector .new-order-item-select-container but result length was 26, with 26 of those :visible

Once you have your front end pages all good for automation readiness, you can start by defining a folder structure for keeping your automation tests.We keep this under project_root/test/automation .

Here is sample folder structure that we followed — this is just indicative. As long as the page finds its elements and commands — it should work fine.

project_root/test/automation
--conf (for configuration files)
--lib (for pageobject models)
---pages
   ----commands
   ----elements
   ----sections
--mocks (for mocks if applicable)
--scripts (bash/csh scripts for command line)
--tests (where the actual tests are)

Here are the npm dependencies that are needed for this to work (configured under optionalDependencies.

"async": "0.9.2",
"chromedriver": "2.35.0",
"config": "1.30.0",
"dpro": "1.2.0",
"event-stream": "3.3.4",
"jsonlint": "1.6.2",
"nightwatch": "0.9.9",
"phantomjs": "^2.1.7",
"phantomjs-prebuilt": "2.1.16",
"ps-tree": "1.1.0",
"selenium-server": "3.9.1",
"testarmada-magellan": "10.1.1",
"testarmada-magellan-admiral-plugin": "^3.0.0",
"testarmada-magellan-local-executor": "2.0.0",
"testarmada-magellan-nightwatch-plugin": "7.0.0",
"testarmada-nightwatch-extra": "5.0.0",
"testarmada-renv": "4.1.0",
"webpack-dev-server": "1.16.2"

The two most important configuration files are

magellan.json — Describes how magellan should execute. Magellan provides lifecycle hooks from where we can add bootstrapping code to start the Webapp and mocks if necessary — this is defined in the setup_teardown.

magellan.json — snippet

nightwatch.json — Describes how the tests and selenium should interact.

nightwatch.json

Note on the Nightwatch.json

1. src folders — indicates the directory where the tests lie
2. output folders — is for the test run reports
3. custom_commands_path — indicates the folder where the page object commands are — optional depending on if this was separated out or not
4. custom_assertions_path — indicates the path for custom assertions if any — optional depending on if they are any custom assertions
5. page_object_path — directory where the page objects are
6. globals_path — page where any global variable file are
7. selenium configuration — selenium server and driver configs

The Test!

The test is an instance of the BaseTest written in BDD style functions.

Sample Test

Test Inheritance

We can add additional functions to the Test Object, by inheriting the Test and adding functions to its prototype. This is useful to have some common functions across all the Tests. Additionally the Test prototype has the before , beforeEach , afterEach and after functions that can be overridden.

Debugging Test Failures

If the test fails and you are unsure why, you can verify the correctness of the CSS query selector by installing the Selenium IDE plugin for Chrome and run the test from there. All of the WebDriver api calls are asynchronous. So checking the result of the call helps verify the correctness of test.

Large Scale Testing using SauceLabs

Testing massive amount of parallel tests requires infrastructure and resources that the local work station might not suffice. At Walmart we use SauceLabs to run our tests as part of our CI CD Pipeline. This is a hosted platform on which you can run tests.

In Conclusion

At WalmartLabs, we are making automation testing a first class requirement for development in which developers will add the automation test cases for every feature they develop. This requires a culture shift that is now gaining momentum. Hopefully this post would have helped in understanding how the underlying components work together and provide a quick bootstrap guide for automation testing.