Adamantium — Self Healing Automated End to End Tests
What is automated end-to-end testing?
Automated end-to-end testing is a software testing technique that uses special end-to-end testing software tools to execute a test case. On the contrary, manual testing is performed by a human sitting in front of a computer carefully executing the test steps.
The tests will simulate user interactions ensuring our flows look and work as expected across our supported browsers.
What are the problems we face?
Elements we use to script our test suites and locate the right elements to interact with can be updated constantly in every sprint and that may cause locator changes. This leads to instability in your tests, potentially rendering them useless as the overhead to maintain them can be difficult to manage.
Here at VoucherCodes we do try to alleviate this issue as much as possible by adding a data-qa attribute to the element we need to interact with. Within the test, we then reference this attribute to limit the instability that can creep into automated end-to-end test suites. If software engineers notice this attribute in a template they are changing it is a good indication the quality assurance team needs to be involved to tweak some tests. However, if the locator update has been missed for whatever reason and the quality assurance team finds out too late the test will just fail. The team then has to retroactively fix the test, potentially causing delays to the rest of the engineering team if they are dependent on them.
Solution?
Self-healing automated end-to-end tests — Adamantium (This will be more relevant to you X-Men fans out there).
Adamantium
This idea came to life in a hackathon project, which in itself is a great example of how quality assurance can get involved in hackathon projects instead of trying to figure out what to test.
Applications are updated constantly, which may cause locator changes. Adamantium uses a type of machine-learning algorithm to analyze the current page state for changes, handle NoSuchElement test failures, and fix broken tests at runtime by replacing the failed locator with a new value that matches the best and performs an action with the new element successfully. After the test run, Adamantium provides a detailed report with the fixed locators and screenshots, and can commit directly to the codebase.
When selecting the nearest locator, logic can be applied here to match within a certain area to limit the number of matches e.g. the same div tag, if a parent element then only the children of that element. We use Nightwatch.js as our testing framework but this can be applied to any automated end-to-end testing framework.
Running a test
If you run a test with locators, but its associated element has been updated, the test throws an error due to a NoSuchElement failure, with Adamantium the test will be rerun with alternative locators.
- [data-qa=”el:vcLogoLink”] to [data-qa=”el:vcLogoLink_123"]
- [data-qa*=”el:nhsHeader”] to [data-qa*=”el:nhsHeading”]
- [data-qa=”checkbox”] to [data-qa~=”checkbox”]
- [data-qa=”el:searchContainer”] to [data-qa=”el:searchwrapper”]
If the test passes on a rerun the locator can be committed to the codebase and is reported to the team for future reference (e.g. Slack notification). However, if the test fails again the alternative locators are not committed to the codebase but are still reported to the team to investigate further, with more context around the reason for the failure at least.
Adamantium will decrease the time and effort required to write reliable automated end-to-end tests, as well as the number of test cases that fail due to test defects. As expected of a hackathon project it is rough around the edges but we are confident we can bring this into our quality assurance process and benefit from it going forward!
Have you heard? We’re hiring at VoucherCodes! Check out our careers page here.
