How To Effectively Analyse Automated Test Results
Quickly identify issues and regressions in your test runs.
Automated testing is a great way to ensure that there are no regressions in your most important user journeys . It is what allows your organisation to continuously deliver a software of the highest quality. If you’ve read my guide on how to get started automating your tests, then you know how to set-up your scripts easily and execute them on the cloud using Asayer.
After test execution, the results are extremely important: they guide you towards issues that need to be addressed. That is why, for test results to be effective it needs several elements:
- Readability: whether it has failed or passed, the test coverage and the nature of the test must all be available and easy to understand quickly.
- It needs to be easily triggered and accessible: integrated to your CI/CD flow, anyone must be able to see and understand the results to promote collaboration.
- Comprehensive: it must contain all the information you need to understand which test failed, why it failed, and the steps you need to take to reproduce and fix the error.
- Alerts or notifications when a run failed so that you can be reactive and deal with the problem before it gets released and affects your customers.
That being said, what do you need in a report for it to be efficient, effective and make you save time all-around?
Timeline
This is a quick overview of the result of your test, its success or failure and why. It should take less than a few seconds for your developers to understand what went wrong during the test run. What is the point of saving time by automating your tests if you have to spend ages manually sifting through lines of logs to understand where the regression is?
Let’s take a look at the above example: it is a test suite for a reservation workflow created on Asayer. Before even opening that tests’ results, you can see that 17 out of all 22 tests failed. By opening a timeline you understand which steps failed and get a breakdown of how long each case in the suite took to complete.
The step that failed is highlighted so that you do not have to waste time digging through the logs to find the issue. The precise error that occurred during the test execution is detailed and even comes with a screenshot.
Log
To look at the error in more detail, all Selenium HTTP messages of the test run are recorded and found in the Log tab. To uncover the specific issue that was found during the test, there is no need to scroll through the entire log, just select the highlighted problem on the Timeline page and it will redirect you to the relevant area.
In our example, clicking on the timeline’s highlighted webdriver error redirects us to this line of the log:
"url_logs": "https://s3.eu-west-3.amazonaws.com/an-logs
/[anonymous]/171D3d57DdD1ef35faf04d2EFDBFE18/cd114-dcbc-4126-ac27-a403f096c/log-cd003a8830a40976e.txt"This way, your developers knows exactly when and why the test failed — they can quickly spot the regression and resolve it.
Console
The browser console logs everything that happens in the back-end during the test. That lets you know what happened at the browser level, what errors might have occurred, see your developers’ logs, etc.
Again, in the above example, you find a console log of this type:
{"level":"SEVERE","source":"network","message":"https://admin:n%3E1sZnn%2C%25%3F%5EDI%3EX(%2BCrA@preprod.[anonymous].com/https://admin:n%3E1sZnn%2C%25%3F%5EDI%3EX(%2BCrA@preprod.[annoymous].com/https://admin:n%3E1sZnn%2C%25%3F%5EDI%3EX(%2BCrA@preprod.[anonymous].com - Failed to load resource: the server responded with a status of 404 ()","timestamp":1559396392287}Here, we can see that the server failed to load the resource and displayed an HTTP Error 404, the server could not find what was requested.
Video recording
To reproduce a bug or a regression that was spotted during the automated test run, you have several tools. You can first assess what went wrong in the timeline and the more in-depth Selenium logs. You can also look at the level of the console to see what the back-end response was. But to wrap it all together and get a 360 view of the test execution itself, a video recording is necessary.
By replaying the execution, you get the entire context behind the issue that occurred. You can see exactly how it happened and, along with all the other information in the results, you get a complete understanding of why the issue occurred and what you can do to reproduce it. Just press play!
Source Code
Normally, writing code is the first step when dealing with test automation. With a software like Asayer, all you need to do to create a test script is to select the steps you want in your suite directly from recorded user journeys, and convert them. That way, you don’t have to spend as much time designing and writing the test, which is very important when you are already trying to save time by using Automation.
In the same example, the 3 steps found in the timeline are translated into approximately 180 lines of code. You can export the source code into another automation software for additional, more complex testing.
Seamless execution and results
At Asayer, our mission is to help engineering teams optimize their automation and debugging tasks by simplifying their entire agile process.
For that, the tests and their results are all contained within the same software so that there is cohesion between test creation, test run, execution and results. In itself, the result of a test is readable, accessible, user-friendly and contains everything you need to understand what went wrong.
If you want your software to always be of the highest quality, and your customer experience to be optimal, then you need to start implementing an effective automation strategy. And for that, your automation results must be top-notch and comprehensive as well so that they are a time-saver, not a pain!
