Reporting automated test results

Given my profile, I obviously think Tesults (https://www.tesults.com) is the best way to report build status, automated test results, spot regression, store artifacts and test case files.

However since not everyone can use Tesults (such as companies where web/cloud services are not allowed) I want to explain why good test reporting is so important and talk about who benefits and why you should at least consider having a system in place to do the basics of what Tesults does. I will also suggest how to report your results in a way that has greatest impact.

Relevance today

One of the top takeaways from the 2018 Stack Overflow Developer Survey (https://insights.stackoverflow.com/survey/2018) published earlier this month was that DevOps is an important trend in the software industry today and that there has been a rise in the usage of the languages and frameworks associated with it.

There is a clear business reason for this trend, it leads to faster time to market and more frequent deployments. To be successful with this faster development process there needs to be continuous integration infrastructure in place with automated tests. Based on the survey this is increasingly the case.

Regression testing in particular is critical here. Without it the project manager or tech lead authorizing a deployment is either taking a huge risk or needs to slow down the dev cycle to allow for adequate quality checks.

Who benefits

Ensuring build status and automated test results data is visible and accessible benefits the whole team. Especially in large teams or teams where the frequency of check ins is high with building and testing taking place several times an hour.

Project managers need to know the current technical health of the project in order to assign tasks, know what issues are being fixed or introduced and whether there are recurring issues or regression that requires changes to process.

Software engineers responsible for fixing issues need to know which change introduced the problem, what the reason for the issue is, and have access to logs, crash dumps and screen captures. They also need access to performance data when working on optimizations.

Quality assurance teams need a complete picture of what tests are running, how often and whether there are any coverage gaps from a functional perspective.

Build and release engineers benefit from knowing all expected jobs are running and can quickly identify build failures and their causes.

Important reporting basics

Effective test results reporting requires that you store results for every test run, across all branches, platforms and build flavors. Having this data allows your system to perform useful analysis after every test run. You should compare results between the latest run with previous runs to identify regression and fixes and notify team members.

If you are collecting performance or key metrics data you should provide analysis tools to make this data easily comparable between test cases over runs. Your team should be able to easily access all files and artifacts for each build and for every test case. Looking around on remote servers or trying to figure out which build server still has the artifacts on the disk is not good enough here because this often requires a specialist to get at, slows things down and wastes time, this information needs to be available at your whole team’s digital fingertips.

You should display your results in a way that is accessible, ideally in a web browser or app everyone on your team has installed. Layout the results in a way that makes it easy to see the current status. Make it easy to drill down and up from a high level project overview and to switch between project platforms, branches, build flavors, test runs, all the way down to an individual test case.

Keep design simple in order to provide clarity. At Tesults we find using traffic light colors, while cliched, is highly effective because the symbolism of these colors is widely understood and they provide quick and meaningful feedback at a glance. You may also want a way to provide a way to assign tasks to team members but if you are using something like JIRA or a standalone bug tracker this may not be as important to you.

Consequences of implementing reporting as described

Visible and accessible results data has a transformative effect on teams in a way that is not immediately obvious if you have not tried it before. Witnessing this transformative power was a major inspiration that led to the creation of Tesults. If you realize how critical automated testing is but are having difficulty getting developers to author, maintain or fix tests then high visibility results data can make these problems go away.

Everyone wants the work they do to be meaningful. The impact of feature code work is clear for everyone to see but without visible test results, automation effort does not always get the recognition it deserves. Visible results data changes that. If one of your developers has added and maintains test cases that check core application features keep running change after change, ensuring there are no detrimental effects for your users and customers, those test cases get listed, displayed and are visible for everyone on the team to see. The engineering effort gets the recognition it should and it becomes obvious how valuable and important these tests are. This encourages further test authoring to ensure functional coverage remains high, along with maintenance and updating of existing test cases.

If failures are highly visible and being clearly highlighted your team is either making a decision not to address particular failures (perhaps deliberately in some cases) or are negligent or incompetent. General awareness like this is not possible if your results are stuck in a log on a server somewhere no one but the build engineer can access. This is crucial to making the faster development cycle possible and ensures your team moves forward fast without breaking things as they progress. Avoid firefighting and disaster containment, keep your team and customers happy.

Poor reporting reduces the value to be gained from automation

Reporting is an aspect of automated build, test and continuous integration that has historically been overlooked or done poorly. Besides the significant financial and time cost this can have if you end up deploying a problematic build, superior reporting is able to provide clarity on the true value of the return in investment in automation and DevOps. Not having solid reporting is operating without KPIs and taking value returned on faith.

Making feature code testable, adding test hooks, writing automation infrastructure, writing and maintaining automated test cases is all a significant engineering investment. If you are doing all of this it is great but to have poor reporting to tie everything together reduces the value you can extract from the effort that has already been made.

Automated testing has taken time to become relatively commonplace beyond the big tech companies to the wider industry but as the Stack Overflow survey demonstrates, it is happening. Some managers have not realized how important the reporting component is and have failed to make it a priority, focusing instead only on traditional project management reports such as the number of open issues and tasks assigned. The rapid rate and massive volume at which test data can be generated today makes it absolutely necessary to automate the reporting too.

Fortunately once great test reporting is adopted it does not take long to see that the development team saves time by having logs and artifacts easily accessible and realize that the automatic analysis helps identify and resolve issues faster. There is greater team clarity on the status of the technical health of the project. Process changes can be introduced based on real data suggesting they would help. Better reporting helps extract greater value from the automation and improves productivity.

Solid reporting is critical for software engineering teams working in shorter development cycles as is increasingly common and you should take the steps outlined to make your own reporting more effective. Store and collect the right data, put in the right analysis tools and make it all visible and accessible.