Addressing “netto” tests execution duration in Test Reports
Preface
Test Reports used to be the weakest part of SQA Mate test cases management tool. That’s because I knew that “the data first”. I needed tests first, then I need tests executions, and only after that I have had data to build test reports.
Time passed by. Required data has been collected. So, I started to build test reports. With JTBD concept in mind, of course. I also started to write an article “What job do we hire Test Reports for?” to show you ideas behind the great Test Report. The article is still under construction:
But I want to show you something interesting from that article.
Spectrum
When I look at a test report, I want to know, how much time did it take to execute it. And I can measure, for example, the difference between first and last executed tests dates. And I’ll get “brutto”, dirty duration. This duration can include weekends, holidays, etc.
To say, test execution lasted 5 days. And here is execution statuses list. What can I do here? Can I see how exactly execution passed by during these 5 days? I mean, can I build any graph?
And I tried to build a “spectrum” graph:
where total length — is “brutto” duration (our “dirty” 5 days) and black vertical lines — are test runs timestamps.
Based on that visual info I can arrange these spectral lines into “clusters” and approximate “netto” time that the team needed to run the tests. It is clear from this picture, that it was not dense 5 days testing. Now I know exactly how much “non-stop testing” hours was needed to complete the task.
Nice, right? And here it is how this “spectrum” line gracefully lies in SQA Mate test report (you can try it yourself already):
What’s next?
These days I develop Test Reports in SQA Mate and I plan to continue experiments with this “spectrum”:
- add colors to the lines — so I could see when “not passed” tests have been found;
- add such spectrum timelines for each executor — so I can evaluate contribution of everyone to make even better approximations;
- improve UX to show test run details on mouse hover over each line of the spectrum;
- other insights (maybe yours?)
If you are interested, try this in SQA Mate now, provide your feedback to me and help to make this tool even even better! (SQA Mate is available for free trial for 1 month).
