Choosing the right Test Case Management software

How Insider Inc found a great tool to implement exploratory testing

Whilst we deploy and manage highly automated test pipelines at Insider, we were missing a Test Case Management solution that would centralize our test execution data across frameworks and enable us to practice effective exploratory testing.

We needed to find a suitable tooling solution in a market that offers many similar choices. Here are some things to consider which may help guide you in finding the best solution for your company and save you some time.

Do your research

A lot of tools on the market offer similar product features but differ by offering their own sleek UI. Some have cool built-in features that can add real value to your testing. There is a wealth of information and expert opinion on blog sites, and some will rank tools for you from 1–10. Whilst we found this useful it was not fully reflective of what we knew Insider needed.

Some tools will integrate with every automation framework and some will integrate with few. Most tools will provide access to their API and enable some form of single sign on “SSO”. All tools will offer basic, advanced, and enterprise licensing options and the enterprise license version can cost anything up to $3K monthly. What we found important was to engage with staff that would use the tool and consider scenarios that could play out in the wild. Crucial for us was finding a tool that provided a workflow for session based exploratory testing and few of the tools on the market actually provide this.

Identify your key criteria

We had lots of relevant criteria when we started, but we focused on three main areas which helped remove a lot of noise. At Insider, we strive to build fast, simple, and optimized solutions which deliver trustworthy and helpful content to our readers. We were looking for something similar in a Test Case Management solution:

• A solution must enable Exploratory Testing.
• A solution must integrate with Jira.
• A solution must integrate with our Automation DevOps tools.

We narrowed the field to focus on two preferred options which would effectively allow us to deliver dynamic testing approaches to support our feature teams and products.

Engage end-users

Know who your users are and engage with them throughout. You may or may not be surprised to realize that it’s more than just your Test Engineers that will use the tool. At Insider, we embrace the narrative that everyone is a tester. With that in mind, we were conscious to speak with a range of feature team members to gain user feedback. But it is imperative to ensure that your Test Engineers are broadly supportive of the tool which you will use.

Keep stakeholders informed

Your stakeholders are not just there to sign off on budget or to ensure that legal implications have been considered and approved. Along the way I received valuable input and advice from my senior leadership at Insider. This ensured that the tool we chose included Enterprise SSO, could scale as necessary at cost, and could potentially reduce our vendor spending elsewhere through interesting features like code coverage and the ability to identify and provide data around flakey tests.

Finally …

Read the small print to ensure that the price point is within your allocated budget and that security and authentication protocols are in place. If you wish to scale in future, that could bring additional cost; therefore it’s important that you are able to project any additional cost in accordance with a monthly or yearly subscription model.

In the end we opted for Testmo, which effectively allows us to capture session-based exploratory testing and enables full integration with our automation frameworks and DevOps tools via a slick and fast UI. I’m confident we have made a good, strategic choice and Testmo should become an integral part of our testing strategy as we grow.

An important piece of advice that I received was not to try to achieve perfection immediately. Like a lot of things, it will take some time to implement fully to your satisfaction all good practices and processes which are in line with your own test engineering vision. As quality is a journey and not a destination, finding a vendor partner which is open to adapting their product offering in line with your changing needs is a welcome bonus.