Conduct tool POC in 5 steps — how?
Remember our previous article on how to select automation tools? After you’ve shortlisted your candidate tool(s), it is always recommended to follow up with a Proof-of-Concept (POC) before committing it/them to your company/project(s).
Because never try, never know (what problems may surface).
Discovering issues after full implementation is far more costly than finding issues during the POC process. In addition to that, the time and effort spent on procuring the tool(s) and setting it/them up will be wasted.
How a POC should be conducted varies on a case-by-case basis; it can be a scenario whereby no automation tool has been implemented before or replacing existing tool(s) with something more suitable. The candidate tool(s) can vary from one to multiple tools, depending on the objective and timeline available.
So, how should we begin?
Test Processes VS POC Steps
We’re assuming that most of you are familiar with the fundamental testing processes. The steps to conduct a POC are pretty much similar to the different phases of testing processes! Let’s take a look below in more detail.
Set Goals/Expectations
The first stage is, of course, to understand what you want to accomplish (set measurable goals whenever possible!) by the end of the POC. What you wish to accomplish may be determined by the current pain points faced. For example, is the existing tool unable to handle certain technologies? There’s a resource constraint, hence we’re seeking for tools to ease some parts of testing.
Setting quantitative goals provides more clarity on where the tool stands, especially when you want to compare against different tools. For instance, when comparing time taken for manual testing vs. using automation tools, set goals like “To be able to reduce testing time by X%” rather than just a generic “The tool is faster/slower.”
Define POC Scope & Action Plan
Know your timeline and who should be involved in this POC (typically comprises the stakeholders, testers who will be using the automation tool and tool vendors, if there’s any). Afterwards, create an action plan and indicate clearly who are the parties involved at what point of the POC. Lastly, identify/design the test scenarios/cases to be used. Here are some things to take note of:
- Be conscious that we aren’t automating the entire project at this point of time
- Don’t just go for the easiest happy path only
- Select a few critical and varied test cases which can help you to determine if the tool is appropriate. For example if your application integrates with several systems/technologies, it is good to incorporate them as part of the POC scopes to test how well the tool can run them seamlessly.
- Don’t forget about failed test scenarios to check if the tool can pick up or capture defect(s)
Preparation & Setup
Needless to say, we need to ensure the test environment is ready, test data is prepared, and all the necessary access rights and connectivities are in place before any execution. Don’t forget to download and install the tool(s) too!
Scripting & Execution
Before any execution can be done, we must always write the scripts! Create automation scripts according to the test plan and scopes, also to include validations/assertions! Once the scripts are done, test the various capabilities of each tool (especially their special features which differentiate themselves from the other tools), and record down all your observations. Here are some areas to take note of:
- Usability — ease of use
- Stability of tool (e.g. the tool supports web UI testing very well but is buggy when it comes to mobile testing)
- Any encounter of scenarios whereby the tool can/cannot handle well
- Any bug(s) found
Reporting & Presentation
After the execution of each tool candidate(s) is done, consolidate all the finding results and experiences (e.g. stability of tool, ease of use, any bug/concern), draft up a report on the pros and cons of each tool, and tabulate comparisons of different tools (if applicable). Present this report to stakeholders and highlight the key features and each of its capabilities in improving testing quality. Finally, provide a conclusion on the verdict of the tool(s).
Is tool(s) POC really necessary?
Every project’s environment/application/technologies used are different, hence what seems to be theoretically correct or works for others may not be the same for your case. It is your responsibility to assess if the tool is suitable for your company/project.
Conducting a POC allows you to get a hands-on experience of the usage and feasibility of the tool when it comes to practical usage. For open-source tools, feel free to download them from the internet safely to begin your evaluation. As for the commercial tools, you can either download the trial version or engage with the salesperson to see how you can collaborate with them for a POC.
This article aims to provide a general idea on how to carry out a POC for which you can further tailor according to your needs.
That’s all for our Christmas present to you. Wishing you an early Merry X’Mas and we will see you next year!
🧙🏼 Team Merlin 💛
Application security is not any individual’s problem but a shared responsibility.
