Can’t Decide On An Automation Tool? Here Are 5 Common Considerations
What problem are you trying to solve with automation?
Break down traditional QA silos and empower developers to own quality checks.
You want automation, and currently don’t have automation in place?
Or you’re planning to switch, but you can’t decide on which one to go with?
Then read on, and hopefully the following 5 considerations should help you to weigh your options:
* What problem are you trying to solve?
* What's your budget?
* Your infrastructure and ease of setup.
* Create a POC.
* Maintenance.
What Problem Are You Trying To Solve
First, figure out what problem it is you’re planning to solve. Open a text editor or grab a pen and paper and bullet point the actual problems you want this new automation tool to help solve.
Using those bullet points as a point of reference when you’re looking at a new tool can help weigh the pros and cons. If that tool doesn’t satisfy more than 50% of your listed points, then investing time into it is not worth the time.
Choosing a tool prematurely with little consideration won’t reflect well when you’re using it. For example, if the automation tool is to help with regression testing, then be specific and ask yourself: what type of regression testing? is it more visual and UI heavy, or is it focused more towards technical implementations like an API? Asking and answering questions like this will help you to zone in on something more suitable for your needs.
Price-Value Ratio
Let’s start with open-source, it’s free! Open-source projects are great, the collaboration and communities that exist around these open-source projects are worth considering.
If the tool you want comes with a price tag, then it’s nearly impossible for me or any article out there to guide you on your companies expenses. Enterprise solutions for example can be expensive.
Understand your budget and weigh the Price-Value ratio. “Price” is based on your budget (i.e. how much can you spend per/month/year). The “Value” determined can be derived from items that you’ve written down on your “list”.
For example, assume you have 5 requirements you want your desired tool to satisfy, if the tool only gives you 2 of the points, BUT, it has a whole bunch of other fancy features that you won’t make use of, then it’s hard to argue you’re getting full value for your money.
Your Infrastructure & Ease Of Setup
How do you plan to run the automation scripts you create?
Should they execute within a CI/CD pipeline or is it enough for the scripts to run on a single persons machine? Or even a slave agent which all developers can testers can access?
If it requires a lot of technical setup, ask yourself if you have the expertise to do so? Or do you have a Dev-Ops team that is there to help to integrate the automation tool into your pipelines?
Understanding your current internal infrastructure can help you to weigh the effort that is required on your end. For example, some enterprise solutions provide a fully functional GUI where the setup is quick and simple, but those same enterprise solutions may have limited integrations, or limited options for customising.
Create A P.O.C
A Proof Of Concept is a great way to test the waters if you’re stuck between two different tools. For example, if there are a number of features a product offers, you may realise that during the POC that some of those features are only available with an additional cost. From there, you can decide if it’s worth proceeding, or if you should look for another tool.
Here are some of the benefits of creating a POC:
- Identify technical issues (compatibilities, integration limitations).
- Knowledge gaps for setup or maintenance.
- Product feature limitations.
- Help to reduce the exposure to risk (i.e. by not going all in upfront).
- Help with $cost forecasts.
Document your POC process set a timeframe, and evaluate once you’ve reached your intended timeframe.
Maintenance
Automation itself is useful but remember to consider the potential disadvantages that automation brings, one being maintenance.
Consider that maintenance will be rather small during the beginning, and will amplify at different rates as your automation suite grows. Assuming you had 500 tests, ask yourself how easily could you navigate the tool/framework to find the one you’re after?
It’s hard to properly extrapolate the amount of maintenance required, however, assessing with a high level overview can help. For example, what would it look like to update 5 tests that share code? If something went wrong, are there clear errors that can help you in finding the problem? What would that debug process look like? Or would updating 5 tests, require you to change the code in 5 different places?
Use maintenance as a metric during your POC. For example, write a number of tests of different lengths and complexities. Then go ahead, and measure the time and effort you’ve taken to update the tests. Without sounding like a broken record, it’s hard to properly extrapolate the amount of maintenance, but this approach can help to understand effort and time required.
