Automation — Choosing & Changing Tools & Framework

If the task has to be repeated everyday, it quickly gets monotonous. So, we try to automate this task to run automatically to make our lives easy. Likewise, in software development cycle, product is enhanced by adding new features or by updating the technology to run the product. All these changes in product require end to end testing which we call Regression testing to make sure the product runs with no difficulties or issues. It gets difficult to perform regression testing manually for every change, especially for a complex project with several product flows. Hence, we automate manual use cases of the existing product to make sure its not broken and this also, provides more confidence in the product that is built.

Now that we know that the use cases should be automated, we need to choose the right tool & framework. There are many tools, framework and languages available in the market. Each one of them has its own pros and cons. It is very important to decide the right one for the organization. And, while choosing there are lots of factors that should be considered. In this article let’s see what factors should be considered when choosing the framework and how changing frameworks impacts the QA’s and organization.

Few of the Automation frameworks that are in market are:

• Cucumber
• Codeception
• WATIR
• Capybara
• Cypress
• Behat
• QTP
• Appium
• Katalon Studio

Now, lets see what are the factors that we should be considering when choosing the tool, language and framework

• Are the tests written by QA’s and also by developers, if yes, the language supported by the framework should be known by QA’s and developers?
• If the tests are written only by QA, do all QA’s know the language that tool is supporting?
• Can it be easily understood by QA’s and implement test cases quickly?
• Does the framework has good documentation?
• Is it free/open source or is it paid?
• Does the framework supports multiple browser testing?
• Does it supports implementation with multiple languages?
• Does the tool support Android and iOS?
• Does it support automation web UI and API?
• Is the support community driven or a paid support?
• Are there improvements done to the tool very often?
• Can the reports be generated easily?
• Are the results generated from the tool are always reliable?
• Execution of the test cases should not take a more time?
• Does it support parallel runs?
• Configurations of the execution should be easy to implement?
• Can the tool be integrated with other testing related tools easily? e.g.: Testrail to update manual test case results.
• Are the updated plugins available in Jenkins or similar tool for execution?

Why should the existing tool or framework be changed?

• The libraries supported by the tool which are used in testing are frequently being deprecated
• There are no improvements done to the tool to support the latest technologies
• Existing framework is difficult to maintain with the huge changes done in the applications that are being tested
• If the test runs are taking lot of memory and time for executions and the results generated are not reliable
• If the existing application is completely revamped and updating existing framework takes same as creating new one.

The above reasons are genuine and the tool, framework should be changed in organization. However, sometimes there are cases where the tool and framework are changed for no reason.

Impact on changing frameworks

Case 1: Move completely to new framework

• Existing 100’s or 1000’s of test cases should be implemented in a new framework and this would take a lot of time.
• In case the existing framework has stopped working, all the testing should be done manually. Due to this, there could be a chance of regression issues in the system.
• All the Jenkins jobs, integrations with other tools should be updated.

Case 2: Keep existing framework & new features test cases in the new framework

• Both the framework should be updated when required.
• QA’s in organization will have to learn more tools and languages if the framework is changed often and existing frameworks are also retained.
• It will be difficult to write the expectations of skills when choosing a candidate as QA
• If different teams in an organization use different frameworks, it will take sometime for QA to understand frameworks when QA switches the team. This would result in slower implementations of automated test cases.
• At some point, it becomes difficult to maintain all frameworks in organizations
• QA joining the organization will not know which framework to learn.

Choose the right tool & framework which is easy to implement, maintain, reliable, cost effective. In the next blog, let us see which tool we chose and how we started working with it within the organization.