The ultimate automated software testing tools guide

Test automation allows software project teams the ability to create repeatable, reusable test scenarios that can be executed as often as needed to drive software quality up and manual labor down.

Though, success in test automation depends on the type of tool chosen. Different considerations matter for mobile vs. web testing. How do you choose? We couldn’t resist an automated software testing tools comparison reviewing 11+ along with a list of questions you can use to help you decide.

Download our complete guide to software testing.

When should you automate?

Important to note in our automated software testing tools comparison — not everything is possible to automate, and all the scenarios possible aren’t necessarily worth automating. Software project teams choose to invest the upfront time in setting up automation test cases for a few reasons:

1. Reports required. The client/product owner/stakeholders involved require extensive daily/weekly documentation. Using automation, reports can be run repeatedly with no additional workload.

2. Length. The project is expected to span over many months. For a two-month project, the cost to invest time in automation isn’t justified. The level of complexity doesn’t warrant the additional coverage. Now, for a two-year project? Investing in automation upfront can ensure that as features are developed, tested and the team moves on, new functionality can be fully regression tested.

3. Focus. For comparison against test automation, most manual tests are end-to-end. Using automation they are more difficult to maintain and stabilize. That’s why teams who opt to automate break up tests into more digestible pieces keeping the focus on individual or small combinations of functions. That way, when code is updated, only the test automation with related functionality needs to be updated.

How software teams choose an automation test tool

Test automation offers the possibility to validate mobile apps instantly and effectively. Once tests have been automated, they can be executed quickly and repeatedly, again and again. In almost all cases, this is the most cost-effective method for regression testing software products that have a long maintenance life.

In fact, test automation of any mobile app is the best way to increase the effectiveness, efficiency and coverage of the testing process. Generally, teams setting up test automation are focused on selecting a tool that allows them to maximize the following success criteria:

Reliability: High rates of issue detection

Maintainability: 1) Minimum sensitivity to application and test case changes 2) Test case definition separate from automation code

Scalability: The ability to efficiently expand test coverage and automation framework if needed

There are even more considerations once teams make the leap to mobile test automation. The following factors make executing test automation for mobile even more complex:

  • Platform diversity
  • Device diversity
  • Tool diversity
  • Tool immaturity
  • Rapid development cycle
  • Rate of technology change
Need to kickstart your software test planning? Download our free tool.

When selecting a test tool, platform compatibility, flexibility, available support and cost should be top-of-mind. Here’s how we recommend assessing possible tools along each dimension:

Platform compatibility

Verify the tool’s ability to interact with all supported mobile and desktop platforms.

Functional/flexible

Check that the tool being considered supports the most important application for your teams (client, admin, etc.)

Available support

Compare the support teams, features or communities in place to help teams or clients overcome technical hurdles.

Cost of ownership

Assess the long-term investment purchasing/setting up or supporting the tool requires compared to the expected return.

?’s to ask in your automated software testing tools comparison

  • Does the tool support multiple OS platforms?
  • How quickly can the tool support new mobile OS releases?
  • How can you use the tool to execute a single test across multiple platforms?
  • How does the tool interact with devices remotely?
  • How many scripting engines does the tool support?
  • Is the tool visual or object-based?
  • Does the tool require jailbreaking or rooting?
  • Can it be used for automation of web, hybrid and native mobile applications?
  • Does it require a code change for automation support?
  • How much does the tool cost?
  • How much support is offered by the tool?

Web automation tools

Selenium IDE

This Firefox plugin provides the ability to record test cases. Best used for testing web apps, functionality is limited and unsuitable for complex web applications. The tool does not allow teams to write tests for flash video games, music, UI/UX tests or file uploads.

Selenium WebDriver

When using Selenium WebDriver, test cases can be written in a wide variety of programming languages including (C#, Java, Ruby, Python, etc.). This open source market leader has a large following and formidable training resources/support articles available. It can be easily integrated with Cucumber/SpecFlow.

Sikuli Script

Sikuli can accommodate recording and testing web and desktop applications. The tool uses image recognition to identify and control GUI components. It is particularly useful when easy access doesn’t exist for a GUI’s internal or source code.

Visual Studio (CodedUI)

Teams can rely on Visual Studio to test web and desktop applications. Again, it is useful when no easy access to GUI’s internal or source code exists. Though, test cases are written in C# and are not easy to maintain.

Mobile automation tools

Appium

According to the Appium website, the tool offers “mobile app automation made awesome.” We couldn’t agree more. The open source tool supports iOS and Android applications, along with native and hybrid applications. The test cases can be written in a wide variety of programming languages including (C#, Java, Ruby, Python, etc.). Different design patterns can be used to minimize test cases maintenance. Tests can be integrated with Cucumber/SpecFlow.

Appium is cross platform meaning tests can be written against multiple platforms using the same API.

Pros

  • Web, hybrid and native app support
  • Good community-based technical documentation
  • Supports Android and iOS
  • Minimum code change required for automation
  • Supports different language scripts

Cons

  • Initial setup time
  • It can only run one instance per Mac

Many know Appium as the mobile version of Selenium. Because the two tools are so similar, test teams can create one framework for both rather than two. That means if teams are testing in both, there’s much less code to maintain over the course of the product.

Calabash

Calabash is an open-source option that can be used for iOS and Android projects. Test cases are written in Ruby, which can pose a problem for staffing in the lesser known language. Different design patterns can be used to minimize test cases maintenance.

Robotium

The free tool allows users to record native and hybrid Android tests. It is supported by Google, can be integrated with Cucumber and written in Java.

Desktop automation tools

TestStack.White

TestStack.White is a free tool used to support Win32, WinForms, WPF, Silverlight and SWT (Java) applications. Test cases are written in C# and can be integrated with Cucumber/SpecFlow.

SpecFlow

SpecFlow is a free offering that provides behavior driven testing capabilities for languages in the .NET family.

Cucumber

Like SpecFlow, Cucumber offers behavior driven testing capabilities only for Ruby and Java.

Service automation tools

SoapUI

The resource offers both open source and paid versions and provides a means to create load and security test cases. It is used for REST API testing, but functionality can be expanded with Groovy scripts.

HttpWebRequest

The HttpWebRequest can be integrated with SpecFlow and Cucumber. It is a .NET library which makes the REST API request directly. Can be easily integrated with Cucumber/SpecFlow.

Additional consideration: Testing in the cloud

We would be remiss to exclude cloud testing options from our automated software testing tools comparison. As screen variety increases, testing in the cloud continues to appeal to businesses. Options including Xamarin Test Cloud, Amazon AWS Device Farm, Sauce Labs, Testdroid and SOASTA allow businesses to expand test coverage and test on all possible devices without physical limitations.

Pros

  • No growing device lab
  • Equally accessible to all team members in a distributed team
  • Optimize speed, efficiency and quality
  • Increase of test coverage
  • Regression testing after bug fixing or further development of software
  • Testing of software on different platforms or with different configurations
  • Data-driven testing (running same test cases using many different inputs)
  • Test automation allows performing different types of testing efficiently and effectively
  • Expedite the release cycle by reducing time for regression testing
  • Quality improvement

Cons

  • Often the application under test must be accessed by a VPN. If the cloud-based device has difficulties accessing the code, it means more setup time for the development team.
Need a tutorial? Watch our demo of mobile test automation with Xamarin Test Cloud.

Test automation success factors

Planning. Building adequate time into the product lifecycle is a core differentiator for successful projects. Identify where your QA team will fit into the process. Will they test every user story? Or will you attempt a combination risk-based/automation testing strategy? These are all things to consider before the team begins development.

Clarity. Clearly define/document test cases and ensure all members testing understand the requirements/functionality being validated.

Control. Ensure your quality analysts/engineers have dedicated testing environments and full control over test databases.

Timing. Our team has found QAs can test most effectively if they are validating/reporting on code developed one sprint behind the development team. Without the lag, the code would be too fluid to test effectively; changes would continually break any automation.

Integration. Integrating your automation and development teams will save your project countless hours of ineffective churn. Open the lines of communication early, so they can directly address any inefficiencies. Take for example web element IDs. If your developers aren’t adding these critical elements to their code, there will be nothing for your QA team to call in their test scripts. Digging deeper, there is an additional way to access the web elements, but it’s more difficult and also slows down the tests. Connecting the quality and development perspectives will eliminate wasted time allowing the team to focus on higher-value work.

Want more from us? Click here to access all our free resources and to learn more!