Is Automatic The Way Forward?

When I see that ‘manual’ has been replaced by ‘automatic’, I instantly think of two things, progression in technology and less manpower for a task. Some people agree with this advancement, others disagree, simply because it creates less jobs for people. But sometimes it is necessary to accept this move forward because it helps us create quality and quicker results, flawlessness in the final product and allows us to advance firmly into the future of development.

In this article I want to focus your attention on Automatic Software Testing (AT) and share with you the advantages together with the importance of such testing and the results we can achieve without a person sitting behind a computer for a long time looking for software bugs.

So Why Choose AT?

Automatic Testing for the user can be more simple to use (as “simple” as it is). It’s as easy as connecting your iPad to your desktop computer which is running the test script. You can then sit back or focus on something else meanwhile the test is operating.

As we know, testing is an integral step before a new version of software is released as new functionalities could potentially damage old versions. Also apps are becoming more complex and test plans are growing in size and making the task much more mesmerising for the human. Overall this is subconsciously pushing us towards the need of AT.

The AT process is split into these 4 parts:

1. Setting of the testing environment (usually 1 MD = 8 hours)

2. Creating and launching depends on the extensiveness that’s included — “test preparation is needed to ensure that all possible situations that can occur within the app are covered.”

3. Keeping the tests ready to use, which only uses 25 % of the working day hours.

4. Launching and evaluating tests only consumes 25% of the working time. For example, 1 MD includes 2 hours of maintenance and another 2 hours evaluating resumes.

Example: During the first phase everything is focused on creating the definition steps and doesn’t consist of much work regarding maintenance, but after this is completed (after executing many steps and scenarios) a new version of an app is released and a lot of features are changed. In this case, we have to re-write a lot of tests. More tests mean that we need more time to check failed test results.

It also requires less time to create new steps because we can use the copy and paste method during some stages as common practice, many only need to be created once.

Advantages, Advantages, Advantages

The obvious plus side to performing the task with AT is that we cut the time spent before the release cycle, making hundreds of tests that would take weeks or months to get through now only taking minutes or hours to fulfill.

Using AT during regression testing is a plus because of frequent code changes and therefore allowing the test to run in a smooth manner.

Load testing is another bonus factor because of pure efficiency.

Another basic reason to move to AT is that they are immune to those annoying human errors that sometimes slip past us, causing the release cycle to delay further. So unnecessary!

A big time saving advantage is the availability of test frameworks. At Inloop we use ‘Calabash’, a cross platform automated acceptance testing software for IOS and Android devices, allowing us to produce tests in an OS agnostic style so that we can duplicate our work across all major platforms.

Keeping all of the above in mind, you will definitely discover the most appropriate approach for you according to the test situation involved within your time and budget.

Disadvantages

  1. Future stages of maintenance and time pile up as more test evaluation is required.
  2. In case of changing the functionality of the app, we need to edit all tests too, therefore we should start with automation during later phases of development when the application will not be changed.
  3. Due to mobile technology diversity, tests might not work on some specific types of devices. In this case more reliable web tests can be achieved via Selenium.
  4. When testing iOS devices we must also operate on a Mac computer. So investing money into this is NOT an option.

Our Experiences

Here’s the basic math. In one working day we can achieve the number of tests that would usually take six days manually. Investing into AT does really pay off quickly and we realised this after just 2 weeks.

During the workflow cross check we discovered that AT allowed us to detect much earlier if both platforms were reacting and behaving the same way.

The pressure of trade-offs have been reduced significantly as we don't have to rely on how much remaining time we have to test the product before its release. This of course then allows us to produce a full quality test every time.

“Using AT on our product gives us a warm fuzzy feeling about the development..”

Changes?

Adapting to this process has had us facing many challenges. So far we have learnt that introducing AT to our cycle has opened many new doors including prioritising the implementation, running tests on the tests and slightly adjusting the alignment of our traditional workflow. There is still a lot to learn but we can strongly recommend the investment in this powerful procedure to produce high quality results in less time.

Who wouldn't want to do that?

Many thanks and a shout out to Adam Okruhlica & Matej Kolimar!