Mobile Application Automation — Where to begin?
This article is a brief introduction to the world of mobile automation and is especially helpful for those who are new to mobile automation and for those who want to start, but do not know where to begin. At the end of this article, we will look at an example of automating an Android application.
When beginning to automate mobile applications, the most important thing is to research the problem and try to answer every question that comes up before writing a line of code or deploying CI/CD:
- Why do you need automation, and what do we want to automate?
- What tools will we use to solve this problem?
- Where will we run autotests, what devices do we need for this, and who do we need for this?
- How will we get the best result and who will analyze the reports?
- Where to begin? How should you start your first project?
In fact, it is very important to answer each of these questions, which we will do in this article. If you take a different path, as most beginners do (I am no exception), then you can spend a lot of time learning the tools. You can then realize that we do not have enough resources to develop and support autotests, or that we need expensive servers to run tests on several emulators. I repeat when approaching each large task, it is important to explore everything that is necessary to solve it now (for example, launching 1 autotest) in as much detail as possible and think about the future (for example, launching 1000 parallel autotests on several platforms). This will save your company a significant amount of time and money.
Why do you need automation, and what do we want to automate?
Before starting any automation, there must be a clear working process for manually testing the products that we want to automate. Many articles have been written about QA processes, but I will voice some important points.
- Functional completeness of the product: what does it mean? The main functionality of the product is written, and most of the UI does not change. This will allow us to write documentation (which won’t have to be completely rewritten every release). This will also reduce the risk of meaningless failure of autotests (where is the element? Stolen);
- Documentation is a super document, each tested product must be covered with test cases or checklists. Autotests will be written according to such a document. If there is no such document, then it is better not to start automation;
- Passed several cycles of regression testing (or smoke testing). Why do we need it? Firstly, we will be sure that the test cases and checklists used for testing are up to date. Secondly, we must understand how long it takes for the complete regression of our product and complex testing cases.
How do you know what needs to be automated?
In mobile testing, there are usually 2 platforms: Android and IOS. Initially, we need to lay down the architecture of autotests so that they are cross-platform. The same set of tests should run on different platforms. The only difference is how the app is delivered and installed on devices and different item locators.
Does everything need to be automated? No, because it is necessary to analyze the current regression and smoke, there are cases that are very difficult and time-consuming to automate, and there is a big risk that they will often fail.
What should not be automated?
- There is no need to automate cases that have the lowest priority for the product that is not tested often, since automating such cases can take a significant amount of time, but will not bring benefits to the product;
- There is still no need to automate such a testing area as user experience (UX) because autotests will not be able to check how convenient the UI is for the user, as such cases are performed manually;
- An unstable product (early development stages) does not make sense to automate, since manual testing of such a product will take less time, and autotests will be unstable due to their frequently-changing functionality.
How should you start to automate testing?
The best place to start automation is with smoke testing. Smoke is usually launched very often, as it contains tests of the main functionality of the product. Thus, running smoke autotests daily will allow you to quickly find critical bugs and improve the quality of testing.
The next step is to automate regression testing, from higher to lower priority.
Why do we need automation?
Automation is needed to reduce the time for smoke and regression testing of the product and to get a quick response on the presence of bugs in the build and its readiness for release. Thus, the end user will receive a quality product much faster.
The introduction of automation at each stage of development (for example, running tests on developer branches, or running tests when features are merged into a common dev branch) can significantly improve the quality of testing.
What tools will we use to solve this problem?
There are quite a lot of programming languages and automation tools, so choosing the best one is not an easy task. I’ll just suggest a stack of tools based on a real project:
IntelliJ Idea — development environment
Java — programming language
Maven — project management system
Selenide — action automation tool
TestNG — test management tool
Appium — automation Driver
Allure — reports
Android Studio( SDK) — android emulators + application tools
Git — version control system
Xcode(SDK) — simulator iphone + application tools
IntelliJ Idea
The development environment has a free version for non-commercial development. For Java, this is the best IDE available. The environment supports a large number of plugins that allow you to solve any problems and make life easier for developers.
Git
This is a common version control system. Usually, version control systems are used by developers to collaborate on code, track who changed the code and when it was changed, and work on several features at the same time. But in test automation, it is also customary to use version control systems, as the autotest code needs to be stored somewhere and several people can also work with it at the same time. Autotest code is stored either in the same repository as the product being developed, or a separate repository is created.
Selenide
This is a framework developed to automate testing using Selenium Web Driver, which allows you to easily use methods to work with elements. Selenide allows you to start creating tests very quickly. If necessary, you can use a pure Selenium WebDriver using Selenide (you can get the driver and work with it directly).
Java
This language is very convenient for beginners, since it is one of the most common programming languages, there is enough information on the Internet about it, and there are ready-made solutions to any problems.
TestNG
A convenient framework for tests, suite.xml files allow you to flexibly configure the launch of autotests. Also, the framework has a sufficiently large number of annotations, which in turn allow you to flexibly work with data before and after the test run.
- @BeforeSuite, @BeforeTest,@BeforeGroups, @BeforeClass, and @BeforeMethod will be run one time before the test suite, test group, and test class before each test. This gives us the ability to assign preconditions to our tests;
- @Test allows you to specify which methods will be our tests;
- @AfterSuite, @AfterTest,@AfterGroups, @AfterClass, and @AfterMethod will be run one time after the suite with tests, a test group, and a test class after each test. This gives us the ability to assign postconditions to our tests;
- @DataProvider, and @Parameters — are required to pass parameters for tests and for implementing DDT testing.
Maven
This uses a pom.xml file to connect the necessary libraries for tests (for example, connecting the TestNG framework library) and also allows you to set the launch of tests for execution and additional parameters for testing. Maven allows us to set up running tests from the command line, which is very handy for running tests via CI/CD.
Appium Server
In simple words, this is a server through which commands are sent to the device. Appium allows you to test native, hybrid, and web applications, as well as test automation on physical devices, an emulator, or a simulator. It offers cross-platform application testing, i.e. one API works for both Android and iOS platform testing scripts.
Appium server has no operating system dependency for mobile devices. This is because APPIUM has a wrapper that translates Selenium Webdriver commands into XCUITest (IOS) or UIAutomator (Android) commands depending on device type, not OS type.
Appium Inspector
Allows you to explore the application interface for finding locators. Before we start writing autotests, we need to understand how we can get locators for the elements we want to act on. The Inspector allows you to connect to an application on a device and get a tree of elements that we can work with. Next, we will look at this with an example.
Allure
This is a framework for creating simple and understandable reports. After running the tests, we need to get a report on successful and unsuccessful tests, as well as understand why the tests fell, see the testing stages, description, screenshot, and error. Allure will help us with this. You can add files with application and browser logs to the report.
Android Studio(SDK)
To develop autotests on an Android app, we can use emulators or physical devices. If we use physical devices, then it is enough for us to use the Android SDK (adb command that allows you to work with connected devices).
If you want to use emulators, then I recommend installing Android Studio, since it can be used with the Android SDK and Virtual Device Manager (VDM). Android Studio is quite heavy, and many developers simply don’t want to have multiple development environments on their computers. However, its installation is simple and makes it easy to interact with emulators. In the future, when you understand all the tools, Android Studio can be removed and the Android SDK can be used from the command line. During the initial stage, it is better not to over complicate your work.
Xcode
Environment for developing iOS applications. A macOS device is required to develop iOS autotests. In Xcode, you can run iOS simulators or connect a real device to macOS and run autotests. In this article, we will not consider an example of running autotests of a mobile application for iOS using this tool.
Where will we run autotests, what devices do we need for this, and who do we need for this?
Test automation is quite expensive and time-consuming, but the result of this work sooner or later pays off. To develop autotests for a mobile application, we need at least one automator who knows his stuff. A manual tester, of course, can be engaged in automation, but it will take quite a lot of time to train him.
Testing mobile applications, both manually and automatically, is best done on real devices. Usually, the QA department asks to purchase a certain set of devices for testing. Testing can be done on emulators, but running one Android emulator requires a lot of computer resources, not to mention running multiple emulators at the same time. In order to use the iOS simulator, a PC with macOS is required. Also, testing on emulators and simulators does not guarantee that a bug will not be detected on the same real device. It is also worth noting that in order to develop and debug iOS application tests, we need a device with Mac OS, such as MacMini.
First, let’s decide which platforms we are testing — Android or iOS, or is the company developing an application on both platforms? If so, where are we going to run our tests?
ANDROID
- Physical server (emulators)
Using the Selenoid tool, which is a program that allows you to control browsers and Android emulators using special drivers. It is able to run each of them in isolation in a Docker container.
The main advantage of this solution is good scalability, allowing us to run several independent emulators in parallel and run our tests on them.
The main disadvantage is the high cost of the solution, as we need a fairly powerful server, which must always have KVM virtualization. Also, emulators have a number of limitations, such as working with the camera and calls; - Own farms (physical devices)
The second option is to purchase real devices for testing and create your own farm of such devices.
The main advantage is that autotests run on real physical devices, which reduces the risk of missing a bug (unlike emulators).
The main disadvantage is the high cost of the solution and the complexity of support. You may wonder why it is expensive, as there is no need to paint here, and the complexity of support lies on the fact that you need to manually discuss this farm; - Cloud solutions (physical devices and emulators)
Using cloud solutions such as BrowserStack. This service allows you to run our tests on your own emulators, simulators, and physical devices (ready-made farms).
The main advantage is that we don’t spend a lot of time setting up, we just connect to the desired device and run our tests there. It is very convenient and does not require the help of DevOps (as in previous versions).
The main disadvantage is that it’s expensive. Perhaps even the most expensive option of all listed, depending on your tasks. If we need to run tests on one device, then this option is great, but if we need 10, 20, or 50 devices, then we need to calculate how much it will cost us.
iOS
- Use of simulators
Tests are run on IOS simulators.
The main advantage is that we will test the IOS application.
The main disadvantage is the high cost of the solution, we need Mac OS to run simulators. Simulators also have a number of limitations, such as working with the camera and calls; - Cloud solutions (physical devices and emulators)
Using cloud solutions such as BrowserStack. The pros and cons have been described above; - Own farms (physical devices)
The main advantage is that we will have our own farm for testing IOS applications on physical devices.
The main disadvantage is the high cost of the solution, we need Mac OS to create our farm, as well as physical IOS devices.
What to choose if we want to test the application on both platforms?
- Cloud solutions (physical devices and emulators)
This is probably the best option, using cloud solutions such as BrowserStack; - Emulators, simulators, physical devices
In this option, we can either create a farm from Android and IOS devices or use simulators and emulators at the same time. But this can only be done on Mac OS, so we need a fairly powerful MacMini to solve this problem.
The main advantage is that we will be able to run our tests on both platforms at the same time.
The main disadvantage is the limited resources of MacMini, when running tests in parallel on several different emulators and devices, we will lose speed and performance.
It is worth noting that this is the most important section of this article, as it is necessary to decide in advance whether we have the resources needed to implement our plans, as well as how and where we want to run autotests so that they are useful for the product, and the company. Creating a test and running it locally is not a problem, the most important thing is to introduce automation into testing processes so that it brings benefits and improves the quality of the product.
How will we get the test result and who will analyze the reports?
It is important to understand how your team will receive the results of the Autotest run. Tests can be run using any CI/CD. Allure allows you to accumulate the results after each test, and then puts everything in the allure-results folder, which you can then use to generate a report. A link to the report can be provided to your team after analyzing the results.
Who analyzes the results?
Usually, this is done by a QA team, and the main task is to analyze the bugs found by the tests and understand how honest the bug is. Perhaps the test fails because it could not find the required locator. If the bug is honest, then it is further reported to the developers.
The screenshots above show an example of an Allure report. The total number of tests displays the results, if you expand the failed tests, you can see the test steps, errors, and screenshots. A detailed description of tests and screenshots allows any member of your team to analyze the failed tests.
Where to begin? How to start your first project?
First, you need to install all the tools for writing tests. There are a lot of articles on the Internet that help you configure the tools, so I will only attach links to official sources from where you can download them since the installation and configuration for different operating systems (Windows, Linux, Mac OS) are different.
Useful links to tools:
Java
IntelliJ IDEA
Appium Server
Appium Inspector
Allure
Maven
Android Studio (SDK tools at the same link)
Git
Checking the functionality of commands
After installing all the tools, you need to check that the commands work correctly. Open a command prompt and type in:
The output of each command must be a version output. If we get a different result, then we need to check the correctness of the installation or set environment variables.
Getting the appPackage and appActivity of the app
After we have everything installed, we need to request the application file with the .apk extension from the development team. Next, we need to know the appPackage and appActivity of our application. How to do it?
- The easiest way is to ask the app developers.
- Install the application on the device and use the adb command from SDK tools:
3. Look in the logs:
- Open the app on your device;
- Execute:
- Arbitrarily do some operations on the app;
- Ctrl + c ends adb command;
- Open log.txt file and find appPackage and appActivity.
Start Appium Server
The Starting Appium Server is quite simple (starting the Appium Server application is not covered here as it differs on different OS).
By default, the server rises along the path
Using the Appium Inspector
- Appium server started;
- The application (which we will test) is open on the device;
- Launch the Inspector (launching the Appium Inspector application is not described here, as it differs on different OS);
- We connect the device to the computer (physical device or emulator) and execute the command on the adb devices command line. The result of the command execution will be a list of connected devices. It is important to remember the device name;
- Install Desired Capabilities. Add to the JSON Representation block:
Or fill in the fields with data manually;
- Starting the session.
As a result, a tree with elements is displayed. When you click on application elements, the properties of the selected element are displayed. Appium Inspector allows you to search through the tree (for example, you have an XPath ready for an element) to check the correctness of the locator path.
Download, open, configure, and run a test project
- On the command line, execute git clone — link to GitHab Project
- Open the project in IntelliJ IDEA (you must specify Java 11 in the project settings in the development environment);
- After you can build the project if there are no errors, then go to the file along the path src/test/resources/config/data.properties. In this file, we need to change the value of the deviceName variable to the name of your device (the result of running the adb devices command);
- You can run tests from a file along the path src/test/java/TestCalculator by clicking on the green arrow (opposite the class name or opposite each test). Or we can run the entire test suite by right-clicking and Running in the drop-down menu on the src/test/resources/testng.xml file.
- If you already have a project, you can find several useful classes and methods for yourself here
Running tests without using the development environment
Go to the test project and open the command line. We must execute mvn commands from the place where the pom.xml file is located, or we must specify the path to it. Such a launch is needed for CI / CD tools so that we can run our tests from anywhere
Useful commands:
- Run the test suite specified in the testng.xml file:
- Run class with tests:
- Run a specific test:
- Run tests passing the deviceName parameter (this parameter is defined in the src/test/resources/config/data.properties file):
- Run tests from anywhere, you just need to specify the path to pom.xml:
Allure Records
Upon completion of the tests (when launched via the mvn command), the folder with the test results is located along the path target/allure-results. We need this data folder to build a report. Commands for building a report:
- This command brings up the server locally to display the report. The report automatically opens in the browser. If you terminate the command, the report will not be displayed. While the server is alive, the report is alive:
- This command generates a report:
- This command opens the generated report(you can open it at any time):
TOTAL
We have analyzed all of the important steps for implementing mobile testing automation, as well as the necessary tools. Based on the current project, you can write tests for your application, and improve the test architecture. This project is an example of how you can get started, but whether it’s right for you depends on the product you’re testing. Mobile automation is expensive and requires high qualifications from a specialist. The key to successful autotests is good planning at the initial stage of writing tests.
