How to create an optimal environment for automated UI testing
There is no doubt that people can invest a lot of money in a project, deploy it, but contrary to the expectations, receive negative reviews in the end and sales decline. Situations like this appear when the project team doesn’t pay enough attention to UI testing.
User Interface testing is a procedure of increased importance since it enables to make sure that the future product will be apprehensible and beneficial for the users. The interface — is what the users see first. If it’s not working well or doesn’t look right, or there is something else that causes discomfort to the users, the whole product fails automatically no matter how outstanding the functionality is.
When it comes to UI testing for mobile applications, there can be two options: test it manually, which requires a lot of time, patience and attention to small details, or to automate this process. Luckily, nowadays there are many options to choose from: some of them are not very good and are suitable only for very small insignificant projects, some of them just have a limited tool-set, and some of them — the hidden jewels — can really cover all the needs that your project has.
So what basically is automated UI testing? When we execute automated User Interface Tests, we imitate the users’ actions. And the main goal of these kinds of tests — to make sure that all the components smoothly and correctly interact with each other. This includes even testing such things as:
- Font type & size — automated scripts search for the screens, where the fonts don’t match the standard;
- Colors — tests check for deviations from the accepted style guide;
- Icon styles — there cannot be different styles for the same icons, tests will check it;
- Visual inconsistencies;
- Dialog boxes — all the dialog boxes should have the same style, font, etc;
- Progress bars;
- Confirmation dialogs;
- Error messages and many other things.
We in Smartym Pro have always been using manual or half-automated UI testing, but since the number of mobile development projects increased and the need for tests increased subsequently, we came to a conclusion that we need to find a tool to boost our UI testing processes in order to let the QA-engineers concentrate on more responsible and difficult tasks.
We made a little research in order to create the most suitable environment for automating UI testing and are willing to share the results with you.
The first thing we had to consider is to choose a tool that will automate our mobile testing scripts. So basically, you can either use the frameworks provided for a particular mobile platform — Espresso for Android and XCUITest for iOS. Or, you can take advantage of utilizing cross-platform tools. Out of many cross-platform solutions for script automatization, we chose Appium as one of the most successful instruments on the market to automate UI testing.
Appium is an HTTP server, written on Node.js, that creates and processes WebDriver-sessions. Appium has the same approach as Selenium WebDriver — it receives HTTP-requests in JSON format from the clients and transforms them according to the platform they work on (Android or iOS).
Why using a cross-platform framework Appium is better than using separate platform-specific ones?
So, as we see, Appium has a lot of benefits compared to the integrated tests. One of them is the black-box principle and the fact that can write tests using one cross-platform language, which saves a lot of time. The black-box approach means that we don’t have to get direct access to the application’s source code to build UI tests.
We decided to use Appium for the UI tests automation since it supports hybrid, native and cross-platform applications, and, thus, gives the opportunity to write tests for several platforms using one API, which, consequently, saves a lot of time and other resources.
Having decided which UI testing framework to use, the next step was to determine the implementation method. There were several options: execute automated tests locally on real devices or use the emulators. Also, the whole UI testing process could be deployed either on a local machine or via a cloud service. Let’s compare the options with each other:
Undoubtedly, cloud implementation is a much better choice, especially when it comes to working with massive projects. Cloud UI testing platforms offer:
- Video-recordings of the testing process;
- Screenshots of each step of the test;
- Tools for comparing the test execution time on different devices;
- CPU and RAM load indicators throughout the whole test execution process;
- Results export tools etc.
Okay, so we chose the technology stack — hybrid framework Appium for tests automation and Cloud solution as a suitable environment for effective automated UI tests execution. Next step is to figure out what Cloud platform exactly to utilize. We tried different solutions, various platforms and here is what we got:
- Firebase TestLab
Firebase TestLab was the first thing we tried to implement. We created tests for Android UI testing via Espresso framework. We created a test user, built a test flow that checks the majority of one of our project’s functionality (important to mention that it was one of the biggest and most complex projects we had, that involves a lot of UI nuances).
Pricing: has a free version that doesn’t allow to use multiple databases, multiple buckets and multiple sites per project. Includes up to 5 Gb of storage, 10 tests per day on virtual devices and 5 tests on real devices. Provides 20 testing devices. There is also a basic paid plan for 25$/month and a flexible pay-as-you-go plan.
Why we didn’t choose this option: unfortunately, Firebase TestLab doesn’t support Appium framework, it works only with native Espresso and XCUITest.
2. SauceLabs
The second solution we tried as a cloud service for UI testing automation on real devices. It is a major service with solid documentation and a range of devices. However one of the biggest disadvantages it has is very little trial time that is not enough even to set up the environment.
SauceLabs has an API and has tools for CI implementation.
Pricing: free trial gives only 100 minutes and 2 devices. The service offers a lot of flexible packages starting from 19$ per months and ending with 7200$ per month for a full unlimited package.
Why we didn’t choose this option: During the process of setting up this cloud solution we found a major con: SauceLabs works flawlessly on both iOS and Android platforms for web and hybrid applications. However, it works well with Android automated UI testing only when it comes to native mobile apps. Because of that using this Cloud solution is useless in our case. Moreover, with hybrid apps, it works only on emulated devices, while our main goal was to test apps on real devices.
3. Browser Stack
Browser Stack has the same advantages and disadvantages as Sauce Labs when it comes to UI/UX testing. BrowserStack also has an API.
Pricing: free trial gives only 100 minutes and 2 devices. The service offers a lot of flexible packages starting from 29$ per months and ending with 4000$ per month for a full unlimited package.
Why we didn’t choose this option:
- In the process of setting up the environment and executing tests we came into some problems connected to dividing the service for real devices and emulators;
- Some documentation contains irrelevant information;
- The long launch of the tested application on a real device. We tried to contact the support — but all we got was a link on their documentation;
In the end troubles with the tested mobile on an iOS device and little trial time made us switch to searching for another solution.
4. BitBar
This is the service we ended up with for our UI automated testing procedures. Among major advantages: great relevant documentation and a month trial, without limits for tests execution time.
We created a Maven project on Java to launch, built the scripts for each OS (Linux, Mac, Windows) where we determined the Appium server parameters. The script constructs an archive with an .apk file, Desired Capabilities script and the test flow, then this archive is being sent to the BitBar server via apiKey. When launching a test in a console, we indicate some additional parameters — OS, path to the application. We utilized the Maven project both for iOS and Android.
However, when trying BitBar, we came into some obstacles:
- Appium works with WebDriver capabilities, so the driver we’ve been using working with native components, — is an advanced WebDriver;
- Because of that solution, Appium couldn’t always determine which type of data it will have to interact with. Of course, there is an opportunity to set up a selector in Desired Capabilities, however, Cloud services particularly demand a specific option (e.g. XCUITest for BitBar);
- As a result, it incorrectly determines the dialect used for elements search. In a specific example, the dialect was incorrectly defined as OSS instead of W3C.
In the end, we got a Maven project on Java, that sends the script to Appium server, which determines the UDID of the remote device, makes screenshots and uploads them into a local project folder after the script execution. Video record is provided by BitBar functionality.
Pricing: free trial for one month, unlimited time for tests, 2 devices. There are 3 paid packages including an option for enterprises. The packages are relatively cheap compared to other Cloud platforms.
The work-flow in the chosen environment
- Planning. We tried to determine some common rules, that developers should stick to in order to take into consideration all the tests integration nuances and thus save a lot of time. For example, one of the rules is to use the same ID names for the same elements on different mobile platforms during the development process. It is essential to use IDs for each clickable element. Also, we thought that will be a nice bonus solution to add accessibility_id on both iOS and Android (it can come in handy, for example, when we need to find an XPath locator — however, this will also lower the speed of tests execution).
- Preparation for tests writing. To test iOS applications, we need a macOS machine with an XCode installed on it. Any OS will be suitable for Android applications tests, including macOS. So, obviously, we chose to use one macOS computer for testing both platforms. To enable iOS apps testing, we need to install XCode, XCode command line, Node, Carthage and Appium command line. Appium doctor is also a useful tool that helps identify the components’ status and check whether all the needed components are installed. To work remotely with real devices, we installed libimobiledevice and iOS deploy. To interact with UI elements on iOS, we will use an Apple library — UIAutomation API, for Android there is framework UIAutomator.
- Setting up the Appium Server. It’s important to note that if there is a necessity to adapt Appium to a specific cloud platform, it will make sense to configure and deploy a local server or a console version, or GUI version first. To start working, we launch an Appium session with a unique ID. To determine the specific parameters for this particular session we set Desired Capabilities. Pay attention to the parameters, that each Cloud Platform demands in Desired Capabilities, they are different from one cloud solution to another, so you should check it carefully in the documentation. The next thing we need, if we want to use real devices, is UDID.
- Tests. To launch a test on a remote device, we transfer the test-flow, Desired Capabilities and a link to the application (or its ID in the cloud storage, if there is such an option). Then, depending on the mobile platform of the device, we determine the driver type to enable using the available locators in the most effective way. And don’t forget a Ui testing checklist!
With BitBar we have the opportunity:
- To track the results live on the screen;
- To do preparatory works on the devices, for example, upload a back-up, alter the device settings;
- To set a complex sequence of actions with the interface via Touch Action tool;
A little tip on how to do UI testing: in order to perform the most effective UI testing session, choose the most popular devices on the market, with different screen resolution and aspect ratio, various processor architecture and RAM size.
Conclusion
UI testing automation has a lot of benefits like precise regression errors search, saving time for QA specialists to let them concentrate on more responsible tasks, tests execution 24/7, etc.
Now Automated UI testing is one of the core elements of testing cycles in Smartym Pro, since we have a very responsible approach to the quality of the products we do. We carefully choose the tools to work with in order to provide the best and cost-effective service and product development experience.
We chose Appium for UI tests automation because it has a great advantage — a black-box approach that erases the necessity to modify the source code. Moreover, it enables us to use the same tests for both iOS UI testing and Android mobile platform due to the fact that Appium is a cross-platform server.
Out of all the cloud platform for UI tests we tried, BitBar turned out to be the most suitable according to our company’s expectations and requirements. We have already used for some of our major projects and this Cloud service has proved its benefits and improved the overall quality and productivity of mobile testing process as one of the main steps in the whole software development cycle.
