Getting started with automated (in-house) testing on Android smartphones using STF
So you found yourself in the situation where your mobile app gains traction, delivering consistent quality becomes more important, testing manually on your own phone is not scalable enough..
You research, you find a bunch of paid services and you find Appium. Eventually you may find openSTF (which fittingly stands for “Smartphone Testing Farm”) and you are awe-struck by what their nicely designed homepage promises.
Why even go through the hassle of setting up your own smartphone testing farm?
Confidentiality, you might not want to install your app on devices that you don’t control.
Multiple uses of the devices themselves: you can always take a phone from your setup to manually verify something, reproduce a bug or similar things.
It might come in handy having an array of phones of your choosing on your hands.
An example use-case could be finding out how a certain feature performs or looks like on a certain device.
So get yourself a cup of your favorite beverage and get going with that Proof of Concept..
Getting started
Sifting through STF’s documents for getting started and deploying, you will get an idea on how it works.
A decent job has been done explaining the services and how they work together, so I will spare you these details and include a nice drawing they provide instead.
As you can see there are a lot of services and it is very overwhelming. The reason behind that can be found in the name “STF” which is an acronym for “Smartphone Testing Farm”, so naturally this is distributed enough to be deployed at scale.
Given this architecture you can run so-called “provider” hosts which are merely the physical machines where you connect your phones to.
These will just run an adb (android debug bus) service called adb and a provider service.
At scale you will have multiple of these hosts, they will all connect to a so-called “app” server which runs the other 12 services that make up the STF infrastructure.
As you can imagine, it can be quite overwhelming and maybe overkill if all you are after is giving it a go, having a PoC or even if you are going to start out with a handful of phones only, as scaling can be done at a later point with the method I will show you in this article.
Assumptions for a successful proof of concept
Mind you these are biased and are depending on what you are planning to do with your setup.
- a linux computer running docker (I chose an Intel NUC because of the punch it packs at its small size)
- a decent USB Hub and cables
- knowledge of docker and docker-compose
- and of course: a bunch of phones
Setup of the machine
- Enter the BIOS and look for settings regarding USB, disable power-saving and similar features, turn off anything that might interfere with the connection.
- If your motherboard has any additional wireless features on-board, specifically WiFi, Bluetooth, Infrared and so on, I would recommend to turn them off, as it will lead to interference eventually.
- Install a Linux distribution of choice, i suggest Ubuntu Server.
- Get Docker by following their instructions, get docker-compose
If you can successfully run the docker hello-world example docker run hello-world and docker-compose is installed you should be good to go.
Get it up and running
For the sake of making this approachable Icreated a compose file and nginx configuration to enable local deployment on a single machine.
- Get it here https://github.com/nikosch86/stf-poc
- Decide what IP will be used to access the STF WebUI, typically that will be the main IP address of your host.
- Choose a secret for the communication between the services and update the
.envaccordingly.
Running docker-compose up -d --build will take care of starting the services and an nginx reverse-proxy.
Reverse Proxy
By default the reverse-proxy will listen on port 80 of your machine and is your passage way to the micro-services that make up STF.
The provider service will listen on the port range 7400–7700, these are used to establish adb remote connections from your workstation or laptop.
Authentication
STF supports multiple ways of authentication.
For this PoC the mock provider is enabled.
That means anyone can login by providing a name and e-mail.
Other options include OAuth, LDAP and SAML.
Phones
It is time to connect some Android phones.
Use good quality cables and a powered USB hub.
It might also be advisable to use ferrite cores or rings on the USB cables to reduce noise in the cables.
Make sure the developer mode and USB debugging is enabled
This can be done by selecting the ‘About phone’ item from the settings menu, scrolling to the build number and tapping that item 7 times, after that, select the Developer options menu and enable USB debugging.
The service that actually establishes the connection to the phones is sorccu/adb and comes with a debug key pre-installed, so once you accept that debug key on each of the devices, STF will proceed to install its STF service on them.
Docker exposes the USB bus system to this container so it can be accessed directly via/dev/bus/usb .
Using STF
Once you log in you will be greeted by an overview of the phones connected.
Anything that can be accomplished using an adb connection, can be done through the STF host on any of the phones.
To do that, you need to select the phone and use the adb connect line shown in the “Remote debug” box in the bottom right corner.
General unit testing is done through webdriver as described here in STF’s document
Scaling STF
Given it’s micro-service architecture and the convenience of having docker at hand, scaling can be done by simply deploying more instances of the adb service together with a provider service.
The provider service needs to connect to the triproxy and the storage service.
The configuration needs to be altered accordingly, taking the compose file as an example.
Without going into too much detail, this could be solved by running docker swarm with node affinity for certain services, which would eliminate the need to expose more ports to the network.
This could be the topic of another article.
Pitfalls
We experienced a couple of issues with the setup over the months.
All of them were hardware related.
- dodgy USB connections
- experiment with different USB hubs
- look out for fitment issues with the plugs - interference / bad WiFi
- disable any unused wireless connectivity features the phones have (bluetooth, cell service, IR.. )
- place the WiFi AP near to the phones, preferably in the same room
- run the USB cables in a tidy manner, avoiding loops - stability issues with phones
- regularely restart the phones (adb reboot, issued from adb service) - stability issues with the USB root of the host
reset the USB subsystem of your host (we do it nightly)
I wrote a helper to assist you in doing that:
https://github.com/nikosch86/usbtools
if you like working with cool stuff like this or if you are working on a similar project and are interested in collaborating, please get in touch
