A Robot Should Be Running Your Appium Tests
Man has often sought to make his life easier through the use of machines. From the wheel to the computer, advances in technology have reduced the amount of manual labor we as a species are required to perform. Software is no different. While we developers already don’t do much manual labor as it is, we are instantly excited by the possibility of doing less and less.
“I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it.”
-Bill Gates
Manual testing on mobile devices is one of the few non-typing, non-clicking tasks left for developers. Between unit testing and functional testing frameworks like Appium, much of this task can also by syphoned away to machines. However, some tasks cannot be performed by these frameworks for various reasons such as limitation of the APIs provided by the hardware vendors. For example, Apple’s UIAutomation API does not support things like, app-switching or pressing the home-button. On another level, triggering simulated user-interaction via an automation API is fundamentally different from actually doing the action. In that spirit, I added tapster integration into Appium.
Tapster
Tapster is a robot that can touch and tap at your command. It has one robotic finger which you can move about by sending commands over a USB interface. It’s 3D printable and fully open hardware, meaning the designs are public and you can print your own at home using a 3-D printer. It was created by Jason Huggins a few years ago and demoed at many conferences playing Angry Birds on phones and tablets.
Tapster Code + 3D Printer Files:
https://github.com/hugs/tapsterbot/
Appium
Appium is a mobile automation framework that supports automating iOS and Android applications and websites. It uses the same protocol as Selenium (a popular web automation framework) and is also completely free and open source. Appium can load applications on to phones and inspect and manipulate the user interface. It wraps the vendor-supplied frameworks to provide a more user-friendly and flexible interface for programmers.
Appium: http://www.appium.io
Integration
The goal was to replace any gestures that Appium would send through software automation APIs and replace them with actual physical interactions performed by the robot. In order to accomplish this 3 problems needed to be solved. You can see Appium driving Tapster to Send a tweet below.
#1 — Moving The Finger in the Real World
The first is how to make tapster move in a human understandable way. Tapster has three servos (motors) that control three arms. Moving the servo will rotate the a certain number of degrees which will cause the stylus attached to the robot to also move, but in a very different way. Tapster is a delta-robot and fortunately some smart guys have already derived the equations which can translate the adjustment of the angles of the servos holding the arms of a delta robot to the corresponding number of centimeters the robot will move in the x, y, and z directions in the real world. Tapster ships with a working inverse kinematics library.
Fore more info on delta robot kinematics:
http://www.ohio.edu/people/williar4/html/pdf/DeltaKin.pdf
#2 — Mapping the Screen to the Real World
The second problem that needed to be solved was how to map those real world movements into actions. This was solved by building a mobile app which displays the coordinates that user touches on the screen.
We combined the app with a calibration routine. First, the routine lowers the pen until the it touches the screen of the mobile device. To do this, we start the finger at the origin and lower it one centimeter at a time. Underneath the finger is a phone running our calibration app as part of an Appium session. Each time we lower the finger, we query the user interface using Appium to see if any UI elements are displayed. If a UI element is displayed then the finger has contacted the screen. Once this point is found we save the z-axis position of the robot so that we know how many centimeters beneath the origin the device is positioned.
Next, we grab two points by placing the finger at our contact z-axis position with different offsets on the x and y axises. Once the finger has touched the screen we can use Appium again to read the coordinates. This time, we store the x and y coordinates displayed on the screen. Now that we have these two points we have all the data we need to build a transformation function to transform coordinates from the centimeter-based real-world robot coordinate system to the pixel-based device screen coordinate system.
Using matrices we can build an equation to represent the problem. You can think of the two data points that we gathered as equations. We can put these equations into a matrix and solve for the transformation function which we can then use to convert any two points between the 2 systems. You can ignore the fact that the robot space is in three dimensions as we only are concerned with whether or not the pen is touching.
For more specifics on the math read: https://msdn.microsoft.com/en-us/library/jj635757(v=vs.85).aspx
#3 — Orchestrating the Interactions
So now we know how to move the robot, and we know where in the real world user interface elements on a mobile device lie, so all we need to do is put it together. To put it together we split out the implementation of the commands from into a robot server. The server knows how to calibrate the robot to a device and how to tap and swipe and coordinates on that phone’s screen.
Next, I modified Appium to know about this robot server and issue it commands. In Appium, I added a command line flag to specify the address of a robot server to execute our gestures. If the flag is supplied Appium will take all gestures and instead of sending them to the underlying automation API will send them as a REST API call to the robot server. The server will tell the robot to perform the commands and then reply back to Appium.
The great part about this approach is that now all of your existing tests automatically work with a robot. Since we have Appium do the conversions, any test written in Appium automatically supports robots out of the box.
Putting it All Together
Now that you understand how it all works, let’s get it going. The only preqrequisite you will need is a properly configuratied installation of appium and node.js
1. Connect your tapster and place an iPhone connected to your computer underneath the finger2. Clone the tapster source code from GitHub:
git clone https://github.com/penguinho/tapsterbot/3. Install the required packages
cd $TAPSTER_SRC/software
npm install4. Start the robot server
cd $TAPSTER_SRC/software/src
node server.js &5. Start the Appium robot calibration app using Appium
appium --pre-launch --app Appium.RobotCalibration -U <YOUR_DEVICE_ UDID_HERE> --platform-name iOS --device-name “iPhone 6” --platform-version 8.3 &You must download and build the robot calibration app onto the device from the source code here:
https://github.com/appium/sample-code/tree/master/sample-code/apps/RobotCalibration6. Run the Calibration Script
node calibration.js -o calibration.json7. Relaunch the server with the calibration data
node server.js -c calibration.json &8. Launch Appium with robot support enabled
appium --robot-address=127.0.0.1 --robot-port=4242
Now you’re robot is ready to run any test. Just reboot the server with the -c option and the path to the calibration.json file you just generated
Next Steps
The whole point of this approach was to unlock additional functionality and bridge the gaps missing in current automation frameworks. Now that the robot knows how to tap any point on the device you can use visual automation tools to find locations to tap when Appium is not available. There are several video out solution for iPhone. You can couple these with image recognition software or existing visual automation tools like Sikuli (http://www.sikuli.org) or Geist (https://github.com/thetestpeople/Geist) and tap and swipe to your heart’s content.
