Building a Mobile Automation Lab (Part 1) — We’re not in Kansas Anymore
There’s lots of reasons to automate end-to-end mobile testing. You’ll find many companies selling cloud solutions.
So why did we decide to build a device lab in this day and age? ¯\_(ツ)_/¯
- Mobile applications have physical experiences — We interact with phones in a different way than we use our computers. This impacts our automation (duh!) but also how the OS and app integrations work.
- We have an audio app! — We build an awesome fitness app where users have over a thousand (and growing) fitness classes with awesome coaches. The quality of our audio is very important and many of the leading products don’t support for audio.
- Infrastructure makes (or breaks) your test automation — A great test automation framework mixes client and API automation with data validation to maximize coverage. An in-house device lab gives us greater control when accessing development environments.
- 💰💰💰💰 — Cloud solutions are expensive and also limit when and how you can upgrade your devices!
Over the next few weeks, we’ll be documenting the life of our mobile automation lab from inception through implementation to usage and maintenance. Where possible, we’ll try to provide examples, resources, and code snippets to our readers!
Mapping out the Yellow Brick Road
Before starting on our journey towards a functioning device lab, we sat down and asked ourselves to sketch out the end state. This helped us come up with questions (and answers!) that helped in the planning conversation.
- What’s the architecture of our test automation? — We write our end-to-end test automation using Python 3 (py.test, modularized code in a private Pypi repo) and Jenkins jobs to execute our tests.
- What kind of application do we have? — Aaptiv has iOS and Android native applications.
- What kind of devices do we need? — There’s a few different axis we used for selecting the right mixture of devices. First and foremost, we used different analytics tools (iTunes, Google Play, and third-party analytics tools like Mixpanel) to identify the most commonly used devices and OSes. We then ensured we had sufficient coverage of form factors (a small screen, a phablet screen) and hardware (low/mid/high specs). Finally, we identify the top devices where it made sense to purchase dedicated test automation phones
- What mobile test automation tool(s) do we want to use? — We felt that Appium was a great foundation for our mobile automation due to its cross-platform nature and basis on the tried-and-true WebDriver protocol. We know this is probably a blog post of its own!
- How do we want to run our end-to-end (e2e) tests? — We felt it was important to run mobile tests in parallel. It lets our testers and developers get feedback faster!
- Where are we going to put the mobile lab? — Like many scaling startups, there’s a high chance that we’ll reorganize seating (about once per month) and a pretty good chance we’ll move offices in the next 6 months. As such, we decided to build our mobile test lab so it could easily be packed up and moved around.
- How are we going to use the mobile lab? — A mobile test automation lab is an awesome tool for promoting mindfulness and a focus on the product! We wanted the devices oriented vertically for easy viewing/interaction. This added some challenges in design — we want to make sure our devices were securely fastened but still accessible!
With our major questions answered, we were able to quickly design an architecture for our mobile automation lab.
Each node consists of:
- A Mac desktop — We used a more powerful machine (iMac) to run the emulator/simulator and general infrastructure (Slack bot, Jenkins in a docker container). The other nodes use Mac Minis (2.6 GHz dual-core Intel Core i5’s, 8 GB memory).
- A powered USB hub — We found the Plugable USB 2.0 7-port Speed Charging Hub was great for charging multiple devices (we ran seven devices off a hub for a few weeks!). It also gave us room for future expansion.
- An iPhone & cabling
- An Android phone & cabling
We primarily connect to the Mac desktops using SSH and VNC. Even so, we keep some hardware around (wireless keyboard/touchpad for working at awkward angles, monitor & HDMI cable) in case we need to do some maintenance. To complete iPhone activation, we purchased a prepaid SIM card — which did the trick even if we didn’t activate it!
Prior to XCode 9/Appium 1.7.0, there wasn’t great support for parallel execution. We’re definitely going to try this out in the future.
Toto, we aren’t in Kansas anymore
So we have an architecture, stacks of Macs and phones, and more complimentary cords and chargers than you can shake a stick at.
So how do we put together the lab? In a world where cloud solutions and containerization are becoming the norm, this was definitely the most unexpected step during which we were very grateful for our well-stocked toolbox.
We decided to use an A/V cart with a flat screen mount as the chassis for our lab. We looked for models with a wide base to give us room for our desktops, wiring, and hubs. A few models had security cabinets for those who would like to store peripherals near the lab — though these were generally more expensive. This model struck a good balance between features and price.
We mounted a pegboard and used pegboard hooks to construct “cradles” to hold each of the phones. A pair of rubber bands pulled across the hooks kept the phones securely in place while still allowing good access for viewing and administration.
Anticipating future movement and maintenance, we used colored zip ties and post-its to differentiate the cabling for each node. A little pre-emptive organization saves a lot of time!
And here’s the completed lab. It’s been stable enough to scoot around the office a few times, a great cornerstone for our end-to-end testing, and a great way to encourage everyone to keep an eye on the quality of our app!
In Part 2, we’ll transform our newly wired hardware into nodes using Appium, Jenkins, and some good, old-fashioned shell scripts.
