This blog post is a continuation of Monkeys to Machines where we at Life360 have transformed from being a total manual testing company to an automation-driven development organization, where we do a very stable high quality release every two weeks.
Chapter 4: UI Automation
Now that we have server backend automation and we started our research for UI Automation tools. We are having same focus for UI automation tools
- Android and iOS Support in single framework
- Multiple programming language support preferably Object oriented language support
- Real device automation support
- Open-source, active contributions 24/7
- Easy to integrate with our agile ticketing ecosystem
- Test driven development
- Easy to use continuous integration tools
From our research we went through different tools like Calabash and Frank we have not chosen them as they needed a change to the app source code which we did not want to make as we want to automate the app which is closer to the end user, so we finally choose the open source tool Appium with Java as our coding language which doesn’t require a source code change.
As a proof of concept we have implemented some scenarios and automated one test each for Android and iOS. On top of it Appium uses Selenium webdriver agent so it’s the best option available for us and we went with it.
While developing the POC project we encountered some roadblocks like Map did not show up on the Android Emulator after app was installed as it needs a specific emulator and keystore values to unlock the map to be displayed in the emulator. iOS simulator was crashing as we are trying to install the release version and trying to automation , Apple doesn’t allow automating the release version of the app so we ended up using debug version of the app to automate.
We have far too many dependencies we have to configure on our local machines like node, appium , ruby, Facebook webdriver agent and ant etc. After all the configurations and roadblocks were resolved we were able to run our first test on both platforms.
This is how our first test used to look.
In the above screenshot of the code if you see we have the Appium capabilities listed as part of the test case itself and we have all the element paths listed in the same test case. We knew this approach will not work if we want to expand our framework and to maintain the code.
So, we came up with the Approach of having different screen classes for Android and iOS and separating out the Appium class which will handle the configuration part based on the provided parameters.
To keep it simple we add a screen class for every screen with the elements and all the actions we can perform using the elements in the screen class. Then we have Test classes where we using those screen class methods and write the test cases to write the desired automation steps. This is somewhat close to page object model but not 100% page object model approach.
Ok now we have a UI automation framework and we have added Pre Commit suites for both iOS and Android. We have then informed our development team to run the test locally on their branch before posting the PR in github. This is worked pretty good and we were finding regression issues , which could have sneaked into production build.
Now that we have working framework we wanted to take the advantage of our server backend automation and reduce some of the steps in UI automation. For example we created the test users on server using backend automation and used them in UI automation instead of going through UI to create the users this obviously added some speed to the framework.
All this is good but there is still a manual process of developers running automation on there machines and we wanted to eliminate that manual labor also.
Chapter 5: Machines have wires
We had a clear vision of what we wanted when we had the automation in place — Run the pre-commit tests on each of the PR to ensure that there is nothing breaks because of that specific PR
We started building our machine lab and we listed the things we want to achieve in this process.
- Bamboo server to interact with the machines and github
- Mac Mini’s which interacts with Bamboo
Bamboo CI tool is already been used in the infrastructure engineering team and it came to us as a no brainer as we already the tool setup in the company and we can just use it out of the box. Then we purchased the Mac Mini’s we needed to handle the different automation suites.
The process of setting up these Mac Minis is very tiresome and required lot of manual labor as you know now we hate doing manual labor. But this is for greater good that’s right so we went with it.
Some of the major issues we faced was having a separate dedicated ethernet port for each machine and also the regular common cables like power , HDMI , ethernet all on this one normal working desk. This is how it looks underneath this desk.
Imagine if there is any issue with a machine we have to go through all these wires to make sure all the connections are connected fine. Also we have daily nightly workplace cleaning crew who we had informed more than once to not touch I mean to not clean under this desk.
Anyways we have the machines up and running and we are able to integrate the Bamboo with github so that every PR will trigger a Bamboo request to run a particular automation suite on these machines and once completed results are posted back to github this gave us good confidence on the code which is checked in.
Chapter 6: Cloud is God
It was amazing to have automated tests to run on these machines. We as a company got a hang of it and enjoyed it until we experienced the following scenarios — We got nightmares beacuse of them
- Electricity going down in the building
- Internet provider changing IP
- Ports communication with Bamboo getting queued up
- Machines getting hotter in the summer and shutting down
- Remote management on these machines
- MAC OS X Upgrades introduced automation software conflicts
With all these issues, the Test Engineering started spending more time on maintaining these machines and it was a nightmare for us.
So in light of these learnings we started researching the cloud CI tools available in the market which are suitable for mobile development. From our research we opted the Circle CI as they offered mobile focused environment with very easy setup for the development. We were able to move forward with Pre commit suite as part of the job in Circle CI. This helped us to take steps to decommission Mac Mini lab we have in our office.
But we are not totally satisfied yet, as we have no visibility of the status of the tests running on these machines and they appeared to us a black box environment. When it comes to automation, we are spending way too much time in installing the software we need to run these tests as part of the CI jobs.
We need a automation testing focused environment which will give us better results and faster process in running our tests , when you think of our existing frameworks which use Appium and selenium there is no other better option other than Sauce Labs. Yes, we have planned on moving from running our automation scripts on Circle CI tool into Sauce Labs.
Which means we build the app code in Circleci and then upload the app file into Sauce Labs cloud and run the tests on Sauce labs machines in the cloud then report back results into Circleci. Now we have everything running in cloud even the Sanity suites and other build generating jobs.
All worked out great with what we were expecting as a end goal , as of now we run close close to 5000 tests per week in UI automation and experience very little to completely zero downtime.