Monkeys to Machines

This blog post will be a two-part blog post of how we at Life360 have transformed from being a total manual testing company to an automation-driven development organization, where we do a very stable high quality release every two weeks.

Chapter 1: A Manual Mess

Let’s go back in time a few years. We were a team of five in Test Engineering. It was a time where all we would do was test Jira tickets after developers put them in “Testing” status. Hmm, that’s normal right? That’s what we thought at the time. Obviously we knew we were doing it wrong and that was not the right way to do it. Remember, we were still monkeys.

Then came the fun part. We started growing and more engineers joined the company. We started seeing that the app was less stable. We were hotfixing quite a few times after submitting the apps to the App Store and Play Store.

We were shipping the app once every couple of months. We asked ourselves, why? Because we were not confident and wanted to test the same thing over and over again, like monkeys. Wait people stop, how many times do we have to test over and over???

Maybe you’ve felt exactly the same way.

I know what you’re about to ask, what about backend APIs? It’s easy, right? We deploy, and if it breaks, we revert. This caused us direct impact on our user experience. If users are not happy and they lose trust, it’s very hard to rebuild that trust.

We knew manual testing was not the right thing to do, and it was costing us many manual testing hours. Oh by the way, our product is a mobile app and there are a gazillion number of mobile phones out there. How are we going to test all those phones? Forget about all the phones — what about the highly used phones our users use? Okay this was getting ridiculous and we were losing our minds.

We know there is one solution to all these problems. Automation!

Oh wait, we have three different platforms: iOS, Android and server backend. Do we need three different frameworks? Or can we use one single framework?

Chapter 2: Learning from Failures

We knew what we wanted to do. We had too many new features going out and wanted to do automation for all the existing features and the upcoming features. We all knew that’s a lot to accomplish, so we decided to hire an outsourced company to provide the technology we needed.

Two automation frameworks were built for iOS and Android. Tools the outsourced company used for iOS was built on Apple’s old XCode UI-Automation framework using Javascript. For Android they used Android UIAutomator framework again using Javascript. Now that we have the framework, we still need the tests.

We then sent the app to the outsourced team and requested for them to write the automated tests for existing features. They took a long time as we anticipated and we eagerly waited for them to deliver.

Once we got the framework and the first phase of automated tests was delivered we were thrilled to run them in-house and started digging into the framework. Once we started to understand it, we discovered that it’s a complete blackbox framework. All of the main components which drive the framework are in a Jar file which cannot be accessed or changed if the framework needs improvements.

Any change we needed had to be requested to the outsourced company. We had to have lots of meetings with their tech support to try to solve all the issues, which costs us money and time.

In the meantime we still had API automation to complete. We had started building a framework for API endpoints using ROBOT framework which turned out to be a nightmare and we made a great effort to extend the capabilities of the framework, but failed.

In light of all these learnings, as a team we have decided to implement our own framework from scratch and started exploring the market standards for tools needed to automate API endpoints, iOS, and Android.

We do not want to reinvent the wheel by using a never-used or newly-created tool but what we want is to use an open source tool and expand its capabilities to match our requirements. This was a learning curve for us as a team and we seriously wanted no more monkey testing for our app.

Chapter 3: Serving Servers

We made the list of the things we want in a framework before we start spending any of our time and energy. Things we expected from the framework:

• Open-source, active contributions 24/7
• Multiple programming language support
• Should be able to work with third-party frameworks like TestNG, Firebug, etc.
• Easy to integrate with our agile ticketing ecosystem
• Test driven development
• Easy to use continuous integration tools

As we have a good understanding of what is needed from an automation framework, it was very easy for us to list the frameworks which would help us. After some research and exploration, we found that Selenium with TestNG using Java will be a perfect match for our requirements to automate API endpoints. We did not want to take any wrong steps at this point and wanted to choose the right tool which could be used by everyone in the company with a very low learning curve. Eventually we wanted to run them as part of our development process.

Now that we made the right choice in selecting the Selenium TestNG framework for API automation, we started implementing the automated tests for the most critical endpoints in our API.

We implemented a pre-commit suite in the Selenium TestNG automation framework for our endpoints. Since we want this to be a part of our development process we used the continuous integration Bamboo tool, and now it runs on every server pull request.

It’s all good. Now that we have a working framework we want to take advantage of this and do 100% automation coverage for API endpoints. It was a learning curve for the team but presently we have 95% of all our API endpoints automated. Also we automate every new endpoint before we deploy it to production. Isn’t it awesome? We can now get the status of our endpoints in production in under 30 minutes. (We know that there is room for improvement for speed. I will elaborate in the upcoming chapters).

We run our automation Sanity suite with 95% coverage of our endpoints every hour to make sure we are up and running in production.

Coming up next:

From Machines To Cloud

Stay tuned for interesting solutions to automate our mobile apps, building out our machine lab, and moving to the cloud. Boom!

See you next sprint!

Working with Us

Life360 is creating the largest membership service for families by developing technology that helps managing family life easier and safer. There is so much more to do as we get there and we’re looking for talented people to join the team: check out our jobs page.