Some thoughts about testing on Android

Ricardo Loo
Aug 2, 2016 · 4 min read

Recently, I wrote about Android testing on my blog. Since then, I have had a few more thoughts about testing on Android. That I’d like to share. The articles explain how to test Android apps that use Azure Active Directory, but the findings apply to most 3rd party services that use OAuth 2.0.

I wrote two articles:

The articles are the result of some research that I’m conducting for my team. We need to add tests to our code to validate that apps still work with the latest version of dependencies or with external contributions.

Our work is made of apps for Android, iOS, Windows as well as web applications on Node.js, ASP.NET, Ruby and other technologies.

Since I’m working on the Android platform, I started testing there, but extrapolated the findings to other platforms.

Coding workflow improved

This is what I had to do before to test a very simple app:

  1. Update my codebase.
  2. Launch the app in my emulator.
  3. Click on a button.
  4. Type username and password, if I remembered them. If not, look for credentials, then type them.
  5. Click on another button.
  6. Visually check that the result is what I expected.

After I added the tests, my workflow looks like the following:

  1. Update my codebase.
  2. Type ./gradlew connectedCheck or ./gradlew test depending on the type of tests.
  3. Check the test result.

That is a great improvement over the previous workflow. Not to mention that I don’t have to reach for my mouse nor remember useless credentials. I know I need those brain cells for something else that I can’t remember anymore.

Additionally, I can still visually check that everything is working if I’m invoking UI Automated tests with ./gradlew connectedCheck.

I use Visual Studio Team Services, but you can use whatever you want

The tests are loosely integrated with Visual Studio Team Services, and the only requirements are the following:

  • Trigger a build on pull requests.
  • Store values in environment variables

You can accomplish pretty much the same with Travis, as shown in figure 1.

Figure 1: Travis support for environment variables/build on pull requests

Instrumented tests or Unit tests

Note: I used instrumented tests as a starting point to run UI Automated tests. In the same way, I used unit tests to run integration tests. Basically I was trying to achieve the same with both types of tests.

The main difference is the context where the tests run. Instrumented tests run in an Android device. Unit tests run on the local machine, as in your development machine.

This brings several considerations to take into account when designing your test strategy:

  • Coverage: Instrumented tests work better for tests that have Android dependencies since your tests are already running in an Android environment. This means less objects to mock and more realistic scenarios. In the article about unit tests, I bypassed the authentication module in my app, leaving that code untested.
  • Test requirements: Since instrumented tests require an Android environment you need an Android device or emulator to run your tests. In comparison, you barely need a JVM to run your unit tests. Running unit tests is easier in build systems where you might not have full control of the agents running the tests. The agents usually have all the requirements in place to run unit tests, this is usually not the case for instrumented tests.
  • Test complexity: Generally, instrumented tests are more complex to code. You have to make sure that the UI is in the expected state, sometimes wait for animations out of your control or device-specific issues. This is not a problem with unit tests frameworks so you might have an easier time coding your unit tests. Note: Android Studio 2.2 includes a test recorder that might make coding your instrumented tests easier.

Last comment I have is, testing is fun. I mean, running tests automatically is way more fun than having to manually go through the same flow over and over.