Android: Local or Instrumented tests?
The Android platform is traditionally very hard to test. Since the environment in which we code is a different OS than the one our application will run, traditional testing practices does not apply very well to Android development. To circumvent this issue we have the following approaches:
- Deploy the application we want to test and send “commands” to it as if it were an user
- Make a clone of the environment that the app runs under the same environment we code
Instrumentation tests
The way we can send commands to an application during tests is through another application that “instruments” the one we intend to assert on. These are always deployed tests, that is: they are run in a real device or emulator.
The platform has always provided this first approach with the name “instrumentation tests”. Those “commands” are run by an instrumentation framework. You have classes like android.test.ActivityInstrumentationTestCase2 or android.test.InstrumentationTestCase. These provide the base API for creating tests with instrumentation commands.
When you run these tests it is important to know that in reality you are deploying two applications: your real application and the test application. The later is the code you put under androidTest folder. These are whole APKs and not only the test classes. That means they have resources, AndroidManifest.xml, assets and classes.dex in them.
Nowadays, Google does not advise us to use the android.test package directly but rather the support test libraries. They contain several enhancements over the normal package including a more recent compatibility with the JUnit 4 testing Java library. This also brings android.support.test.InstrumentationRegistry which is a facility class to access the instrumentation context. The methods there, though, may seem a bit odd at first.
In android.support.test.InstrumentationRegistry we have:
getContext(): contrary to what it might seem, this is not the main application context. This is the test application context. Through it we can access the assets, resources and etc of the test application. Remember that in this kind of test we actually deploy two complete APKs and this is the context that references our test APK.getTargetContext: this is the application context. The application we are testing.
Beware of the difference. If you are running into issues like it can’t find the asset then be certain you are using the proper context.
Local tests
Another kind of tests in Android is when we run pure Java tests. In Android, we don’t actually execute any Java bytecode. Java is a language that compiles to an intermediate instruction that is understood by the runtime, in this case, the Java Virtual Machine. Android is similar but different. It has its own virtual machine called Dalvik or ART (Android Runtime) that understands a different bytecode.
The fact that both virtual machines understand different bytecodes makes things a bit more complicated. Sometimes we want to run tests on the JVM (Java Virtual Machine) because it is faster than deploying two whole APKs and let the system start a context for each. Running the tests on the same JVM that our development environment usually makes things orders of magnitude faster than instrumented tests.
In order to do that we can try to make our classes free of any Android dependencies using things like MVP or other architectures. This usually boils down to a great deal of bureaucracy in code. Having several interfaces and classes that its sole purpose are to delegate implementations (and quite often only to delegate to test implementations during testing). When we have classes that are not dependent on Android then we can test it on a JVM without problems.
But what happens if we do have a dependency and it gets called? You will receive a RuntimeException with a message “Stub!”. So, dead end?
Not really! The alternative is to make the bytecode compatible! How? Well, let’s re-implement the whole Android stack for our tests :) Actually, we can skip some parts and just propagate others to real Java counter parts. The fact is: there is already a library that pursuits just that. Robolectric is your friend.
So, when using Robolectric we can run tests locally on the JVM as if they were normal Java code being tested. In the end, inside your application, the virtual machine will be a different one and this is crucial. Don’t forget that! Things might go wrong in your JVM failing tests that simply won’t fail on real Android. Things like cryptography are really different amongst platforms.
Which test should I use?
Well, that is the golden question right? As is the case with those: there is no right answer. Normally, instrumentation tests are focused on user interaction assertions. That is, you send commands as if you were performing what a real user of the application would try. Then you make assertions based on the results.
On the other hand, JVM tests are focused on method invocation assertions. That is, you execute a given method with given inputs and assert on outputs. You don’t need the whole lifecycle of things or handling of UX (user experience) events. You just shoot with several inputs and expect some outputs.
As stated, the first approach is better suited for interface testing. What happens to the UX if some button is clicked or if some list is scrolled? That is normally what you want to test with instrumentation.
General Java tests want to find bad logic in your method implementations. This is often logic bound. You want to check if your interests calculation is taking care of all corner cases.
Normally, Android applications should not carry much logic in them. They are the front end of some service. Examples are: Gmail, Facebook, Spotify and etc. They have a lot of code to deal with UX and not that much to deal with business logic. That is because business is handled in the backend, not the frontend. So, generally, there should be more instrumentation tests than JVM tests.
That is not to say that instrumentation tests are always preferable. They do carry some issues with them. For instances: they are hard to make non-flaky. Instability is a general concern with them as they can have memory issues, external apps dependencies and so on. In the past, this made them almost impossible to have. Current tools like Espresso, TestButler from LinkedIn and the InstrumentationTestRunner make them a lot more reliable. Though not as much as JVM tests.
But I’ve heard about BDD and things like specification by example. Aren’t they the same thing?
Absolutely not! They are also crucial in the quality gateway of a complete product. First, let’s make it clear what we are refering to:
One thing is unit testing where you nail down all possibilities of a single unit of your code or your UI. This is normally the target of our instrumentation and local tests in Android. We don’t normally test a whole scenario just to be sure that clicking a button gives us the expected message. We can simply start the screen we want (Activity, Fragment or View) and perform the action directly without a scenario context.
That is not the way BDD or SBE usually works. They have a different target that is the user story. You normally have a complete end-to-end scenario that works just the same as the user would do in the application. What is really nice about them is that focus on a user story allows them to be cross channel. That means: probably the same story should be used to test iOS, Android and Web applications. When something goes badly the message should be consistent amongst platforms. We can’t do that with instrumentation or local tests in Android.
The focus of BDD and SBE are the end-to-end insurance of quality. They do not use mocks for anything. They traverse the app as a user of the system will do just the same and naturally will hit your backend just the same. Therefore, normally, they demand some environment in which they can buy things or whatever transaction your app does.
Ok, now a silver question (golden has been taken already): should I write tests if we have BDD or SBE?
You probably already know the answer. Of course we must write tests! As stated on the last point, tests we write are different than BDD or SBE. We are in full control of a unit of code or interaction. That means you can grab objects and ensure they are in the state you want them to be. That means you don’t need an API for ensuring your code will handle failure of external dependencies in a nice way. You SHOULD mock the APIs you are using to ensure you are testing your code.
A downside of BDD and SBE is that they take a lot longer to execute than instrumentation tests. When the suite is somewhat big, it is normal for these tests to take hours to execute.
What is important here is building several layers of quality. Mobile developers make tests for their code. Quality assurance developers make tests for the product user stories. There should be other layers like performance testing, security assessment and so on.