Week 6/7: Testing 1, 2, 3
These weeks were focused on testing our MVP and making sure it’s ready to deploy to the app store on November 16th.
Here’s what we did:
- Added instrumentation tests with Espresso (Monsurat)
- Static code analysis with Lint (before & after changes) (Rebecca)
- Usability Testing (Anna)
- Added Fabric Crashlytics for crash management (Rebecca)
Instrumentation Tests
Starting out we had 0 tests. Now we have tests grouped by Activity with a total of 9 tests! It was really fun writing tests to interact with the app and to watch the emulator run the tests live as if someone was actually interacting with it.
Some of the tests included:
- does selecting a site toggle the check mark overlay?
- does pressing the action button navigates to the map screen?
- does clicking the fab button on a site card start the navigation?
- can you scroll through all the cards?
Writing instrumentation tests for GridViews and RecyclerViews was hard. It was almost like learning a new language with a bunch of restrictions on what you could and couldn’t do. I used this, this, and this to get a better understanding of how to use Espresso. For the RecyclerView I had to write my own matcher in Espresso because you can create whatever view you want in a RecyclerView and because it’s meant to be flexible there isn’t a way to make a standardized matcher for it like there is for ListView. With great personalization comes great responsibility to test properly! Of course with some help from stack overflow:
One weird thing I ran into was that after deleting an old test and running all the remaining tests Android Studio would still try to run the test I had already deleted. Turning my green all tests passed status to red for a test that didn’t even exist anymore! To fix this I had to delete my old test configuration and make a new one, which was kind of annoying. Hopefully there’s a better way and I just don’t know it yet.
Another problematic thing I noticed was when I added a test to test if scrolling was working on the main screen where you select sites to visit (there is a known bug on this screen that prevents the user from scrolling back to the beginning) the test never terminates. Espresso just hangs on that test forever without accepting or rejecting because my test asks it to try and click a card that it can’t get to after scrolling to the end of the list due to the bug. I would think that there would be a timeout threshold where we can consider the test failed but that didn’t happen here and I had to manually terminate it. This has some interesting implications. Maybe there is a feature that allows me to time box my tests. If there isn’t that would be a nice feature to add.
Overall, writing instrumentation tests was interesting and informative. While there are some things in our app we couldn’t test with espresso like “is the image from this url the image displayed in the app” what we were able to test gives us a good foundation for getting quickly notified when a code change breaks some of the UI functionality in our app.
Static Code Analysis
We had some struggles using lint. The build would fail with the error “Could not find com.android.tools.build:gradle:3.0.0.”
After some Googling, I found this thread on StackOverflow, which led me to a solution — I needed to add some things to gradle:
buildscript {
repositories {
...
// You need to add the following repository to download the
// new plugin.
google()
}
dependencies {
classpath 'com.android.tools.build:gradle:3.0.0'
}
}Finally, lint ran and output a beautiful HTML file riddled with things we need to fix. A sample for your perusal is included below:
We only had one error (in a file we weren’t using), which is great news! I deleted the file and addressed all the warnings I could.
We got a clean bill of health on the second pass.
Usability Testing
The original plan was to do usability testing with three people. I had problems running our MVP on my computer so we had to do a little compromise and just do one usability test. Granted it was only one, we found out a lot about our usability.
I gave our tester minimal information about our app. I told them it was a historical walking tour app for the neighborhood surrounding Tufts and asked them to use our app. I asked our user to talk aloud while navigating the app. Let’s walk through their comments on each of the navigations.
Screen 1 — Site Selection
On our landing page, the tester asked “What is this?” Since there is no home page or instructions on the first page, they had to guess to figure out what this page was. The heading was also irrelevant and didn’t help the user figure out that this was a page to select the sites to visit. Though the user had to guess to find out what the purpose was, the icons were clear enough that they eventually figured out they had selected some sites to visit. As Ming said when we presented our MVP, the app requires an on boarding experience.
Screen 2 — Site Map
This page really confused our user. They said that they were unsure how the route was selected, and how pressing go on the cards at the bottom related to the pre-selected route. They also said that the heading, “Start Navigation” was unclear and that they would have preferred something more informative about what’s displayed on the screen.
Screen 3 — Navigation
We conducted our usability testing on Android Developer Studio, so we acknowledge that we didn’t actually test the usability of this app while walking around the city. Hopefully, we will be able to conduct these studies in the future.
Screen 4 — Site Description
For this page, I interjected to turn the volume up on the computer. That disrupted our findings as I don’t know if he would have known that they need to check the sound if they don’t want to read it. After pressing the play button and pausing, they commented that they were unsure what happens if they finish hearing the entire site description. “Would it repeat the text again? Would it redirect to a new page?” Based on this, I think that a replay button may be helpful to distinguish between pausing in the middle of the description and reaching the end.
Crash Management (Fabric Crashlytics)
Installing Crashlytics seemed like it would be easy. I signed up, installed the plugin, and followed the instructions, but I got stuck on an onboarding screen. Even when I went to the website, all pages redirected to onboarding. Crashlytics just wouldn’t recognize that all the required code had been inserted.
I restarted Android Studio a couple of times, but that didn’t help. After a lot of cursing at Fabric’s UX team, I restarted my computer and suddenly Fabric recognized the app. I could see the dashboard and everything.
So frustrating! I expected more of you, Google.
I added some test crashes to our first activity to make sure that Crashlytics was tracking them, and it worked.
The next step? Figuring out why our second activity was crashing for real.
Additional Bugs Found
Our second activity, a beautiful map of selected locations, crashed on launch repeatedly. The cause seemed to be telemetry related to mapbox:
11–06 21:53:30.016 13220–13220/org.medfordhistorical.overthemystic V/MapboxTelemetry: Initializing telemetry.
11–06 21:53:30.042 13220–13220/org.medfordhistorical.overthemystic V/MapboxTelemetry: Enabling telemetry.
11–06 21:53:30.048 13220–13220/org.medfordhistorical.overthemystic I/art: Rejecting re-init on previously-failed class java.lang.Class<com.mapbox.services.android.telemetry.location.GoogleLocationEngine>
11–06 21:53:30.049 13220–13220/org.medfordhistorical.overthemystic I/art: Rejecting re-init on previously-failed class java.lang.Class<com.mapbox.services.android.telemetry.location.GoogleLocationEngine>
11–06 21:53:30.049 13220–13220/org.medfordhistorical.overthemystic I/art: Rejecting re-init on previously-failed class java.lang.Class<com.mapbox.services.android.telemetry.location.GoogleLocationEngine>
11–06 21:53:30.050 13220–13220/org.medfordhistorical.overthemystic A/art: art/runtime/thread.cc:1321] No pending exception expected: java.lang.NoClassDefFoundError: com.mapbox.services.android.telemetry.location.GoogleLocationEngine
Crashlytics didn’t recognize the crash, so we were on our own for debugging. Upon further research there seems to be some issues with a few of Mapbox’s SDK’s we’re using and other people are experiencing some of the errors we get while troubleshooting as well. Just from last week:
Our current approach to fixing this is looking for a workaround and keeping close tabs on the nightly builds and new releases from Mapbox that might contain a fix. i.e. https://github.com/mapbox/mapbox-gl-native/issues/10379
