The Art of Android DevOps

The DevOps movement has been making quite a buzz in the development community in the last few years — and even though it is usually mentioned in the context of web development — it is not the only thing that can benefit from it. Let us take a look at what DevOps is, what it means for mobile developers and how we can use it to deliver better Android applications.

DevOps (short for Developer and Operations) is a term coined to describe the approach of using tools, metrics and processes to help your team deliver high quality experiences — using automation to minimise mistakes and maximise efficiency in product delivery. It is focused on enabling developers to use and manage software operations processes and infrastructure without needing a separate operations team. “DevOps culture” is here to make your lives easier and ensure that your users get the highest quality product in the users’ hands and speed up your development process.

Here at Undabot, our mobile DevOps practice evolved as a result of natural needs and it consists of four parts:

Continuous Integration

Continuous integration enables us to collaborate on the same codebase and to be confident that integrating our changes and modifying the codebase did not break the build. There is no more checking out a branch and building it locally just to confirm it works, or to see if the latest merge/rebase broke the development branch, just look at the branch status on the CI and you know what is going on.

Automated testing

The best way to ensure your code works is to test it — and I do not mean manually tapping every button in your app — but writing tests. Running tests on each check-in into the codebase enables us to detect errors early and gives us confidence that the check-in did not break any parts of the system.

Code quality checks

Keeping the codebase healthy means keeping the code clean of pesky bugs, unnecessary complexity and keeping technical debt at minimum. We can achieve that by using static analysis tools that can help detect possible bugs before they are pushed to production and by tracking metrics that let us know the current state of the codebase to see our progress.

Continuous deployment

Deployment for Android is bothersome — change to proper flavours, up the version, push, wait for the build, download the APK, send it via email/upload it to your beta deployment tool/upload to app store, write the change log. And there is always the bus factor and horror stories of lost keystores, deploying wrong versions or/and test flavours. Continuous deployment enables us to always deploy the proper versions to the testers, clients or users without having to worry about it.

So, how do we achieve all of this?

Continuous Integration:

There are many options for setting up the CI server. From setting up your own by using the feature rich and powerful Jenkins to using one of the many services that offer a “click-and-go” type of CI. Here at Undabot, our weapon of choice is Gitlab CI. I am going to skip this part since my colleague Renato already wrote an awesome post about setting up a CI using Gitlab! Check it out here: Working with YAML in Gitlab CI from Android perspective

Automated testing

Unit Testing is the basis of all good software (ask Uncle Bob if you don’t believe me!) since it gives you insight exactly into which piece of your code works and which one is broken. Running unit tests on your CI is really simple, but there is one important metric we can gather here — test coverage. Luckily, gradle comes with an already built-in JaCoCo test coverage plugin to which we have added the JaCoCo Android Plugin. It is really easy to enable — just add it to your gradle file and choose which type of reports you would like to have:

And that’s it! Running jacocoTestReport task will now generate code coverage reports for your project.

Another metric we can grab here is running mutation tests. Mutation tests are “tests for your tests” that introduce multiple changes to your bytecode (so called “mutations”) and run the tests against those mutations. If a mutation survives the test, it means your test is flaky and should be improved. To run mutation tests, you can use PIT tool (if you wonder what PIT stands for — it stands for PIT. Just another recursion joke :) ). For Android, PIT is really easy to set up. Just use the Gradle plugin:

After you’ve setup, you can run a pitest task to start your mutation tests.

Note: Depending on the size of your test suite, mutation testing can take a really long time, so it may not be the best thing to incorporate in your regular CI pipeline. I would suggest running it once in a while as a manual step.

Instrumentation tests:

Instrumentation tests are tests that are run on Android devices so they can access the Android framework API. They are usually slower than unit testing and require devices to be ran, usually an emulator or a device farm — be it in the cloud or your own. There are many services providing cloud device farms but if you have got enough devices in your office, you might look into building your own with the help of Open STF | Smartphone Test Farm

To run instrumentation tests, we start an emulator and then use the scripts from our Gitlab CI blog post to wait for them to start up and kill them after the test are run.

Code quality checks

To ensure the quality of our code, we use static analysis tools that help us catch bugs before they are checked into the codebase. Setting them up is pretty easy, since they all have gradle plugins:

Findbugs

One of the most popular static analysis tool for Java. Unfortunately, it stopped being maintained so we are looking to replace it with its successor called SpotBugs. Setting it up is quite easy:

To increase the amount of issues FindBugs can discover, I suggest you add these plugins to your findbugs setup. They expand your rule set quite a lot:

fb-contrib™: A FindBugs™ auxiliary detector plugin

Find Security Bugs

PMD

Another popular source code analyser. It has some overlap with findbugs, but also covers some issues which findbugs does not. Integration is also really easy and similar to findbugs:

Checkstyle

One of the most important part of keeping your codebase neat and tidy is making sure everyone can read it and reducing time spent on resolving git conflicts which occur due to different styling and formatting between developers. Having an agreed upon code style makes it easier for everyone to navigate your codebase, to do code reviews and to check-in readable code. Checkstyle helps you by reporting code style violations and you can set it up to follow your code style:

Android Lint

Android Lint is another code scanning tool that does not just detect your usual Java problems and bugs, but also Android-specific ones — from internalisation to common mistakes, it is pretty important for us as Android developers — the best advantage it has is that a lot if Android Lint rules are integrated with IntelliJ lint, so you can see warnings while you write the code. It also supports setting up a baseline for your warnings, which means you can set it to report only new issues added to the codebase.

Except these, there are a few more non-standard ones that you could look into.

• ErrorProne
  This one actually falls into the build step of your CI, since it breaks the build if it discovers issues.
  ErrorProne is a tool by Google that hooks into your Java compilation step and throws warnings and errors for common issues, while also giving you fix suggestions. But its biggest advantage is its extendability and its Patch and Refaster tools. Patch automatically fixes all of the issues Errorprone found with its suggested fixes, while Refaster is a refactoring tool which allows you to quickly refactor all kinds of expressions and code blocks. This can be quite useful if you are planning on deprecating an API.
• Ktlint
  Ktlint is a linter tool for kotlin that is pretty similar to gofmt tool from the world of golang. It finds style violations and formats your code according to the official code style from kotlinlang.org. If you use Kotlin, definitely look it up.
• Detekt
  Detekt is a static analysis tool for kotlin but with an added complexity analysis. This allows you to not just find which classes have code smells and issues, but also to find out which are the most complex classes in your codebase that can point out possible system design problems.

So now that we have added these, what happens? In an ideal world — we get reports with a dozen bugs, read them and fix the common issues. But, in the real world, there is a legacy code to cover, new features to build and rarely enough time to spend on ensuring that the tools we are using are completely satisfactory, so developers mostly just set them to report issues as warnings so they do not break the builds and end up forgetting about them as time passes.

That does not satisfy us — so we decided to solve that problem. Luckily, besides HTML reports, all of the tools are also able to create XML reports, which means they can be easily parsed. So we decided to create a small script which copies the old reports on our gitlab cache (its per-branch) and then runs our quality check task for the new code. Then, we wrote a small tool that parses and diffs the old reports and the new ones, throwing an exception if the diff tool finds new issues that did not previously exist and generating small human-readable reports pointing to the newly discovered issues. That enables us to know if there have been any new issues that should be fixed.

Deployment

Submitting manually is stressful — upload a wrong APK to the App Store and it will take days until your fix reaches the users which can mean losing customers, money, releasing features that were not supposed to be released and more. Send the QA or the client a wrong flavour or build and you will waste time in the feedback loop trying to find out what is going on. Automating that process lets us reduce stress and amount of work to be done and increases bus factor for people who can build and deploy the app to the App Store. Here at Undabot we decided to use the fast lane for deployment. fastlane is a ruby-based tool for automatic deployment which enables you to write scripts that enable you not just to deploy, but also update versioning, take screenshots, update change logs and more.

Find out how to setup Fastlane in the official guide

Our fastlane deployment script is based on one important method called `pipeline`. It is a single method with multiple arguments, allowing us to easily add new actions to all of the lanes or just to one. Our pipeline method is quite generified so it can easily be modified on a per-project basis. So let’s take a look at how an internal release method looks and take a dive inside.

Let us go over it, line by line: Name tells us what this lane will do — deploy an internal release test version Pipeline is the method that checks the flags and invokes proper methods

qa: true - Tells fastlane if it should perform quality checks task (the same one we use for tracking our app’s quality)

unit_tests: true - Tells fastlane if it should run the unit testing task bump_version: true - Reads the versioning file and bumps the version

badge: true - To make QA testing easier, we have an overlay over the app icon with the app version info. This tells fastlane to run the task that overlays the app icon with the version badge.

build_type: 'release' - Tells fastlane which version of the app to build app_icon: production_icon - To differentiate from debug and release versions, we use different icons. The argument is a predefined variable which is a path to the icon it should apply, for example:
production_icon = "/**/src/main/**/app_icon.{png,PNG}"

As we use Crashlytics to deploy test versions, we use this flag to upload the APK to crashlytics and tell it which group to deploy to.

crashlytics_deploy: true,
crashlytics_groups: ["devs-group"],

vcs: false - When working with release versions, some things need to be committed (like the new version number). This tells fast lane to do a new commit after deploying.

And of course, we notify the team via Slack that the build is done. To do that, we use these flags:

slack_icon: "https://undabot.com/super_secret_app.png",
slack: true,
channel: "secret-android-ninja-turtles"

Now, let us dive into the method itself - first we check if our git status is clean, checkout the whole branch and then check if it is the branch we wanted - this prevents us from deploying the wrong branches to the wrong teams:

After we are sure we have checked out the correct branch, we can run the gradle tasks we want — we do this by checking if the option for the task is set to true and running it with gradle fastlane action, for example:

gradle(task: "test") if options[:unit_tests]

After we are done with all the tests and the QA checks, we clear the build directory by running gradle(task: 'clean')and we’re ready to build our app.

First things first, we check if we need to bump the version — if we do, we do it by using one of the predefined tasks from our gradle versioning script that reads the version file and writes them back into the file depending on the bump type. For example, here we have predefined build types for bumping build and patch versions:

Next up, we add the badge to the app icon using the badge - fastlane docs badge action and after the build is done, we run revert_original_icons to revert it back (we might want to commit the version bump, but we do not want to commit the app icon)

Finally, we can build the app by running the gradle task with the build variant and flavour we want:

gradle(
      task:'assemble',
      flavor: flavor,
      build_type: build_type
  )

And we are ready to deploy it to our crashlytics beta group or play store:

To upload to Play store check the Fastlane docs on supply to see how to setup your supply task.

The only thing left to do is commit the changes (we bumped the version, remember?) and notify the team that the release is done!

How does it all tie together?

To tie all of these tasks together, we have to add them to our CI pipeline (for us that is Gitlab CI). Let us take the YAML script we used in our previous post and edit it a bit.

Instead of just running unit tests, we will replace the script part of unit_tests task to run JaCoCo instead of - ./gradlew test so it looks like this:

unit_tests:
  stage: test
  script:
  - ./gradlew jacocoTestReport

Now let us rework the static_analysis task - first, we need to enable per-branch caching so we can compare our old results to our new ones. We can do this by adding just a few lines to our script with our cache key and path to files we are caching:

cache:
    key: "$CI_COMMIT_REF_NAME_qa"
    paths:
      - app/build/reports

Our script for this task will be the gradle task that backs up old reports, runs the tools and then generates the report diff:

script:
    - ./gradlew backupOldCodeQualityResultsAndRunChecks

NOTE: The “$CI_COMMIT_REF_NAME” is GitLab’s environment variable which is the branch or tag name.

To deploy, we just add a task that calls the lane we want, in our case, let us use our internal release test lane again. To ensure it is deployed only from master and development branches we will add their names under the only tag and set this only to manual deployment since we do not want to have this version built automatically on every push:

And we are all done! Now, deploying to any branch will start the set of checks, tests and quality assurance profiles, with deployment available on your master or development branches.

The end?

As DevOps is more of a culture than a set of things you check off the list and forget about, our process continuously keeps evolving and updating. Since the introduction of Kotlin, our analysis toolset will need to be expanded with ktlint and detekt to cover the kotlin part of our codebase. Since we would love to have a good overview of the state of our codebase at all times, we are also looking at introducing SonarQube to replace our riffing tool and help us visualise the complexities and bugs. As time passes, our filters are getting refined and we are adding more content to them. The important thing to keep in mind here is to keep improving and adapting the tools to your needs and making the life of your team just a bit easier.

Thanks to Sinisa Cvahte for the design.

Thank you for reading. Please comment, like or share it with your friends and we hope to see you soon.

Would you like to join us? Check out the open positions at our Careers page.

Undabot and Trikoder are partner organisations. We analyze, strategize, design, code and develop native mobile apps and complex web systems.