Smart build android modules on CircleCI

Continuous integration (CI) proven to be a crucial part in software development. It helps automate many processes while iterating your software and make sure everything is working properly. If you do not know about it you can read it here.

In another hand, long running on the CI server does slow down the software development process. In this article I will show one of the way my team use to decrease the building time of CircleCI in Android modules.

What are benefits of continuous integration?

With the help of continuous integration, we can achieve:

1. Making sure the source code can be built, so that everyone in the team can build it on his/her local machine
2. Making sure all the tests and quality checking of the code base is green
3. Automating deployment process of the application
4. The CI servers ofter base on containers technology → they are really reliable in testing and building

What are disadvantages of continuous integration?

Beside advantages, I found some disadvantages from my experience:

1. Long running CI slows down your software development. For example: we have to wait for CI to pass before doing any merging — this may take hours
2. Building our own CI needs significant amount of resource
3. Depending on other CI providers sometime slow us down (for example: the CI containers are down)

For the small team I always recommend using CI providers (like CircleCI) instead of trying to build your own CI containers.

The problem

I am currently working at Sentio — the company try to make you productive on the phone by transforming your phone into a laptop.

We have many repos in Github, one of them contains source code of all the applications like Calculator, Browser, FileExplorer, etc. The next requirement is to separate Browser and FileExplorer so that we can build them as standalone Android apk. We have to choose between two option:

1. Separate them into their own repo
2. Put them into individual module within the same repo

Because the size of the team is small, we decided to go with option (2). Using option (2) also help us reuse a significant amount of shared codes between modules. In another hand, creating many modules will slow down the CI build time.

The solution

After doing some discussion our team decided to implement some scripts to tell CircleCI what modules to build and test with the help of git diff command

We choose python to implement all the scripts due to its simplicity. I will use an example to show you how it works. You can find the source code here. In the example we have three modules:

1. Browser (Android module)
2. FileExplorer (Android module)
3. Data (Android library module)
4. Browser and FileExplorer will depend on Data

The idea is: whenever a build is about to start we will run gif diff command to get all the file changes from the current commit comparing with the latest commit on develop branch:

1. If the changes contain only files in the Browser or FileExplorer we gonna build and test on that module only
2. If the changes contain files in the Data module, we build everything
3. Otherwise, we build everything too

Starting with the function to detect module changes

def detect_changed_modules(compare_branch):
    result = set()
    all_modules = get_modules("../settings.gradle")
    for dir in directory_changes(compare_branch):
        print dir
        if dir.startswith("Data/"):
            print "Data changing rebuild all"
            return all_modules
        for module in all_modules:
            if dir.startswith("{0}/".format(module)):
                result.add(module)
                break
    if len(result) == 0:
        result = set(all_modules)
    return list(result)

• get_modules: is a function that parse settings.gradle to get all the modules (without Data module)
• directory_changes: is a function that get all the file changes (only get the name of the file, we don’t care about what changes) in the current commit compare to another branch (develop) — a note in here is we only care about what new changes in the current commit compare to develop no new changes from develop

Next we have a function to run a command and stream the result to command line. This function also returns a result code (zero means success)

def run_command_on_subprocess(command):
    process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    for line in iter(process.stdout.readline, ''):
        print line
    return process.poll()

Finally we have the build.py script:

from utils import *
import sys


def run_command(cmd):
    print "Running command {0}".format(cmd)
    return run_command_on_subprocess(cmd)


def main():
    modules = detect_changed_modules("origin/develop")
    print "Building on these modules {0}".format(modules)
    for module in modules:
        result = run_command("./gradlew :{0}:check".format(module))
        if result != 0: sys.exit(result)
        result = run_command("cp -r ${{HOME}}/${{CIRCLE_PROJECT_REPONAME}}/{0}/build/reports $CIRCLE_ARTIFACTS".format(module))
        if result != 0: sys.exit(result)


if __name__ == "__main__": main()

And the circle.yml file:

dependencies:
   pre:
     - if [ ! -e /usr/local/android-sdk-linux/build-tools/25.0.3 ]; then echo y | android update sdk --all --no-ui --filter "build-tools-25.0.3"; fi;
     - if [ ! -e /usr/local/android-sdk-linux/platforms/android-25 ]; then echo y | android update sdk --all --no-ui --filter "android-25"; fi;
     - if ! $(grep -q "Revision=43.0.0" /usr/local/android-sdk-linux/extras/android/m2repository/source.properties); then echo y | android update sdk --all --no-ui --filter "extra-android-m2repository"; fi;
   cache_directories:
     - /usr/local/android-sdk-linux/build-tools/25.0.3
     - /usr/local/android-sdk-linux/platforms/android-25
     - /usr/local/android-sdk-linux/extras/android/m2repository

machine:
  java:
    version: oraclejdk8
  python:
    version: 2.7.10

test:
  override:
    - python ./script/build.py
    - if [ $CIRCLE_BRANCH = 'master' ]; then ./gradlew assembleRelease; fi
    - if [ $CIRCLE_BRANCH = 'master' ]; then cp -r ${HOME}/${CIRCLE_PROJECT_REPONAME}/Browser/build/outputs/apk $CIRCLE_ARTIFACTS; fi
    - if [ $CIRCLE_BRANCH = 'master' ]; then cp -r ${HOME}/${CIRCLE_PROJECT_REPONAME}/FileExplorer/build/outputs/apk $CIRCLE_ARTIFACTS; fi
  post:
    # - bash <(curl -s https://codecov.io/bash) -t [YOUR_KEY]

deployment:
    production:
        branch: master
        commands:
            # - ./gradlew githubRelease --info

The result

With the help of the python scripts building the Browser only take about 5 minutes (for building and testing):

In the other hand, building the whole repo may take up to 11 minutes:

I have created a sample repo containing all the python scripts here, checkout if you want to see the code.

Thank you for spending time reading and happy coding!!!