How Github actions supercharged our QA pipelines

At DocPlanner our mobile applications are built using React Native, this is a brilliant framework if you want to quickly build a native application for many platforms using the famous React.js provided by Facebook. And the developers who primarily focus on the mobile application are front end developers who normally write applications using JavaScript and web technologies.

With the use of React native, we need to deploy to multiple platforms during the quality assurance(QA) and the production release processes. We need one pipeline for android to build the new application version and then to send the new version to a distribution application called HockeyApp. And another pipeline to test the new version of the iOS application that then gets sent onto the Apple equivalent distribution application called TestFlight.

Originally this was a manual process using a Microsoft product called AppCenter for the iOS build and Microsoft Azure pipelines for the Android build. The way we used AppCenter was quite manual at the time. When building a new QA version of the iOS application, we manually configured the build, set the certificates Apple requires to sign the application and once the pipeline was completed, we downloaded the generated application file (the .ipa) and then uploaded the new version of the application using a tool Apple provide on Mac OS X called the Application loader.

The Application loader was used to release the application to TestFlight, and once done then we told our QA team to install the new version of the application and test the new feature we introduced.

It was a bit of an insane process and got old real quick once we started building more features. Also we had to have meetings to educated new developers on how this process worked.

After a while of repeating this process a few times, we all knew this flow was broken and had to be fixed.

What we wanted to achieve

First we looked into what was working quite well. We noticed that the Android QA pipeline was a lot more smooth, because all that it required from us was to manually trigger the pipeline in Azure pipelines. This showed a lot of promise.

So we went to the drawing board and wrote down the ideal workflow to easily and efficiently release new versions of our application and to keep us sane.

- It was to be automated.

- It was to be fast.

- And would only build once a pull request was approved.

- Or when a pull request was re-approved after some changes.

We wanted to restrict the builds since they were resource expensive and to reduce the amount of noise the QA team would receive in our distribution apps, HockeyApp and TestFlight.

Possible solutions

1. One option was just to trigger the Azure pipelines using the Github integration, this would work well and would be effortless to get working, but it did not have the capability to only run once the pull request was approved. We could have implemented some logic into the pipelines themselves to ensure a certain amount of approvals were met, but this would introduce the need to access the Github API from Azure pipelines and having to manage another access token.

2. Another option was to just trigger the pipelines ourselves manually using the Azure pipeline comment, this looked promising but would involve the manual step of remembering to post the comment once the pull request was approved. We considered this in the early stages of Github actions and looked into a way to automate this comment being posted.

Then we found Github actions.

This looked promising and opened the door to a lot of possibilities. At the time Github actions was in closed beta, this was a concern but we continued to investigate if this was a good solution. We also liked the visual diagram Github actions provided, it helped explain to other developers and people in devOps how our pipeline worked.

What are Github actions?

Github actions is a workflow automation platform built on top of the Github API.

It listens for certain events in Github and triggers a number of docker instances once the event has been fired. The actions are sequential and can return a code to determine if the workflow should continue or if it should stop.

This platform really provides another layer to the Github flow for managing your repositories, for example you could trigger the workflow when a user pushes some code, the workflow could then trigger a task to run `npm build` and then `npm publish`. This does not require another CI server to handle this, instead its completely managed from Github, it also provides the execution of theworkflow in a visual way where you can very quickly find out what went wrong and where in the workflow it happened.

Since an action is built using a Dockerfile and a bash script, the possibilities are endless.

At the time of implementation of our solution, Github actions was in closed beta. We knew this would cause a number of issues along the way and some unexpected results. But we continued to try it out.

How we defined the workflow

With an agreement of the internal mobile improvement team to try out Github actions, we proceeded forward with implementing a basic workflow. The plan was to simply post an Azure pipeline comment once the Github pull request was approved.

We bound the workflow to the on-review event, this would then trigger a shell script to access the Github API to pull in the array of reviews for the current pull request.

With this array we would then parse the reviews and count how many approvals the pull request had. If it had the expected amount, then we would post the comment using the Github API.

This worked well as an initial experiment, but once we realised the the pipeline would not trigger because it was trying to start the pipeline on behalf of the commenter, we moved to accessing the Azure pipeline api directly. The commenter was a bot in this case and could not be granted the access required.

What was nice about using Github actions was they provided an access token out of the box without the worry of storing it, for the Azure pipeline token, we used a new feature called Github Secrets. With Github secrets we could easily access the Azure token inside one of our actions in the Github actions workflow.

While developing the workflow for Github actions, we had to find a way to improve the development process and ideally find a way to test these actions locally on a machine instead of pushing each change to the pull request to test if the update worked the way we intended. After some research we found a very useful tool call Act, https://github.com/nektos/act.

Act is a small utility used to allow you to run your Github actions locally on your machine before pushing the changes to your Github repository. It contains a number of valuable commands to assist in building out your Github actions, such as running a specific action, running a specific Github event or just running a dry run of your workflow. It can also provide a visual representation of your workflow.

Act is available on Mac OS X using brew install.

Azure pipelines did the rest

We took a script from the Azure GitHub repository for their GitHub actions to interact with Azure pipelines, https://github.com/Azure/github-actions/blob/master/pipelines/entrypoint.sh.

This had all we needed to correctly trigger both the Android and the iOS Azure pipelines. Once the script was setup locally in our project we then setup both the Azure actions used to trigger the script.

Our workflow ended up looking like this:

From Azure pipelines we would build both the platforms and then post the build number with the Github branch in Slack.

We have since improved the script to only build the first time a commit has been approved, and then if another commit has been pushed, on the next approval to rebuild both the QA pipelines.

In the distribution applications we also improved the structure of the build numbers and we automated the release notes for each build to contain the name of the branch and a prefix to declare the environment the application was build in.

This was another issue we had since in the distribution application it was very difficult to determine what build was what, we just got a random build number. This has been massively improved since we implemented the new workflow.

Preview of a build in TestFlight, before and after

Findings and Issues along the way

Github have done a great job building this tool, and they were quite helpful and responsive when we ran into problems with Github actions. Kudos to the Github support team :).

The tool **Act** really improved the developer experience with building out this workflow, without it the process would have been more painful and would have taken longer to do. Definitely an essential utility when building these workflows.

We stored all the scripts locally in the mobile repository under the `.github/` folder, the current folder structure looks like:

.github/
   actions/
     approve-pipeline/
       Dockerfile
       entrypoint.sh
     azure-pipeline/
       Dockerfile
       entrypoint.sh

This worked well and provided greater flexibility when building the workflow, what was nice was when making a change to the workflow, the pull request was isolated to the intended changes we were adding.

Although one issue currently with Github actions is that it is currently in closed beta, every now and then we notice the workflow randomly failing. This does not happen often but is annoying when the issue is out of our control. Normally we contact the Github support team and they report that their development team is working on a solution. The good news is we dont entirely rely on Github actions and when it happens we just trigger the QA pipelines manually, but this now rarely happens but its nice to know we have a fallback just incase.

Next steps

Moving forward we would like to enhance the way we interact with the established workflow. We want to add a badge to indicate how many approvals must be passed before triggering the Azure pipelines.

So overall implementing Github actions has been an enjoyable experience that has helped us to fully automate and manage our Android and iOS quality assurance builds.