An Introduction to Continuous Integration & GitHub Actions for Julia

Published in

Analytics Vidhya

9 min readJan 27, 2020

The Problem

In open source development, as developers are creating parallel branches of their own work and only merging them when they are complete, the code on an unmerged copied branch will differ more and more from that of the original master branch. The imminent discrepancies between the branches can bring us many challenges, namely introducing bugs, creating merge conflicts, or even bringing the whole codebase to an outright stoppage, making the whole process of developing and releasing software an extremely challenging one.

Developers need to test the commits to ensure that they work properly and efficiently without any bugs, don’t break any packages, and that they run every single platform the project supports across the board. It’s therefore very likely to take a very long time, become very tedious and that not all aspects of the project is tested, giving us a very low test coverage, and could also cause problems in collaboration.

Subsequently, the synchronization between the developers will be out of track very soon. So, to develop it in a more efficient and consistent way, developers have created some strategies to make integration very simple and repeatable.

What is CI and Why

Continuous Integration, CI for short, is a workflow strategy that ensures all the modifications made are merged with the latest version of the master branch very frequently for all developers in a team, in open source development. An example of a service providing CI is GitHub, where each developer can create a fork (an independent copy) of the repository to make a Pull Request to ask for their work (commits) to be merged, reviewed with comments and suggestions, or closed. This not only makes everyone’s life easier to debug the bugs earlier in the process and reduces the possibility of having merge conflicts (competing changes in the code where Git cannot resolve), but also enhances the communication between developers to address problems. Here, we encourage each developer to integrate their code with the main branch more frequently instead of working on some features in isolation for long periods of time before integrating them. Essentially, this allows everyone to get the latest version of the project in more frequent intervals, and thus reduces the chance of conflict and the time it takes for it to get reviewed. Furthermore, it would be beneficial for everyone to test every small change they make to reduce the risk of error and catch potential bugs early on so that it won’t affect the branches made later on, as well as having automatic testing across multiple platforms to catch unexpected issues on platforms not tested earlier and create instructions for specific platforms, to increase the test coverage by having it automated, of which some of them could be missed by testing it manually, and to remove the hassle of having to test the programming manually every time, which is sorted out by Continuous Deployment. Every change will push a notification to the team which allows it to be then both cross-checked by other developers easily whether they want to leave comments or suggestions.

In most situations, Cl is practiced by a team simultaneously with automated testing using a dedicated server or CI service. Here, the code will be automatically built and tested by the server to check whether it both works and can be integrated with the code from the main branch. Then, an output with the results of the build will be given by the server containing all the specifics as to whether or not the branch passes all the requirements for integration into the main development branch.

A Peek on CI/CD

As CI gives us all the details about the build and test information for every commit made on every branch, it helps us accomodate for what is known as Continuous Delivery, the practice of developing software in such a way that you could release it at any time. Continuous Deployment is an extension of continuous delivery and allows you to deploy newly developed features into production with confidence. A place where you can learn more about CD for free is GitHub Actions, which has interactive courses for anyone to learn more about the automation of workflows and testing on GitHub repositories, from which you can build, test and deploy your code.

Image 1: Wikimedia Commons; licensed under CC A-SA 3.0 Unported

CI on GitHub!

What developers do for integration is that they make changes locally and push those changes to GitHub, known as commits, and CI’s role here is to use automated testing to determine whether or not they will integrate with the latest master branch. First, when an activity occurs (eg. when someone opens a pull request), CI servers will parse the message sent by Webhooks on GitHub and grab the latest copy of the branch so it can build it and run the tests. When that’s done, GitHub uses the information sent back from the CI server to display the checks information in the pull request, along with links that you can click on to go back for more details, like the image below:

Image 2: Screenshot from Pull Request #977 on FluxML/Flux.jl on GitHub

This gives everyone, whether they are a contributor, member, or owner of a repository, to have a clearer idea of which changes can be integrated to the master branch and which ones might not (or need more changes). In a basic CI/CD setup, we can create a system that allows every new commit on the master branch to be deployed on the master branch by the CI provider, such as GitHub Actions. Check this out for more: https://lab.github.com/github/hello-github-actions!

Why CI/CD is important for Julia

Now that we all know that Continuous Integration is a great way for everyone to manage a workflow of a project involving multiple people working on different aspects of it at the same time, we can talk about why this is a big deal for Julia package developers. This comes especially handy when it comes to developing packages in the Julia ecosystem. You may find CI/CD perhaps being used more often by Julia package developers compared to other programming languages, as Julia has a very extensive collection of packages, and thus it makes it very hard to manage, test, and review such a large number of packages being merged into the ecosystem every time. In addition to the fact that most packages in Julia have a lot of code being reused and composed, there is more of a risk of having conflicts in the code, and thus could break dependent packages, or worse, the whole project! As this is such an important aspect of package developing in the Julia programming language, developers have created bots to help automatic this process and make testing easier. An example of this is NewPkgEval.jl, which evaluates the packages to determine whether the commits to Julia would break the ecosystem itself or not. There’s also a bot called Registrator.jl that automates the creation of registration pull requests for julia packages to the general registry.

Setting up a basic CI script for a Julia repo using GitHub Actions

Assuming that you have a repository with Julia programs that you would like to run, you can use GitHub Actions (as mentioned before) to help automate your CI workflows and deploy the code straight from GitHub. Today we won’t be delving into too much detail but instead we’ll just go through the basic to get you up and running. One of the essentials of having a CI workflow or process in Julia is this GitHub Action called “setup-julia” that helps set up a Julia environment for use in actions by downloading a specific version of Julia and adding it to PATH so that the script can be run easily by just calling julia.

If you head over to your repo, you should be able to see a tab that is called “Actions”. Click on that and select the option “Set up a workflow yourself” on your right.

What this does is that it creates a .yml file inside a workflows folder inside a .github folder. By default, it is called main.yml but feel free to change the name of it while keeping the .yml extension at the end.

When that is done, you may create another folder called src to place everything else in your repo (excluding the new .github folder) inside it. It is not necessary to do so but it might make the repo more organized, especially if you have a lot.

You can now return to the edit the .yml file. First, you can change the name: section to the name you would like to call your script. Afterwards, we can setup the schedule for deployment. Replace the on: section with the following code:

on:
  schedule:
  - cron: "*/8 * * * *"

The example shown above allows the file to be run every 8 minutes. Feel free to change the number to an arbitrary one. Click here to learn more about how you can customize the cron:.

Now let’s move on and modify the jobs section used to run the code. Paste the following code as a template and we will modify it afterwards:

jobs:
  build:    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v1
    - uses: julia-actions/setup-julia@v1
      with:
          version: '1.0.4'
    - uses: julia-actions/julia-buildpkg@master
    - run: julia --project file.jl

The runs-on value here specifies the OS the Action will be running on. In this example, ubuntu-latest was used. You can change the runner to other operating systems such as Windows and macOS. Click here for more information. Then, you can change your Julia version by changing '1.0.4' with the version you would like to use, eg. '1.2.1'. At the end, remember to change file.jl with the path to the program that will be initiating this workflow. An example might be main.jl or src/main.jl if you created a src file.

The Julia on the machine we’ll be using will not have all the packages installed. So, please add the following 2 lines to the top of your program above using <Package> to install the packages we need (feel free to skip if no packages are used!):

import Pkg
Pkg.add(["package1", "package2", "package3"]) #list of package names

Here’s an example:

import Pkg
Pkg.add(["OAuth", "HTTP", "JSON"])
using OAuth, HTTP, JSON

Well done! We have successfully created an automation for our program! Feel free to have a look at this example to see how this is done.

Before you leave…

I just want to say that I highly recommend anyone who is looking to join the software development industry to have a peek at how CI/CD works, if you haven’t done so already. It’s not actually a very complex idea: you just have to get your head around it. I know how annoying it sounds at first glance to have to merge your commits several times a day, maybe because you can’t be bothered, or think that you time can be used more efficiently on working on the project itself, but just try it: this is part of the project as well! It will instantly change the work environment into a very productive one. Thanks for reading!

References: