GitHub pulls its tentacles into CI/CD and artifact management

In May 2019, GitHub announced the release of the Package Registry service. After that, in August, the support for CI/CD in Actions was announced.

In this article, I’ll tell you about these services and will show how can we use it on the example of a small pet project on GitHub.

What’s that?

GitHub Actions is a platform that allows you to manage the software life cycle, for GitHub-based source code. In fact, this is a new competitor for TravisCI, CircleCI, and many other free CI/CD platforms.

GitHub Package Registry is the central repository of artifacts. It supports five different types of artifacts:

• NPM packages (Javascript)
• Docker-images
• Maven artifacts (Java)
• NuGet packages (.NET)
• Ruby-gems (Ruby)

Actually, this is a cool opportunity to have all artifacts in one place, because it is not always desirable to run your own Nexus or Artifactory server. You don’t need to use external integrations for GitHub, now you have all the services in one place. I think it’s extremely convenient, but it’s up to you.

GitHub is becoming more and more similar to GitLab, but somewhere it has already surpassed an opponent. For example, GitLab doesn’t support yet NuGet packages and Ruby gems.

How to try?

Both services are currently in beta, and you can sign up for a beta test on these pages.

Migration from other services is very simple, I migrated several of my pet projects from TravisCI and DockerHub to Actions and Package Registry.

Now I’ll show you how it looks in one example. The project is pretty simple, I wrote about it in that article. Nothing complicated, the usual LaTeX code, with the help of which artifacts are collected (2 PDF files), they are published in GitHub releases. In order not to download a pack of LaTeX packages, I wrote a Dockerfile so that you can conveniently work on any OS.

Package Registry

There are only two simple steps to begin to work with Package Registry instead of DockerHub.

Create token in GitHub settings page with write:packages and read:packages scopes.

GitHub token creation

Then we can log in with this token and push artifacts, extremely simple, isn’t it?:

docker login docker.pkg.github.com --username amet13
docker tag docker-latex:0.0.1 docker.pkg.github.com/amet13/master-thesis/docker-latex:0.0.1
docker push docker.pkg.github.com/amet13/master-thesis/docker-latex:0.0.1

Note: I set my GitHub username in lowercase, otherwise you will get an error from Docker:

Error parsing reference: "docker.pkg.github.com/Amet13/master-thesis/docker-latex:0.0.1" is not a valid repository/tag: invalid reference format: repository name must be lowercase

Here’s what it looks like in the GitHub UI:

Package Registry page

In addition to instructions for downloading the latest image, download statistics are available. You also could download a separate Docker layer via the UI, image download history is also available.

Actions

This is a bit more complicated service, but for those who have ever worked with any other CI/CD system, it won’t be difficult to understand. The configuration in Actions is described in declarative YAML, although HCL was used before.

Some basic concepts:

• Workflow — the process which controls software’s life cycle (build, test, package, release, deploy) in the repo
• Workflow file — the file with Workflow description, it’s located in the repo’s directory .github/workflows/
• Job — is every Workflow’s run, Jobs are run by triggers, there could be many Jobs at the same time
• Step — every Job contains Steps, you can run commands or Actions in the Step
• Action — existing “plugin”, you can find a lot of plugins in the awesome-actions repo
• Virtual environment — where do we run the Job (VM on Windows, macOS, Linux)
• Runner — an environment in the runtime, only one job could work at the same time on the Runner
• Event — runs the Workflows, for instance: Push, Pull Request, Webhook, Cronjob, etc
• Artifact — binary files, logs, pictures, etc

Jobs list for the Workflow

Limits and restrictions:

• 20 Workflows for the one repo
• 1000 API-calls/hour for all Actions in the repo
• maximum Job lifetime — 6 hours
• 20 Jobs could work in parallel (for all Workflows in repo)
• forbidden to use Actions for crypto mining and serverless computing

Here is the most relevant documentation for Actions.

Logs for the Job

Example

Let’s get back to my example. Here is my config for Actions, I want to show it to you in detail.

Setting the Workflow’s name and describing a trigger (list of all triggers in docs):

name: master-thesis
on: [push]

On which Virtual Environment do we run Job:

jobs:
  build:
    # ubuntu-latest, ubuntu-18.04, or ubuntu-16.04
    # windows-latest, windows-2019, or windows-2016
    # macOS-latest or macOS-10.14
    runs-on: ubuntu-latest

First Step, in the name: scope setting the Step’s name (optional), in the uses: scope — which Action do we want to use, in my case — clone the repo:

steps:
    - name: Checkout repo
      uses: actions/checkout@v1

On the next Step I don’t use Action but the set of my commands where I log in to the Package Registry, build Docker-image and push it to the Registry. In the env: scope I set environment variables, the one them I’ve got from secrets:

- name: Build docker image and push it to the registry
      env:
        GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
        DOCKER_IMAGE_ORIGIN: "docker.pkg.github.com/amet13/master-thesis/docker-latex"
      run: |
        # Pull submodules
        git submodule init
        git submodule update --remote
        # Login to GitHub Packages and build Docker image
        docker login docker.pkg.github.com -u amet13 -p ${GITHUB_TOKEN}
        docker pull ${DOCKER_IMAGE_ORIGIN}:latest
        docker build -t ${DOCKER_IMAGE_ORIGIN}:${GITHUB_SHA} .
        # Generate PDF artifacts
        docker run --rm -i \
          -v ${PWD}:/master-thesis:Z ${DOCKER_IMAGE_ORIGIN}:${GITHUB_SHA} \
          bash -c "latexmk -xelatex -synctex=1 -jobname=master-thesis main.tex"
        docker run --rm -i \
          -v ${PWD}:/master-thesis:Z ${DOCKER_IMAGE_ORIGIN}:${GITHUB_SHA} \
          bash -c "cd presentation/ && latexmk -xelatex -synctex=1 -jobname=presentation main.tex"
        # Publish Docker image to GitHub Packages (with latest tag)
        docker tag ${DOCKER_IMAGE_ORIGIN}:${GITHUB_SHA} ${DOCKER_IMAGE_ORIGIN}:latest
        docker push ${DOCKER_IMAGE_ORIGIN}:${GITHUB_SHA}
        docker push ${DOCKER_IMAGE_ORIGIN}:latest

I’m completely sure that will be an Action for automatic Docker build, tag and push in the nearest future, instead of running manual commands.

Adding a secret with GitHub token

The next step, after the PDF files, build I need to create a GitHub release and include generated files to this release. For automatic release creation, I use third-party Action. You can also use a scope if: for condition — run only on tag creation:

- name: Create GitHub release with artifacts
      uses: softprops/action-gh-release@v1
      if: startsWith(github.ref, 'refs/tags/')
      with:
        files: |
          master-thesis.pdf
          presentation/presentation.pdf
        name: "Build ${GITHUB_SHA}"
      env:
        GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}

Key takeaways

Despite the beta status, both services work well, I’m sure that many things will be finished before the stable release. At some points it can be inconvenient, there are no global variables, but this can be done with some hacks.

I also liked GitHub’s decision with migration from HCL to the YAML. There are many host types for Virtual Environments, sparing limits, let’s see how it will be. Generally, this stack works very well for simple pipelines in public repositories.

This article in Russian on habr.