How to Integrate GitHub Workflows into Your Project

Image made by the author — GitHub Workflow

Your source code repository is hosted on GitHub and now you’d like to learn how to automate processes for developing, testing, building, and release of your software. Then you’ve come to the correct post! Let’s begin.

GitHub provides many functionalities that make your life easier as a developer and I think you should take advantage of these benefits.

In this post, I’ll explain what GitHub Actions are, what they do, and how you can integrate them into your workflow to automate repeated tasks.

End of this post, We’ll be ready to set up a Workflow with GitHub Actions.

We will go one step further and create a Reusable Workflow to use by each repository in your organization.

Image made by the author — CI/CD Pipeline

Actions vs. Workflows: Understanding the Difference

Actions are the portable building block used to perform complex tasks that you may call multiple times. They are called inside a job as a step. You’ll see more detail about “what a job is” in the continuation of the post. By using an action, you can reduce the amount of repetitive code needed to write a workflow. You can create your actions or you can reuse open-source actions from the GitHub Marketplace. Here are some example actions; building an npm package, publishing an npm package to an artifactory, and scanning an npm package to find out if there is any vulnerability.

Little tip; Many of the functionality we might want to implement into our workflow is probably already published by someone else as an action, so take a look at to GitHub Marketplace before reinventing the wheel. In addition, it is always possible to modify existing actions.

On the other hand, Workflows are a collection of job definitions that get executed when they are triggered by an event. Workflows can also be triggered manually by the user. The clear answer for the difference between an Action and a Workflow is, Actions are used as a step in jobs inside workflows.

A repository can have multiple workflows, as well as a workflow can be re-used by multiple repositories. For example, one workflow can build and test your application when a new pull request is created, while another workflow can deploy your application when a new release is created. And if you have multiple repositories they can re-use the same workflow for their CI/CD process. They are called Reusable Workflows and that’s the topic we are going to go deeper in this post.

Before we dive deep into the workflows, I’d like to explain a bit about the components that we’ll use a lot in this post;

Acomposite action is one of the types of custom GitHub Actions that can be created. A composite action bundles multiple steps into a single action but is presented as one step when it is invoked inside a workflow. For example, building and publishing an npm package can be combined into one Composite Action and run sequentially. They can be handy to divide large jobs into multiple files. Composite action is stored in a different file than the reusable workflow.

Little warning; If your repositories belongs to a private organization then you may need to create an ssh key and share it with the other repositories wants to checkout composite action during workflow run. Because private repositories are not publicly accessible. Here you can find how you can create an ssh key for your repository. You can find example usages of both (with and without an ssh key) in the following topic. If it’s an internal repository then you don’t need to checkout the composite action, you can call it directly.

Another important thing is Reusable Workflows can use secrets by inheriting them from a repository while Composite Actions cannot. You need to pass each secret via input parameters.

An event can be anything that can happen on a GitHub repository. For example, pushing a code, opening a pull request, or creating a release. You can check the list of events here.

A job is a series of steps that gets executed inside a workflow based on the trigger conditions. Each step is either a script or a GitHub Action. The steps under the job are run sequentially but the jobs run in parallel by default. To run the jobs sequentially, you need to define dependencies between them. Each job runs separately by a specific runner which will report the results of the job back to GitHub. And the results can be checked directly from the Github UI.

A matrix is a strategy to run multiple jobs that has the same job definition with different parameters. When you create a job definition with a matrix strategy, it will automatically create multiple runs with the same job based on the combinations of the parameters defined by the user. For example, you can build and publish multiple npm packages under one repository based on the directory of the package.json file or you can build your project in multiple node versions. Here you can find what kind of combination you can use for the matrix.

It’s time to see how you can use Reusable Workflows and Composite Actions in your project!

Reusable Workflows: How to Implement and Utilize Them

Workflows are written in YAML syntax and are stored in .github/workflows directory at the root of the repository. The difference in Reusable Workflows is they need to be located in a different repository than your applications located so they can be called by multiple repositories.

Rather than copying and pasting from one workflow to another, you can make workflows reusable to avoid duplication. Your organization can build up a library of reusable workflows that can be centrally maintained.

The only thing you need to check is if your repositories belong to a private organization then you need to update the access permissions of your reusable workflows repository settings to share them within the organization.

Settings of the repository where reusable workflows are located

There are some keywords in the syntax of a workflow you should get familiar with. Here you can find the full syntax.

Now we’ll see how Reusable Workflows and Actions work together. We’ll create one workflow for the repository where your application is located and one workflow for a reusable one. I have added comments for the steps in both files. Don’t worry we’ll deep dive into composite actions that we used in those examples in the upcoming topic.

Application workflow under .github/workflows/ directory in the Application Repository

Reusable workflow under .github/workflows/ directory in the Reusable Workflows Repository

Useful Tips for Workflows & Actions👌

The environment variables that you defined at the beginning of the workflow can not be usable at the job condition level. But they are reachable at the step condition level. If you would like to use them in “job.if” level then you need to create an output in the previous job and use it in the next one. In this example, you can see both “step.if” and “job.if” level context availability.

In GitHub documentation, you can see all the context availability here.

env:
  DEPLOY: ${{ github.event_name == 'pull_request' && github.event.action != 'closed' }}

jobs:
  init:
    name: Init
    runs-on: ubuntu-latest
    outputs:
      DEPLOY: ${{ steps.setvars.outputs.DEPLOY }}
    steps:
      - name: Checkout Source Code
        uses: actions/checkout@v3

      - name: Set Environmet Variables
        id: setvars
        run: |
          if [[ "${{ github.event_name }}" == "pull_request" && 
                "${{ github.event.action }}" != "closed" ]];
          then
            echo "DEPLOY=true" >> $GITHUB_OUTPUT
          fi

deploy:
    name: Build & Deploy CDK
    needs: 
      - init
    if: needs.init.outputs.DEPLOY
    runs-on: ubuntu-latest
    steps:
      .
      .
      .
      - name: Deploy
        if: ${{ env.DEPLOY == 'true' }}
        working-directory: ${{ env.WORKING_DIRECTORY_DEPLOY }}
        run: |
          npx npm-assets .
          npx cdk deploy --context config=local --require-approval=never

When you create a dependency with the “need” keyword you have to wait for the job you specified under the keyword but the result of the job doesn’t need to be successful. You can run the next step in any case. Or you can add more conditions for each result of the step.

    needs: 
      - init
      - unit-tests
      - build-nuget
    if: |
      always() &&
      needs.init.result == 'success' &&
      (needs.unit-tests.result == 'success' || needs.unit-tests.result == 'skipped') && 
      (needs.build-nuget.result == 'success' || needs.build-nuget.result == 'skipped')

Or

    needs: 
      - init
      - unit-tests
      - build-nuget
    if: |
      always() &&
      !contains(needs.*.result, 'failure') &&
      !contains(needs.*.result, 'cancelled')

You can use the outputs of a job as an environment variable for another job. I prefer this approach if the output of the previous job is going to use only one specific job.

    # In this example shows that IMAGE_VERSION and IMAGE_PATH
    # variables set by init job. build-docker job wants to use it
    # as an environment variable. Instead of using long and 
    # complicated format like ${{ needs.init.outputs.IMAGE_VERSION }}
    # you can add those as a job environment variable and
    # you can use more clear variable names inside the steps.
    build-docker:
     name: Build Docker Image
     needs: 
        - init
      - unit-tests
    if: inputs.DOCKER_PROJECT_NAME
    defaults:
      run:
        working-directory: ${{ env.WORKING_DIRECTORY_NUGET }}
    env:
      ARTIFACTORY_SECRET: ${{ secrets.ARTIFACTORY_KEYS }}
      IMAGE_VERSION: ${{ needs.init.outputs.IMAGE_VERSION }}
      IMAGE_PATH: ${{ needs.init.outputs.IMAGE_PATH }}
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Source Code
        uses: actions/checkout@v3

      - name: Authenticate Artifactory for Docker
        run: |
          USERNAME=$(jq -r '.username' <<< "$ARTIFACTORY_SECRET")
          API_KEY=$(jq -r '.apiKey' <<< "$ARTIFACTORY_SECRET")
          echo $API_KEY | docker login --username $USERNAME --password-stdin sample.artifactory.io

      - name: Build image
        run: |
          docker build -f ${{ inputs.DOCKER_PROJECT_NAME }}/Dockerfile \
          --build-arg "NUGET_REPO=${{ env.ARTIFACTORY_NUGET_URL }}" \
          --build-arg "NUGET_USERNAME=$(echo $ARTIFACTORY_SECRET | jq -r .username)" \
          --build-arg "NUGET_PASSWORD=$(echo $ARTIFACTORY_SECRET | jq -r .apiKey)" \
          --tag "$IMAGE_PATH:$IMAGE_VERSION" .

      - name: Push Image
        run: docker push "$IMAGE_PATH:$IMAGE_VERSION"

And last, here’s a list of some actions created by the community that I’m using in my pipelines.

alexcasalboni/aws-lambda-power-tuning: Using to optimize our Lambda functions for cost and performance way.

thollander/actions-comment-pull-request: Using for adding the lambda power tuner results directly to the Pull Request as a comment.

microsoft/variable-substitution: Using for updating variables in JSON file inside the repository during the workflow run.

dorny/test-reporter: Using for generating .Net Unit Test results as a report directly to the workflow run.

Composite Actions: How to Implement and Utilize Them

You already know how you can call an action inside a workflow from the previous topic. Now we are going to use actions provided by GitHub, the GitHub community, and the organization we are in.

composite action under prep-npm/ directory in the Reusable Workflows Repository

composite action under unit-tests/ directory in the Reusable Workflows Repository

Action Secrets: Passing them to Workflows

Actions Secrets are located under Settings of the Repository

There are 3 types of secrets: Repository Secrets, Environment Secrets, and Organization Secrets. There are also tiers within the Secrets such as Actions, Dependabot, and Codespaces. Organization Secrets are created by the organization automatically when you create a repository under this organization. You can specify which repositories these organization secrets are shared. Repository Secrets and Environment Secrets are mostly added by the developer. I definitely recommend adding your hidden values like Api Keys, Tokens, etc to the Action Secrets.

Environment Secrets

Repository Secrets

Organization Secrets

When you are calling a reusable workflow if you add the “inherit” keyword then all the secrets in the application repository will be inherited to reusable workflow during the workflow run.

jobs:
  call-workflow:
    uses: Sample-Organization/reusable-workflows/.github/workflows/build.yml@master
    with:
      SOLUTION_NAMES: '["Solution1", "Solution2", "Solution3"]'
    secrets: inherit

Inside reusable workflow, you need to define which secret you want to inherit by.

on:
  workflow_call:
    inputs:
      SOLUTION_NAMES:
        required: false 
        type: string
    secrets:
      REUSABLE_REPOSITORY_SSH:
        required: true 
      ARTIFACTORY_KEYS:
        required: true
      SONAR_TOKEN:
        required: true

Thank you for bearing with me till the end 🙋

I hope you found the Reusable Workflows and GitHub Actions useful and that you’re ready to integrate them into your projects. The workflows that I shared in this post are just simple examples of how you can use them. There are a lot of other use cases and benefits to it. I strongly recommend you read the full GitHub documentation on it and experiment with how you can apply it to your use case.

By the way, I’m really curious about the useful actions or tips you found, feel free to share with me in the comments.

Happy gitting! 👋