Scheduling AMI builds: A dynamic approach in Gitlab CI

Published in

Version 1

9 min readMay 9, 2024

Problem Overview

Building idempotent AWS AMIs is I’m sure an area that many infrastructure teams would be familiar with. Typically, we use tools like Packer and Ansible to install, configure and generate the AMIs used to deploy a product or service.

Usually, this is automated via a CI pipeline; perhaps you pass in a variable with the release version of the product you wish to build, and your CI pipeline does the rest, baking your AMI and dropping it neatly into your AWS account.

This is exactly the setup our team uses for many of the products we deliver. It works well, but still requires developer intervention to kick off the pipeline and pass in the details required to initiate the process.

We wanted to extend this further by having the pipeline simply run on schedule, determine if there was any new release package available to build, then if there was, kick off a child pipeline to do the work. Our fresh AMIs would be waiting for us in the morning, ready to test.

In order to understand our starting point, let me give a little overview of our tooling:

We already had a pipeline capable of building our gitlab-runner images, we use these both internally in the development team, but also make them available to our Gitlab user base to allow them to execute their own pipeline jobs.
The existing pipeline was capable of not only building multiple versions of gitlab-runner, but also deploying those versions onto multiple base OS versions.
From a template, Terraform is used to dynamically generate a pipeline build job for each of the versions to be built. Ansible is used to pull the package from a YUM repository, install and configure it and Packer would bake the AMI for us.

Setting up a schedule in Gitlab to run this overnight is easy, just a few clicks and you’re done. The challenge lay in two areas; determining if there was a new package available to be built, and tracking if the build had been successful and to avoid re-attempting it the next time the pipeline is run.

In addition to scheduling the pipeline, we wanted to retain the ability to manually build AMIs, for development and/or testing purposes, this would require only a subset of the jobs required by the scheduled pipeline.

Version 1 is a Premier AWS Consulting Parnter. Click here to read more.

Tracking Success — One Solution

To simplify matters, I’m going to concentrate on the process flow for only the scheduled pipeline and I’m not going to go into the specifics of the AMI build jobs themselves, but focus more on the data persistance and build tracking aspects of this solution.

Controlling which jobs are executed based on how the pipeline was launched is simple enough by including a suitable rule in your job definition:

rules:
  - if: $CI_PIPELINE_SOURCE == "schedule"

In order to only launch the build process when a new gitlab-runner release becomes available, it is neccesary to track the versions that have previously been built and compare that to the packages that are currently available, in our case, stored in a YUM mirror of the official gitlab-runner repository. A high level process flow is outlined below.

In our case we have terraform generate our build jobs from a template, this allows us to build multiple AMIs from a single parent pipeline.

Persisting Data — Secure Files

In order to achieve the above, three distinct elements are required to pull the tracking functionality together:

Build a list of available gitlab-runner packages.

2. A list of gitlab-runner packages that have been built previously.

3. The delta between 1 & 2.

Furthermore, in order to track successful builds from one pipeline run to the next, the data has to be persistent. There are many ways this could be accomplished, for example job artifacts or DynamoDB, but in the interests of simplicity, we opted to use a Gitlab native feature; Secure Files. This approach also decouples the tracking from the pipeline itself, so there is no requirement to track pipeline or job IDs in order to retrieve artifacts.

This is project level file storage that sits outside of the git repository, but within the Gitlab project and is accessible via the GUI (Settings > CI/CD > Secure Files) and most importantly, via the Gitlab API.

Planned high-level pipeline structure:

parent_pipeline:

build-list-of-available-packages
diff-list-against-secure-file
launch-child-pipeline-if-there-are-builds-to-do

child_pipeline:

dynamically-generate-multi-ami-build-pipeline
trigger-generated-pipeline

triggered_pipeline:

build-ami(s)
update-secure-file

Some Example Code

Generating the list of packages available on the YUM mirror and making it available as a job artifact:

get-package-list:
  stage: get-package-list
  script:
    - curl -L $PACKAGE_REPO | grep $PRODUCT-$MAJOR_VERSION | sed -e 's/.*href="//g'| sed -e 's/">.*//g' > packages.txt # extract package name from within HTML tags
  artifacts:
    paths:
      - packages.txt
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

Diffing the output from the previous job with the content of the persistent file:

# This job compares the artifact from the previous job to the package list previously saved into Secure Files (CI/CD Settings > Secure Files)
diff-packages:
  stage: diff-packages
  script:
    # If we are on a feature branch, replace forward slashes in branch name with underscores so we can use it as the file name, else use CI_COMMIT_TAG
    - >
        if [[ -n $CI_COMMIT_BRANCH ]];
        then
          BRANCH_REF=$(echo $CI_COMMIT_BRANCH | tr "/" "_")
        else
          BRANCH_REF=$(echo $CI_COMMIT_TAG)
        fi
    # Get the file ID of the current tracker file, using the branch name as reference
    - >
        FILE_ID=$(curl -s --header "PRIVATE-TOKEN: $SECURE_FILE_TOKEN" https://<your gitlab server FQDN>/api/v4/projects/$CI_PROJECT_ID/secure_files |jq --arg BRANCH_REF "$BRANCH_REF" '.[] |select(.name == "\($BRANCH_REF).txt")|(.id)')
    # Download the file from Secure Files
    - >
        curl -s --request GET --header "PRIVATE-TOKEN: $SECURE_FILE_TOKEN" https://<your gitlab server FQDN>/api/v4/projects/$CI_PROJECT_ID/secure_files/$FILE_ID/download --output $BRANCH_REF.txt
    - diff -n $BRANCH_REF.txt packages.txt | grep $PRODUCT > diff.txt || true
    # If diff output is zero bytes in length, exit the pipeline
    - >
        CHECK_DIFF=$(stat --printf="%s" diff.txt);
        if [ $CHECK_DIFF -eq 0 ];
        then echo "There are no new packages to be built. Exiting pipeline." ;
        exit 1;
        fi
  dependencies:
    - get-package-list
  artifacts:
    paths:
      - diff.txt
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

The artifact diff.txt created above is passed to Terraform to generate a pipeline file, if the file contains multiple versions of gitlab-runner to be built, Terraform will generate a pipeline with multiple build jobs.

Pulling this together, you should end up with something like this in your .gitlab-ci.yml. The ami-build-trigger job is used to call the child pipeline which actually does the AMI build. This only gets executed if the diff-packages job determines that there is work to do.

variables:
  PRODUCT:
    value: "gitlab-runner"
    description: "Name of the product being built"
  MAJOR_VERSION:
    value: "16"
    description: "The Major Version of the product"
  PACKAGE_REPO:
    value: https://<your local YUM mirror>/repos/gitlab-runner
    description: "The URL of your YUM repository"

stages:
  - get-package-list
  - diff-packages
  - ami-build-trigger

# This job pulls a list of packages from a remote repo and strips off any HTML tags
get-package-list:
  stage: get-package-list
  script:
    - curl -L $PACKAGE_REPO | grep $PRODUCT-$MAJOR_VERSION | sed -e 's/.*href="//g'| sed -e 's/">.*//g' > packages.txt # extract package name from within HTML tags
  artifacts:
    paths:
      - packages.txt
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

# This job compares the artifact from the previous job to the package list previously saved into Secure Files (CI/CD Settings > Secure Files)
diff-packages:
  stage: diff-packages
  script:
    # If we are on a feature branch, replace forward slashes in branch name with underscores so we can use it as the file name, else use CI_COMMIT_TAG
    - >
        if [[ -n $CI_COMMIT_BRANCH ]];
        then
          BRANCH_REF=$(echo $CI_COMMIT_BRANCH | tr "/" "_")
        else
          BRANCH_REF=$(echo $CI_COMMIT_TAG)
        fi
    # Get the file ID of the current tracker file, using the branch name as reference
    - >
        FILE_ID=$(curl -s --header "PRIVATE-TOKEN: $SECURE_FILE_TOKEN" https://<your gitlab server FQDN>/api/v4/projects/$CI_PROJECT_ID/secure_files |jq --arg BRANCH_REF "$BRANCH_REF" '.[] |select(.name == "\($BRANCH_REF).txt")|(.id)')
    # Download the file from Secure Files
    - >
        curl -s --request GET --header "PRIVATE-TOKEN: $SECURE_FILE_TOKEN" https://<your gitlab server FQDN>/api/v4/projects/$CI_PROJECT_ID/secure_files/$FILE_ID/download --output $BRANCH_REF.txt
    - diff -n $BRANCH_REF.txt packages.txt | grep $PRODUCT > diff.txt || true
    # If diff output is zero bytes in length, exit the pipeline
    - >
        CHECK_DIFF=$(stat --printf="%s" diff.txt);
        if [ $CHECK_DIFF -eq 0 ];
        then echo "There are no new packages to be built. Exiting pipeline." ;
        exit 1;
        fi
    - ls -l
  dependencies:
    - get-package-list
  artifacts:
    paths:
      - diff.txt
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

ami-build-trigger:
  stage: ami-build-trigger
  trigger:
    include:
      - local: your-ami-build-jobs-go-here.yml

Included in the child pipeline is a job to update the secure file. This job is only executed if the build job was successful. Unfortunately, it is not possible to update this file in place, so it has to be downloaded, amended and replaced. If there is more than one build job to be run, it’s possible that both jobs may attempt to modify the file simultaneously (if your gitlab-runners have concurrency enabled), hence a random sleep is introduced, which shouldn’t cause too much concern if you’ve scheduled this to run overnight. If it does, best leave concurrency disabled in your project’s gitlab-runner.

# This job downloads, amends and uploads the tracker file from/to Secure Files
.update-secure-file:
  stage: update-secure-file
  script:
    - >
        if [[ -n $CI_COMMIT_BRANCH ]];
        then
    # Our branch name may have a forward slash in it, if it does, replace with an underscore
          BRANCH_REF=$(echo $CI_COMMIT_BRANCH | tr "/" "_") 
        else
          BRANCH_REF=$(echo $CI_COMMIT_TAG)
        fi
    # Going to sleep for random time between 0-180 seconds, to reduce chance of a race condition when writing to secure file when num_build_jobs > 1
    - time sleep $(( RANDOM % (180) ))
    - >
       JOB_ID=$(curl -vs --header "PRIVATE-TOKEN: $SECURE_FILE_TOKEN"  https://<your gitlab server FQDN>/api/v4/projects/$CI_PROJECT_ID/pipelines/$PARENT_PIPELINE_ID/jobs |jq '.[] |select(.name == "diff-packages")|(.id)')
    - echo "Retrieve the diff.txt from the upstream diff-packages job"
    - >
       curl -vs --header "PRIVATE-TOKEN: $SECURE_FILE_TOKEN" --output artifacts.zip --location https://<your gitlab server FQDN>/api/v4/projects/$CI_PROJECT_ID/jobs/$JOB_ID/artifacts
    - unzip artifacts.zip
    # Get the file id for our branch specific package build tracker
    - >
        FILE_ID=$(curl -s --header "PRIVATE-TOKEN: $SECURE_FILE_TOKEN" https://<your gitlab server FQDN>/api/v4/projects/$CI_PROJECT_ID/secure_files |jq --arg BRANCH_REF "$BRANCH_REF" '.[] |select(.name == "\($BRANCH_REF).txt")|(.id)')
    # Download the file so we can edit it
    - >
        curl -s --request GET --header "PRIVATE-TOKEN: $SECURE_FILE_TOKEN" https://<your gitlab server FQDN>/api/v4/projects/$CI_PROJECT_ID/secure_files/$FILE_ID/download --output $BRANCH_REF.txt
    - echo "Delete the tracker file from Secure Files"
    # Delete the remote copy
    - >
        curl -s --request DELETE --header "PRIVATE-TOKEN: $SECURE_FILE_TOKEN" https://<your gitlab server FQDN>/api/v4/projects/$CI_PROJECT_ID/secure_files/$FILE_ID
    # Update the content of the package build tracker
    - cat diff.txt | grep $VERSION >> $BRANCH_REF.txt
    # Sort the file in place, so we can diff accurately in future
    - sort -V -o $BRANCH_REF.txt{,}
    # Push the updated file up to Secure Files (check it via CI/CD Settings > Secure Files)
    - >
       curl --request POST --header "PRIVATE-TOKEN: $SECURE_FILE_TOKEN" https://<your gitlab server FQDN>/api/v4/projects/$CI_PROJECT_ID/secure_files  --form "name=$BRANCH_REF.txt" --form file=@$BRANCH_REF.txt
 
update-secure-file-dev/${ami}/${version}:
  extends: .update-secure-file
  rules:
    - if: $INITIAL_TRIGGER == "schedule" && $CI_COMMIT_BRANCH =~ '/^feature\/.*$/'
      needs:
        - job: build-dev/${ami}/${version}
          artifacts: true

update-secure-file-release/${ami}/${version}:
  extends: .update-secure-file
  rules:
    - if: $INITIAL_TRIGGER == "schedule" && $CI_COMMIT_TAG =~ '/^v.*$/'
      needs:
        - job: build-release/${ami}/${version}
          artifacts: true

API Calls + Tokens

The API calls used in updating the Secure Files are documented here.

Note that Gitlab’s standard CI_JOB_TOKEN variable does not yet have the ability to make all of the API calls required to Secure Files, hence the usage of $SECURE_FILE_TOKEN in the example code above, you should set this up in your project with Role ‘Maintainer’ and a scope of ‘api, api_read’

Challenges

Whilst the code above may give you starting point for pulling together your own automated build pipeline, there are some challenges to consider. Firstly, the code presented does not include functionality to generate the secure file. Where one does not exist already, you need to manually create a ‘seed’ file before the first run, named to match your git branch, for example feature_mybranch.txt, remember to replace any forward slashes with an underscore. You can the upload it via the GUI.

The content of this file will depend upon what you’re building and the format that YUM returns the package names in, but one package per line format is expected. If your pipeline has the ability to build multiple AMIs concurrently, you will probably want to avoid leaving the file empty, otherwise you may end up building an AMI for every package your YUM repo holds!

Runner concurrency can result in a race condition when updating the secure file, hence the sub-optimal random sleep demonstrated here. Depending on the number of AMIs you expect to be building and the time your build job takes to complete, I’d suggest possibly configuring your project gitlab runner with concurrency disabled. After all, if you’re scheduling overnight you may not be in any hurry and can pick up your new AMIs in the morning.

Obviously, the code presented here is unlikely to work perfectly in your use case, but hopefully it gives you some insight into one particular solution. Maybe you can adapt it to suit your needs or perhaps it’s inspired you to develop an alternative approach entirely. Good luck!

About the Author:
Fraser McAlister is a Senior AWS DevOps Engineer at Version 1.