Using JSON in your GitHub Actions when authenticating with GCP

If you try to input raw JSON into your GitHub action, it’s likely that you’re going to be in for a few surprises. Especially if your JSON contains spaces or extra encoded characters, like the new line character in your GCP service account JSON file (private key).

Those unexpected characters can cause commands that seem perfectly well formatted to fail with errors that can leave you scratching your head for hours.

You can try to find the perfect jq, sed, awk incantation to deal with this, but an easy way to avoid all of this headache is to encode your json into a base64 string. It is in fact the recommended way to deal with longer form secrets in Github actions.

To encode your json you can use the base64 utility like this cat <your-creds-file> | base64, and then use that base64 encoded string as your Github action secret.

When you need to decode this secret, you can do so with echo '$YOUR_BASE64_ENV_VAR' | base64 --decode | jq > gcp-credentials.json.

This base64 encoded string can be used as is in your Github actions. Some examples below:

env:
  PREFECT_API_KEY: ${{ secrets.PREFECT_API_KEY }}
  PREFECT_API_URL: ${{ secrets.PREFECT_API_URL }}
  BASE64: ${{ secrets.BASE64 }}
  GCP_RESOURCE_REGION: ${{ secrets.GCP_RESOURCE_REGION }}
  GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
  GCP_ARTIFACT_REPOSITORY_NAME: ${{ secrets.GCP_ARTIFACT_REPOSITORY_NAME }}
  FLOW_IMAGE_NAME: ${{ secrets.FLOW_IMAGE_NAME }}

# authenticating with Gcloud example
- name: Authenticate to Google Cloud
  uses: google-github-actions/auth@v1
  with:
    credentials_json: ${{ secrets.BASE64 }}

# authenticating with Docker example
- name: Build and push image to GAR
  run: |
    RAW_IMAGE=$GCP_RESOURCE_REGION-docker.pkg.dev/$GCP_PROJECT/$GCP_ARTIFACT_REPOSITORY_NAME/$FLOW_IMAGE_NAME
    IMAGE_URI=$GCP_RESOURCE_REGION-docker.pkg.dev/$GCP_PROJECT/$GCP_ARTIFACT_REPOSITORY_NAME/$FLOW_IMAGE_NAME:$GITHUB_SHA
    
    docker build -q \
    --build-arg PREFECT_API_KEY=$PREFECT_API_KEY \
    --build-arg PREFECT_API_URL=$PREFECT_API_URL \
    --build-arg BASE64=$BASE64 \
    -t $RAW_IMAGE \
    -f ./Dockerfiles/flows.Dockerfile .

    docker tag $RAW_IMAGE $IMAGE_URI
    docker push $RAW_IMAGE
    docker push $IMAGE_URI
    echo "IMAGE_URI=$IMAGE_URI" >> $GITHUB_OUTPUT
  shell: bash

And if they are needed in a Docker container that’s built in your Github workflow, it is as easy as converting them with the command mentioned above.

RUN echo "$BASE64" | base64 --decode | jq > gcp-credentials.json

Another neat trick to try when dealing with JSON credentials files, is to consider how these JSON objects can reduce the number of secrets that you create in your workflows. You can consider creating an extra step at the begining of your workflow which saves individual keys of your JSON file as environment variables. You can also mask these values to make them extra secure. There’s a nice post about this from Anna Geller here.

Hope that this saves someone a few headaches!

[In case you’re interested in the repo where I’m using this code, it is here].