Publishing Python Packages to AWS CodeArtifact Using Github Actions

Published in

Jamf Engineering

6 min readFeb 24, 2022

Our use case

My team at Jamf builds serverless applications using Python. We write our code locally, push to GitHub, and automatically deploy into our AWS accounts. The shared packages we write for internal use need to be hosted in a location that is not only accessible from all these locations but is also secure.

AWS provides a service for us to manage our packages: AWS CodeArtifact. It is a fully managed repository service that requires little operational overhead and works with the package tooling our team uses (namely pip and twine).

Our developers have the ability to interact with CodeArtifact using their IAM roles to obtain temporary credentials for the AWS CLI and there is no need to manage IAM user access keys. In October 2021, GitHub announced support for OpenID Connect enabling GitHub Actions to authenticate to cloud providers and obtain credentials. This eliminates any secrets management in GitHub for accessing AWS resources.

Let’s walk through the full process for setting up the trust between GitHub and our AWS account, creating the CodeArtifact repository, and allowing GitHub Actions to both publish to and install packages from that repository. All of the examples in this post will be basic CloudFormation templates. I will also point to where in the AWS console you can do this manually (but without screenshots).

This post assumes you are familiar with using CloudFormation, GitHub Actions, and Python’s packaging tools.

Establishing trust between GitHub and AWS

First, we need to add GitHub as an identity provider in IAM. Deploy the template below to create it. The thumbprint in the template was calculated following AWS’s documentation here.

Copy and save the ARN (Amazon Resource Name) for this new “token.actions.githubusercontent.com” identity provider (the CloudFormation stack will have this as an output).

You can add identity providers in the console by navigating to IAM > Identity Providers > Add provider. View the GitHub documentation here and the AWS documentation here. The console can obtain the thumbprint for you.

We are now in a position to create IAM roles that can be assumed using a token issued by GitHub.

CodeArtifact

We will now create the repository where we will store our packages. To create a repository, we also need to create a domain for it. Domains are used for managing repositories. Repositories can contain packages and artifacts of any supported type (i.e. Python and npm packages can be in the same repository).

The template below will create a domain and a repository within it named “main” by default. This repository is connected to the public PyPI. Requests for packages such as requests can be made here and the packages will be pulled in and cached from upstream.

Note which AWS region you created your repository in. This will be needed later when configuring the GitHub Actions.

To create the domain in the console navigate to CodeArtifact > Artifacts > Domains > Create domain. A domain must be created before you can create a repository. To create the repository navigate to CodeArtifact > Artifacts > Repositories > Create repository. You will be presented with options for upstream repositories to connect to.

Using the CodeArtifact Repository

AWS provides detailed documentation for how to use CodeArtifact with the supported languages and tools here. Some of the options will set or modify configuration files (writing tokens, setting repository URLs, etc.).

To configure pip you would run:

aws codeartifact login --tool pip --domain my_domain --domain-owner 111122223333 --repository my_repo

In these commands the --domain-owner value is your AWS account ID.

If you want to avoid configuration changes, or have a use case that doesn’t work with automatically configuring pip, you can obtain a CodeArtifact token and pass it into the index URL:

CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain my_domain --domain-owner 111122223333 --query authorizationToken --output text`# https://aws:$CODEARTIFACT_AUTH_TOKEN@my_domain-111122223333.d.codeartifact.region.amazonaws.com/pypi/my_repo/simple/

This allows us to use our repository while developing locally. We now need to ensure GitHub can do the same for our CI/CD process with GitHub Actions. This includes both the automated publishing of package versions and the ability to install those package versions in our projects.

Publishing Packages

This template will create an IAM role with the minimal CodeArtifact permissions to publish packages. The IAM role will be tied to the GitHub org/repo you enter and cannot be used by others (you would create a role for each package repo in GitHub).

Note that you can optionally pass a branch name (this is repeated in the later IAM role template for installing packages). You can lock down an IAM role to only be assumable by a GitHub Action that was triggered by a matching branch. In our case, we normally set this to main so only code in the main branch can be built and pushed. No other branch will be able to trigger an action that can assume this role and gain these permissions.

Deploy the template below to create the IAM role. Copy and save the ARN for this IAM role (the CloudFormation stack will have this as an output).

To create the IAM role in the console navigate to IAM > Roles > Create role. View the GitHub documentation here and the AWS documentation here.

Now that the IAM role has been created, we can use it in our GitHub Action. Below is an example workflow file for building and publishing a Python package from the repository. For our team, we normally run two different workflows. The first is on pull request to perform testing and validation before allowing merging to main. The second, this workflow, triggers on the merge to main only and will publish the package.

Inside the GitHub repository this file would be located in a .github/workflows/ directory.

Note the added id-token: write permission for this job. This is required to allow the action to request a token from GitHub’s OIDC. See GitHub’s documentation here.

The configure-aws-credentials action is maintained by AWS and handles the request to obtain temporary credentials using the issued token. The credentials are stored in environment variables used by the AWS CLI and SDKs. In this step you will want to set the correct AWS region for where you created your CodeArtifact repository and provide the ARN of the IAM role it will assume. role-session-name will default to “GitHubActions” if you don’t set it.

The rest of the workflow is straightforward. Python is setup, twine is installed, the package is built, and then in the last step we use the AWS CLI to configure twine before uploading the package.

Installing Packages

For projects that use the packages we publish to CodeArtifact, the process is nearly identical for allowing installation within GitHub Actions. The following template is very similar to the publisher IAM role template above, but provides the minimal permissions for installing packages. It otherwise works just like the other with the relationship to the GitHub repository conditions.

Deploy the template below to create the IAM role. Copy and save the ARN for this IAM role (the CloudFormation stack will have this as an output).

The GitHub Actions workflow below closely matches our publisher workflow with a some notable differences. One is this workflow is only triggered on (all) pull requests. The other is using the AWS CLI to configure pip instead of twine just before we install the dependencies. This is something you would adjust to match how you handle dependencies within your own projects.

Summary

With some up-front work, AWS CodeArtifact provides a great solution for hosting packages for both local development and CI/CD workflows. Paired with GitHub’s OIDC provider we can do all of this without storing or managing long lived credentials.