AWS CodeStar Pipelines with Multiple CodeCommit Sources

This article will cover cloning two different CodeCommit repository into a single AWS Codestar Pipeline build.

The typical AWS Codestar Pipeline will have the following sequence of operations.

CodeStar Pipeline Stages

The source stage pulls code into the CodeBuild container. A problem with the Source Stage is that it doesn’t download the .git folder. Since git submodules depend on the metadata in the .git folder they will not be included when the Source Stage of the pipeline downloads code from AWS CodeCommit. To work around this problem we use git clone in the build file to mimic the operations of git submodule add in a normal git environment.

To start configuring the CodeStar project for two sources, first clone the main CodeCommit repository locally then add the second repository (PythonCommon in this case) using “git submodule add” as shown below.

Cloning the inital AWS CodeStar Project showing the resulting file structure

To test if the AWS CodeStar Pipeline accepts the PythonCommon repository we import it into index.py.

Import the module PyhtonCommon.parsers from a completely different AWS CodeCommit repository.

and then push the update to the CodeCommit repository triggering the deployment pipeline.

The Pipeline output in the AWS console shows a failure in the CodeBuild stage.

Clicking on “Details” to open the CodeBuild log shows that the PythonCommon module was not found.

The pipeline will fail until the following conditions are met:

  • git submodule add is replaced by git clone in buildspec.yml
  • AWS credential helpers is invoked in buildspec.yml
  • Permissions are added to the pipeline IAM role to give git clone access to the second CodeCommit repository.

First, substitute the operation of git submodule add with git clone and add AWS credentials helper as shown in bold in the buildspec.yml file below.

version: 0.2

phases:
install:
runtime-versions:
python: 3.8
commands:

# Upgrade AWS CLI to the latest version
- pip install --upgrade awscli
- git config --global credential.helper '!aws codecommit credential-helper $@'
- git config --global credential.UseHttpPath true
- git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/PythonCommon


pre_build:
commands:

# Discover and run unit tests in the 'tests' directory. For more information, see <https://docs.python.org/3/library/unittest.html#test-discovery>
- python -m unittest discover tests

build:
commands:

# Use AWS SAM to package the application by using AWS CloudFormation
- aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template template-export.yml

# Do not remove this statement. This command is required for AWS CodeStar projects.
# Update the AWS Partition, AWS Region, account ID and project ID in the project ARN on template-configuration.json file so AWS CloudFormation can tag project resources.
- sed -i.bak 's/\$PARTITION\$/'${PARTITION}'/g;s/\$AWS_REGION\$/'${AWS_REGION}'/g;s/\$ACCOUNT_ID\$/'${ACCOUNT_ID}'/g;s/\$PROJECT_ID\$/'${PROJECT_ID}'/g' template-configuration.json

artifacts:
files:
- template-export.yml
- template-configuration.json

Next, the Pipeline needs permissions to access the second CodeCommit repository from the git clone command added to the buildspec.yml file. To access permissions in the Pipeline dialog click settings then the service role arn as shown below.

Location of permissions role associated with Pipeline

At the bottom of the IAM console note the name of the permission boundary then click the remove button to remove it.

Next click “Set” to add the same permission boundary again except with modifications which will be explained below.

Next select and expand the permissions boundary that you just removed and click “Edit Policy.”

To give the Pipeline permissions to git clone another repository add that repository’s ARN to the list of resources as shown below

Click through to save and set the boundary until you see that the original, updated boundary policy is displayed in the pipeline ToolChain role as shown below.

The pipeline should now run souring two repositories in one AWS Codestar Pipeline.