Photo by Sebastian Unrau from Unsplash

Improving Security of CI/CD Pipelines on AWS by Using Temporary Access Credentials

I am currently in the process of improving the security of cloud operations for one of my clients and I wanted to share how I was able to improve the security of all of our build and deployment pipelines using temporary AWS access credentials.

Insecure design

We have about 40 CodeBuild projects, some of which require AWS credentials to access certain AWS resources, and almost all of them require read access to our CodeCommit repositories. Repositories are cloned over NPM by specifying SSH Git URLs in the package.json file.

Before I began the migration the setup looked as follows:

1. Access to packages from CodeCommit was provided by using a pre-created SSH key that was installed into custom docker images that were used in the CodeBuild projects.
2. A set of long term AWS access keys was created using a single IAM user, and plugged into CodeBuild environments right through environment variables in more than a dozen of CodeBuild projects.

In terms of security, there are a lot of problems with this design and a lot of room for improvement.

1. First things first, reusing the same set of access keys, which are the private SSH key and the set of long term AWS credentials in our example, across multiple places is an obvious anti-pattern. If a key or credential leaks somewhere somehow multiple resources will get exposed and become vulnerable because the incident will not be isolated.
2. The second issue here is the use of long term credentials in itself. Credentials need to be rotated to reduce the window of exposure in case of a leak event and to ensure that any past security accidents don’t affect the security of the company’s future cloud operations.
3. Another glaring issue is the passing of security credentials through environment variables. First, environment variables are not encrypted. Second, they can be easily accessed in plain text from the CodeBuild project settings in the AWS console, by using AWS CLI or AWS SDK, or from inside the OS image by examining system environment variables.

The following sections will describe a solution that won’t require to pass any access credentials through environment variables. However if you absolutely need to do so, checkout AWS Secrets Manager service.

Secure design

Long term access credentials should be replaced by short term credentials available in the instance metadata profile. This comes with several benefits:

1. Short term credentials are bound to the IAM role attached to a resource meaning that you still continue to retain the granular control over permissions granted to the resource.
2. By default, short term credentials are valid for only 1 hour, after which they expire and get rotated. This setting can be adjusted in the IAM role settings to a maximum value of 12 hours.
3. Temporary credentials also allow us to clone Git dependencies from CodeCommit without using any SSH keys or HTTPS credentials which is great because they are just another form of a long term key. To set this up we’re going to use AWS CLI credential helper.

You can find out more about temporary credentials in this chapter of IAM service documentation. Check it out, it’s a great read!

Migration to temporary AWS credentials

The first part of the migration was to update our own code to use temporary access credentials. We have several shell scripts using AWS CLI and a few applications that are using AWS SDK internally.

Overall switching to temporary credentials was not a difficult task since both AWS SDK and AWS CLI have a special way of looking up the access credentials from the environment they find themselves in. One of the places that they look into is the instance profile metadata.

If you didn’t know, instance profile is a large collection of useful metadata information available to each running EC2 instance, that is accessible at the 169.254.169.254 link-local address. One of the entries in an instance profile is a set of temporary AWS credentials that are automatically retrieved for us by EC2 from STS using the IAM role attached to an instance.

Typically, in order to make sure that these temporary credentials are going to be picked up by your application you have to ensure that:

1. Your CodeBuild project does not define AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
2. You do not pass any access keys to AWS SDK constructor or constructors of any of its SDK subclasses.
3. You don’t store any credentials in ~/.aws/credentials file on the system.
4. You do not inject any AWS access credentials into your applications in any other way.

In my case, after making sure that all traces of long term access credentials were gone, AWS SDK and CLI have automatically picked up the temporary credentials from instance metadata profiles and continued to work as expected.

You can read more about how AWS CLI and AWS SDK lookup credentials from the surrounding environment in this chapter of AWS SDK documentation.

Configuring Git and NPM

The second part of the migration was to get rid of SSH keys used to clone CodeCommit packages through Git, and set up CodeCommit access over temporary AWS credentials.

Before, we were using SSH Git URLs in our applications’ package.json files:

/* package.json file */
{
    "name": "my-app",
    "version": "1.0.0",
    "author": "my-team",
    ...
    "dependencies": {
        "custom-package": "git+ssh://git-codecommit.region.aws...",
        ...
    }
}
// full URL example
"git+ssh://git-codecommit.region.amazonaws.com/v1/repos/my-repo"

In order to be able to clone those dependencies over SSH, a private SSH key was installed under ~/.ssh directory under id_rsa filename. In addition to that, ~/.ssh/known_hosts were configured with publicly known CodeCommit hostnames, to make sure that an installation process did not require any interactivity from a user.

To switch to temporary credentials I had to do a few things:

1. First, I needed to configure Git to use the AWS CLI credential helper. This step allowed Git to communicate with CodeCommit using temporary STS credentials only.
2. Then, in order to make sure that NPM was going to use the new Git set-up, I needed to update Git URLs in package.json files across all projects to HTTPS format.

To configure Git to use AWS credentials helper, I’ve included following commands in our custom CodeCommit Docker files:

# omit RUN if not using Docker;
# additionally, if running this on Windows omit [&&], [\] and
# use double-quotes to wrap [!aws ... $@] part;
RUN git config --global credential.helper \
        '!aws codecommit credential-helper $@' && \
    git config --global credential.UseHttpPath true

Then I’ve updated CodeCommit URLs in all of our package.json files:

/* a package.json file */
{
    "name": "my-app",
    "version": "1.0.0",
    "author": "my-team",
    ...
    "dependencies": {
        "custom-package": "git+https://git-codecommit.region...",
        ...
    }
}
// full URL example
"git+https://git-codecommit.region.amazonaws.com/v1/repos/my-repo"

And that was it! After completing the migration, I was able to run npm install and custom dependencies from CodeCommit were installing just fine without requiring any access credentials whatsoever!

Conclusion

Solution that I demonstrated in this article should be applicable to any kind of pipelines built on CodeBuild, not only build and deployment types as I suggested in the beginning :). Let me know, however, if you have any issues or questions in the comments below and I’ll be happy to answer them!

Thanks so much for reading this article! If you would like to see more content like this in the future please leave a like and share this article. Till next time!