Minimise the risk of misuse in CI/CD pipeline
When having a CI/CD pipeline setup, it is generally configured to be run on commit pushes, pull request creation, etc. Developers are granted access to start the workflow either through customer managed policies or the cloud provider managed policies. It is required that malicious actors don’t misuse the access to tamper with the application and, take control of the servers by executing unwarranted commands using the granted privileges.
In this article, we will be discussing the steps you can take to minimise this threat. We will be giving examples of GitHub Actions and AWS CodeBuild as the CI/CD pipelines with the resources residing within the AWS ecosystem. The same principles can be applied to other workflows as well.
Threats
- Exfiltration of secrets and build artifacts compromising the integrity of the system.
- Injection of malicious commands into the buildspec tampering with the application.
- Taking control of sensitive resources in the system.
- Becoming the weakest link in the application security.
Prevention Strategies
Be strict with the access
When using an AWS IAM role or user in your CI/CD pipeline, be specific with the access it needs. Be as strict as possible and grant the least privileges. For example, if the role needs to download the file from Amazon S3, grant only the s3:GetObject permission instead of granting full access to S3. Even with the read permission, restrict the access to the paths it needs. Instead of granting access to all the files in the bucket, restrict the access to only the files your role needs.
The Condition JSON element allows you to provide the least privileges. You can use it to restrict access to requests originating from a certain IP address. An example of a strict policy using the Condition element is
Do not hardcode secrets
Never hardcode the secrets. The secrets can be API keys or IAM user credentials or any other sensitive information. If you are using GitHub Actions, take advantage of encrypted secrets to encrypt the secrets and store them securely. If using AWS CodeBuild, take the advantage of AWS Secrets Manager or AWS Systems Manager Parameter Store to load the environment variables securely into the build environment.
In AWS CodeBuild, you can load sensitive keys from Secrets Manager like below (ensure that the CodeBuild service role has access to fetch and decrypt the secret)
Diligently do code review
Perform proper code review and ensure that no comprising piece of code is getting merged in. Pull requests making changes to the workflow files must be reviewed by the code owners before they can be merged in. That way, any attempt to mess with the deployment servers would be averted.
Ensure that only trustworthy URLs are accessed to download packages or data.
On GitHub, you can take advantage of CODEOWNERS. For all the CI/CD related files, you can add a code owner so that no change is merged in before explicit approval from a code owner. An example .github/CODEOWNERS may look like the following making the GitHub user username as the owner of all the files in the directory deployment.
Adding an extra layer is also preferable for buildspec and even deployment files (AWS CloudFormation templates or Terraform files). You can add a check to ensure that no critical resource is being modified in those templates. The check would fail if
- a critical resource is being deleted.
- suspicious permissions are being granted to an IAM user.
- a critical resource is being modified.
and many more depending on your use-case. To do so for AWS CloudFormation templates, you can create a stack policy like the one below which disables all updates to the production database.
Restrict build triggers
Restrict the workflow to be run only on certain trigger events. For example, if you have a workflow that deploys your stack, configure it to run on pull request merge events and not all pull request events. Identify when and how frequently the workflows need to run and specify the trigger events accordingly. For example, if you are maintaining a Python library, you must be having a workflow that deploys the library to pip. In such cases, you can configure the trigger as creating a new release or a tag instead of a commit push.
While some workflows are required to be run on commit or pull request events, some may be user actions specific. For those, the workflows should be restricted to certain users. In my last article, I created a GitHub Actions workflow to merge pull requests. The trigger event was a user commenting /merge on a pull request. For such events, there should be an additional check that ensures that the comment is coming only from designated users. This ensures that no one accidentally triggers a workflow that they should not be allowed to.
Restricting workflow start events at a branch level is also a good practice. You can configure the workflows to be run only if the event was triggered on a specific branch. For example, you may have a deployment workflow that is run on pull request merges. You can restrict it such that the workflow is run only if the base branch was of your deployable branches.
With GitHub Actions, you can take advantage of the if tag to restrict access like
Such fine control is not possible with AWS CodeBuild, to bypass, you can use bash scripts like
where the conditional.sh is like
Require manual approval
For deployment related workflows, especially the ones targeting production, you would want to have an additional layer of validation. In AWS CodePipeline, you can add a manual approval before the stack is finally deployed and available to the public. This adds an additional layer of checks that goes through the people of higher authority ensuring that broken/unsatisfactory builds are not deployed.
Have environment-specific IAM roles/users
It is common practice to use a single IAM role or user and grant it access to the necessary resources in all the environments. While it results in maintaining fewer IAM roles and their permissions, it exposes a flaw that the role can access resources out of its scope. It is advisable to have a role for each environment so that the role for the development environment cannot access or modify the resources of the production environment.
With GitHub Actions, the same can be achieved like
We hope the above steps helped you in strengthening the security of your CI/CD pipeline. If you have any other tip that can help in securing the pipeline even more, please share in the comments so that the others can also benefit from it.
Have suggestions that can improve the process? Come join us! Juvoxa is hiring. Send an email to hr@juvoxa.com.
