Code, Commit, Secure: Mastering Secret Detection with GitHub Actions

Security remains paramount in the ever-evolving software development landscape, especially when managing sensitive data like secrets and credentials. Our products, which revolve around building robust and secure applications, are no exception. Recognizing the critical need to safeguard our codebase from potential security breaches, we embarked on a journey to directly integrate an effective secret detection mechanism into our development workflow.

Decision to Use detect-secrets

• Community-Driven Development: Being open-source detect-secrets benefits from community-driven development, ensuring continuous improvement and updates.
• Transparency and Trust: The open-source nature allows for transparency in the tool’s functionality and algorithms, fostering trust and reliability.
• Comprehensive Secret Detection: Capable of detecting a wide range of secret types, including API keys, tokens, and passwords.
• Customizable Scanning: Offers the ability to customize scanning rules and patterns, making it adaptable to various project needs.
• Permissive Licensing: The MIT license allows for free use, modification, and distribution, making it ideal for commercial and private projects.
• No Copyleft Restrictions: This avoids the complexities and obligations often associated with copyleft licenses.
• Seamless Workflow Integration: Easily integrates with GitHub Actions, allowing for automated scanning within CI/CD pipelines.

Challenges Faced

Throughout the development process, we encountered several challenges:

• Secrets in Code: Unintentionally committing secrets to source code was a significant risk, especially with multiple contributors.
• Workflow Integration: We needed a solution that seamlessly integrates with our existing CI/CD pipeline without disrupting the development flow.
• Scalability and Efficiency: The solution had to be efficient and scalable, capable of handling a growing number of repositories and code changes.
• Utilizing OIDC for Secure AWS Access: To address the security concerns around storing AWS credentials, we adopted OpenID Connect (OIDC). OIDC allowed us to authenticate with AWS securely without storing sensitive AWS credentials in our GitHub repository. This approach significantly reduced the risk of credential leakage.
• Using S3 Bucket for Centralized Secret Baseline: We chose to use an AWS S3 bucket to store our centralized .secrets.baseline. This decision was driven by the need for a secure, centralized location for our baseline file, which is crucial for the effective functioning of detect-secrets. The S3 bucket provided a scalable and accessible storage solution, ensuring that our baseline file was consistently up-to-date across all scans and accessible to our GitHub Actions workflow.
• Efficient Notification Across Multiple Repositories: With over 100 repositories in our project, ensuring timely and efficient notification for any security issues was daunting. We needed a solution to alert us immediately whenever a potential secret was detected in any of these repositories. Manual monitoring was not feasible due to the sheer volume and frequency of updates across these repositories.
• Integrating Slack for Real-Time Alerts: We integrated Slack into our GitHub Actions workflow to overcome the challenge of efficiently managing notifications. This integration allowed us to receive instant alerts whenever a secret was detected in our repositories. By funnelling notifications through Slack, we ensured that our team was promptly informed about potential security issues, enabling quick response and resolution. This approach streamlined our notification process and significantly enhanced our ability to manage and respond to security incidents across our project landscape.

Understanding OpenID Connect (OIDC) in GitHub Actions

OpenID Connect (OIDC) offers a more secure alternative to traditional static credentials. It’s an authentication layer on top of the OAuth 2.0 protocol and allows clients to verify the identity of an end-user based on the authentication performed by an Authorization Server. In the context of GitHub Actions, OIDC enables secure and temporary access to AWS resources without storing sensitive credentials in the GitHub repository.

How OIDC Works in GitHub Actions

• Role Configuration in AWS: We start by configuring an IAM role in AWS that trusts GitHub as an OIDC identity provider. This role includes policies that grant necessary permissions to access specific AWS resources.

GitHub OIDC role

• GitHub Actions Workflow Setup: In our GitHub Actions workflow, we use the aws-actions/configure-aws-credentials@v4 action to facilitate OIDC authentication. This action retrieves a temporary token from GitHub's OIDC provider and uses it to assume the pre-configured IAM role in AWS.

• Temporary Credentials for Secure Access: By assuming the IAM role, the workflow gets temporary credentials that are valid only for the duration of the job. These credentials are used to securely access AWS resources, such as downloading or updating the .secrets.baseline file from an S3 bucket.
• No Static Credentials Required: The key advantage is that we don’t need to store static AWS credentials or GitHub Actions secrets in our GitHub repository. This significantly reduces the risk of credential leakage and enhances the overall security of our CI/CD pipeline.

Github action Flow-diagram

Workflow Description

• Trigger Conditions: The workflow is triggered on push events to the master branch, pull requests to the master branch, a scheduled cron job on weekdays at 10 AM and 6 PM SGT, and manually through the workflow dispatch.

• Permissions: We set specific permissions for the workflow, allowing it to write ID tokens, read repository contents, and write pull request comments.

• Checkout Repository: The workflow checks out the current repository, ensuring it has access to the latest code.
• Cache pip Dependencies: We cache the pip dependencies to speed up the workflow. This step checks for a cache hit and restores the cache if available.

• Install detect-secrets: We install detect-secrets and its gibberish plugin, which is crucial for scanning the code.
• Set up AWS OIDC Authentication: This step configures AWS credentials using OpenID Connect (OIDC), allowing secure access to AWS resources without storing sensitive credentials in the repository.
• Fetch Centralized .secrets.baseline from S3: The workflow fetches the centralized .secrets.baseline file from an S3 bucket, ensuring we have the latest baseline for comparison.
• Scan PR for Secrets: This critical step scans each file changed in the pull request or push to master. It uses detect-secrets-hook to check for secrets and aggregate any findings into an environment variable, AGGREGATED_MESSAGE.

• Scan Entire Main Branch at Scheduled Time: The workflow scans the entire main branch on a scheduled basis. It compares the new scan results with the centralized baseline and updates the baseline in S3 if new secrets are detected.

• Notify Slack if New Secrets are Detected: If secrets are detected, the workflow sends a notification to Slack, including details about the secrets and a link to the pull request.

• Comment on PR with Detected Secrets: Additionally, the workflow posts a comment on the pull request detailing the detected secrets. This step uses a while loop to process each detected secret and format the information into a readable comment.

• Cleanup: The workflow cleans up by removing the local copy of the .secrets.baseline file to maintain a clean working environment.
• Fail the Job if Secrets are Detected: Finally, if any secrets are detected, the workflow fails the job. This step is crucial as it prevents merging potentially harmful changes into the master branch. It is a final check, ensuring no secrets slip through the cracks.

This workflow represents a comprehensive approach to secret detection, integrating seamlessly into our development process. It not only automates the detection of secrets but also ensures that any findings are immediately communicated to the team through Slack notifications and direct comments on the pull request.

Results and Benefits

By implementing this workflow, we’ve achieved an effective and automated secret detection system in our CI/CD pipeline. This has significantly reduced the risk of accidental secret exposure, enhancing our security posture. The automated scans ensure that every code change is scrutinized, providing peace of mind that our codebase remains secure.

Our Github Action is in action.

Notifying users via comments:

Failed the job, so PR can not be merged.

Notify the respective teams on Slack.

Conclusion

Implementing detect-secrets in our GitHub Actions has been a pivotal step in our journey towards a more secure and reliable development process. It exemplifies our commitment to security and serves as a model for others in the community. Our experience and insights will inspire and guide others to enhance their project's security.