Are Your CI/CD Processes Compliant?

How you can build your CI/CD process to address compliance requirements

CI/CD pipelines are critical in the modern software lifecycle. They help to deliver applications faster, in more consistent way. But speed is not enough, you also want to ensure that release went through necessary tests (unit, functional, performance), does not have security vulnerabilities or was not deployed by unauthorized person.

Today many industries have regulatory obligations. Companies need to be compliant to organizational and regulatory guidelines. Usually it is not easy to have such data at hand and up to date, as it is not gathered continuously.

It is very important to ensure that your pipelines are compliant with these guideline as non-adherence may lead to fines, profit loss, security breach, or damage to your brand / reputation.

In this story, you will learn how to get continuous insights in CI/CD processes and verify if they adhere to organization guidelines.

What is compliance in the CI/CD context?

When you plan and implement pipelines to build and deploy your services you already are following company standard and best practices. But compliance is more that just having a standard, you need to have evidence (tracking data) that this standard is continuously followed and easy way to verify not just initially but also ongoing.

Typically this is done without any automation - policy or the compliance officers are checking policies at the end of the deployment process, or only ‘by request’, when you are facing an audit process.

When you are constantly deploying hundreds of services, manually handling compliance data is not sufficient and scalable. It is also very time consuming, boring and annoying.

Your DevOps team need to collaborate with your security and compliance teams to ensure you are tracking necessary checks, approvals and gates are built in the pipelines from the start. Moreover the tracking data must be available at hand for auditors without any additional involvement of the DevOps team.

Security and Compliance Center (SCC) service in IBM Cloud allows you to manage and view your compliance controls. There are a predefined set of control checks related to DevOps compliance that you can use:

You can create a ‘compliance profile’, which will contain set of controls that are relevant to your requirements.

Lets see how you could create evidence for some of these controls in your pipelines.

Shift left DevOps— early insight in application security

Typical, simple process to deploy containerized application is focused on building the code, creating container image and deploying that to container runtime. Any security vulnerabilities are checked late in the cycle, after the image is built, or not integrated in the pipeline at all (done via separate tooling and process). Such checks are also usually limited to operating system and most common services (database, web server) weaknesses, leaving the application code and its dependencies untouched.

You can put these checks early in the cycle by utilizing Code Risk Analyzer (CRA) feature, which is a part of IBM Cloud Continuous Delivery service and can be used in DevOps Toolchains. Toolchains are utilizing Tekton pipelines and CRA functionalities are available as Tekton tasks. These tasks are available in the Open toolchain tekton catalog along with many others.

CRA feature allows you to discover vulnerabilities in your application (Python, Node.js, Java) and OS stack (base image) based on threat intelligence from Snyk and Clair, and provides fix recommendations.

It can also discover misconfigurations in your Kubernetes deployment files based on industry standards and community best practices.

It finally produces a bill of materials that reflects the composition of a deployment and allows you to provide evidence what was checked.

You would typically put these checks in the pipeline that executes during processing of the pull request to detect any issues early in the development cycle and prevent them from going to the production. CRA tasks will put comments with analysis results into the PR:

It also gives you quick summary where you can see which checks were not successful:

If any issues or risks are found, you can see and resolve them early. Once the issues are fixed and all checks are successful, you can merge the PR or even configure to automatically merge if all checks are successful.

Merging of the PR would automatically trigger the delivery pipeline, which usually additionally to scans, performs the following: builds the service, runs tests, creates the container image using your Dockerfile, pushes the built image to container registry, scans the image and deploys to container runtime in staging and/or production environment.

If you are building your application in IBM Cloud, you can utilize IBM Cloud Container Registry service, Vulnerability Advisor scanner in the container registry, and deploy your applications to Red Hat OpenShift on IBM Cloud service.

When your pipeline is executed, from the compliance perspective, it is important to also gather evidence for your tests and scans. IBM Toolchains service has Tekton tasks that create such evidence for DevOps Insight service and Security and Compliance service.

These tasks will help you fulfill following controls:

• Ensure Continuous Delivery toolchain continuously scans source code to identify critical and high vulnerabilities
• Ensure Continuous Delivery toolchain source code meets Center for Internet Security benchmarks
• Ensure Continuous Delivery has Code Risk Analyzer configured that collects a bill of materials for pipeline run
• Ensure Vulnerability Advisor scans images for OS vulnerability detection
• Ensure Continuous Delivery toolchain has unit tests that are continuously run to validate source code changes

Policy based promotions

When your pipeline is executed, various facts are gathered that can be used to make decisions about pushing your application to the given environment. In the pipeline I have added step called evaluate-gate-task, which makes that decision.

How does it work underneath?

The facts about each build are gathered and published to the DevOps Insight service:

Then based on these facts, you can build a policy that will be evaluated during pipeline run. In this example I’m only looking at results from unit tests and code scan results to build the policy. You can add more rules if needed.

Creating centralized polices allows you to enforce checks across different pipelines and not to implement these policies again and again, manually in each pipeline.

Continuous visibility and compliance

Now, that you have implemented security and compliance checks in your pipeline, you can have policy driven deployments. But how can your security or compliance officer check that it is really being followed in day by day operations?

Facts and evidence of checks are gathered during toolchain executions. This data is available to Security and Compliance Center. Your security officer can schedule environment scans, or perform ad-hoc on-demand scan to check current state of the environment.

When the scan is finished, you, acting as the compliance officer, have quick access to how your toolchains are meeting your compliance control goals and which goals are failing or missing:

In this case, you can see that many controls are passed successfully (green color), there are no failures (red) of the ones that were implemented, but there are still some checks not implemented yet (yellow). So depending on your requirements, this toolchain may be already compliant or may still need some additional work to be done, but you can see that clearly and make corrective actions.

Benefits of adding compliance to your CI/CD processes

1. Continuous assurance that your process is compliant.
2. Significant effort reduction to gather data for auditing and regulatory purposes.
3. Instant visibility of the CI/CD pipelines for auditors.
4. Faster and safer software delivery thanks to automated checks and validations which builds confidence in delivering securely.

In this story you have learnt about the basics of the CI/CD compliance and how such solution could be designed in IBM Cloud.