Complete Guide to end-to-end DevSecOps CI/CD pipeline with open sources

DevSecOps emerges as a pivotal approach in software development, gaining rapid traction as organizations increasingly acknowledge the indispensable role of integrating security practices into their development processes. By amalgamating the collaborative and automated principles of DevOps with stringent security measures, DevSecOps instills a culture of security throughout the software development lifecycle.

Representing a substantial evolution in software development practices, DevSecOps seamlessly blends foundational principles and tools from both software development and IT operations. This integration facilitates the swift delivery of new application features and services, prioritizing security standards throughout. In this article, we delve into the core tenets of DevSecOps and their profound impact on modern software development. We explore strategies for constructing comprehensive DevSecOps pipelines, encompassing a diverse array of security practices and tools, thereby fortifying organizations against vulnerabilities while maintaining operational agility and efficiency.

Why DevSecOps is Important?

Modern organizations are migrating from monoliths to microservices, i.e., a series of loosely coupled services. Microservice architecture provides the ability to be agile, deploy code faster, and gain a competitive advantage. However, there is only one downside to microservices. They are not secure by design. Let us look at the following cases:

• Enterprises use a host of browsers and mobile-based applications that follow microservice architecture. And those microservices, by nature, expose APIs to communicate, which means much more attack surface for hackers.
• There are tons of services used on the microservices, like databases, message brokers, service mesh, etc., which means even more applications and more surface area for a security attack. Microservices use containers, which further raises security concerns. For example, if one Kubernetes pod is breached due to any security issues, all other pods can also be easily attacked (also known as complex attack vectors).
• Using microservices, containers, cloud platforms, and Kubernetes makes applications prone to attack and ensconces security as a fundamental problem in IT organizations.

On the other hand, security teams are not well equipped to handle containerized applications. They need to learn and understand the security implications of all the new technologies (microservices, containers, Kubernetes, etc.) in order to identify the issues. And there is a steep learning curve. Thus, it’s important to make a security shift left in the CI/CD pipeline, i.e., to practice DevSecOps.

Three essential considerations to achieve DevSecOps

Security cannot be treated as siloed operations; it has to be a shared operation between the product team and the security team. To mitigate software risks and frequently deploy software into production, integration of security best practices into all stages of deployment is essential. Refer to the image below:

1. Integrating security as part of CI/CD Pipeline

Every application has to go through a list of security tests: static application security test (SAST), dynamic application security test (DAST), fuzz testing, dependency scanning, container scanning, license compliance, and secret detection.

The security team uses various tools (mentioned below) to understand the security posture of the project or the group of tasks.

• SAST: Sonarqube, Bandit, HCL AppScan, etc..
• DAST: HCL Appscan, OWASP ZAP
• Container Image scanning ( Kubernetes or VM Scanning): Aquasec

The security team developers and project managers should have common access to the security scan reports containing the issues and vulnerabilities. These tools have to be part of the delivery pipeline so that managers can view the information and make decisions to halt the deployment pipeline in case of genuine and critical errors. This way, developers can immediately iterate the merge request (without any context switch) until all the vulnerabilities are resolved.

Let’s see some examples of how DevSecOps pipelines are robust and secure:

Potential vulnerabilities like SQL injection vectors are found during a static application security test (SAST) to scan application source code (like JSP, XML, CSS, etc.). In this case, a delivery pipeline has to fail automatically.

Or, during Dynamic Application Security Testing (DAST), the security tool scans the dependency and finds the application is using older versions of libraries. In this case, a delivery pipeline has to fail automatically.

2. Identifying the risk of a software release in the early stages of software delivery

The risk of a software release is mitigated when it is detected early in the delivery cycle, starting with the build process. However, risk assessment of releases requires a manual review of a large number of logs and metrics produced in the build, test, deploy, and production stages. This can be a manually intensive and extremely time-consuming process. For example, using log analysis, checkmarx can identify SAST regression by identifying changes in communication patterns, i.e., if there are changes to communication between services in terms of talking to new services or using different protocols than before (say, HTTP rather than HTTPS).

Or, developers can use the CheckMarx APIs integrated into the separate dashboard to view the results after the CI/CD pipeline is complete.

3. Enforcing governance and compliance in the delivery pipeline

Application security does not only mean securing the application code-wise or monitoring its risks. It includes issues related to compliance and policies. There are new policies formed to comply with growing regulatory requirements for limiting breaches of application security. For example, the policy to have no port other than 443 should be exposed, application service can be deployed only from the approved registry, or application cannot be deployed in certain regions.

Apart from numerosity, the policy definition changes from one region to another, like GDPR (General Data Protection Regulation) rules.

And security and compliance managers should define policies in time to improve compliance, security enforcement, and developer efficiency. They need a tool to protect the application and software delivery process at every stage. An example of a policy can be an application that cannot be considered for deployment if the testing is not completed.

The main steps in the pipeline are as follows:

1. When a user commits code to a CodeCommit repository, it triggers a Jenkins event, which in turn initiates CodePipeline.
2. CodeBuild compiles the build and uploads the artifacts to either an S3 bucket or JFrog repository. Authentication details, such as scanning tool tokens, are retrieved from the parameter store.
3. CodeBuild conducts code scans using open-source Software Composition Analysis (SCA) and Static Application Security Testing (SAST) tools like OWASP Dependency-Check and SonarQube/PHPStan, respectively. Users have the flexibility to select these tools during deployment.
4. If vulnerabilities are detected during SCA or SAST analysis, the API is invoked, and the scan results are made available. Furthermore, the scanning outcomes are presented on a dedicated dashboard.
5. In the absence of vulnerabilities, the pipeline initiates deployment events, and the build along with Docker images are deployed in the pods.
6. Subsequent to successful deployment, CodeBuild triggers dynamic application security testing (DAST) scans utilizing open-source tools such as OWASP ZAP.
7. If vulnerabilities are identified during DAST scanning, the APIs are triggered to integrate the results into the dashboard.
8. In the absence of vulnerabilities, an approval stage is initiated, prompting an email to the designated approver.
9. Upon approval, CodeDeploy proceeds to deploy the code to production.
10. Slack bots or other open-source bots capture the entire build, deployment, and security state changes, subsequently sending email notifications to subscribed users via SNS notifications.
11. The system tracks API calls and sends notifications for critical events occurring in the pipeline and CodeBuild projects for auditing purposes.
12. Security within the pipeline is maintained through SCA, SAST, and DAST security checks, with the option to incorporate Interactive Application Security Testing (IAST) techniques by combining SAST and DAST stages.
13. Encryption is strongly recommended for both code and artifacts, whether at rest or in transit.

In the following section, we will elucidate the deployment and execution of the pipeline employing the CloudFormation template used in this instance. Additionally, references to open-source alternatives for each tool or service will be provided. When deploying infrastructure via CloudFormation templates, it is advisable to utilize linting tools like cfn-nag to scan for potential security vulnerabilities.

Conclusion

In this post, I presented a DevSecOps pipeline that includes CI/CD, continuous testing, continuous logging and monitoring, auditing and governance, and operations. I mentioned how to integrate various open-source scanning tools, such as SonarQube, PHPStan, and OWASP Zap, for SAST and DAST analysis. I explained how to aggregate vulnerability findings in Security Hub as a single pane of glass. This post also talked about how to implement security in the pipeline using open-source tools.