How to incorporate security in CI/CD pipelines?
Learn how we reduced the attack and risk surface in our services and applications in production, implementing eterSAST.
By Andrés Ruiz & Mauro Pappatella, Information Segurity Engineers, from the InfoSec Team at etermax
DevSecOps at etermax
DevOps has made it possible to develop applications faster and with quality, aligning development, operations and testing. However, in many cases, security is not built into these practices, so the result can be an application with vulnerabilities.
This is where DevSecOps comes in, which involves incorporating security as a significant component of DevOps practices.
We could say that DevSecOps invites us to integrate security tools within the DevOps framework in an automated way.
At etermax we decided to start our journey in these practices by incorporating security controls in the CI/CD pipelines of new and existing projects.
eterSAST: the challenge
With this in mind, we began an investigation and found different ways to implement security controls early and in an automated way, as well as a large number of commercial, free and open source tools.
We set ourselves the challenge of developing a solution that met the following requirements:
- It needed to cover different use cases like back-end projects, front-end projects, and even infrastructure as code.
- CI/CD Jobs of short duration so as not to increase the total pipeline time significantly.
- Easy to implement for developers.
- Scalable and flexible: we wanted to be able to modify the tools we used for each type of scan or add new ones as needed.
- Centralized control: we wanted to make changes and improvements without having to make changes to all the code repositories and CI/CD pipelines that included the tool.
- Global and per-project exceptions: we wanted to have the possibility to accept risks when necessary through exceptions.
- Have visibility of the number of projects that use the tool.
With these objectives in mind, eterSAST was born.
Inside eterSAST
All projects within the company use Gitlab CI/CD as an orchestrator, so the best way to start was with a Gitlab template that could be integrated into other pipelines. In this way, developers only had to add a stage called “infosec” to their .gitlab-ci.yml file and refer to the corresponding template as follows:
stages:
- infosec
include:
- project: ‘etermax/security/infosec-sast’
file: ‘infosec.gitlab-ci.yml’
This is what the developer has to put in their CI/CD yaml. No other changes are needed in their repository.
The following diagram summarizes the architecture of eterSAST:
In the template, to which the developers refer in their projects, different jobs are defined that perform security analysis of different types. Right now, we have four jobs that analyze the following:
- Secrets in the code (sensitive data such as passwords, tokens, SSH keys, etc).
- Dependencies with high/critical severity vulnerabilities as long as they have a fix.
- Errors and bad security practices in configuration files. (Kubernetes, Docker, Terraform).
- Docker images with critical vulnerabilities.
The scheme gives us the flexibility to be able to try and choose different tools to perform such tasks in each job, for example to detect secrets we use Gitleaks but nothing prevents us from modifying this tool for another similar tool such as Talisman . The same goes for the other scans, we started using Trivy but nothing prevents us from replacing or trying other similar tools that do the same thing or cover other use cases.
In general we lean towards free and open source tools. However, what we consider important when choosing one of the many tools on the market to perform a security scan or test is:
- The technology it supports.
- If it has a command line utility to easily add it to our pipelines.
- The format of the report output, we generally prefer JSON to automatically process detections but some tabular structure is also important to display detections in the console to developers.
- The false positive rate.
- The flexibility to customize scans and extend functionality.
Continuing with the flow of eterSAST operation, when one of the jobs detects a vulnerability, a report is uploaded in JSON format to an S3 bucket. This event triggers a lambda function that is responsible for parsing the document and applying some filters so that it later creates a “finding” in the AWS SecurityHub service, which is where we centralize alerts and findings from other AWS solutions like GuardDuty.
From the perspective of the developer, it obtains summary information of the finding in the logs of the job in question. As an example we can see a detection in the Docker image scanning job:
When a new finding is generated in Security Hub, a notification is sent via another feature to our private Slack alert channel. When the alert arrives to this channel, an analysis begins and the team that owns the project that generated the alert is assisted to remedy the vulnerability, if applicable.
There is another important part of the solution that runs in parallel and it has to do with monitoring. Particularly, we were interested in being able to know on a daily basis which etermax projects were the most active (with more pipeline executions), which had eterSAST and which did not, as well as knowing which ones had the most detections.
For this we use Gitlab webhooks that allow notifications to be sent to a user-defined API in the face of certain events that occur in a group or project within the organization. In this way we created an API that receives notifications every time a pipeline runs, with all the details of the associated project and the pipeline. This information is stored in an RDS database so that it is very easy to consult this information from Grafana in real time.
In turn, the database is also fed by the detections that we make when scanning the repositories with which we can cross the data, correlate them and obtain much more complete dashboards.
Conclusion
Incorporating DevSecOps practices in a company takes time and effort, and there are many aspects to consider and ways to implement them. One of the best ways to get started and add value quickly is by adding automated scans to CI/CD pipelines. This gives us early information about any vulnerability or potential security issue before services or applications reach a productive environment where the security takes a more reactive role.
With eterSAST we managed to start making the famous “shift to the left” in security and in this way reduce the attack surface and the risk in our services and applications in production, without the need for large investments or invasive modifications in all the pipelines of existing CI/CD.
Finally, the concept of MVP (minimum viable product) was very important for the implementation of the solution to be successful. We worked in short iterations, where the first iterations sought to complete the minimum necessary functionalities to have something functional that added value so as to get early feedback. In this way we had valuable inputs of information on the use of the tool with each increment of the product that shaped it in successive iterations.
