How do we secure ManoMano applications with our DIY pipeline ?
Securing ManoMano applications before they get delivered to the production environment is of utmost importance. Due to the whopping number of micro-services, automation of security checks is a crucial need for us. Many companies have adopted commercial tools to enforce the security compliancy; however, at ManoMano, we prefer doing it in our way. In this article, we will present our DIY security pipeline that relies exclusively on open source tools. In the first section, we will make an overview of our monitoring architecture, then we will review the different security components that are required by each ManoMano project to be security compliant, in the third section we will present how we managed to integrate the security checks in the CI/CD, and in the last section we will see how do we enforce the security checks.
Overview of the monitoring infrastructure
At ManoMano, we rely on the Gitlab CI/CD — Continuous Integration and Continuous Delivery (CI/CD) — to lint, test, and release out the code to the production environment in an automated way. What a better place to integrate the security components than in the pipeline itself as shown in Figure 1. Whenever a git push is triggered by a developer (1), a CI/CD pipeline gets executed along with its security components (2), namely gitleaks for detecting hard-coded secrets, Gemnasium for scanning dependencies, SonarQube for reviewing the code quality and finally Trivy for performing a scan of the Docker images. A detailed overview of the tools will be provided in the next section. Upon execution, the security components will issue a report in the CI/CD itself and upload the logs on our logging platform (DataDog) that houses all our security logs (3). In order to upload this information on DataDog, each tool has been slightly updated to integrate a component that interacts with the DataDog collector. The DataDog monitors will trigger and send the alerts to our SIEM (4), which based on the criticality will forward the alerts to the Defect Management System (DMS). A SIEM is a security monitoring platform where all the security events are collected and processed (see our article for more information). Eventually, the DMS will create and assign JIRA tickets to the corresponding team (6) and (7).
It is worth noting that adding the security checks to the pipelines allows us to embed the project name, team name, and domain name in the logs and later on design security dashboards customized per domain.
Security requirements
A security check in the CI/CD pipeline monitors a given stage of the software development and ensures that any security issue is prevented and reported to the security team as well as the developer before the code goes to production — in a nutshell the objective is to minimize the security risks. Whenever a security check is added to the CI/CD pipeline, beforehand we ascertain the efficiency of the tool based on the false positive and true positive metrics so we feel confident that the tool best suits the company needs both now and in the future. Our DIY security pipeline is composed of the following tools:
- Software Composition Analysis Tool (SCA)
The SCA analyzes the dependencies of the projects and makes sure those dependencies are not vulnerable. The tool parses dependency files, e.g. composer.lock for PHP projects, extracts the list of dependencies along with their version, and verifies if a given version matches their vulnerability database. At ManoMano, we rely on a modified version of Gemnasium for scanning the dependencies. In Figure 2, we present an excerpt of a Gemnasium report featuring the severity of the vulnerability, the CVE identifier, the report URL for further information, the vulnerability scanner, a short description of the issue, the affected package and its version, the solution if found, the path to the vulnerable library, and the file where the dependency has been found. In addition to the normal Gemnasium report, we do highlight the path from the parent library symfony/framework-bundleto the vulnerable library symfony/http-foundation. This path sheds the light on the parent library that needs to be updated or upgraded. It is worth noting that all the vulnerabilities are sorted out by criticality ranging from Critical to Unknown for sake of clarity.
- Static Application Security Testing (SAST)
The SAST reviews the code quality and spots the security best practices violations. For instance, using the hashing function “MD5” instead of “SHA256” might cause a hash collision or setting up an endpoint without restricting the origin might hide a potential vulnerability underneath. At ManoMano, we opted for a light code analyzer (SonarQube) as complex SASTs turned out be inaccurate, see our previous article for further information. A tool that provides too many false positives loses credibility to the developer’s eyes. We also think that training developers based on our human first strategy is more beneficial — developers spot vulnerabilities more easily but also avoid introducing them. To this end, all the ManoMano developers are enrolled in a training course related to the OWASP top 10 with practical exercises.
- Docker image vulnerability scanner
Most of ManoMano code is running in containers, except for the legacy code which represents a tiny part of our code. Relying on containers also implies using a distribution and thus tracking and updating vulnerable packages becomes necessary. To this end, a vulnerability scanner of Docker images based on a modified version of Aquasec Trivy tool has been embedded into our CI/CD pipeline, which performs a scan of all the freshly built Docker images and reports any vulnerable packages. In addition to scanning packages, trivy also addresses the distroless distribution by e.g. analyzing the libraries found inside Go like program. Nevertheless, the tool is not able to scan Java / PHP / Node.js standalone executables yet. An issue and a pull request were opened a few weeks ago.
- Detect secrets in source code
Secrets hard-coded in the source is something prohibited at ManoMano no matter the environments including integration and sandbox environment. By using Gitleaks tool — a SAST tool for detecting hardcoded secrets like passwords, API keys, and tokens in git repos — we easily spot developers’ sins. The tool is composed of two types of profile that are composed of regular expressions: one profile for parsing source code files, e.g. Java files and another profile for parsing the markdown and configuration files, e.g. YAML file. Splitting the configuration into two profiles allowed us to cut out the false positives and at the same occasion improve the accuracy. Whenever a false positive is reported, the commit associated with is whitelisted; and on the contrary, developers are required to clean up their project.
How did we integrate the security check in our CI/CD ?
Every project embeds a file named “.gitlab-ci.yml” at its root folder that contains the different Gitlab CI/CD jobs. The “.gitlab-ci.yml” file inherits from a core template, which defines the different jobs that will be executed by the CI and the CD. Using core templates allows us to standardize all the pipelines, i.e., same nomenclature, same stages, etc. It is worth noting that the “.gitlab-ci.yml” file requires a code owner approval before merging the updates to the main/master branch, preventing a malicious developer to disable permanently the security checks. In Figure 3, we took an example of a Python project’s .gitlab-ci.yml that inherits from the core template “python.gitlab-ci.yml”, which itself depends on (1) “python/ci-init.yml” for downloading the dependencies, (2) “python/ci-build-test.yml” for linting and testing the application, and (3) “security/ci-python.yml” for running the security checks. For sake of simplicity, the CD jobs are not present in Figure 3.
How do we enforce the security checks?
Blocking the pipelines of projects that are not security compliant is a tricky question. When we were envisaging this possibility, we faced a strong no go from developers’ side. One of the issues that were raised — if we need to release out a hotfix urgently in the production environment and one of the security checks fails because there is no security patch that can address a security issue, how shall we proceed ? To this end, we provided to the developers both an environment variable SKIP_SECURITY_JOBS and a commit message pattern to allow them to disable the security steps. Although we strongly discourage the use of such variables, we were afraid that it would become a bad habit and that by default all the pipelines would skip the security jobs. To tackle this problem, we made the decision of not blocking the pipelines but instead making developers, IT leads, and engineering managers aware of the security issues along with their criticality via defects. However, some IT leads have decided to enforce the blocking mode. In our case, whenever a vulnerability with a severity critical or high is detected, a defect is created and passed on to the corresponding team. We leveraged the DMS that is used for tracking the bugs to also pass on the vulnerabilities. Figure 4 displays a graph representing the evolution of number of defects per week by means of candles.
In addition to integrating security tools, we are also cooperating with developers to get their feedback in the aim of improving the tools. Whenever Gemnasium and Trivy identify vulnerable packages, it does not systematically imply that the vulnerable components will pose a security threat, e.g. if the vulnerable functions are not called. When dealing with a few projects, the security engineers have the time to delve into the code; however, it is unthinkable to verify over 200 micro-services manually. As a matter of fact, only critical vulnerabilities will be reported via defects in the first hand and then in the second hand security issues with severity high will be included.
Last but not the least, we also need to (1) ensure that all the new projects run the security checks and (2) track down the projects that skipped the security checks. To this end, all the jobs executed in the CI/CD are forwarded via a Gitlab webhook to a micro-service that verifies whether or not the security checks have been disabled. All the projects that omitted the security checks are reported in the DataDog security dashboards.
Conclusions
We presented you our extensible and scalable DIY pipeline framework. The goal is always the same — bring value to find the right balance between security and agility for developers. If you would like to share your insights about your security pipeline, do not hesitate to post a comment below this article. We also invite you to watch our Youtube Video about our CI/CD strategy.
