Importance of CI/CD Security: Solarwinds & Codecov Aftermath
The solarwinds and codecov attacks led to a lot of upheaval in the cyber industry but shone a light on the fact that the security rigor of securing software delivery pipelines (or CI/CD systems) was woefully lacking.
Doing a bit of deep dive into software supply chain, it can be described as: all the components and processes that go into the creation of software. In Modern software, the software supply chain would be:
- Components:
a. Proprietary Code
b. Open-Source Code Dependencies
2. Processes Source-code to Production Process Going through the CI/CD or Devops Pipelines.
Specifically talking about the CI/CD pipelines, the usage has exploded and software development lifecycle (SDLC) has evolved:
- Software developments has moved from shipping large software at once to multiple deployments/day.
- Led to adoption of specific CI/CD tools like jenkins, Circle CI, Argo CD etc & further, many such tools have now transformed from being self-hosted to cloud-based CI/CD systems like GitHub Actions, GitLab CI etc
- The increasing ease of use of such systems has led to Shift-left of devsecops, with developers writing their own workflow code for their specific purpose.
- Third-party code is now increasingly part of CI/CD, with ~23000 actions on GitHub Actions Marketplace or 1900+ Jenkins plugins and 3500+ CircleCI Orbs.
- Faster & At-Scale deployment means that the CI environment is becoming more complex and hence reliability and performance become important metrics or CI systems.
The CI/CD pipelines are the keys to the cloud kingdom, the above trends add to both the importance & complexity of the challenge of securing the CI/CD systems.
Lot of the recent breaches have included attackers directly attacking either the CI/CD software providers (CircleCI or Teamcity breach) and/or have breached the CI/CDs of organisations to either push in a backdoor(Solarwinds, Codecov) or exfiltrate sensitive information.
While the industry had done a lot of good work in securing networks, production/cloud systems etc but paradigms for securing SDLC pipes were under-invested and the Biden Admin’s EO of 2021 led to a increasing awareness and a push from the industry with google launching SLSA(Jul-2021), OWASP covering CI/CD top 10 risks(Jun-2022) and even CISA publishing a defending CI/CD paper(Jun-2023).
Our belief at KoalaLab is that to solve the important problem of CI/CD security, paradigms from securing production systems like egress-filtering & network monitoring, observability and posture management(for repositories) should be used with the same rigor.
References:
- Circle CI Breach (Jan-2023)
- JetBrains Teamcity CI/CD Breach (March-2024)
- Azure CLI bug leaking secrets in CI/CD (Jul-2023)
- Acuity confirms leak of federal data through GitHub Repos (April-2024)
- Self-hosted GitHub runners to backdoor (Oct-2022)
- Software supply chain security overview