Importance of CI/CD Security: Solarwinds & Codecov Aftermath

The solarwinds and codecov attacks led to a lot of upheaval in the cyber industry but shone a light on the fact that the security rigor of securing software delivery pipelines (or CI/CD systems) was woefully lacking.

Doing a bit of deep dive into software supply chain, it can be described as: all the components and processes that go into the creation of software. In Modern software, the software supply chain would be:

1. Components:

a. Proprietary Code

b. Open-Source Code Dependencies

2. Processes Source-code to Production Process Going through the CI/CD or Devops Pipelines.

Specifically talking about the CI/CD pipelines, the usage has exploded and software development lifecycle (SDLC) has evolved:

1. Software developments has moved from shipping large software at once to multiple deployments/day.
2. Led to adoption of specific CI/CD tools like jenkins, Circle CI, Argo CD etc & further, many such tools have now transformed from being self-hosted to cloud-based CI/CD systems like GitHub Actions, GitLab CI etc
3. The increasing ease of use of such systems has led to Shift-left of devsecops, with developers writing their own workflow code for their specific purpose.
4. Third-party code is now increasingly part of CI/CD, with ~23000 actions on GitHub Actions Marketplace or 1900+ Jenkins plugins and 3500+ CircleCI Orbs.
5. Faster & At-Scale deployment means that the CI environment is becoming more complex and hence reliability and performance become important metrics or CI systems.

The CI/CD pipelines are the keys to the cloud kingdom, the above trends add to both the importance & complexity of the challenge of securing the CI/CD systems.

Lot of the recent breaches have included attackers directly attacking either the CI/CD software providers (CircleCI or Teamcity breach) and/or have breached the CI/CDs of organisations to either push in a backdoor(Solarwinds, Codecov) or exfiltrate sensitive information.

While the industry had done a lot of good work in securing networks, production/cloud systems etc but paradigms for securing SDLC pipes were under-invested and the Biden Admin’s EO of 2021 led to a increasing awareness and a push from the industry with google launching SLSA(Jul-2021), OWASP covering CI/CD top 10 risks(Jun-2022) and even CISA publishing a defending CI/CD paper(Jun-2023).

Our belief at KoalaLab is that to solve the important problem of CI/CD security, paradigms from securing production systems like egress-filtering & network monitoring, observability and posture management(for repositories) should be used with the same rigor.

References:

1. Circle CI Breach (Jan-2023)
2. JetBrains Teamcity CI/CD Breach (March-2024)
3. Azure CLI bug leaking secrets in CI/CD (Jul-2023)
4. Acuity confirms leak of federal data through GitHub Repos (April-2024)
5. Self-hosted GitHub runners to backdoor (Oct-2022)
6. Software supply chain security overview