Integrating Deception with DevOps
DevOpsSec or DevSecOps is the process of integrating security best practices as part of the development and deployment process. Every organization has its own DevOps methods, and this blog focuses on a few ways to integrate deception into the DevOps cycle to provide internal security monitoring
Jenkins is one of the more widespread continuous integration (CI) and continuous delivery (CD) solutions used by organizations to perform various build and deployment tasks
Since Jenkins is such a popular DevOps engine to manage and operate software delivery cycles, it needs access to various credentials, keys, secrets, etc., to authenticate with external systems to deploy and manage applications
The Jenkins systems itself could be one of the primary targets for attackers to deploy crypto miners, ransomware, targeted malware, and other malicious applications on remote systems. There have been multiple instances of attackers targeting Jenkins via stolen credentials and exploitation.
Targeting Jenkins Servers
A recent post from the NCC group covers hundreds of vulnerabilities that attackers can exploit in Jenkins plugins. Once attackers have access to the Jenkins server, they can steal credentials from Jenkins, create or modify build scripts, decrypt secrets stored in credentials.xml file, or perform other malicious activity.
The credentials.xml file stores both global and system credentials.
Passphrases stored inside the
credentials.xmlfile can be decrypted using the following script.
println hudson.util.Secret.decrypt(“{AQAAABAAAAAgPT7JbBVgyWiivobt0CJEduLyP0lB3uyTj+D5WBvVk6jyG6BQFPYGN4Z3VJN2JLDm}”)Attackers can also modify Jenkins scripts to add users to target systems for persistence
Integrating deception with the build process
DevOps teams should assume the possibility of Jenkins or other CI/CD systems getting compromised and integrate deception into their CI/CD cycle.
These are a few of the possible areas where DevOps teams can deploy deception:
- Deploy decoy Jenkins systems to lure attackers away from production systems.
- Deploy decoy documents to detect intruders stealing organization intellectual property.
- Add deceptive credentials to the Jenkins credential store and monitor for the use of these credentials.
- Deploy decoy Jenkins pipelines which, when executed, build code from decoy GitHub Repos and deploy tasks on decoy systems.
- Deploy decoy Github credentials, Office 365 mailboxes, etc., and monitor for the use of these credentials
- Integrate deceptive workloads as part of production workloads by deploying decoy databases, containers, file servers, etc., along with production assets.
- Deploy deception as part of their full-stack deployments. DevOps team can package deception as part of Cloud formation stacks, MEAN stack, LAMP stack, ELK stack, etc., and distribute breadcrumbs pointing to deceptive stacks
Passing deceptive credentials to applications and lambda functions
Developers pass credentials, secrets, keys, and other access tokens in various forms to applications to get access to backend resources like databases, file servers, etc. Similarly, DevOps teams can pass deceptive credentials to production workloads to lure attackers to decoy systems.
Passing deceptive servers, credentials, or secrets as environmental variables. Pass deceptive hostnames or credentials as environment variables, parameters, in configuration files, etc. Attackers will target running processes and configuration files to search for database and file server credentials.
Passing deceptive servers or credentials in docker config files. Attackers who are in the network can search for parameters passed to docker containers. DevOps teams can plant deception in configuration files and misdirect attackers to decoy systems.
Passing deceptive key-store parameters as command line arguments. . Applications use key-stores to hold and retrieve secrets and credentials. DevOps teams can deploy deception in key-stores and pass deceptive key-store parameters as command-line arguments or environmental variables and monitor if any attackers are reading and using this deceptive data.
Strengthening Defenses Against API Attacks
DevOps teams can deploy API deception to learn and gather intelligence on how adversaries target critical APIs. Most of the APIs are external facing and subject to different types of attacks
The decoy API endpoints can be instrumented with additional detection mechanisms to identify various types of attacks.
During the deployment of Web apps, developers or security administrators can inject fake website paths accessing deceptive database entries.
/backup/backup.php
/manage/dbusers.htmlDevOps teams can monitor for attackers targeting deceptive paths and exploiting vulnerabilities, to learn their attack methods.
Deception for Backup & Restore
Many organizations are moving to the public cloud for backup services. DevOps teams can deploy deception to detect any attacks originating from inside the network or externally on the organization’s public cloud infrastructure that arise due to misconfigurations.
Conclusion
As the defacto DevOps engine, Jenkins needs to be secured and configured correctly. Exploitation, stolen credentials, and misconfigurations can result in an exposure of the various secrets stored in Jenkins. Attackers can modify pipelines and deploy malware or add accounts for persistence. DevOps teams must, therefore, look to new security controls such as deception technology to defend Jenkins and other CI/CD solutions from compromise.
Additional reading on dumping credentials and attacks on Jenkins systems:
