An exercise that helps prioritize security actions based on actual risk and helps us discover weak points is a tabletop exercise (TTX); a tabletop exercise is a simulated discussion or walk-through of a planned response to a hypothetical crisis or disaster scenario. Participants role-play and evaluate their response plans, procedures, and decision-making processes.
Developing TTX based on confirmed incidents allows us to define more probable scenarios and deliver solutions that tackle risks with higher impact or likelihood.
Early this month, CircleCI alerted their Customers about a security incident they were experiencing. After the incident, CircleCI disclosed precise details we could use to improve our defenses.
First of all, based on the information they shared, let’s analyze tactics and techniques used by the adversaries:
- 16/12/2022, Initial access: An engineer had their computer compromised by malware. Because of the names of the files, the affected OS was MacOS. It was likely some phishing (Phishing, Technique T1566) where the adversary deceived the engineer into downloading PTX-Player.dmg and installing it.
- 16/12/2022, Defense Evasion: There is no data about what the malware did (I couldn’t find the hash in Virustotal or other tools), but the antivirus software did not catch it.
- ?/12/2022, Lateral movement: The adversary stole a valid, 2FA-backed SSO session to gain access to production systems (Remote Service Session Hijacking, Technique T1563).
- ?/12/2022, Privilege Escalation: The adversary generated production access tokens to access and exfiltrate data (Valid Accounts, Technique T1078)
- 19/12/2022, Discovery: The adversary engaged in reconnaissance activity. There is no information about what techniques the adversary used for discovery.
- ?/12/2022, Collection: The adversary extracted encryption keys from a running process and data from a subset of databases and stores, including customer environment variables, tokens, and keys.
- 22/12/2022, Exfiltration: The adversary exfiltrated all collected data. There is no information about the technique used, but they probably downloaded the data from the cloud.
- 29/12/2022, Lateral movement: The attacker used the information collected to attack other clients because a CircleCI customer alerted them about suspicious GitHub OAuth activity (Valid Accounts, Technique T1078).
There are already some recommendations for Circleci’s users in the article. Still, we can take a step forward and consider different scenarios and what we should review and improve based on what the adversaries did.
We could begin by analyzing the impact of the continuous integration platform we use having the same breach. Reviewing the continuous integration platform is especially important, as we have seen this happening frequently.
- Do we know what secrets are stored?
- What would happen if all the secrets were invalidated?
- How fast can we regenerate new secrets?
- What information can an adversary obtain from the platform?
Because we know that in the incident analyzed, the adversary compromised a computer without the AV detecting the compromise, another scenario we could explore is what would happen if an engineer’s computer got compromised and the AV did not see it.
- What sessions might the engineer have, and for how long are they valid?
- Apart from the sessions, can the adversary get tokens or ssh keys?
- If the adversary tried to steal passwords, what password would be able to steal, and are all the accounts protected with 2FA? What kind of 2FA?
- What information can be found on the engineer’s computer?
- Is the computer part of an AD, and the adversary might be able to move from there laterally?
Similarly, as we know that the tokens from the continuous integration platform were stolen and used to access customers’ environment, we could analyze what would occur if an adversary was able to steal a token from our continuous integration platform:
- Can we detect connections from different places, not from the CI/CD tool?
- Do we know what actions are part of the day-to-day activity and detect actions that are not regular?
- What permissions will the adversary acquire, and will it be able to move laterally or gain persistence based on the permissions the tokens have?
- How easy is it to invalidate the token?
- What sensitive information can be accessed, and can we track all requests to know what information was accessed?
Those are just examples of scenarios we can discuss based on the incident. We can also simulate one of the scenarios. By doing so, we will be able to confirm how effective are our security measures and guarantee that all that we assumed is true.
Simulating an incident has a higher cost as it requires more time. For this reason, we can do multiple tabletop exercises and, from these exercises, decide what scenario we want to simulate.
As each company is different, it is impossible to know what companies should be looking into after an incident like this; actions that can arise from an activity like this are, for example:
- Reduce the number of secrets in CI/CDs by deleting unused ones.
- Rotate secrets periodically or, if possible, use OpenID Connect (GitHub, GitLab, CircleCI).
- Reduce the session expiration of critical systems.
- Ensure engineers have only the permissions they need.
- Ensure engineers can’t escalate permissions ( For example, by stealing tokens from the CI/CD).
- Ensure the AD is appropriately configured.
- Create new customized alerts.
- Look for the IOC shared by CircleCI in our network.
- Add IP restrictions when possible.
- Hire a pentest based on the assumed breach model.
- …
Through a TTX exercise, each company will be able to determine what actions they need to prioritize and why.
