AWS CloudTrail Logging Logic Flaw

Published in

Stage 2 Security

4 min readJul 23, 2020

Please support us via participating in our hands-on Cloud Hacking (AWS/Azure/GCP/etc.) training course!

TL;DR: AWS’s logging service ( CloudTrail ) buffers logs before writing the logs out to a storage location (e.g. an S3 bucket), enabling a malicious attacker who has compromised an AWS account to redirect logs to a data source they control w/o being detected (e.g. the “DeleteTrail” log entry in CloudTrail can be hidden from defenders).

Demo

This demo will walk you through the steps to reproduce the attack…

The content below describes in detail how this attack can be leveraged by an attacker to defeat detection via AWS CloudTrail logging, with a commonly used SOC detection workflow.

CloudTrail Overview

CloudTrail is AWS’s primary logging service, which enables an organization to log most of their interactions with the AWS control plane (e.g. AWS’s APIs) in order to audit activity within an AWS account (e.g. which user is doing what, when, and where in AWS).

Typically, CloudTrail is configured to output logs into an S3 bucket. For example, the following trail named “awstraining059-CloudTrail-GTEGFTMG7KS4” is set to output to an S3 bucket named “awstraining059-s3bucket-16z0wyp3gs55d”:

If we inspect the files within the S3 bucket, we will see gzip compressed files, which contain log data in JSON format.

Generally these log files are stored in S3 buckets and then subsequently they are transmitted, processed, and analyzed to detect suspicious and malicious behaviors by various detection technologies (e.g. AWS Lambda, Splunk Notables, etc.)…

If an attacker were to compromise an AWS account, access the CloudTrail service, and then delete the trail generating these logs, a “DeleteTrail” event would be generated and written to these log files. Then if subsequent detections (e.g. AWS Lambda, Splunk Notables, etc.) looked for these events, they would be alerted that an attacker has most likely compromised the AWS account.

Due to the design of the CloudTrail service, events are queued up into a buffer before being written to disk in the S3 bucket…

This presents a slight window where if an attacker moves quickly, they may be able to redirect the “DeleteTrail” event to an S3 bucket they control, which is not being monitored by a security team, in order to disable logging and avoid detection for a prolonged period of time…

At first, this may sound complex, but due to the API first design of AWS services, it’s quite easy for an attacker to execute. If the attacker has obtained the correct permissions within the AWS account, they can merely copy and paste multiple cli commands in a row via the command line to avoid detection, based on our current testing.

Additional Research

Additional research in this area may include cleaning out the logs being written to the attacker’s S3 bucket (e.g. removing any entry with a specific IP address)and then writing the logs back into the target’s S3 bucket. Much of this cleaning workflow has been covered in our training course for the last few years.

Recommendations

At a high level, the S3 bucket that CloudTrail is writing logs into, should ideally be in a separate AWS account (e.g. not hosted in the same AWS account being monitored). This does require a slightly more complex setup, but is generally worth the extra time and effort. It prevents many different attack vectors where AWS logs may be tampered with, if an AWS account is compromised. This recommendation does not mitigate the above attack but is a good practice for preventing many of the basic AWS logging disruption & cleaning techniques.

At the current time, I do not see a reliable way to mitigate the above logging disruption technique, as the technique is really leveraging the way CloudTrail was designed to work. I foresee that AWS will probably have to make some changes to the way that the CloudTrail service operates on the backend, in order to effectively mitigate this attack scenario.

Contact Us!

We do cybersecurity engineering (DevSecOps) for many large tech companies and tech start-ups, reach out to us on our website if you have any questions!

Contact Us: Stage2Sec.com