Visualize AWS Cloudtraillogs with ELK stack and Kinesis.

There are multiple solutions how to do it, but we will consider only delivering through the Kinesis Stream.

• a) CloudTraillogs → s3 ← logstash → elasticsearch ← Kibana
• b) CloudTraillogs → CloudWatchLogs → KinesisStream ← logstash → elasticsearch ← Kibana

a) First option is using s3 bucket.

1. Enable AWS Cloudtraillogs write logs to s3

AWS console → Cloudtrail → Trails → Create Trails
At this point you can configure logstash with s3 input plugin to pull cloudtrail logs from s3 bucket. However such solution has a few limitations:

• Currently s3-plugin doesn’t support a nice way to keep state of which files are already pulled from the bucket , so if you are using multiple nodes of logstash in your infrastructure then probably you get into the issue.

b) Second way is using CloudWatchLogs and KinesisStream

1. Configure Cloudtraillogs send logs to CloudWatchLogs:

Here is how you can do it via console/cli:

Sending Events to CloudWatch Logs - AWS CloudTrail

or easily with Terraform module https://github.com/cloudposse/terraform-aws-cloudtrail

2. Create Kinesis Data Stream (cloudtraillogs_stream)

Creating and Updating Data Streams - Amazon Kinesis Data Streams

3. Create CloudWatchLogs Subscription Filter

Using CloudWatch Logs Subscription Filters - Amazon CloudWatch Logs

4. Pull Cloudtrail logs from KinesisStream with logstash-kinesis plugin , parse and push them to the Elasticsearch.

Example of logstash config file is below:

input {
kinesis {
id => “kinesis_cloudtrail_logs”
kinesis_stream_name => “cloudtraillogs_stream”
region => “eu-central-1”
application_name => “ls_cloudtraillogs”
profile => “logstash_cloudtraillogs”
codec => “cloudwatch_logs”
tags => [“kinesis”,”cloudtrail”]
}
}

filter {
if “kinesis” in [tags] and “cloudtrail” in [tags] {
json {
source => “message”
skip_on_invalid_json => true
}
mutate {
remove_field => [ “message” ]
}
if [sourceIPAddress] =~ /.+/ and [sourceIPAddress] =~ /^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/
{
geoip {
source => “[sourceIPAddress]”
add_tag => [ “geoip” ]
}
}
}
}
if “cloudtrail” in [tags] and “kinesis” in [tags] {
elasticsearch {
id => “cloudtraillogs_output_plugin”
hosts => “elasticsearch_hosts”
index => “cloudtraillogs-%{+YYYY.MM.dd}”
user => “elasticsearch_user”
password => “elasticsearch_password”
}
}

5. Next step is creating Visualization and Dashboards in Kibana with data which we delivered to the Elasticsearch.

Example of Visualizations:

Geographical location by geoip address field

Top errors

Count by EventName field

Top users by their activity(requestParameters.userName field)

There are numerous fields in cloudtrail logs by which you can create variety of visualisations and put all it into a single dashboard.

Example of number/variety of fields in Kibana UI

Count events by eventType field.