Visualize AWS Cloudtraillogs with ELK stack and Kinesis.

Illia Sh
Illia Sh
Jan 9, 2019 · 3 min read

There are multiple solutions how to do it, but we will consider only delivering through the Kinesis Stream.

  • a) CloudTraillogs → s3 ← logstash → elasticsearch ← Kibana
  • b) CloudTraillogs → CloudWatchLogs → KinesisStream ← logstash → elasticsearch ← Kibana

a) First option is using s3 bucket.

1. Enable AWS Cloudtraillogs write logs to s3

AWS console → Cloudtrail → Trails → Create Trails
At this point you can configure logstash with s3 input plugin to pull cloudtrail logs from s3 bucket. However such solution has a few limitations:

  • Currently s3-plugin doesn’t support a nice way to keep state of which files are already pulled from the bucket , so if you are using multiple nodes of logstash in your infrastructure then probably you get into the issue.

b) Second way is using CloudWatchLogs and KinesisStream

1. Configure Cloudtraillogs send logs to CloudWatchLogs:

Here is how you can do it via console/cli:

or easily with Terraform module https://github.com/cloudposse/terraform-aws-cloudtrail

2. Create Kinesis Data Stream (cloudtraillogs_stream)

3. Create CloudWatchLogs Subscription Filter

4. Pull Cloudtrail logs from KinesisStream with logstash-kinesis plugin , parse and push them to the Elasticsearch.

Example of logstash config file is below:

input {
kinesis {
id => “kinesis_cloudtrail_logs”
kinesis_stream_name => “cloudtraillogs_stream”
region => “eu-central-1”
application_name => “ls_cloudtraillogs”
profile => “logstash_cloudtraillogs”
codec => “cloudwatch_logs”
tags => [“kinesis”,”cloudtrail”]
}
}

filter {
if “kinesis” in [tags] and “cloudtrail” in [tags] {
json {
source => “message”
skip_on_invalid_json => true
}
mutate {
remove_field => [ “message” ]
}
if [sourceIPAddress] =~ /.+/ and [sourceIPAddress] =~ /^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/
{
geoip {
source => “[sourceIPAddress]”
add_tag => [ “geoip” ]
}
}
}
}
if “cloudtrail” in [tags] and “kinesis” in [tags] {
elasticsearch {
id => “cloudtraillogs_output_plugin”
hosts => “elasticsearch_hosts”
index => “cloudtraillogs-%{+YYYY.MM.dd}”
user => “elasticsearch_user”
password => “elasticsearch_password”
}
}

5. Next step is creating Visualization and Dashboards in Kibana with data which we delivered to the Elasticsearch.

Example of Visualizations:

There are numerous fields in cloudtrail logs by which you can create variety of visualisations and put all it into a single dashboard.

aws-elk

Visualize AWS with ELK stack

aws-elk

Visualize AWS with ELK stack

Illia Sh

Written by

Illia Sh

aws-elk

Visualize AWS with ELK stack