Visualize AWS Cloudtraillogs with ELK stack and Kinesis.

Illia Sh
Published in
3 min readJan 9, 2019


There are multiple solutions how to do it, but we will consider only delivering through the Kinesis Stream.

  • a) CloudTraillogs → s3 ← logstash → elasticsearch ← Kibana
  • b) CloudTraillogs → CloudWatchLogs → KinesisStream ← logstash → elasticsearch ← Kibana

a) First option is using s3 bucket.

1. Enable AWS Cloudtraillogs write logs to s3

AWS console → Cloudtrail → Trails → Create Trails
At this point you can configure logstash with s3 input plugin to pull cloudtrail logs from s3 bucket. However such solution has a few limitations:

  • Currently s3-plugin doesn’t support a nice way to keep state of which files are already pulled from the bucket , so if you are using multiple nodes of logstash in your infrastructure then probably you get into the issue.

b) Second way is using CloudWatchLogs and KinesisStream

1. Configure Cloudtraillogs send logs to CloudWatchLogs:

Here is how you can do it via console/cli:

or easily with Terraform module

2. Create Kinesis Data Stream (cloudtraillogs_stream)

3. Create CloudWatchLogs Subscription Filter

4. Pull Cloudtrail logs from KinesisStream with logstash-kinesis plugin , parse and push them to the Elasticsearch.

Example of logstash config file is below:

input {
kinesis {
id => “kinesis_cloudtrail_logs”
kinesis_stream_name => “cloudtraillogs_stream”
region => “eu-central-1”
application_name => “ls_cloudtraillogs”
profile => “logstash_cloudtraillogs”
codec => “cloudwatch_logs”
tags => [“kinesis”,”cloudtrail”]

filter {
if “kinesis” in [tags] and “cloudtrail” in [tags] {
json {
source => “message”
skip_on_invalid_json => true
mutate {
remove_field => [ “message” ]
if [sourceIPAddress] =~ /.+/ and [sourceIPAddress] =~ /^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/
geoip {
source => “[sourceIPAddress]”
add_tag => [ “geoip” ]
if “cloudtrail” in [tags] and “kinesis” in [tags] {
elasticsearch {
id => “cloudtraillogs_output_plugin”
hosts => “elasticsearch_hosts”
index => “cloudtraillogs-%{+YYYY.MM.dd}”
user => “elasticsearch_user”
password => “elasticsearch_password”

5. Next step is creating Visualization and Dashboards in Kibana with data which we delivered to the Elasticsearch.

Example of Visualizations:

Geographical location by geoip address field
Top errors
Count by EventName field
Top users by their activity(requestParameters.userName field)

There are numerous fields in cloudtrail logs by which you can create variety of visualisations and put all it into a single dashboard.

Example of number/variety of fields in Kibana UI
Count events by eventType field.



Illia Sh

Recommended from Medium


See more recommendations