ELK v5.1.1 in Amazon Container Service. AWS CloudTrail Integration.

There are some products , that offer CloudTrail logs aggregation on their SaaS ELK solutions, for instance http:\\logz.io. And even Amazon provides it’s own, ELK stack.

But what if we don’t want to use any SaaS solutions and paying extra for Amazon ElasticSearch is not the best option for us (also considering the fact that their offers are nearly outdated: ElasticSearch v2.3 and v1.5).

We decided to go our way and build recently released ELK v5.1.1 stack on our AWS environment, which will process CloudTrail logs and present them as nice statistics\graphs in Kibana web interface. Furthermore we wanted to get it running in containers on AWS ECS.

Planned data processing scheme with ELK stack

Implementation and gotchas

Official ELK images from Docker Hub are used in this setup. We didn’t have to modify Kibana image, it’s just perfectly fit our needs. As for ElasticSearch we have to pay attention to, so called, “Ulimits”, one of those “vm.max_map_count” is being set on EC2 machine at launch:

sudo sysctl -w vm.max_map_count=262144

Another one “nofile” can be set in ECS task definition.

However LogStash require a little more attention, (I will tell the whole story futher in this article).

CloudTrail data

AWS CloudTrail is a web service that records AWS API calls for your account and delivers log files to you.

This is what guys at Amazon say, but what’s hidden behind “delivers log files to you”. Well, not really to you as a person, but to a JSON-formatted files packed in GZIP archives to specific S3 bucket on your AWS account. JSON format is native to ElasticSearch, sounds like nice coincidence, nuh? Yes, same format but each CloudTrail data archive is a bunch of JSON objects in an array and ElasticSearch doesn’t expect any arrays. Therefore we need to figure out how to:

• Pull archived logs from S3;
• Unzip files;
• Modify CloudTrail files so not an array, but a list of JSON objects will be processed through Logstash to Elasticsearch.

Logstash games

I tried few options to make Logstash send data from S3 to Elasticsearch DB. And also had to modify official image.

First try: “dark side of the Force”
The idea was to set up a script on EC2, that will run every 5 minutes, read last file from S3, modify JSON with help of JQ and save data to local JSON file, that is picked by LogStash with a file{} input plugin. Later on, it turned out a little bit superfluous.

Second try: “The Force”
After playing with scripts and googling around I found a couple of Logstash plugins that allowed my idea to come true.

• S3 input plugin, that is installed in official docker image and works out-of-the-box and allows Logstash to read from S3.
• CloudTrail codec plugin. This guy was a bit tricky.

First of all, CloudTrail codec plugin was not installed in official image. Well, not a big deal, I tried to install it and got a whole lot of errors on my screen. I didn’t really want to get back to my first option “dark side of the Force”, therefore dug a bit deeper. All of the errors were swearing about dependencies mostly about version of “logstash-core-api-plugin”. All Plugins are nothing but Ruby gems hosted on RubyGems.org. And apparently, plugin, we wanted to install, was not updated yet on RubyGems.org. Ok, we have a possibility to install plugins locally. Updated Logstash plugin source code was available and grabbed from this GitHub repository, then built with RUBY:

# gem build logstash-codec-cloudtrail.gemspec
WARNING: ...
...
  Successfully built RubyGem
  Name: logstash-codec-cloudtrail
  Version: 3.0.0
  File: logstash-codec-cloudtrail-3.0.0.gem

At the beginning I tried to use all three official docker images for ELK , not to rebuild any of those. But later on, I realized I’m at a tipping point where this is now something that we can’t avoid. We had to cover three steps:

• Copying custom Logstash configuration;

input {
 s3 {
  bucket => "testcloudtrail"
  access_key_id => "Access_key_here"
  secret_access_key => "Secret_here"
  delete => false
  interval => 60 # seconds
  type => "cloudtrail"
  codec => "cloudtrail"
  prefix => "AWSLogs/AMAZON_ACCOUNT_ID_HERE/CloudTrail/"
  sincedb_path => "/usr/share/logstash/data/cloudtrail_s3_sincedb"
 }
}
filter {}
output {
 elasticsearch { hosts => ["elasticsearch:9200"] }
 stdout { codec => rubydebug }
}

• Instructing Logstash to use this configuration on start;
• Copying and installing, locally built, CloudTrail codec plugin.

All files were prepared and uploaded to our GitHub repo. Then I just built docker image and uploaded it to Amazon EC2 Container Registry, in our case it was the best place where to store docker images, since we planned to use Amazon EC2 Container Services to run this ELK stack.

PS C:\temp\elk_v_5\logstash> docker build -t logstash5 .
Sending build context to Docker daemon 12.29 kB
Step 1 : FROM logstash
latest: Pulling from library/logstash

386a066cd84a: Pull complete
75ea84187083: Pull complete
...
Status: Downloaded newer image for logstash:latest
 ---> 0ec18fcb5f14
Step 2 : COPY docker-entrypoint.sh /
...
Step 3 : COPY logstash.conf /opt/
...
Step 4 : COPY logstash-codec-cloudtrail-3.0.0.gem /opt/
...
Step 5 : RUN /usr/share/logstash/bin/logstash-plugin install /opt/logstash-codec-cloudtrail-3.0.0.gem
 ---> Running in 7f4c7c21cdfc
Validating /opt/logstash-codec-cloudtrail-3.0.0.gem
Installing logstash-codec-cloudtrail
Installation successful
 ...
Step 6 : CMD -f /opt/logstash.conf
...
Successfully built 4f12af16f6c7
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' pe
rmissions. It is recommended to double check and reset permissions for sensitive files and directories.

Even though in this article I used docker for Windows to build image, I strongly recommend to do it on *NIX machine instead to avoid issues with a permissions (see last notification above).

Persistent data storage

We use Amazon EFS to store Logstash “sincedb” (file, with data of last read from S3)

EFS is mounted to EC2 instances (created from pre-baked AMI) with bootstrap script “user data” (can also be done via fstab: equivalent solutions) and then mapped (with R\W permissions) to containers. So in this manner we’ve created persistent storage and now we don’t have to worry about possible data loss when restarting containers.

One of gotchas: I had an issue when setting this up, container data was not written to EFS, it was caused by “device lock“. I might have started some processes that could have been locking my EFS mount point and wasted a couple of hours trying to fix this. If you ever face the same situation, just give up and restart your EC2 instance. After restart problem has never occured again.

The icing on the cake

Task definition is a one of fundamental AWS ECS components, and you will have to create the one for sure, so I’m pushing our task definition for ELK v5 stack to our GitHub repo.

So at the end we’ve got a following picture:

What’s left is to go to flexible Kibana interface and configure dashboard for your taste with nice visualizations of things, happening in your Amazon infrastructure. Enjoy!

References:

• AWS EC2 AMI for Auto Scaling group;
• AWS ECS and ECR;
• AWS EFS;
• AWS Cloutrail;
• Official docker images:
• ElasticSearch;
• Logstash;
• Kibana.