If you want to capture detailed information about the requests sent to your application load balancers, how fast those requests processed by the target and what HTTP error codes were returned by the target and ALB then you want to consider enabling ALB access logs. Some of this information already available in Cloudwatch monitoring but if you want to dig deep in to each individual request (trust me you will) you need ALB access logs.
ALB access logs are not enabled by default. To enable the logs you need to provide the target S3 bucket and update the bucket policy as given in the AWS documentation. Make sure that you pick the correct AWS-Acccount-id for the region your ALBs are deployed. One thing to keep in mind is that this account ID not your account ID but an account ID provided by AWS. (I believe AWS ALBs are hosted in separate AWS account by Amazon for all the customers though they appear in our account). See the AWS documentation for setting up a bucket and bucket policies.
When it comes to pricing, Access Logging is free but we have to pay the S3 storage cost.
Once this is setup, we can use AWS Athena to query the access logs but we needed much more elegant solution than querying the logs using Athena’s SQL interface so we decided to push those ALB logs in to Elasticsearch cluster and create lovely dashboards and alerts if thresholds breach.
To read the logs, we setup Logstash instances and use S3 Input Plugin to read from S3 and push to Elastic cluster. To run a Logstash cluster, easiest approach would be running the standard Docker image that will read, parse and then index them.
However we noticed that running Logstash inside docker was not working as expected due to:
- We had more than 10 ALBs in our production setup generating large number of files every day. When we started our logstash cluster (I remember we had 5 docker instance) we couldn't index the access logs at the rate it was written. We could add more and more Logstash agents to read the S3 bucket but there is a catch to this (see the next point).
- Logstash keeps the track of files that it scans in a local database called sincedb and if we are to launch multiple Logstash agents they would all read S3 bucket from the beginning and create duplicate records in our index. Simply put there is no simple way that Logstash agents in a cluster could share single sincedb.
- If we are using Docker the sincedb file is configured to write in to docker instance and restating the docker would mean we would lose the sincedb state. (To overcome this, we can create EFS mount in the EC2s and have that mount available for docker. then we can change the sincedb path in our dockerized Logstash to share the same — but this is too much of work)
So we came up with a simple solution to overcome this
Firstly we decided to configure different S3 bucket path for each ALB. For example, my User Service ALB creates access logs in a separate path called mys3bucket/alblogs/user-service. We did this for all the ALBs.
With this approach we can now create separate pipelines for each ALB defining the logs path as the suffix then Logstash pipeline would start reading files with the suffix not the entire bucket. This takes away the burden of single pipeline reading large number of files.
Then, as I mentioned in the previous section, to overcome sincedb getting refreshed every time we relaunch docker instance, instead of docker we decided to have a dedicated EC2 instance to run Logstash.
In this instance we ran logstash as service and rather than running multiple agents, we used logstash pipelines for each ALB suffix. When running pipelines we need to make sure that we have enough CPU power in EC2 instance. We used c5.4xlarge instance and you may choose whats best for you. You can check the CPU utilisation from Cloudwatch and then change the instance type accordingly.
Lets dig deep in to Logstash installation and pipeline configurations
To install logstash in your EC2 instance (we used Amazon Linux). Please refer to official Logstash documentation here if you need full details. In summary here are the steps:
Download and install the public signing key:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearchThen create the following to /etc/yum.repos.d/logstash.repo file
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-mdFinally install logstash using YUM
sudo yum install logstashWhen you instal logstash this will, it will register logstash as a service and get started automatically everytime EC2 instance restarted. The configurations are located in /etc/logstash directory and you may update the startup.options file with all startup configurations. In this file you can see LS_HOME is /usr/share/logstash and it’s where you pipelines can be placed.
Please make sure when you create new files in /usr/share/logstash, you create them with logstash user’s ownership since logstash instance runs under logstash user.
Now lets create some pipelines:
As mentioned above, pipelines are useful when defining different configuration files (in our example, different S3 bucket suffix per ALB) and also difference performance requirements (in our case, our processing WAF ALB logs requires more CPU power hence we need to define different pipeline.workers count for that).
So we have logically separate each ALB to a different pipeline and created one logstash configuration file per each ALB. See below for an example, and you can see how we use Grok filter to split fields in to useful names when indexing. You may choose different Grok filters depending on your requirement.
We created a directory called pipeline in /usr/share/logstash and placed individual files there.
input {
s3 {
bucket => "my-alb-logs-bucket"
prefix => "alb-prefix"
add_field => {
"doctype" => "aws-application-load-balancer"
}
}
}filter {
if [doctype] == "aws-application-load-balancer" or [log_format] == "aws-application-load-balancer" {
grok {
match => [ "message", '%{NOTSPACE:request_type} %{TIMESTAMP_ISO8601:log_timestamp} %{NOTSPACE:alb-name} %{NOTSPACE:client} %{NOTSPACE:target} %{NOTSPACE:request_processing_time:float} %{NOTSPACE:target_processing_time:float} %{NOTSPACE:response_processing_time:float} %{NOTSPACE:elb_status_code} %{NOTSPACE:target_status_code:int} %{NOTSPACE:received_bytes:float} %{NOTSPACE:sent_bytes:float} %{QUOTEDSTRING:request} %{QUOTEDSTRING:user_agent} %{NOTSPACE:ssl_cipher} %{NOTSPACE:ssl_protocol} %{NOTSPACE:target_group_arn} %{QUOTEDSTRING:trace_id} "%{DATA:domain_name}" "%{DATA:chosen_cert_arn}" %{NUMBER:matched_rule_priority:int} %{TIMESTAMP_ISO8601:request_creation_time} "%{DATA:actions_executed}" "%{DATA:redirect_url}" "%{DATA:error_reason}"']
}
date {
match => [ "log_timestamp", "ISO8601" ]
}
mutate {
gsub => [
"request", '"', "",
"trace_id", '"', "",
"user_agent", '"', ""
]
}
if [request] {
grok {
match => ["request", "(%{NOTSPACE:http_method})? (%{NOTSPACE:http_uri})? (%{NOTSPACE:http_version})?"]
}
}
if [http_uri] {
grok {
match => ["http_uri", "(%{WORD:protocol})?(://)?(%{IPORHOST:domain})?(:)?(%{INT:http_port})?(%{GREEDYDATA:request_uri})?"]
}
}
if [client] {
grok {
match => ["client", "(%{IPORHOST:c_ip})?"]
}
}
if [target_group_arn] {
grok {
match => [ "target_group_arn", "arn:aws:%{NOTSPACE:tg-arn_type}:%{NOTSPACE:tg-arn_region}:%{NOTSPACE:tg-arn_aws_account_id}:targetgroup\/%{NOTSPACE:tg-arn_target_group_name}\/%{NOTSPACE:tg-arn_target_group_id}" ]
}
}
if [c_ip] {
geoip {
source => "c_ip"
target => "geoip"
}
}
if [user_agent] {
useragent {
source => "user_agent"
prefix => "ua_"
}
}
}
}output {
elasticsearch {
hosts => [ "http://elastic-cluster" ]
index => "alb-index-name"
user => "user"
password => "password"
}
}
Now you need to add these files to pipeline configuration. Update the /etc/logstash/pipeline.yml with those files. For example, your pipeline.yml will look like this
- pipeline.id: waf-alb
path.config: "/usr/share/logstash/pipeline/waf-alb.config"
pipeline.workers: 5
- pipeline.id: service1-alb
path.config: "/usr/share/logstash/pipeline/service1-alb.config"
pipeline.workers: 3You can start and stop logstash service as below
initctl stop logstash
initctl start logstashYou may check the logs in /var/log/logstash-stdout.log and /var/log/logstash-stderror.log for the status of indexing.
Also it will worthwhile to create an index template prior to indexing ALB access logs if you are planning to use geo location fields. Otherwise, Elasticserach default them to number field. Here is a good article on setting up an index template for ALB access logs.
Visualisations in Kibana
Now we have ALB access logs in Elasticsearch we can start creating graphs for various use cases. For starters, we can create TSVB visualisation to see if all the logs and being injected to our index. In this example below I used, Group by alb_name.keyword.
Logstash allows us to get the geo-location based on the IP address, and if you have correctly configured index template to accept geoip.location field as geo_point field, then a lovely heat-map like below can be generated using Coordinate Map visualisation
In addition you can create visualisations to present HTTP errors, response time, Operating system of the customer, Browser name, etc.. Refer to AWS documentation on what each field in ALB access log means and come up with your own visualisations.
Below is the fullscreen view of how it will be shows when we put everything together.
Summary
ALB Access logs are very useful when you want analyse customer requests and present them in a usable manner using your favourite visualisation tool. When you index them in to an Elasticsearch cluster what you can with those data is limitless. Kibana provides good set of visualisation tools and you are limited by your imagination only 😊. And when you use Logstash the correct way, you can index any amount of logs in to your Elasticsearch cluster in no time.
I hope this will help you to setup your own ALB access logs dashboards. Tell me how you are using ALB access logs in your production ALBs.