Decoding the ELB Log Entries
We have taken a close look at all the features of the AWS Elastic Load Balancer and now understand how crucial it is to have a load balancer as a part of your cloud environment. In this blog we will go a step further and explore another feature of this almost magical service – The ELB Access Logs.
For the ELB to capture and deliver access logs, one needs to enable access logging manually.Though disabled by default the access logs can come really handy while trying to track user access patterns, latency, response time and even identify errors in the application.
One of the best features about these access logs is that they can be automated to notify the user of the log entries in a particular interval (the intervals can be anything between 5 minutes and 60 minutes). For example, if you set your interval at 5 minutes, then after every 5 minutes the logs will be delivered to you. (B default this interval is set at 60 minutes.). The ELBs can also deliver multiple logs for the same period under some circumstances like high traffic, log intervals being very short or presence of multiple load balancer nodes.
While ELBs generate log file names that give you a thorough address of the log file (Format as shown), there is a lot more stored in the log entry itself.
{Bucket}/{Prefix}/AWSLogs/{AWS AccountID}/elasticloadbalancing/{Region}/{Year}/{Month}/{Day}/{AWS Account ID}_elasticloadbalancing_{Region}_{Load Balancer Name}_{End Time}_{Load Balancer IP}_{Random String}.logAccess Log Entry
Log entries are stored in access log files. Each log entry holds information of only one request made to the ELB. Shown below is the standard Access Log Entry format.
timestamp elb client:port backend:port request_processing_time backend_processing_time response_processing_time elb_status_code backend_status_code received_bytes sent_bytes “request”The table below will explain the each field/ element in the log entry name in detail.
Field Name Description timestamp Recorded in UTC it represents the time at which a response was sent back to the client. elb Display the name of the load balancer used. client:port Represents the IP address of the client making the request along with its port. backend:port Represents the IP address of the backend instance that processed the client request along with its port. request_processing_time This is the total time taken by the ELB to receive the request from the client and send it to a back end instance for further processing. (recorded in seconds) backend_processing_time This is the total time taken by the backend instance to receive a request from the ELB and start sending the response headers. (recorded in seconds) response_processing_time This is the total time taken to receive a request from the client, process it by a backend registered instance and send a response back to the client. elb_status_code Shows the HTTP status code from the ELB. backend_status_code Represents the HTTP status code from the registered instance. received_bytes This field tells you the size of the request in bytes received from the client. For TCP requests, the bytes include the header as well as the request body, whereas for HTTP requests only the request body is included. sent_bytes This field represents the size of the response in bytes sent to the client. Similar to received bytes, the TCP requests include the request body and header but HTTP requests do not include the header. “request” This field shows what was requested from the ELB and is stored in the following format. “HTTP Method + Protocol: //Host header:port + Path + HTTP version”. (For TCP requests, the url will not be populated and will show three dashes in double quotes and separated by spaces instead e.g: “- – -”)
Cloudlytics is an analytics tool that allows AWS users to monitor the access log files and extract more detailed insights from the data stored. Here are a couple of ways in which Cloudlytics has helped many.
Geographical data: Cloudlytics will use these log files and help track the user access patterns with reference to their geographical locations and present in the form of heat maps thus giving the user a better idea of the request origins.
Backend Status Reports: with Cloudlytics analyzing the log files, one can identify any application issues. For example, if a recurring 404 error (a downtime error) is logged for a particular instance, then a developer observing the reports from backend will know that this particular instance is either facing a lot of incoming error leading to its down time, of has a single point of failure.
One can also acquire reports of latency, unique clients, data uploaded/ downloaded and more.
Originally published at blog.cloudlytics.com on October 1, 2014.
