Structured Logging for Rails using ELK

  • has this route or controller been used at all in the last six months?
  • how did this advert’s price get set to zero, and by who?
  • we’re seeing a sudden uptick in transactions per second every day at 1 AM — is it one of our API users, or is it a bot trying to scrape us?
  • how we obtain events & emit log entries in Rails
  • how those events are sent to Elasticsearch via Filebeat
  • how Elasticsearch indexes those events via ingest pipelines
  • how we configure Kibana to display those events

Structured Output Format

Our log subscriber outputs one line of JSON per request. We use JSON as it can then be directly ingested in to Elasticsearch with little to no modification. This makes it much more reliable and flexible than a more traditional human readable log file. A reformatted example follows:

  • params contains all parameters that we can safely record or index, including parameters supplied as part of a PATCH/POST/PUT request
  • action and controller are duplicated from route to make it easier to search for them specifically within ELK
  • session_id is recorded partly so we can attempt to replay a user's actions from within ELK
  • original_path includes any query string parameters: this is primarily useful for discovering parameter decoding edge cases in our application or Rails itself

Structured Output Code

The code used to generate this output is split in to two main sections: the subscriber itself, and a routine to flatten the parameters.

Flattening Parameters

Parameters in Rails can be represented internally as objects containing objects, eg:

  • trapping instances of Rack::Test::UploadedFile or Rack::Multipart::UploadedFile and representing them only as [FILE UPLOAD #{filename}]
  • truncating any string longer than 120 characters

Log Subscriber

The log subscriber attaches itself to the ActionController on the process_action and redirect_to methods. process_action handles publishing the log event to ELK, whereas redirect_to sets a thread local variable for process_action to later pick up and register as a redirect.

  • runs the request’s parameters through the flattener
  • attaches exception information, if present
  • attaches relevant headers (eg HTTP_HOST, HTTP_USER_AGENT, etc)
  • obtains a user ID and/or administrator ID from the session, if present
  • attaches overall, database and view runtimes
  • attaches route as a combination of controller and action to make searching easier in ELK
  • adds a @timestamp
  • encodes the result as JSON and emits it to the log

Filebeat

Filebeat is part of the “beats” suite of software, published by Elasticsearch as part of their ELK stack. Filebeat reads logs files, applies some basic filtering, and sends the contents on to Elasticsearch.

  • we tell Elasticsearch to use the ingest pipeline named rails on each event
  • the events from elk.json.log are tagged rails, which means they're then sent to the logs-rails-* indices
  • json.keys_under_root means we treat each log entry as the entire event, rather than embedding its contents in a json object when we send it to Elasticsearch

Elasticsearch

Apart from regular Elasticsearch configuration, there are two main things we had to consider when setting up ES to handle Rails log data:

  • Ingest Pipelines
  • Index Templates

Ingest Pipeline

We created a simple ingest pipeline named rails. It uses the ingest-geoip and ingest-useragent plugins to find a location for the request's IP address and decode the HTTP User Agent respectively.

Index Template

Our index template for indices matching logs-* is based heavily on the defaults that filebeat provides for its indices, with some modifications:

  • we limit the index to one shard and zero replicas, to save space on our single machine Elasticsearch setup
  • we copy the geoip and user_agent mapping definitions from the nginx-access default mappings into our rails section
  • we specify dynamic mapping rules for our rails parameters:

Kibana

Once the data is in Elasticsearch we can use Kibana as a dashboard to analyze it. We found it useful to set up Kibana before setting up filebeat: its development tools allow for easy interaction with the Elasticsearch API.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store