Container Logging @ Practo

If you’re running Kubernetes production clusters, container logs are important for debugging and monitoring purposes. However, these logs do not persist outside the pod lifecycle, and hence a solution is warranted for debugging later. This article describes the implementation of cluster logging adopted at Practo Engineering.

Kubernetes cluster logging
As with distributed systems, especially kubernetes, logging too is complex, but not complicated, and once set up properly, there’s little maintenance involved.
There are various types of logs that need to be parsed and consumed accordingly. In addition to container logs, there are docker logs, syslogs, kubelet and logs for other kubernetes components.

As described in the docs here, there are various approaches to docker logging. The most popular of these is a node-level logging agent, whereby a pod runs on each node via daemon-set, mounts the `/var/log and /var/lib/docker/containers` directory for every node, parses them, and distributes it to the output store.

Thankfully, Fluentd works well with kubernetes, taking care of all of this for you, and all you need to decide before hand is just…

“ Where will the logs be stored? ”

Lessons with Elasticsearch
Elasticsearch (coupled with Kibana) is the seemingly obvious store for logs, especially for read-heavy use cases. However, if like us, your intended usage of the logging solution is debugging, and lacks frequent querying, it might make more economical sense to push to s3.

S3 archiving
So we decide the output should be s3. What next?
fluent provides templates with different sample configurations here. We’ve used `v1.1/debian-s3`.

To run the fluentd-kubernetes daemon sets , an additional IAM role with access to the bucket is required. (You can also use a key-secret, but that is not recommended.)

Additionally, if you use kops, you can add an additional policy in your cluster config.
Edit your cluster via kops edit cluster ${CLUSTER_NAME} and add the following to the spec:

additionalPolicies:
    node: |
      [
        {
          "Effect": "Allow",
          "Action": ["s3:*"],
          "Resource": ["arn:aws:s3:::<bucket-name>/*"]
        },
        { "Effect": "Allow",
          "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
          "Resource": ["arn:aws:s3:::<bucket-name>"]
        }
      ]

After editing, run kops update cluster ${CLUSTER_NAME} --yes to have the changes take effect.
Also note that an edit in `fluentd.conf` is needed to use IAM profile for nodes.

<match *>
  @type s3
<instance_profile_credentials>
    ip_address IP_ADDRESS
    port       PORT
  </instance_profile_credentials>
</match>

Some more changes like adding time key to messages, and adding tags for namespace, were made by us to fluentd.conf. All of these can be found in this fork here.

practo/fluentd-kubernetes-daemonset

To deploy fluentd, clone and :
kubectl apply -f fluentd-s3-daemonset.yaml

Consumption
After logs have been pushed to S3, you can use athena to query S3 data.
CREATE DATABASE logs;

CREATE EXTERNAL TABLE `pod`(
  `log` string COMMENT 'from deserializer', 
  `time` string,
  `kubernetes` struct<container_name:string,namespace_name:string,pod_name:string,pod_id:string,host:string,namespace_id:string> COMMENT 'from deserializer')
ROW FORMAT SERDE 
  'org.openx.data.jsonserde.JsonSerDe' 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  's3://<bucket-name>/logs'
TBLPROPERTIES (
  'has_encrypted_data'='false')

And that’s it! You’re all set.

Additional remarks
It is advisable to set an expiration policy on the S3 bucket to delete logs automatically, lest cost increase after a while.

Also note that since container logs only capture stdout and stderr, push appropriate logs to stdout/stderr, not to files.

Summary
There are some great tools available out there for Athena data visualisation, and sharing (Redash ❤️) . You can also directly use Athena APIs to build yourself a dashboard. See more docs here.

This is basically how we ship k8s logs out of the cluster. Do let us know your thoughts in the comments section.

Follow Practo Engineering on twitter for regular updates. If you like this article, please hit the applause icon to recommend it. This will help other Medium users find it. We are also hiring. Visit practo.com/careers to know more.