Container and Service Mesh Logs

Published in

SE Notes by Alexey Novakov

6 min readJan 18, 2019

The other days I have been learning Istio Service Mesh and its sample configuration for Logging with Fluentd. My entire user experience of using Istio is very positive. Service Mesh has been really a missing piece, since people started to do distributed applications. Now we have most of things for free, which people were building in the past themselves:

Logging
Monitoring
Communication Security
Service Discovery
Distributed Tracing
and more

To leverage all the things of Istio, you would need to be already on Kubernetes “train” or at least use Docker-Consul as alternative option.

Service Mesh Logs

Official Logging with Fluentd example shows how Istio Mixer component can forward the mesh telemetry logs to Fluentd daemon. Using logentry configuration, we can describe, which variable needs to be published in the log entry. Official example is here: https://istio.io/docs/tasks/telemetry/fluentd/#configure-istio

A bit earlier on the same Istio documentation page, there is a YAML file for deploying non-production stack of Fluentd, Elasticsearch and Kibana:

Logging with Fluentd

This task shows you how to configure Istio to log to a Fluentd daemon.

istio.io

If we look at Fluentd configuration for data processing, we can find that source of the logs is Fluentd port, which is used by Istio to send its telemetry logs.

forward.input.conf: |-
  # Takes the messages sent over TCP
  <source>
    type forward
  </source>output.conf: |-
  <match **>
     type elasticsearch
     log_level info
     include_tag_key true
     host elasticsearch
     port 9200
     logstash_format true
     ...
  </match>

The effect of such configuration can be checked with any application deployed with Service Mesh. For example, official sample Bookinfo application may produce such logs in Kibana:

Elasticsearch document._source contains those variables, which were sent by Istio as logentry definition. There is snippet from the logentry spec:


  severity: '"info"'
  timestamp: request.time
  variables:
    source: source.labels["app"] | source.workload.name | "unknown"
    user: source.user | "unknown"
    destination: destination.labels["app"] | destination.workload.name | "unknown"
    responseCode: response.code | 0
    responseSize: response.size | 0
    latency: response.duration | "0ms"

See original configuration also here: https://istio.io/docs/tasks/telemetry/fluentd/#configure-istio

Container Logs

Now what is missing are the logs of our applications. Most of the time, when an application is already in Docker container, we just send its logs (info/debug/error/war) into stdout/stderr streams, no specific file appenders. However, we would like to see those console logs in the same Kibana window as well. This would allow us to correlate and debug our distributed application by looking at Service Mesh logs + the logs of our application. In order to get these logs we can go with some of the options below:

Make every application (or microservice, if you wish) to forward its logs to Fluentd daemon, i.e. to its TCP/UDP port.
Use Kubernetes Cluster logs or more precisely is to use Container logs. For example: /var/log/containers/*.log. Idea and configuration was borrowed from this great post by Rosemary Wang: https://medium.com/@joatmon08/application-logging-in-kubernetes-with-fluentd-4556f1573672

Problem of forwarding logs to Fluentd

We would need to “teach” every application of our distributed system to forward logs into Fluentd Daemon port. This may require additional development and basically touching already existing services. This can be painful or even not possible in some cases. Especially, when our services are written in different programming languages. Some of the languages/platforms may already provide an integration or a libraries to work with Fluentd forwarder, but for some of them, we would need to write such writer ourselves. From DevOps perspective we choose option #2 🌟

Cluster logs

In case of my Minikube environment, which is single node Kubernetes cluster, cluster logs are stored here:

You may notice a file name “details…”. It belongs to one of the Bookinfo sample app service. We are going to read these Cluster/Container logs by Fluentd.

Important

Fluentd Kubernetes Deployment spec needs to have the following volumes to be able to read Cluster logs from the disk:

volumes:  ...
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers

2. Service Account and Cluster Role needs to be configured for the Fluentd deployment, otherwise Fluentd Pod won’t start, saying that it does not have a permission to use Kunernetes API.

Here is a link on full spec for this post including ServiceAccount and ClusterRole:

novakov-alexey/kubernetes-specs

Kubernetes Specs. Contribute to novakov-alexey/kubernetes-specs development by creating an account on GitHub.

github.com

Tailing Cluster logs

In order to grab those Cluster logs, we need to setup some additional configuration in fluend.conf file:

<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag kubernetes.*
  format json
  read_from_head true
</source>

<filter kubernetes.**>
  @type kubernetes_metadata
</filter>

<match kubernetes.var.log.containers.**fluentd**.log>
  @type null
</match>

<match kubernetes.var.log.containers.**kube-system**.log>
  @type null
</match>

<match kubernetes.var.log.containers.**kibana**.log>
  @type null
</match>

<match kubernetes.var.log.containers.**istio**.log>
  @type null
</match>

Fluentd configuration above means:

type tail — is core-included input plugin of Fluentd. Shortly, it allows to tail files and get every new line from a file into Fluentd. Every record tag will be prepended with kubernetes prefix, then comes the log file name. An example of the Fluentd tag for the rating service of Bookinfo app:

kubernetes.var.log.containers.ratings-v1–8558d4458d-f5nfs_default_ratings-29110101c1c36a6f1e49a3946a4008151da9e602c5d725e0227978c44e86e9f1.log

2. filter kubernetes — is a filter plugin “kubernetes metadata” to enrich our log entry with Kubernetes metadata. Here is what we have gotten from one of the Bookinfo app service as part of the Elasticsearch _source field document:

    "kubernetes": {
      "container_name": "ratings",
      "namespace_name": "default",
      "pod_name": "ratings-v1-8558d4458d-f5nfs",
      "pod_id": "2dc1ff65-1b01-11e9-a3e7-080027a306ba",
      "labels": {
        "app": "ratings",
        "pod-template-hash": "8558d4458d",
        "version": "v1"
      },
      "host": "minikube",
      "master_url": "https://10.96.0.1:443/api"
    }

From the plugin GitHub page:

This plugin derives basic metadata about the container that emitted a given log record using the source of the log record.

3. matchkubernetes.var.log.containers.**fluentd**.log and similar

Forwarding logs of some of the containers to /dev/null , since we are not interested in those containers logs. Basically, we filter out them, before pushing them as documents to Elasticsearch.

Cluster logs in Kibana

After applying new Fluentd configuration, we can finally see that Fluentd is forwarding new logs into the Elasticsearch and they become available in Kibana:

I have selected only Cluster logs in above Kibana window, however we also have Istio Telemetry logs there. One can see both types of logs in one place.

Thanks for reading till the end!

Summary

We have combined Service Mesh log with all the logs of our Kubernetes-based application. Instead of modifying existing services to forward their logs to Fluentd daemon, we access Kubernetes Cluster logs directly. Basically, what was available in stdout/stderr, can be seen now in Elasticsearch/Kibana. Thanks to nice Fluentd plugins like in_tail and kubernetes_metadata_filter.