Centralised Logging for Istio 1.5 with ECK(Elastic Cloud On Kubernetes) and Fluent Bit

I recently tried to setup centralised logging for Istio 1.5. I was looking for an easy and straight forward way to setup Elasticsearch, Fluentd/Fluent Bit and Kibana. While I was looking around, I came across the ECK — Elastic Cloud on Kubernetes that really got my interest. And the official documentation explains how to setup ECK with Istio.

Elastic Cloud on Kubernetes simplifies setup, upgrades, snapshots, scaling, high availability, security, and more for running Elasticsearch and Kibana in Kubernetes for one or many use cases.

Istio | Elastic Cloud on Kubernetes [1.1] | Elastic

But I wanted to use Fluentd or Fluent Bit with the ECK stack. After looking around a bit I found this GitHub repo

nanmu42/k8s-eck-logging

And an accompanying blog

Setup Elasticseach, Kibana and Fluent Bit for Your Kubernetes Cluster Logging

that explains the steps to integrate Fluent Bit with ECK. But it was for Kubernetes only. It did not have steps to setup for Istio. So I went through these two and merged them to setup Istio 1.5 with ECK Stack and Fluent Bit.

Prerequisites

• GKE cluster up and running
• Istio 1.5 installed with permissive mode

Steps to Follow

Install ECK CRD in your GKE cluster

kubectl apply -f https://download.elastic.co/downloads/eck/1.1.0/all-in-one.yaml

Check if the installation was successful

kubectl get pod elastic-operator-0 -n logging -o=jsonpath='{range .spec.containers[*]}{.name}{"\n"}'

If the output of the above command contains both manager and istio-proxy, ECK has been successfully installed with the Istio sidecar injected.

Clone the following repo

git clone https://github.com/fai555/istio-eck-fluent-bit.git

And apply the following commands

Access Kibana

Get the name of the Kibana CRD you created

kubectl get kibana -n logging

A ClusterIP Service is automatically created for Kibana. The name of the ClusterIP service will be

<KIBANA_CRD_NAME>-kb-http

In our case that will be elastic-istio-kb-http. Use kubectl port-forward to access Kibana from your local workstation:

kubectl port-forward -n logging service/elastic-istio-kb-http 5601

Open https://localhost:5601 in your browser. Your browser will show a warning because the self-signed certificate configured by default is not verified by a third party certificate authority and not trusted by your browser. You can temporarily acknowledge the warning for the purposes of this quick start but it is highly recommended that you configure valid certificates for any production deployments.

Login as the elastic user. The password can be obtained with the following command:

kubectl get secret elastic-istio-es-elastic-user -n logging  -o=jsonpath='{.data.elastic}' | base64 --decode; echo

Use username: elastic and password: 3KJ15o6ujqLbCM25UE13Q50L (acquired from the from the previous command. Yours will be different)

When you login for the first time you will not see any logs or anything. You first have to create an Index Pattern. Your ECK and Fluent Bit setup will match your index pattern and show the matching logs in Kibana.

To create your first Index Pattern, go to the Management tab

Under Kibana, select Index Pattern

Put a * in the Index Pattern field

Click Next Step and in the Configure settings section from the drop down select @timestamp

Hit Create Index Pattern

You have your very first index pattern.

Now go to the Discover Tab and you will see logs coming into your ECK stack.

If you are already familiar with Kibana and index pattern, you can go ahead and do your thing. I am not particularly experienced with Kibana and whole elastic stack. Thats why so many screenshots for people like me.

I also setup APM Server. But that is beyond the scope of this story. I will try to write something about that as well after I get my hands dirty.

Reference

• ECK Official Documentation for Istio integration
• ECK with Fluent Bit Integration Blog Thanks to nanmu42
• Fluent Bit official Website