Setup Highly Available EPK stack with Fluentd in GCP | ElasticSearch , Pub/Sub and Kibana

Suganya G
Suganya G
Sep 8, 2018 · 6 min read

Editor’s note: When you have to run log collection and analytics at scale, most of the times the bottleneck is the ingesting / messaging layer. EKK (K for Kinesis and K for Kibana) stack in AWS world is gaining traction and we thought we will build an equivalent in GCP world. This post explains using Pub/Sub dealing with ingestion at scale.

Searching for a particular log data or error data across hundreds of log files on hundreds of servers is difficult without good tools. A common approach to this problem is to set up a centralized logging solution so that multiple logs can be aggregated in a central location.

EPK Setup :

Elasticsearch is an open source distributed real-time search backend and used for lot of analytics solutions

Kibana is front end of elasticsearch to visualize the elasticsearch data

Cloud Pub/Sub brings the scalability, durability , flexibility, and reliability of enterprise grade messaging layer on GCP. It allows for secure and highly available communication between source and destination and delivers messages in low-latency. It will act as queue until the messages delivered to destination. A publisher application creates and sends messages to a topic. Subscriber applications create a subscription to a topic to receive messages from it.

EPK work flow diagram

Overview of our solution :

  1. All Web/App servers logs published to Pub/Sub using fluentd forwarder agent
  2. logs are persisted in a message store of Pub/Sub until they are delivered and acknowledged by subscribers
  3. The subscriber receives pending messages(logs) from its subscription and acknowledges each one to the Pub/Sub service.
  4. When a message is acknowledged by the subscriber, it is removed from the subscription’s queue of messages.
  5. Fluentd aggregator used to Pull the logs from Pub/Sub and push it into Elaticsearch cluster via load balancer
  6. Kibana will display the real time logs for analysing

Setup Elasticsearch and Kibana :

Let us assume that kibana and elasticsearch cluster has already creaded and added the master instances under internal load balancer and we will point the load balancer’s internal static IP on fluentd aggregator instance later

Create a Pub/Sub topic and subscription :

  1. In GCP console navigation menu, click on Pub/Sub service under Big data
  2. Go to Topic and and click on Create topic icon and create a new topic and note down the topic name

3. Once the the topic crated, look for three dots on right side and click on the New subscription and create a new subscription and note down the subscription name

Creating service account to access Pub/Sub

  1. We will create two service account keys for Publish and subscribe
  2. In GCP console go to the service accounts under IAM & admin and click on the Create service account icon
  3. In create service account page, give the service account name and select the project role as Pub/Sub publisher
Service account for Pub/Sub publisher

4. Download the JSON file and name it as publish.json. We will use this file in fluentd forworder’s configuration

5. As same like Pub/Sub publish service account, create one more service account for Pub/Sub subscription and save the key file as subscriber.json. We will use this file on the fluentd aggregator server.

Service account for Pub/Sub Subscriber

Installing fluentd forwarder on Web/App servers:

  1. We wRun the command to install fluentd-forwarder (td- agent) in all web/App servers

For ubuntu

curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

For Redhat or centOS

curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

2. Install the gcloud pub sub plugin to push the logs to Pub/Sub

/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-gcloud-pubsub

3. Replace the existing fluentd config file with new file

cd /etc/td-agent/
mv td-agent.conf td-agent.conf.old
touch td-agent.conf

In config file, I am going to push the nginx logs to pub/sub

<source>
type tail
path /var/log/nginx/access.log
pos_file /var/log/td-agent/nginx-access.pos
tag example.publish
format nginx
</source>
<match example.publish>
type gcloud_pubsub
project Project_id
topic Topic_name
key /path/of/publish.json
flush_interval 10
autocreate_topic false
</match>

In the above file,

  • type tail → fluentd will start read the tail of log
  • path → location of log file
  • pos_file → will record the position it last read into this file.
  • tag → It used to check the correct match from source
  • format → Mention the parser name of log
  • project →Mention the project id of GCP project
  • topic → Mention the topic name
  • key → Copy the publish.json file in server and mention the location

4. Start the td agent using the below

service td-agent start

Setup Fluentd Aggregator :

  1. Install the td-agent by the running the below command

For ubuntu

curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

For Redhat or centOS

curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

2. Install the gcloud pub sub plugin to pull the logs from Pub/Sub

/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-gcloud-pubsub

3. Install the elasticsearch plugin to push the logs to elasticsearch

/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch

4. Create a new fluentd config file

cd /etc/td-agent/
mv td-agent.conf td-agent.conf.old
touch td-agent.conf

add the below contents in td-agent.conf and replace the values denoted in bold lines

<source>
type gcloud_pubsub
tag example.pull
project Project_name
topic toipic_name
subscription subscription_name
key /path/of/subscriber.json
max_messages 1000
return_immediately true
pull_interval 2
format json
</source>
<match example.pull>
@type elasticsearch
include_tag_key true
host "Elaticsearch internal loadbalancer ip"
port "9200"
logstash_format true
<buffer>
chunk_limit_size 2M
flush_thread_count 8
flush_interval 5s
retry_max_interval 30
queue_limit_length 32
retry_forever false
</buffer>
</match>

5. start the td_agent

service td-agent start

Create an index in Kibana

Once you start getting logs in Elasticsearch, You can create an index pattern in kibana to visualise the logs. We have specified logstash format true , so the by default index name will create is logstash and it will append the date in index pattern , so we will create an index pattern as logstash-* . Follow the below steps shown in pictures to create an index pattern.

Create index in kibana

After created the index, we can see the logs in Discover tab of kibana,

Discover logs in kibana

There goes your EPK stack live. Happy log analytics! :)

Searce Engineering

We identify better ways of doing things!

Suganya G

Written by

Suganya G

Associate Cloud Architect

Searce Engineering

We identify better ways of doing things!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade