Setup Highly Available EPK stack with Fluentd in GCP | ElasticSearch , Pub/Sub and Kibana
Editor’s note: When you have to run log collection and analytics at scale, most of the times the bottleneck is the ingesting / messaging layer. EKK (K for Kinesis and K for Kibana) stack in AWS world is gaining traction and we thought we will build an equivalent in GCP world. This post explains using Pub/Sub dealing with ingestion at scale.
Searching for a particular log data or error data across hundreds of log files on hundreds of servers is difficult without good tools. A common approach to this problem is to set up a centralized logging solution so that multiple logs can be aggregated in a central location.
EPK Setup :
Elasticsearch is an open source distributed real-time search backend and used for lot of analytics solutions
Kibana is front end of elasticsearch to visualize the elasticsearch data
Cloud Pub/Sub brings the scalability, durability , flexibility, and reliability of enterprise grade messaging layer on GCP. It allows for secure and highly available communication between source and destination and delivers messages in low-latency. It will act as queue until the messages delivered to destination. A publisher application creates and sends messages to a topic. Subscriber applications create a subscription to a topic to receive messages from it.

Overview of our solution :
- All Web/App servers logs published to Pub/Sub using fluentd forwarder agent
- logs are persisted in a message store of Pub/Sub until they are delivered and acknowledged by subscribers
- The subscriber receives pending messages(logs) from its subscription and acknowledges each one to the Pub/Sub service.
- When a message is acknowledged by the subscriber, it is removed from the subscription’s queue of messages.
- Fluentd aggregator used to Pull the logs from Pub/Sub and push it into Elaticsearch cluster via load balancer
- Kibana will display the real time logs for analysing
Setup Elasticsearch and Kibana :
Let us assume that kibana and elasticsearch cluster has already creaded and added the master instances under internal load balancer and we will point the load balancer’s internal static IP on fluentd aggregator instance later
Create a Pub/Sub topic and subscription :
- In GCP console navigation menu, click on Pub/Sub service under Big data
- Go to Topic and and click on Create topic icon and create a new topic and note down the topic name

3. Once the the topic crated, look for three dots on right side and click on the New subscription and create a new subscription and note down the subscription name


Creating service account to access Pub/Sub
- We will create two service account keys for Publish and subscribe
- In GCP console go to the service accounts under IAM & admin and click on the Create service account icon
- In create service account page, give the service account name and select the project role as Pub/Sub publisher



4. Download the JSON file and name it as publish.json. We will use this file in fluentd forworder’s configuration
5. As same like Pub/Sub publish service account, create one more service account for Pub/Sub subscription and save the key file as subscriber.json. We will use this file on the fluentd aggregator server.

Installing fluentd forwarder on Web/App servers:
- We wRun the command to install fluentd-forwarder (td- agent) in all web/App servers
For ubuntu
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | shFor Redhat or centOS
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh2. Install the gcloud pub sub plugin to push the logs to Pub/Sub
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-gcloud-pubsub3. Replace the existing fluentd config file with new file
cd /etc/td-agent/
mv td-agent.conf td-agent.conf.old
touch td-agent.confIn config file, I am going to push the nginx logs to pub/sub
<source>
type tail
path /var/log/nginx/access.log
pos_file /var/log/td-agent/nginx-access.pos
tag example.publish
format nginx
</source><match example.publish>
type gcloud_pubsub
project Project_id
topic Topic_name
key /path/of/publish.json
flush_interval 10
autocreate_topic false
</match>
In the above file,
- type tail → fluentd will start read the tail of log
- path → location of log file
- pos_file → will record the position it last read into this file.
- tag → It used to check the correct match from source
- format → Mention the parser name of log
- project →Mention the project id of GCP project
- topic → Mention the topic name
- key → Copy the publish.json file in server and mention the location
4. Start the td agent using the below
service td-agent startSetup Fluentd Aggregator :
- Install the td-agent by the running the below command
For ubuntu
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | shFor Redhat or centOS
curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh2. Install the gcloud pub sub plugin to pull the logs from Pub/Sub
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-gcloud-pubsub3. Install the elasticsearch plugin to push the logs to elasticsearch
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch4. Create a new fluentd config file
cd /etc/td-agent/
mv td-agent.conf td-agent.conf.old
touch td-agent.confadd the below contents in td-agent.conf and replace the values denoted in bold lines
<source>
type gcloud_pubsub
tag example.pull
project Project_name
topic toipic_name
subscription subscription_name
key /path/of/subscriber.json
max_messages 1000
return_immediately true
pull_interval 2
format json
</source><match example.pull>
@type elasticsearch
include_tag_key true
host "Elaticsearch internal loadbalancer ip"
port "9200"
logstash_format true
<buffer>
chunk_limit_size 2M
flush_thread_count 8
flush_interval 5s
retry_max_interval 30
queue_limit_length 32
retry_forever false
</buffer>
</match>
5. start the td_agent
service td-agent startCreate an index in Kibana
Once you start getting logs in Elasticsearch, You can create an index pattern in kibana to visualise the logs. We have specified logstash format true , so the by default index name will create is logstash and it will append the date in index pattern , so we will create an index pattern as logstash-* . Follow the below steps shown in pictures to create an index pattern.


After created the index, we can see the logs in Discover tab of kibana,

There goes your EPK stack live. Happy log analytics! :)

