Monitor your infrastructure with InfluxDB and Grafana on Kubernetes

Tamas Foldi
Nov 22, 2020 · 9 min read
Image for post
Image for post
Grafana in action — Learn how to set it up in your AWS cloud

Monitoring your infrastructure and applications is a must-have if you play your game seriously. Overseeing your entire landscape, running servers, cloud spends, VMs, containers, and the applications inside are extremely valuable to avoid outages or to fix things quicker. We, at Starschema, rely on open source tools like InfluxDB, Telegraf, Grafana, and Slack to collect, analyze, and react to events. In this blog series, I will show you how we built our monitoring infra to monitor our Cloud infrastructure, applications like Tableau Server and Deltek Maconomy, data pipelines in Airflow among others.

In this part, we will build up the basic infrastructure monitoring with InfluxDB, Telegraf and Grafana on Amazon’s managed Kubernetes service: AWS EKS.

Create a new EKS Kubernetes cluster

In case you have an EKS cluster already, just skip this part.

I assume you have a properly set up aws cli on your computer, if not, then please, do it, it will be a life-changer. Anyway, first, install eksctl which will help you to manage your AWS Elastic Kubernetes Service clusters and will save tons of time by not requiring to rely on the AWS Management Console. Also, you will need kubectl, too.

First, create new a Kubernetes cluster in AWS using eksctl without a nodegroup:

I used eu-central-1 region, but you can pick another one that is closer to you. After the command completes, add a new nodegroup to the freshly created cluster that uses only one availability zone (AZ):

The reason why I created a single AZ nodegroup is to be able to use EBS backed persistent volumes along with EC2 autoscaling groups. On multi-AZ node groups with autoscaling, newly created nodes can be in a different zone, without access to the existing persistent volumes (which are AZ specific). More info about this here.

TL;DR use single-zone nodegroups if you have EBS PersistentVolumeClaims.

If things are fine, you should see a node in your cluster:

Create a namespace for monitoring apps

Kubernetes namespaces are isolated units inside the cluster. To create our own monitoring namespace we should simply execute:

For our convenience, let’s use the monitoring namespace as the default one:

Install InfluxDB on Kubernetes

Influx is a time-series database, with easy to use APIs and good performance. If you are not familiar with time-series databases, it is time to learn: they support special query languages designed to work with time-series data, or neat functions like downsampling and retention.

To install an application to our Kubernetes system, usually we

  1. (Optional) Create the necessary secrets as an Opaque Secret(to store sensitive configurations)
  2. (Optional) Create a ConfigMap to store non-sensitive configurations
  3. (Optional) Create a PersistentVolumeClaim to store any persistent data (think of volumes for your containers)
  4. Create a Deployment or DaemonSet file to specify the container-related stuff like what we are going to run.
  5. (Optional) Create a Service file explaining how we are going to access the Deployment

As stated, the first thing we need to do is to define our Secrets: usernames and passwords we want to use for our database.

Next, create some persistent storage to store the database itself:

If you are new to Kubernetes, the way to execute these files is to call kubectl apply -f <filename> , in our case kubectl apply -f influxdb-pvc.yml.

Now, let’s create the Deployment, that defines what containers we need and how:

It will create a single pod (since replicas=1), passing our influxdb-creds as environmental variables and influxdb-pvc PersistentVolumeClaim to obtain 5GB storage for the database files. If all good, we should see something like:

After we defined what we want to run, it’s time for how to access it? This where Service definition comes into the picture. Let’s start with a basic LoadBalancer service:

It tells that our pod’s 8088 port should be available thru an Elastic Load Balancer (ELB). With kubectl get service , we should see the external-facing host:port (assuming we want to monitor apps outside from our AWS internal network).

This is great, but instead of HTTP, we might want to use HTTPS. To do that, we need our SSL certification in ACM with the desired hostname. We can either do it by generating a new certificate (requires Route53 hosted zones) or upload our external SSL certificate.

Image for post
Image for post
Amazon Issued SSL Certs are great but require Route 53 hosted zones. Alternatively, you can import existing SSL certificates.

If we have our certificate in ACM, we should add it to the Service file:

After executing this file, we can see that our ELB listens on two ports:

SSL is properly configured, the only thing is missing to add an A or CNAME record pointing to EXTERNAL-IP .

We all set, our database is running, and it is available on both HTTP and HTTPS protocols.

Installing Telegraf on Kubernetes

We need some data to validate our installation, and by the way, we already have a system to monitor: our very own Kube cluster and its containers. To do this, we will install Telegraf on all nodes and ingest cpu, IO, docker metrics into our InfluxDB. Telegraf has tons of plugins to collect data from almost everything: infrastructure elements, log files, web apps, and so on.

The configuration will be stored as ConfigMap, this is what we are going to pass to our containers:

To run our Telegraf data collector on all nodes of our Kubernetes cluster, we should use DaemonSet instead of Deployments.

Please note that this will use the same influxdb-creds secret definition to connect to our database. If all good, we should see our telegraf agent running:

To check the log messages from the telegraf pod, simply execute kubectl logs <podname>. You should not see any error messages.

Set up Grafana in Kubernetes

This will be the fun part, finally, we should be able to see some of the data we collected (and remember, we will add everything). Grafana is a cool, full-featured data visualization for time-series datasets.

Let’s start with the usual username and password combo as a secret.

Add 1GB storage to store the dashboards:

Define the deployment. As Grafana docker runs as 472 uid:gid, we have to mount the persistent volume with fsGroup: 472.

Finally, let’s expose it in the same way we did with InfluxDB:

Voila, we should have our Grafana up and running. Let’s check the ELB address with kubectl get services , point a nice hostname to its hostname/IP, and we are good to go. If all set, we should see something like:

Image for post
Image for post
I am glad that you made it here, now let’s log on!

Use the username/password combination you defined earlier, and see the magic.

Image for post
Image for post
Home screen for our empty Grafana

Define database connection to InfluxDB

Why this can be done programatically, to keep this post short (it’s already way too long), let’s do it from the UI. Click on the gear icon, data source, Add data source:

Image for post
Image for post
You know where should you click

Select InfluxDB:

Image for post
Image for post

Add http://influxdb:8066/ as URL, and set up your user or readonly influxdb user.

Image for post
Image for post

Adding our first Grafana Dashboard

Our telegraf agent is loading some data, there is no reason to not look at it. We can import existing, community-built dashboards such as this one: https://grafana.com/grafana/dashboards/928.

Click on + sign on the side bar, then Import. In the import screen add the number of this dashboard (928).

Image for post
Image for post

After importing, we should immediately see our previously collected data, in live:

Image for post
Image for post
This is really cool

Feel free to start building your own dashboards, it is way easier than you think.

Next steps

In the next blog, I will show how to monitor our (and our customers) Tableau Server, and how to set up data-driven email/slack alerts in no time.

Starschema Blog

Data contains intelligence that can change the world — we…

Tamas Foldi

Written by

Tamas is co-founder and CTO of data services firm Starschema where he leads the Starschema technical team to deliver results for the most innovative enterprises

Starschema Blog

Data contains intelligence that can change the world — we help people discover, manage and use this intelligence.

Tamas Foldi

Written by

Tamas is co-founder and CTO of data services firm Starschema where he leads the Starschema technical team to deliver results for the most innovative enterprises

Starschema Blog

Data contains intelligence that can change the world — we help people discover, manage and use this intelligence.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store