Monitoring a MariaDB server using Prometheus and Grafana

Prometheus is a systems and services monitoring system originally developed at SoundCloud and now a Cloud Native Computing Foundation project.

In this post we’ll outline the process of adding a Prometheus metrics endpoint to an existing chart, MariaDB chart in our case, and use Grafana to monitor the charts performance. We will do this by adding a Prometheus exporter to our Kubernetes Pods defined in the Helm chart template.

Lets begin by installing the prerequisites…

Installing the prerequisites

We’ll quickly walk through the steps of installing and configuring Prometheus and Grafana on a Kubernetes cluster using Helm charts.

Install a Ingress controller

$ helm install --name ingress stable/nginx-ingress

Note: minikube users can enable the ingress addon using the command minikube addons enable ingress

Install Prometheus

$ helm install --name prometheus stable/prometheus

Install Grafana

$ helm install --name grafana stable/grafana \
    --set server.adminPassword=password,server.ingress.enabled=true,server.ingress.hosts={grafana.example.com}

The above command configures grafana.example.com as the ingress hostname, so add the line xxx.xxx.xxx.xxx grafana.example.com to your operating systems /etc/hosts file, where xxx.xxx.xxx.xxx is the external IP address of the ingress controller. Get the IP address using the command:

$ kubectl get services \
    -l release=ingress,component=controller \
    -o jsonpath="{ .items[0].status.loadBalancer.ingress[0].ip }"

Note: minikube users should use the IP address of the minikube VM. Get the IP address of the minikube VM using the commandminikube ip

Login to http://grafana.example.com using the credentials admin/password and add Prometheus as a data source by navigating to the Add data source dialog in Grafana and specifing the data source Name as Prometheus, Type as Prometheus and the Url as http://prometheus-prometheus-server.default.svc.cluster.local as shown in the screenshot below.

In the Dashboards tab of the dialog you can import the Prometheus Stats dashboard to view the statistics of the Prometheus server.

Adding Prometheus metrics endpoint to MariaDB

We’ll use the MariaDB chart from the official charts repository to install MariaDB to our cluster. At the time of writing, the chart does not expose a metrics endpoint for Prometheus.

Note: Pull request #1330 has been merged, The changes described below are not required for MariaDB chart versions ≥ 0.7.0. You can simply enable the metrics endpoint by specifying --set metrics.enabled=true while deploying the chart.

We’ll use the prom/mysqld-exporter:v0.10.0 docker image which creates a Prometheus exporter for MariaDB server metrics and launch it as a sidecar container in the MariaDB pod deployment. We’ll define port 9104 as a container port since mysql_exporter creates the metrics endpoint on this port.

Edit the templates/deployment.yaml file of the stable/mariadb chart and add the following definition to launch the metrics container.

- name: metrics
  image: "prom/mysqld-exporter:v0.10.0"
  imagePullPolicy: "IfNotPresent"
  env:
  - name: DATA_SOURCE_NAME
    value: "root:{{ .Values.mariadbRootPassword }}@(localhost:3306)/"
  ports:
  - name: metrics
    containerPort: 9104

For Prometheus to discover the endpoint exposed by the MariaDB deployment, edit the templates/svc.yaml and add the following annotations to the Service resource.

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "9104"

With these changes in place, deploy the MariaDB Chart using the command:

$ helm install --name mariadb stable/mariadb/

The MariaDB chart will be deployed and you should notice that the mariadb pod now consists of two containers.

# kubectl  get pods -l app=mariadb-mariadb
NAME                               READY     STATUS    RESTARTS   AGE
mariadb-mariadb-4198203111-rg9w8   2/2       Running   0          10m

Note: A Patchfile, created against chart version 0.6.3, is available in the GitHub Gist ede014d7a36912b4ac57050a64509ff2

Adding a dashboard

We’ll install a Grafana dashboard to visualize and monitor the MariaDB server metrics. The percona/grafana-dashboards repository on GitHub contains a collection of Grafana dashboards for MySQL/MariaDB monitoring using Prometheus.

For an overview of the MariaDB server metrics, we’ll use the MySQL Overview dashboard. Download the dashboard JSON and import it into Grafana from the Dashboards > Import menu option.

Finally, load the MySQL Overview dashboard to monitor the MariaDB server metrics.

Tip: To view the statistics of each node in the cluster add the Node Exporter Full dashboard and specify Prometheus as the data source in the Import dialog

Conclusion

We can easily add Prometheus metrics endpoints to Helm charts and with Prometheus and Grafana we can just as easily monitor the performance of the charts deployed in the cluster.

Visit the Exporter and Integrations page to check if a metrics exporter exists for your favourite application.