Monitor remote kubernetes cluster using prometheus

Amjad Hussain Syed
Nov 3 · 2 min read

We will see how to setup prometheus to monitor kubernetes cluster using kubernetes api and kube-state-metrics.

This article is completely focused on configuring prometheus to monitor kubernetes. There are many tutorials on the web to guide you how to install prometheus.

Kubernetes-api:

Following are the details required to configure prometheus scrape job.

  1. Create a service account which has permissions to read and watch the pods.
  2. Generate token from the service account.
  3. Create scrape job as following.
  4. Import the Grafana dashboard to monitor the metrics.
- job_name: kubernetes
kubernetes_sd_configs:
- role: node
api_server: https://kubernetes-cluster-api.com
tls_config:
insecure_skip_verify: true
bearer_token: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
bearer_token: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
scheme: https
tls_config:
insecure_skip_verify: true
relabel_configs:
- separator: ;
regex: __meta_kubernetes_node_label_(.+)
replacement: $1
action: labelmap

kubernetes-api scrape job
kubernetes-api scrape job

kube-state-metrics:

kube-state-metrics provides metrics related to monitor pods which you cannot find it in the kubernetes api. you can setup alerts when the pod is in CrashLoopBackoff, Failed, etc… or alert on cronjob and many other metrics.

  1. Deploy the kube-state-metrics using their own provided image as POD on to kubernetes. you can refer this link to get example.
  2. Make sure to give read, watch permission to all the required resources to monitor to the service account.
  3. Create a service with type as Loadbalancer(Internal to vpc) or if you have nginx-ingress, create an internal ingress and expose the service so that it can be accessible from prometheus.
  4. Create a prometheus scrape job.
  5. Import the Grafana dashboard to monitor the metrics generated by kube-state-metrics.
- job_name: k8s-kube-state-metrics
static_configs:
- targets: ['kube-state-metrics-example.com']
metrics_path: "/metrics"
Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade