Monitor remote kubernetes cluster using prometheus
Nov 3 · 2 min read
We will see how to setup prometheus to monitor kubernetes cluster using kubernetes api and kube-state-metrics.
This article is completely focused on configuring prometheus to monitor kubernetes. There are many tutorials on the web to guide you how to install prometheus.
Kubernetes-api:
Following are the details required to configure prometheus scrape job.
- Create a service account which has permissions to read and watch the pods.
- Generate token from the service account.
- Create scrape job as following.
- Import the Grafana dashboard to monitor the metrics.
- job_name: kubernetes
kubernetes_sd_configs:
- role: node
api_server: https://kubernetes-cluster-api.com
tls_config:
insecure_skip_verify: true
bearer_token: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
bearer_token: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
scheme: https
tls_config:
insecure_skip_verify: true
relabel_configs:
- separator: ;
regex: __meta_kubernetes_node_label_(.+)
replacement: $1
action: labelmap

kube-state-metrics:
kube-state-metrics provides metrics related to monitor pods which you cannot find it in the kubernetes api. you can setup alerts when the pod is in CrashLoopBackoff, Failed, etc… or alert on cronjob and many other metrics.
- Deploy the kube-state-metrics using their own provided image as POD on to kubernetes. you can refer this link to get example.
- Make sure to give read, watch permission to all the required resources to monitor to the service account.
- Create a service with type as Loadbalancer(Internal to vpc) or if you have nginx-ingress, create an internal ingress and expose the service so that it can be accessible from prometheus.
- Create a prometheus scrape job.
- Import the Grafana dashboard to monitor the metrics generated by kube-state-metrics.
- job_name: k8s-kube-state-metrics
static_configs:
- targets: ['kube-state-metrics-example.com']
metrics_path: "/metrics"