Monitoring kubernetes with prometheus(Part-1) — 100daysoflearning
Combining DAY 11, 12 & 13
This Post is going to be interesting as I am going to talk alot about prometheus and Grafana. So recent Linuxacademy course “monitoring kubernetes with prometheus” is actually very practical course and answers for most of the questions like how to setup Prometheus for Kubernetes cluster and how to do the monitoring.
Starting off the course gives the introduction on how to setup a cluster. This is just the commands more than the explanation.Below is the list of commands copied from instructor TRAVIS this is for one master and one worker node.
“””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””
swapoff -a
====================
vi /etc/fstab
Comment out swap
#/root/swap swap swap sw 0 0
=====================
Add the Kubernetes repo
cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF
=================
Disable SELinux
setenforce 0
================
Permanently disable SELinux:
vi /etc/selinux/config
Change enforcing to disabled
SELINUX=disabled
================
yum install -y kubelet-1.11.3 kubeadm-1.11.3 kubectl-1.11.3 — disableexcludes=kubernetes
================
systemctl start kubelet && systemctl enable kubelet
================
cat << EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
================
sysctl — system
================
Create kube-config.yml:
vi kube-config.yml
Add the following to kube-config.yml:
apiVersion: kubeadm.k8s.io/v1alpha1
kind:
kubernetesVersion: “v1.11.3”
networking:
podSubnet: 10.244.0.0/16
apiServerExtraArgs:
service-node-port-range: 8000–31274
==========================
Initialize Kubernetes
kubeadm init — config kube-config.yml
==========================
Copy admin.conf to your home directory
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
==========================
Install flannel
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
Patch flannel
==========================
vi /etc/kubernetes/manifests/kube-controller-manager.yaml
Add the following to kube-controller-manager.yaml:
— allocate-node-cidrs=true
— cluster-cidr=10.244.0.0/16
systemctl restart kubelet
=================
Setting up the Kubernetes Worker
Do same as the master and then below command:
kubeadmin join < MASTER_IP >:6443 — token < TOKEN > — discovery-token-ca-cert-hash sha256:< HASH >
On the master node, test to see if the cluster was created properly.
================
Get a listing of the nodes:
kubectl get nodes
“””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””
So above should setup kubernetes cluster with one master and one worker node given that you have linux instance with centos and Docker installed.
Next we directly jump on Prometheus Architecture explaining about:
client libraries for application lever monitoring , exporters for application we don’t have source code for , now prometheus need to know how to access exporters and targets so we can do by prometheus config or service discovery :
scraping a request is made to http end point and the result is parsed and sent to storage.
Next section we will setup prometheus on kubernetes cluster. This is a very practical lesson and you might not fully understand all the things as you see because you are expected to have good amount of kubernetes knowledge. So we go through the below steps (taken form the lecture itself):
A metrics namespace for our environment to live in
A ClusterRole to give Prometheus access to targets using Service Discovery
A ConfigMap map that will be used to generate the Prometheus config file
A Prometheus Deployment and Service
Kube State Metrics to get access to metrics on the Kubernetes API.
You can clone the YAML files form Github.
For all the yml files you need to do kubectl apply -f <filename>
For config file you need to add the ip’s of the master and the worker(private is fine).
you can access the prometheus dashboard at :Access Prometheus by visiting https://<MASTER_IP>:8080.
Next up we have the explanation for the prometheus configuration file which can seem too much to grasp at first but as you try to understand peacefully with open mind , things will start making sense. Everything can be linked wrt the documentation :Prometheus Configuration Documentation.
Next up we have the grafana setup from the same github repo mentioned earlier you can now move on to the grafana folder and start deploying the grafana files using same kubectl apply -f <filename>. If you have followed till now you have everything setup but you will be shown the node exporter as down now the below steps have to be performed in order to make the node exporter up (Content copied from the Lecture notes):
Create the Prometheus user:
adduser prometheus
Download Node Exporter.
========================================cd /home/prometheus
curl -LO “https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz"
tar -xvzf node_exporter-0.16.0.linux-amd64.tar.gz
mv node_exporter-0.16.0.linux-amd64 node_exporter
cd node_exporter
chown prometheus:prometheus node_exporter
vi /etc/systemd/system/node_exporter.service
/etc/systemd/system/node_exporter.service:
========================================
[Unit]
Description=Node Exporter
========================================
[Service]
User=prometheus
ExecStart=/home/prometheus/node_exporter/node_exporter
========================================
[Install]
WantedBy=default.target
Reload systemd:
========================================
systemctl daemon-reload
Enable the node_exporter service:
========================================
systemctl enable node_exporter.service
Start the node_exporter service:
========================================
systemctl start node_exporter.service
Add the following to config-map.yaml:
job_name: ‘node_exporter’
static_configs:
— targets: [‘<KUBER_MASTER_PRIVATE_IP>:9100’, <UBER_WORKER_PRIVATE_IP>:9100]
========================================
Do above on both master and worker. In order to have Grafana Dashboard you need to import the JSON from the link : Kubernetes Nodes Dashboard
and the grafana Dashboard will be working.
For now I am leaving here next few days I will complete this stuff and add screenshots and final steps and my feedback on the whole. Till now this course is really very practical with minimal theory so you need to be very focused to understand all the stuff.
Saiyam Pathak
https://www.linkedin.com/in/saiyam-pathak-97685a64/
https://twitter.com/SaiyamPathak
