Kubernetes Multi-Cluster monitoring with Prometheus and Submariner
Multi-Cluster deployments with Kubernetes are becoming very common, due to different reasons like redundancy, performance, scale, and more.
Monitoring such an environment and its services is important. The most common monitoring system used in Kubernetes (at this moment) is the Prometheus, which supplies metrics collection to a time series DB, alerting dashboards (using Grafana and more).
Submariner - as described on the Submariner website
Submariner enables direct networking between Pods and Services in different Kubernetes clusters, either on-premises or in the cloud.
To go through the next tutorial, it is recommended to get familiar with both Prometheus and Submariner, but it is not mandatory as this tutorial shows a toy example in a local environment.
Submariner implements the mcs-api to allow communication between services and pods, on a multi-cluster setup. Combining with Kube Federation (which allows federating resources such as namespace, deployments, and Custome Resources (CRDs) among clusters) — will allow full multi-cluster control for administrators. More about combining KubeFed and Submariner to achieve a really Multi-Cluster setup in my next post!
After this short prolog, let’s dive into our example on how setup a multi-cluster monitoring system. The following steps will show a toy example on how to connect two clusters with Submariner
Step 1 — Setup a local Kindenvironment:Kindis a tool depending on Docker to run multiple Kubernetes clusters locally.
Hence you will need Docker, Kind and kubectlinstalled on your local machine.
- For docker installation refer to Get Docker.
- For Kind installation refer to the quick-start guide.
- For kubectl installation please refer to Kubernetes Install.
I work with a Macbook, so I could install all the tools above on my local machine using Homebrew with one command.
brew update && \
brew cask install docker && \
brew install kind && \
brew install kubectlStep 2 — Spinning up some local clusters:
We will use a boilerplate setup from Submariner.
Just download the Submariner repository, and run make clusters
This will spin up two local Kubernetes clusters.
git clone https://github.com/submariner-io/submariner
cd submariner
make clustersPlease refer to Submariner Quick-Start Guide for installations on different cloud providers.
Step 3 — Deploy Prometheus on the clusters:
There are many ways for deploying Prometheus on your Kubernetes cluster, I have used the kube-prometheus setup (some will recommend Helm which is great as well), which deploys prometheus-operator from this link. But we are going to modify the quick start version to allow Submariner metrics collection and monitoring as well.
Install jsonnet, Again I’m using Mac, so I install using Homebrew
brew install jsonnetDownload pre-defined configuration files and build the
git clone https://github.com/danibachar/submariner-cheatsheet
cd submariner-cheatsheet/prometheus/install
jb init
jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@release-0.8
jb update
sudo chmod +x ./build.sh
./build.shAfter the script finishes running, you will notice several new directories under submariner-cheatsheet/prometheus/install. The one that we will find interesting is the manifast directory and its subdirectory setup under them there are all the yamls defining the Prometheus CRDs and the operator.
Go back to the root directory where we downloaded the submariner repository into. And let's deploy Prometheus
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 apply -f submariner-cheatsheet/prometheus/install/manifests/setup
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster2 apply -f submariner-cheatsheet/prometheus/install/manifests/setup
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 apply -f submariner-cheatsheet/prometheus/install/manifests/
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster2 apply -f submariner-cheatsheet/prometheus/install/manifests/Now you should see Prometheus CRDS, services, and deployments starting to spin up. Run each of these commands to see the relevant Kubernetes resources.
Note that Prometheus is deployed in the monitoring namespace.
Note that this example is using cluster1 config, change the kubeconfig flag to point to the cluster2 config to see the status there.
# Note the new namespace `monitoring` has appearedkubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 get ns# Will show all the new CRDs defined by Prometheus or any other setup (for example Submariner has its own CRDs)
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 get crds# Will show all the relevant service accounts, role and role bindings defined for Prometheus.
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 get role, rolebindings, clusterrole,clusterrolebindings, serviceaccount -n monitoring# Will show the service monitoring and prometheus setups
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 get prometheus --all-namespaces
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 get servicemonitor --all-namespaces
Step 4— Install subctl Submariner CLI:
curl -Ls https://get.submariner.io | bash
export PATH=$PATH:~/.local/bin
echo export PATH=\$PATH:~/.local/bin >> ~/.profileStep 5 — Deploy Submariner and join clusters to the mesh:
1) Define Submariner Broker on cluster1(I really recommend you to go into the Submariner website and read about its architecture and the Broker role in it). In short, one of the Broker roles is to propagate changes in the Submariner datapath to all clusters in the mesh from a central point, avoiding bombarding the system with messages directly between the clusters.
2) Join cluster1 and cluster2 into the mesh
# Deploy cluster1 as the Broker cluster
subctl deploy-broker --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1# joins cluster1 and cluster2 into the mesh
subctl join --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 broker-info.subm --clusterid cluster1 --natt=false
subctl join --kubeconfig submariner/output/kubeconfigs/kind-config-
cluster2 broker-info.subm --clusterid cluster2 --natt=false
You can now run:
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 get servicemonitor --all-namespaces
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 get servicemonitor --all-namespacesAnd see the relevant ServiceMonitor of Submariner
AMESPACE NAME AGE
monitoring alertmanager 6h24m
monitoring blackbox-exporter 6h24m
monitoring coredns 6h24m
monitoring grafana 6h24m
monitoring kube-apiserver 6h24m
monitoring kube-controller-manager 6h24m
monitoring kube-scheduler 6h24m
monitoring kube-state-metrics 6h24m
monitoring kubelet 6h24m
monitoring node-exporter 6h24m
monitoring prometheus-adapter 6h24m
monitoring prometheus-k8s 6h24m
monitoring prometheus-operator 6h24m
submariner-operator submariner-gateway-metrics 6h22m
submariner-operator submariner-lighthouse-agent-metrics 6h22m
submariner-operator submariner-lighthouse-coredns-metrics 6h22m
submariner-operator submariner-operator-metrics 6h23mIf for some reason the Submariner ServiceMonitor objects are not present you can re-deploy them using
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 apply -f submariner-cheatsheet/prometheus/submariner-service-monitors/
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 apply -f submariner-cheatsheet/prometheus/submariner-service-monitors/You can also expose the Prometheus server on one of the clusters and make sure its Targets and Service Discovery is configured correctly
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 \
port-forward --address 0.0.0.0 svc/prometheus-k8s 9090 --namespace monitoringTargets — http://localhost:9090/targets
Service Discovery — http://localhost:9090/service-discovery
I really urge you to go and read about Prometheus and how it utilizes service discovery to discover metrics endpoints. Note that it is important to allow Prometheus access to the submariner-operator namespace using ClusterRoleBingins — our example.jsonnet file allows for this configuration. If you are using another Prometheus setup like Helm or others you will need to understand how to enable this access role.
Step 6 — Exporting the Prometheus server services:
subctl export service --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 --namespace monitoring prometheus-k8s
subctl export service --kubeconfig submariner/output/kubeconfigs/kind-config-cluster2 --namespace monitoring prometheus-k8sThis step is using the Kubernetes multi-cluster API implemented by Submariner and exposes the prometheus-k8s service in both clusters to be accessed from any cluster in the mesh.
When using Submariner you can access an exported service
with the following DNS scheme: service-name.spacename.svc.clusterset.local.
Submariner utilizes the Service Discovery mechanized within Kubernetes (by supplying a CoreDNS plugin that monitors changes in the multi-cluster service and supplies the relevant IP address to this DNS query). While the usage of this scheme is the recommended one, right now the implementation of this DNS plugin (named Lighthouse) will prefer to return the IP of the local service (i.e if we query this address from cluster1 and we have that kind of service deployed there it will always return the IP of that service). Alternately, if the service is not deployed on that cluster but some others, currently (September 2021) there is a Round-Robin mechanism that will return a different IP each time.
For these reasons, when we want to access a certain Prometheus server on a certain cluster we will need to use a slightly different scheme: cluster-name.service-name.namespace.svc.clusterset.local.
For example, to access Prometheus metrics endpoint on a cluster named cluster1in namespace monitoringyou can curl the following: curl cluster1.prometheus-k8s.monitoring.svc.clusterset.local:9090\metrics
Step 7 — Configuring Grafana with each of the Prometheus servers:
OK, so now we are ready to build some dashboards! There are several different ways to expose the Grafana Service from within the Kubernetes cluster. For our experiment purposes, we will just run in a different shell
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 port-forward --address 0.0.0.0 svc/grafana 3000 -n monitoring- Login with admin:admin (user:password)
- Add Datasources
3. Enter the DNS Address with the relevant Prometheus port as explained in the previous section cluster1.prometheus-k8s.monitoring.svc.clusterset.local
cluster2.prometheus-k8s.monitoring.svc.clusterset.local
4. Save and test connection (do the following per Prometheus server)
5. Create your first Dashboard
6. Add an empty panel
7. Define some queries, let’s try and monitor some Submariner metrics (for the full list of metrics that Submariner provides please access this website). Choose the server from the list and add queries, in our example we added simple Submariner metrics for each cluster: submariner_connections and submariner_connection_latency_seconds
Step 8 — Optional — Setup Prometheus Federation:Prometheus offers a federation feature, where one can configure hierarchical or cross-service federation. The main idea behind it is to allow the aggregation of metrics and information from several different Prometheus servers (possibly across different locations). This is where the Submariner multi-cluster setup comes in handy!
To allow this federation we will need to add additional scraping config to our main Prometheus server.
Take a look at the submariner-cheatsheet/prometheus/prometheus-additional.yaml.
- job_name: "prometheus-federate"
honor_labels: true
metrics_path: '/federate'
params:
match[]: ['{job=~".+"}']
static_configs:
- targets: ["cluster2.prometheus-k8s.monitoring.svc.clusterset.local"]You can see the additional configuration, as described in the federation page of Prometheus, we are adding cluster2 a static scraping destination. The match labels basically define to scrape all jobs. Take a deeper look into Prometheus Querying for more info.
Set cluster1 as our main monitoring cluster and add the Prometheus federation configuration to it
kubectl --kubeconfig \
submariner/output/kubeconfigs/kind-config-cluster1 \
create secret generic additional-scrape-configs \
--from-file=submariner-cheatsheet/prometheus/prometheus-additional.yaml \
--dry-run=client -oyaml > \
submariner-cheatsheet/prometheus/manifests/additional-scrape-configs.yamlNote that after running this command a new file will be created under submariner-cheatsheet/prometheus/manifest/additional-scrape-configs.yaml.
Now we want to refer to this file from the main Prometheus definition.
let's edit it:
nano submariner-cheatsheet/prometheus/manifests/prometheus-prometheus.yamlCopy and paste the following to the end of the file we just opened:
additionalScrapeConfigs:
name: additional-scrape-configs
key: prometheus-additional.yamlAnd apply the new configuration, here we just run the whole manifest but you can run it specifically
kubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 apply -f submariner-cheatsheet/prometheus/manifestsorkubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 apply -f submariner-cheatsheet/prometheus/manifest/additional-scrape-configs.yamlkubectl --kubeconfig submariner/output/kubeconfigs/kind-config-cluster1 apply -f submariner-cheatsheet/prometheus/manifests/prometheus-prometheus.yaml
That is it, you can now go to the server targets and see the new federation target configured!
GitHub Repo
You can go into my Submariner-Cheatsheet GitHub repo, under the prometheus folder you will see some resources and scripts that can automate the setup.
If for some reason not all of the Submariner ServiceMonitor has been deployed, you can re-deploy them using the yaml files supplied under prometheus/submariner-service-monitor