Automating uptime monitoring on Kubernetes with Statuscake

Published in

Stakater

8 min readJul 31, 2018

Monitoring uptime and availability of your microservices and receiving alerts when things go down is very critical to meet your SLA and ensure business continuity. StatusCake is a hosted service for uptime monitoring and alerting which can be used with Kubernetes services. It helps you to aggregate alerts in one handy view and efficiently pipe them to the alerting method of your choice.

Statuscake features

StatusCake has the following features out of the box.

Uptime monitoring: Monitor your application availability automatically from over 28 countries worldwide
Domain monitoring: You get an alert when it’s time to renew domain or when domain DNS records have been changed.
Pagespeed Monitoring : StatusCake provides Pagespeed type tests to fully collect and analyse data on load time for websites and individual pages
SSL Certificate Monitoring: You always get alerted when SSL certificates are due for renewal
Server Monitoring: You get alerted when certain RAM, CPU and Disk usage are thresholds are exceeded
Monitor through HTTP, HEAD, TCP, DNS, SMTP, SSH, PING and PUSH protocols
Management of complex alerting rules with PagerDuty
Virus Scanning: StatusCake virus scan alerts you when a threat is detected. It searches for trojans, viruses, malware & other malicious threats on your website.
Reporting: Get detailed uptime & speed tests reports delivered via email manually or by setting these up on a daily/weekly/monthly schedule.

StatusCake gives you many options to decide when and how often to be contacted when there is an alert triggered by Kubernetes Ingress monitoring. Contact method can be Email, SMS, Push notifications for Android & iOS, or through integration with third party application like PagerDuty.

Configuring for multiple services

Kubernetes Ingress is an object that provides external access to the services in a cluster via HTTP. Ingress is often used for SSL termination, load balancing, and name-based virtual hosting to the cluster internal services.

The problem most experience is how to configure monitoring for multiple or many multiples of kubernetes ingresses, especially in microservices architectures, and get alerted when they are down or unhealthy. Ingress Monitor Controller was created to address this problem. It will continuously monitor Kubernetes ingresses liveness/change and automatically update URL endpoint configured in StatusCake.

When Ingress Monitor Controller has been configured, it will always register an ingress automatically when created, so that Statuscake can alert you through email, slack channel, SMS or any other alert means whenever the service may be down. This should help you avoid a 3am call by ensuring quick response to issues.

Monitoring for Kubernetes

StatusCake features that are of more interest to us in relation to monitoring of Kubernetes services are:

Uptime checking with HTTP type tests

StatusCake uptime test ensures that the service being monitored is always up and running without errors. An alert is triggered when the service is down, or when there are errors in the application. Since HTTP is a standard protocol used in Kubernetes cluster service deployment, this is the type of test that will be used with Ingress Monitor Controller.

Alerting to Slack

Slack is a powerful notification service supported by StatusCake. It is integrated with StatusCake for receiving and assigning alerts. The fact that slack runs on PC, macOS, Android and iOS makes it a good choice for sending StatusCake notifications to the user.

With the StatusCake-Slack integration, you easily push alert data to Slack automatically whenever there’s a problem.

Alerting to Email

StatusCake also support notifications via email. This is the easiest method since you only need to add Contact group on StatusCake UI, then provide a list of email address that should receive alerts. When the service is down, you’ll get an email like below:

Deploying statuscake to Kubernetes to monitor a service

StatusCake monitoring of Kubernetes services is done with the help of Ingress Monitor Controller (IMC). IMC is responsible for collection of service metrics on all or specific namespace and pushing the data to StatusCake.

Ingress Monitor Controller

Ingress Monitor Controller is a tool designed by the Stakater team to automate the management of URL endpoint monitors in StatusCake using kubernetes events. Other supported monitors are UptimeRobot and Pingdom, but since the code is opensource and extendable, it is easy to implement support for other uptime checkers by creating a custom monitor service interface.

Below are some features of Ingress Monitor Controller:

It automatically creates new URL endpoint monitors in StatusCake whenever you create a new Ingress, with the required annotation, on your Kubernetes Cluster.
IMC will automatically update URL endpoint monitors in StatusCake when ingresses on Kubernetes are updated.
It will also automatically delete URL endpoint monitors in StatusCake for an Ingress that has been deleted on the cluster.
It can be configured globally or per namespace as well depending on the need

The Ingress Monitor Controller watches Ingresses that are created in the cluster, and if the particular ingress is annotated accordingly, i.e. with `monitor.stakater.com/enabled: “true”`, then IMC registers a monitor for that endpoint with Statuscake.

Depending on the requirement, the Ingress Monitor Controller can be configured at a global level, to monitor all namespaces of the cluster, or it can be configured to watch a specific namespace only. This can be configured via a `watchNamespace` property in the config map.

Configuring Ingress Monitor Controller with StatusCake

I’ll take you through a step by step deployment of Ingress Monitor Controller on your Kubernetes cluster. I assume you already have an active StatusCake account, if you don’t have one, you can signup for one.

Once your account is ready, you’ll need to obtain the following details to be used on Ingress Monitor Controller:

StatusCake Username
StatusCake API Key
StatusCake API URL
StatusCake contact list/group

To obtain API Key, login to StatusCake dashboard and navigate to ACCOUNT > User Details > API Key

At the bottom, you’ll see Endpoint: https://app.statuscake.com/API/, this will be your API URL. Username is on Account > User Details > Username

Note these values since we will use them later when deploying Ingress Monitor Controller to Kubernetes Cluster. Contact Groups are managed under Contacts > Create Group

To ensure we have clean test environment, a test namespace will be used to demonstrate the deployment. I have a three nodes cluster that I’ll use for this demo:

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 43m v1.11.0
k8s-node-01 Ready <none> 15m v1.11.0
k8s-node-02 Ready <none> 10m v1.11.0

Step 1: Create a test namespace

Create a test namespace on Kubernetes using kubectl command line tool:

$ kubectl create namespace test-namespace
namespace/test-namespace created

Check if the newly created namespace is shown on the system:

$ kubectl get namespaces
NAME           STATUS AGE
default        Active 11m
kube-public    Active 11m
kube-system    Active 11m
test-namespace Active 17s

Step 2: Create a deployment for our test web application.

Let’s create a sample web application container image that listens on a HTTP server on port 80:

$ vim http-app-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: http-app
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: http-app
    spec:
      containers:
      - name: http-app
        image: katacoda/docker-http-server:latest
        ports:
        - containerPort: 80

Deploy it using the command below:

$ kubectl apply -n test-namespace -f http-app-deployment.yml
deployment.extensions/http-app created

Since we specified replica count of 3, three pods will be created.

$ kubectl -n test-namespace get pod
NAME                       READY     STATUS    RESTARTS   AGE
http-app-97f76fcd8-vcxc9   1/1       Running   0          33m
http-app-97f76fcd8-vtnkt   1/1       Running   0          33m
http-app-97f76fcd8-xq6cn   1/1       Running   0          33m

Step 3: Expose your Deployment as a Service internally

Create a service for deployment above:

$ cat http-app-service.yml 
apiVersion: v1
kind: Service
metadata:
  name: http-app-svc
  labels:
    app: http-app
spec:
  type: NodePort
  ports:
  - port: 80
    nodePort: 30080
  selector:
    app: http-app

To deploy the service, run the command:

$ kubectl apply -n test-namespace -f http-app-service.yml 
service/http-app-svc created$ kubectl get service -n test-namespace
NAME         TYPE     CLUSTER-IP    EXTERNAL-IP PORT(S)      AGE
http-app-svc NodePort 10.110.42.173 <none>      80:30080/TCP 26s

Kubernetes Engine makes this Service available on port 30080/TCP on all the nodes in your cluster. Also, note that there is no external IP allocated for this Service. You can do a curl to the cluster IP and port 80 to confirm if working:

$ curl http://10.110.42.173
<h1>This request was processed by host: http-app-97f76fcd8-vtnkt</h1>$ curl http://10.110.42.173
<h1>This request was processed by host: http-app-97f76fcd8-vcxc9</h1>$ curl http://10.110.42.173
<h1>This request was processed by host: http-app-97f76fcd8-xq6cn</h1>

We can confirm that the response is coming from three different pods:

Step 4: Create an Ingress resource

Ingress is a collection of rules and routing policies that govern how external users get access to Kubernetes internal services. For this guide, I deployed Kubernetes Nginx Ingress. Learn more about using Ingress on kubernetes.io

Once deployed, let’s create an ingress resource with

$ cat ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: test-namespace
  name: http-app-svc-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    monitor.stakater.com/enabled: "true"
spec:
  rules:
  - host: service.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: http-app-svc 
          servicePort: 80

Deploy it using:

$ kubectl apply -f ingress.yml

This ingress is linked to our service (http-app-svc) created earlier as the backend. Make sure you have `monitor.stakater.com/enabled: “true”` annotation for IMC to monitor it.

Step 5: Deploy Ingress Monitor Controller

Ingress Monitor Controller can be deployed using its manifest file. Download it here.

Open the ingressmonitorcontroller.yaml file you just downloaded, in a text editor to modify configuration.

In the ClusterRoleBinding resource, specify the namespace

namespace: test-namespace

Next we need to edit config.yaml data of the ConfigMap resource to specify configuration parameters for StatusCake.

data:
  config.yaml: |-
    providers:
    - name: StatusCake
      apiKey: your-statuscake-api
      apiURL: https://app.statuscake.com/API/
      username: your-status-cake-username
    enableMonitorDeletion: true
    monitorNameTemplate: "{{.Namespace}}-{{.IngressName}}"

Once the changes have been made, you can deploy the controller to the namespace to be monitored, in our case this is test-namespace using below kubectl commands:

Start by deploying ConfigMap

$ export  name_space="test-namespace"
$ kubectl apply -f ingressmonitorcontroller.yaml -n $name_space
configmap/ingressmonitorcontroller created
serviceaccount/ingressmonitorcontroller created
clusterrole.rbac.authorization.k8s.io/ingressmonitorcontroller-role created
clusterrolebinding.rbac.authorization.k8s.io/ingressmonitorcontroller-role-binding created
deployment.extensions/ingressmonitorcontroller created$ kubectl -n test-namespace get pods | grep ingressmonitorcontroller
ingressmonitorcontroller-84dccb865c-vk45v   1/1  Running   0          11m

Once the Ingress Monitor Controller is in running state, login to StatusCake Dashboard and confirm if uptime test for the ingress was created.

If for any reason the ingress is down, you should receive a notification via Contact group added in StatusCake website.

A sample of down state message with the reason is below:

Here is also a video demonstrating how Ingress Monitor Controller registers monitors for new Ingresses on Statuscake, and removes monitors if the ingress is deleted: