Endpoint monitoring using prometheus, blackbox exporter, microsoft teams and nginx

In today’s digitally-driven world, monitoring your endpoints effectively is crucial to ensure the health and performance of your applications and services. This article will guide you through setting up endpoint monitoring using Prometheus, Blackbox Exporter, and Microsoft Teams for alerting.

INTRODUCTION:

Endpoint monitoring is a critical aspect of maintaining the reliability and performance of your services. By using Prometheus for monitoring, Blackbox Exporter for endpoint probing, and Microsoft Teams for alerting, you can create a robust monitoring system that keeps you informed about the status of your endpoints.

Endpoints you can monitor ranges from websites, backend services and IOT devices. Setup is also adaptable to a private or public network setup. based on preference.

PREQUISITES:

Before we begin, ensure you have the following:

• Basic understanding of Prometheus and its architecture
• A working installation of Docker and Docker-compose
• Admin access to Microsoft Teams

we are going to create a file structure that looks like the below:

endpoint_monitoring
  - blackbox.yml
  - alertmanager.yml
  - prometheus.yml
  - docker-compose.yml
  - rules
    - rules.yml
  - nginx
    - Dockerfile
    - certificates
    - nginx_433.conf
  - prometheus-data   # to persist prometheus data

Blackbox exporter:

The Blackbox Exporter is a specialized tool designed for probing endpoints over various protocols to verify their availability and performance. The primary role of the Blackbox Exporter is to perform active checks against endpoints, such as websites, APIs, or network services, and export the results as metrics that Prometheus can scrape and analyze.

create blackbox.yml and add the code below

modules:
  http_2xx:
    prober: http
    http:
      preferred_ip_protocol: "ip4"
      valid_status_codes:  [200]  
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
grpc:
    prober: grpc
    grpc:
      tls: true
      preferred_ip_protocol: "ip4"
  grpc_plain:
    prober: grpc
    grpc:
      tls: false
      service: "service1"
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
      - send: "SSH-2.0-blackbox-ssh-check"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp
  icmp_ttl5:
    prober: icmp
    timeout: 5s
    icmp:
      ttl: 5

Alert manager:

Alertmanager is a component of the Prometheus monitoring system designed to handle alerts generated by Prometheus servers. It manages the alerts by deduplicating, grouping, and routing them to the correct receiver integrations such as email, Slack, PagerDuty, or custom webhooks. Alertmanager is essential for effectively managing and responding to the alerts generated by Prometheus based on the conditions you define.

create alertmanager.yml and add the code below

route:
  group_by: ['alertname']
  group_interval:  30s
  repeat_interval:  1m
  group_wait:  30s
  receiver: 'prometheus-msteams'
receivers:
- name: 'prometheus-msteams'
  webhook_configs: 
  - send_resolved: true
    url: 'http://promteams:2000/alertmanager'

Promteams:

PromTeams (Teams Proxy) is a tool designed to facilitate the integration between Prometheus Alertmanager and Microsoft Teams. It acts as an intermediary or proxy to handle the routing of alerts from Prometheus to Microsoft Teams channels.

To receive alerts in Microsoft Teams, follow these steps:

• Make sure you have an admin access
• Open Microsoft Teams and go to the channel where you want to receive alerts or create one.
• Click on the “…” (More options) next to the channel name and select “Connectors.”
• Search for and add the “Incoming Webhook” connector.
• Configure the webhook, giving it a name and icon, and copy the URL provided as teh team_token.

Rules:

Sample rule setup to monitor site liveness and ssl expiry. Create rules/rules.yml and paste the code below

groups:

  - name: Monitor mysite.com
    rules:
    - alert: mysite.com is Down
      expr: |
        up{instance="https://mysite.com", job="mysite.com"} == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "mysite.com site is down"
        description: "mysite.com is not reacheable for the last 1 minutes."

    - alert: mysite.com SSL Expired
      expr: |
        probe_http_ssl{instance="https://mysite.com", job="mysite.com"} == 0

      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "mysite.com site SSL Expired"
        description: "mysite.com site SSL exppired."

    - alert: mysite.com SSL <30 days to Expire
      expr: |
        ( ceil((probe_ssl_earliest_cert_expiry{instance="https://mysite.com", job="mysite.com"}-time())/86400) <= 30 )
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: "mysite.com site SSL Expiry soon"
        description: "mysite.com site SSL expires in  {{ $value }}  days from now."

Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit. create prometheus.yml file and paste in the code below:

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
           -  alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
   - "rules/rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

# blackbox exporter itself
  - job_name: "blackbox_exporter"
    static_configs:
      - targets: ["blackbox_exporter:9115"]
# your endpoints
  - job_name: "mysite.com"
    static_configs:
      - targets:
        - https://mysite.com
    metrics_path: /probe
    params:
      module:
      - http_2xx
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115

Nginx:

setup nginx reverse proxy to the prometheus server for access. create nginx folder and add required files as shown in fig 2.

Create nginx/certificates folder and move certificates in.

create a nginx/Dockerfile and add below code:

FROM nginx:1.15.8
RUN  rm /etc/nginx/conf.d/default.conf
COPY nginx_443.conf /etc/nginx/conf.d/default.conf  

RUN mkdir /etc/nginx/ssl/
COPY certificates/your_cert.crt   /etc/nginx/ssl/
COPY certificates/your_cert.key  /etc/nginx/ssl/ 

create nginx/nginx_443.conf and add below nginx conf

upstream prometheus-backend {
    server prometheus_monitoring:9090;
  }

server {
    listen 443 ssl;
    server_name <server_name> ;

    ssl_certificate      /etc/nginx/ssl/your_cert.crt;
    ssl_certificate_key  /etc/nginx/ssl/your_cert.key;

    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;
    client_max_body_size 100M;

    location / {
      proxy_pass http://prometheus-backend;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect off;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";

    }


  }

Complete setup:

create a docker-compose.yml file and paste in the code below to spin up all services required.

version: '3.8'
services:
  blackbox_exporter:
    image: quay.io/prometheus/blackbox-exporter:latest       
    container_name: blackbox_exporter
    ports:
      - "9115:9115"
    volumes:
      - .:/config
    command:
      - --config.file=/config/blackbox.yml
  alertmanager:
    image: quay.io/prometheus/alertmanager
    container_name: alertmanager
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
  promteams:
    image: quay.io/prometheusmsteams/prometheus-msteams      
    container_name: promteams
    ports:
      - "2000:2000"
    environment:
      - TEAMS_INCOMING_WEBHOOK_URL=<teams_token> #replce with your teams token
      - TEAMS_REQUEST_URI=alertmanager
  prometheus_monitoring:
    image: prom/prometheus
    container_name: prometheus_monitoring
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
      - ./rules/rules.yml:/etc/prometheus/rules/rules.yml
    depends_on:
      - blackbox_exporter
      - alertmanager
      - promteams
  nginx:
    build: ./nginx
    container_name: nginx
    ports:
      - "80:80"
      - "443:443"
    depends_on:
      - prometheus_monitoring
volumes:
  prometheus-data:

start monitoring

docker-compose up --build -d

This opensource setup will save you hundreds of dollars on your infrastructure endpoints monitoring.

Thanks for reading the article, hope this helps !