Red Hat 3scale Monitoring with Prometheus and Grafana

Starting with Red Hat 3scale 2.3.0.GA, metrics for Prometheus are provided from the APICast pods. To verify the functionality and which metrics are available, `rsh` into either of the APICast pods and curl port `9421`.

$ oc rsh apicast-staging-3-fkb5l
sh-4.2$ curl http://localhost:9421/metrics
# HELP nginx_http_connections Number of HTTP connections
# TYPE nginx_http_connections gauge
nginx_http_connections{state=”accepted”} 990
nginx_http_connections{state=”active”} 1
nginx_http_connections{state=”handled”} 990
nginx_http_connections{state=”reading”} 0
nginx_http_connections{state=”total”} 990
nginx_http_connections{state=”waiting”} 0
nginx_http_connections{state=”writing”} 1

Additional metrics and labels may be included in future releases, so be sure to
validate for each release.

To depict these metrics in the Prometheus Console, the following steps deploy
Prometheus to the namespace that Red Hat 3scale AMP is provisioned.

NOTE: Ensure You have system administrator access to the OpenShift cluster, and you have prepared the OpenShift cluster by installing Red Hat 3scale.

The following `yml` files are required to complete the installation. These can
be found at my github location here: https://github.com/joelicious/3scale-monitoring

* 3scale-prometheus-crd.yml
* 3scale-prometheus-operator.yml
* 3scale-servicemonitor.yml

Login to OpenShift with administrator permissions.

$ oc login -u system:admin

Install the custom resource definitions necessary for running the Prometheus operator.

$ oc create -f 3scale-prometheus-crd.yml

Install the Prometheus operator to your namespace by using the following command syntax.

$ oc process -f 3scale-prometheus-operator.yml -p NAMESPACE=<YOUR NAMESPACE> | oc create -f -

For example, use this command for a project (namespace) named 3scale-v24:

$ oc process -f 3scale-prometheus-operator.yml -p NAMESPACE=3scale-v24 | oc create -f -

NOTE: The first time that you install the Prometheus operator into a namespace, it might take a few minutes for the Prometheus resource pods to start. Subsequently, if you install it to other namespaces on your cluster, the
Prometheus resource pods start much faster.

Instruct the Prometheus operator to monitor the 3scale’s apicast-staging node
in a project by using the following command syntax:

$ oc process -f 3scale-servicemonitor.yml -p NAMESPACE=<YOUR NAMESPACE> APICAST_STAGING_SERVICE_NAME=apicast-staging | oc apply -f -

For example, use this command for a project (namespace) named 3scale-v24 that includes 3scale’s apicast-staging service.

$ oc process -f 3scale-servicemonitor.yml -p NAMESPACE=3scale-v24 APICAST_STAGING_SERVICE_NAME=apicast-staging | oc apply -f -

Upon completion of the steps, the apicast-staging service has been updated to
reflect that port 9421 is now exposed.

APICast Staging Service

In addition, the OpenShift Console depicts a `prometheus-operator`
and a `prometheus-prometheus`.

OpenShift Prometheus

Expanding the `prometheus-prometheus` pod exposes the prometheus route which can be used to access the Prometheus Console.

Prometheus Console

To verify the integration with APICast metrics, select `Status` and `Service Discovery`.

Prometheus Service Discovery

Under Service Discovery, there should be active target for `3scale-v24/apicast-staging/0`. This active target can further be described by selecting Status from the menu, then Targets.

Prometheus Targets

Provided the target endpoint is `UP`, then individual metrics can be referenced on the main Prometheus Console. For example, by selecting `threescale_backend_calls` in the drop down list and then selecting `Execute` the metrics for that metric is displayed.

Backend Calls Metrics

Selecting `Graph` displays the same information on a graph.

Backend Calls Graph

Grafana is an application that provides metric and analytic dashboards, with support for Prometheus. The steps to install Grafana in an OpenShift cluster can be found at the following location:

NOTE: Make sure to use the branch of `origin` associated to the OpenShift Cluster version deployed.

The `README` for this folder contains the instructions to provision Grafana. For OpenShift 3.11, this requires the execution of a shell script.

Before executing the steps, it may be necessary to create a user and provided
cluster-admin rights to that user.

$ oc login -u system:admin
$ oc adm policy add-cluster-role-to-user cluster-admin <USER>

Then execute the shell script.

./setup-grafana.sh -n 3scale -p 3scale-v24

These steps create a new project named ‘grafana’ which contains a pod for the
grafana console. After selecting the URL to the console, login using the
account that has cluster-admin rights.

Grafana Console

Select “Add Data Source”, then “Prometheus”. Update the name to something
unique, i.e. Prometheus-v24. For HTTP URL, place the URL of the Prometheus
console. Click `Save & Test`.

Grafana Datasource

Once the datasource is enabled, then create `New dashboard` from the main
console. Select `Add Query`, which presents an empty dashboard. In the `Queries to`widget, select the name of the datasource that was created. In the test widget, place a name of a metric that is to be tracked. For example,
`threescale_backend_calls` can be entered.

The title of the graph can be set via the Gear Icon menu on the left side of
the web page.

Grafana Dashboard

Upon completing the configuration, the panel can be viewed.

Grafana New Dashboard

At this point, the panel then can be saved to a dashboard for easy retrieval.
Additional panels can be added to the dashboard for a complete view of
the state of Red Hat 3Scale AMP.

Grafana 3scale