Red Hat 3scale Monitoring with Prometheus and Grafana
Starting with Red Hat 3scale 2.3.0.GA, metrics for Prometheus are provided from the APICast pods. To verify the functionality and which metrics are available, `rsh` into either of the APICast pods and curl port `9421`.
$ oc rsh apicast-staging-3-fkb5l
sh-4.2$ curl http://localhost:9421/metrics
# HELP nginx_http_connections Number of HTTP connections
# TYPE nginx_http_connections gauge
Additional metrics and labels may be included in future releases, so be sure to
validate for each release.
To depict these metrics in the Prometheus Console, the following steps deploy
Prometheus to the namespace that Red Hat 3scale AMP is provisioned.
NOTE: Ensure You have system administrator access to the OpenShift cluster, and you have prepared the OpenShift cluster by installing Red Hat 3scale.
The following `yml` files are required to complete the installation. These can
be found at my github location here: https://github.com/joelicious/3scale-monitoring
Login to OpenShift with administrator permissions.
$ oc login -u system:admin
Install the custom resource definitions necessary for running the Prometheus operator.
$ oc create -f 3scale-prometheus-crd.yml
Install the Prometheus operator to your namespace by using the following command syntax.
$ oc process -f 3scale-prometheus-operator.yml -p NAMESPACE=<YOUR NAMESPACE> | oc create -f -
For example, use this command for a project (namespace) named 3scale-v24:
$ oc process -f 3scale-prometheus-operator.yml -p NAMESPACE=3scale-v24 | oc create -f -
NOTE: The first time that you install the Prometheus operator into a namespace, it might take a few minutes for the Prometheus resource pods to start. Subsequently, if you install it to other namespaces on your cluster, the
Prometheus resource pods start much faster.
Instruct the Prometheus operator to monitor the 3scale’s apicast-staging node
in a project by using the following command syntax:
$ oc process -f 3scale-servicemonitor.yml -p NAMESPACE=<YOUR NAMESPACE> APICAST_STAGING_SERVICE_NAME=apicast-staging | oc apply -f -
For example, use this command for a project (namespace) named 3scale-v24 that includes 3scale’s apicast-staging service.
$ oc process -f 3scale-servicemonitor.yml -p NAMESPACE=3scale-v24 APICAST_STAGING_SERVICE_NAME=apicast-staging | oc apply -f -
Upon completion of the steps, the apicast-staging service has been updated to
reflect that port 9421 is now exposed.
In addition, the OpenShift Console depicts a `prometheus-operator`
and a `prometheus-prometheus`.
Expanding the `prometheus-prometheus` pod exposes the prometheus route which can be used to access the Prometheus Console.
To verify the integration with APICast metrics, select `Status` and `Service Discovery`.
Under Service Discovery, there should be active target for `3scale-v24/apicast-staging/0`. This active target can further be described by selecting Status from the menu, then Targets.
Provided the target endpoint is `UP`, then individual metrics can be referenced on the main Prometheus Console. For example, by selecting `threescale_backend_calls` in the drop down list and then selecting `Execute` the metrics for that metric is displayed.
Selecting `Graph` displays the same information on a graph.
Grafana is an application that provides metric and analytic dashboards, with support for Prometheus. The steps to install Grafana in an OpenShift cluster can be found at the following location:
Enterprise Kubernetes for Developers. Contribute to openshift/origin development by creating an account on GitHub.github.com
NOTE: Make sure to use the branch of `origin` associated to the OpenShift Cluster version deployed.
The `README` for this folder contains the instructions to provision Grafana. For OpenShift 3.11, this requires the execution of a shell script.
Before executing the steps, it may be necessary to create a user and provided
cluster-admin rights to that user.
$ oc login -u system:admin
$ oc adm policy add-cluster-role-to-user cluster-admin <USER>
Then execute the shell script.
./setup-grafana.sh -n 3scale -p 3scale-v24
These steps create a new project named ‘grafana’ which contains a pod for the
grafana console. After selecting the URL to the console, login using the
account that has cluster-admin rights.
Select “Add Data Source”, then “Prometheus”. Update the name to something
unique, i.e. Prometheus-v24. For HTTP URL, place the URL of the Prometheus
console. Click `Save & Test`.
Once the datasource is enabled, then create `New dashboard` from the main
console. Select `Add Query`, which presents an empty dashboard. In the `Queries to`widget, select the name of the datasource that was created. In the test widget, place a name of a metric that is to be tracked. For example,
`threescale_backend_calls` can be entered.
The title of the graph can be set via the Gear Icon menu on the left side of
the web page.
Upon completing the configuration, the panel can be viewed.
At this point, the panel then can be saved to a dashboard for easy retrieval.
Additional panels can be added to the dashboard for a complete view of
the state of Red Hat 3Scale AMP.