Autoscaling applications using custom metrics on OpenShift Container Platform 3.11

Published in

IBM Cloud

11 min readJan 8, 2020

Red Hat OpenShift Container Platform 3.11 can automatically scale up or down your application based on CPU and memory usage. Sometimes, these metrics alone are not enough to properly determine when an application needs to scale. Enter, the Prometheus Adapter.

With this adapter, you can add functionality to automatically scale your application pods based on custom application metrics, in addition to the default CPU and memory usage metrics.

Note: The Prometheus Adapter is not a supported feature on OpenShift Container Platform 3.11 and is only a Technology Preview feature on OpenShift Container Platform 4.1.

The Prometheus adapter is an implementation of the custom metrics API (custom.metrics.k8s.io) that can connect to an existing Prometheus server. By connecting to a Prometheus server through this adapter, the Horizontal Pod Autoscaler, which is responsible for automatically scaling applications, can query the server to retrieve custom application metrics. The autoscaler can then use those metrics in combination with the default CPU, and memory usage metrics to determine when to automatically scale pods.

This blog guides you through a basic setup to gain the functionality from this adapter:

1. Setting up a sample application that exposes custom application metrics.
2. Completing a minimal Prometheus server setup by using an Operator so that Prometheus can monitor the sample application.
3. Setting up the Prometheus Adapter so that your autoscaler can get the exposed Prometheus metrics.

The focus of this blog is to demonstrate how to set up the required components rather than provide a deep dive into how each component works. How certain components function is only briefly described at a high level. For a deeper dive into the concepts and workings of the components that are used and referenced in this blog, review the additional resources that are listed under Further reading.

Prerequisites

Access to an OpenShift Container Platform 3.11 cluster with permissions to create resources.
Basic knowledge of the following concepts and tasks:
* OpenShift operations, including how to deploy resources.
* Kubernetes.

You can explore the capabilities of the adapter with a newly created OpenShift Container Platform 3.11 cluster. The only required component for the adapter to work is Prometheus. If you are using an existing cluster, Prometheus might already be running.

Sample application

The sample application that is used for this demonstration is based on the Bitnami NGINX and Bitnami NGINX Exporter images. This demonstration uses these Bitnami images instead of the official NGINX images as the Bitnami containers are built to run as non-root and the default OpenShift Container Platform security policy runs containers as non-root.

To set up the sample application, complete the following steps:

1. Create a Deploymentfor creating a pod to run the two containers:

oc apply -f app-deploy.yaml

Use the following app-deploy.yaml file with the preceding command:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: sample-app
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: sample-app
      annotations:
        prometheus.io/scrape: 'true'
    spec:
      containers:
        - name: nginx
          image: 'bitnami/nginx:latest'
          ports:
            - containerPort: 80
              protocol: TCP
          imagePullPolicy: Always
        - name: exporter
          image: 'bitnami/nginx-exporter:latest'
          command:
            - /usr/bin/exporter
            - '-nginx.scrape-uri'
            - 'http://127.0.0.1:8080/status'
          imagePullPolicy: Always

2. Create a Service to expose the pod so that you can later navigate to the web server and so that Prometheus can scrape metrics from the exporter container:

oc apply -f app-svc.yaml

Use the following app-svc.yaml file with the preceding command:

apiVersion: v1
kind: Service
metadata:
  name: sample-app
  labels:
    app: sample-app
  annotations:
    prometheus.io/scrape: "true"
spec:
  ports:
    - name: nginx
      protocol: TCP
      port: 80
      targetPort: 8080
    - name: exporter
      protocol: TCP
      port: 9113
      targetPort: 9113
  selector:
    app: sample-app

3. Optional: If you want to navigate to the web server externally from the cluster, create a route for the service:

oc expose service sample-app

To see the address that resolves to the web server, run the following command:

oc get route | grep sample-app

When you go the sample-app route with a browser, you can see a result similar to the following example:

Prometheus setup

For this demonstration, Prometheus is set up with the use of the Prometheus Operator to simplify the procedure, however you can use another method if you have a preferred set-up procedure. If you choose to use another method, this demonstration requires Prometheus to be set up properly to scrape metrics that are exposed by the NGINX Exporter.

Installing the Prometheus Operator framework

The Operator Framework is a Technical Preview feature in OpenShift Container Platform 3.11. To install the framework, you need to meet the following requirements:

1. Access to an OpenShift Container Platform cluster with an account that has cluster administration (cluster-admin) permissions.
2. Ansible playbooks that are provided by the latest Ansible installer for OpenShift Container Platform 3.11.

Procedure:

1. Add the following line to [OSEv3:vars] within the inventory file that you used to install and manage the cluster:

openshift_additional_registry_credentials=[{'host':'registry.connect.redhat.com','user':'<your_user_name>','password':'<your_password>','test_image':'mongodb/enterprise-operator:0.3.2'}]

Use the same username and password credentials that you use to log in to the Red Hat Customer Portal.

2. Run the registry authorization playbook by using your inventory file to authorize your nodes. Use the same credentials that you used in the preceding step.

ansible-playbook -i <inventory_file> playbooks/updates/registry_auth.yml

The ansible-playbook command might be in your playbook directory, for instance, /usr/share/ansible/openshift-ansible .

3. Install the Operator framework by using the Ansible playbook:

ansible-playbook -i <inventory_file> playbooks/olm/config.yml

4. Once the installation is complete, go to your cluster web console. Within the web console, you can see a result similar to the following example:

Setting up Prometheus

Now that the Operator framework is installed, you can use the framework to set up Prometheus to monitor the sample application. You can set up Prometheus through the OCP CLI tool or the web console. The steps to complete the set-up with either method is similar. This blog outlines the steps using the CLI approach.

1. Create a Prometheus Operator by creating a Subscription object for the Operator:

oc create -f prometheus-operator.yaml

Use the following prometheus-operator.yaml file with the preceding command:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  generateName: prometheus-
  namespace: default
spec:
  source: rh-operators
  name: prometheus
  startingCSV: prometheusoperator.0.22.2
  channel: preview

2. Create a ServiceMonitor object as a custom resource that is provided by the Prometheus Operator:

oc apply -f service-monitor.yaml

Use the following service-monitor.yaml file with the preceding command:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: sample-app
  labels:
    app: sample-app
  namespace: default
spec:
  selector:
    matchLabels:
      app: sample-app
  endpoints:
    - port: exporter

3. Create a Prometheus object as a custom resource that is provided by the Prometheus Operator:

oc apply -f prometheus.yaml

Use the following prometheus.yaml file with the preceding command:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: example
  labels:
    prometheus: k8s
  namespace: default
spec:
  replicas: 1
  version: v2.3.2
  serviceAccountName: prometheus-k8s
  securityContext: {}
  serviceMonitorSelector:
    matchLabels:
      app: sample-app

4. Create a service to expose your Prometheus pods so that the Prometheus Adapter can query the pods:

oc apply -f prometheus-svc.yaml

Use the following prometheus-svc.yaml file with the preceding command:

apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  ports:
  - name: web
    port: 9090
    protocol: TCP
    targetPort: web
  selector:
    prometheus: example

Prometheus Adapter setup

With Prometheus set up and running, set up the Prometheus Adapter by running the following command:

oc apply -f prometheus-adapter.yaml

This command applies the following prometheus-adapter.yaml resource definition file to create the adapter:

kind: ServiceAccount
apiVersion: v1
metadata:
  name: custom-metrics-apiserver
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-metrics-server-resources
rules:
- apiGroups:
  - custom.metrics.k8s.io
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-metrics-resource-reader
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  - services
  verbs:
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: custom-metrics-apiserver
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: custom-metrics-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: custom-metrics-apiserver
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics-resource-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-metrics-resource-reader
subjects:
- kind: ServiceAccount
  name: custom-metrics-apiserver
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hpa-controller-custom-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-metrics-server-resources
subjects:
- kind: ServiceAccount
  name: horizontal-pod-autoscaler
  namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: default
data:
  config.yaml: |
    rules:
    - seriesQuery: '{__name__="nginx_http_requests_total",namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
          service: {resource: "service"}
      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"
      metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.alpha.openshift.io/serving-cert-secret-name: prometheus-adapter-tls
  labels:
    name: prometheus-adapter
  name: prometheus-adapter
  namespace: default
spec:
  ports:
  - name: https
    port: 443
    targetPort: 6443
  selector:
    app: prometheus-adapter
  type: ClusterIP
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  service:
    name: prometheus-adapter
    namespace: default
  group: custom.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheus-adapter
  name: prometheus-adapter
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-adapter
  template:
    metadata:
      labels:
        app: prometheus-adapter
      name: prometheus-adapter
    spec:
      serviceAccountName: custom-metrics-apiserver
      containers:
      - name: prometheus-adapter
        image: directxman12/k8s-prometheus-adapter-amd64
        args:
        - --secure-port=6443
        - --tls-cert-file=/var/run/serving-cert/tls.crt
        - --tls-private-key-file=/var/run/serving-cert/tls.key
        - --logtostderr=true
        - --prometheus-url=http://prometheus-operated.default.svc:9090/
        - --metrics-relist-interval=1m
        - --v=4
        - --config=/etc/adapter/config.yaml
        ports:
        - containerPort: 6443
        volumeMounts:
        - mountPath: /var/run/serving-cert
          name: volume-serving-cert
          readOnly: true
        - mountPath: /etc/adapter/
          name: config
          readOnly: true
        - mountPath: /tmp
          name: tmp-vol
      volumes:
      - name: volume-serving-cert
        secret:
          secretName: prometheus-adapter-tls
      - name: config
        configMap:
          name: adapter-config
      - name: tmp-vol
        emptyDir: {}

This file includes the following key sections for defining the adapter: role-based access control, adapter configurations, Kubernetes API server extensions, and the Deployment resource.

The role-based access control (RBAC) section is the first and largest section within the file. These definitions are used to create the necessary service account (ServiceAccount), roles (ClusterRole)and the role bindings (RoleBinding, ClusterRoleBinding) for the adapter to work.

The next section is the adapter configurations (ConfigMap) that specifies how metrics from Prometheus are discovered, and the resources that are associated with those metrics. The configurations name the metrics and define the queries for the metrics. In this demonstration, the adapter queries the number of HTTP requests that the NGINX web server has received. The configurations define this query with the following line:

- seriesQuery: '{__name__="nginx_http_requests_total",namespace!="",pod!=""}'

By changing the seriesQuer value, which is just a PromQL query, the adapter can retrieve (discover) any metrics that Prometheus scrapes. The metrics that are specified in the seriesQuery field are then associated with the Kubernetes resources that are defined in the resources field. The name field renames the Prometheus metric to a name that the adapter exposes for the HPA to use. Finally, the metricsQuery field is used to apply any functions and operations to retrieve metric values. In this case, a rate and sum operation are performed, which requires nginx_http_requests_total to be renamed to nginx_http_requests_per_second.

For more information on the configurations, see Prometheus Adapter Config.

The following sections of the file includes the registration to extend the Kubernetes API server with an APIService object, which points to the adapter Service. Finally, aDeployment resource is defined for creating the adapter.

Test automatic scaling

Now that your adapter is created, you need to verify that the adapter is working by confirming that the adapter is configured to retrieve your application metrics.

1. Confirm that the adapter is configured to retrieve your application metrics, which for this sample application is the number of HTTP requests per second (nginx_http_requests_per_second). Run the following command to view the metrics that are retrieved:

kubectl get — raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

The jq flag is optional, but you are recommended to include the flag to produce a better formatted output:

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "namespaces/nginx_http_requests_per_second",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "pods/nginx_http_requests_per_second",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "services/nginx_http_requests_per_second",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

Search for the section with the name pods/nginx_http_requests_per_second in the output. If you specified any other metric for the adapter to query, ensure that the output includes sections for those other metrics.

2. Next, verify that the sample application metric, HTTP requests per second, is being retrieved:

kubectl get — raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/nginx_http_requests_per_second" | jq .

Output:

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/nginx_http_requests_per_second"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "namespace": "default",
        "name": "sample-app-674b6d4b6c-4bnnv",
        "apiVersion": "/v1"
      },
      "metricName": "nginx_http_requests_per_second",
      "timestamp": "2019-10-02T21:32:27Z",
      "value": "33m",
      "selector": null
    }
  ]
}

The resulting value uses Kubernetes-style quantities (SI units). A value of 33m is 0.033, which means that the sample application is getting 33/1000 HTTP requests per second and that these requests are originating from Prometheus. This number may not make much sense as is, so as an example, imagine that the value is 1 (1000m). This means that the web server is receiving one request every second, for example, if someone browses to the the home page and refreshes every second.

You can increase this requests per second value by refreshing the page. Use your browser to navigate to the web server with your previously created route for the value and then refresh the page a few times. You can also use curl or a similar command to send HTTP requests to the sample-app service ClusterIP. You might need to wait approximately a minute for the new value to be detected as the adapter queries Prometheus at 1-minute intervals. If needed, you can configure this interval.

At this point, you can now describe when your sample application is to scale up or down when the number of requests to the web server exceeds or drops below a threshold.

3. Create an Horizontal Pod Autoscaler HorizontalPodAutoscaler resource to query the application metrics that are exposed by the Prometheus Adapter:

oc apply -f hpa.yaml

Output: hpa.yaml

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: sample-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sample-app
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Pods
    pods:
      metricName: nginx_http_requests_per_second
      targetAverageValue: 70m

4. Navigate to your NGINX web server with your browser and send a few requests to the server in quick succession. As an alternative to using your browser, you can use curl or any HTTP client to send requests.

The Horizontal Pod Autoscaler should detect the increase in traffic after approximately 1 minute. Once the rate surpasses 70m, the autoscaler begins scaling up your application. When the rate drops back down, the autoscaler scales your application down accordingly. The autoscaler can take approximately 5 minutes to start scaling down your application. If needed, you can configure the interval. For more information, see Algorithm Details.

Summary

After following these steps, you now have the minimal setup needed for you to use to learn more about using the Prometheus Adapter. Once the adapter is set up, you can begin using the Horizontal Pod Autoscaler to scale applications based on your custom application metrics in addition to your typical CPU and memory usage metrics.