How to Install Application Monitoring Tools on cf-for-k8s

Published in

Cloud Foundry Foundation

5 min readFeb 23, 2021

Cloud Foundry aims to make development, deployment, and maintenance of applications faster and easier on Kubernetes. In addition, engineers want to deploy performant apps — and keep them that way. Application monitoring tools help them achieve this.

This post provides the steps that an engineer would need to take to deploy Prometheus and Grafana in order to monitor applications that are running on Kubernetes clusters, deployed using cf-for-k8s.

The benefits of Prometheus and Grafana

Prometheus and Grafana converge application performance monitoring (APM) and observability needs into a single toolchain.

Together, Prometheus and Grafana are the two most extensively used application monitoring tools in the DevOps world. These two tools complement each other very well, and help collect data and visualize what’s transpiring under the hood of your cloud applications. Another main advantage is that both the tools are available open source.

Prometheus, with its targets and pull model, helps collect metrics from the underlying system. Applications can be configured with the right kind of telemetry data to take advantage of supplying the right metrics to the Prometheus instance.

Grafana’s core competence is its diverse set of data visualization tools and charting interfaces. Grafana supports ingestion from a wide variety of data sources, with Prometheus integrations being a core feature. Dashboards built using Grafana often act as the control center for platform operations.

How to use cf-for-k8s with APM tools

The Cloud Foundry community is no exception when it comes to having a preference for using Prometheus and Grafana as part of their DevOps automation tools.

For all users of cf-for-k8s, the contributors have provided a simple way of including support for both Prometheus and Grafana. By including a couple of additional config files when creating the YAMLs that will deploy cf-for-k8s to Kubernetes clusters, pods for both these APM tools will be created and available for use.

Steps to install Prometheus and Grafana on cf-for-k8s

Follow these steps to learn how to install Prometheus and Grafana on your cf-for-k8s instances. The install steps are the same as deploying cf-for-k8s on any Kubernetes cluster (details available here). They just include additional config files at one stage.

Before getting started

Step 0. Prerequisites:

Make sure that you have the tools above installed on your machine before you proceed.

Clone the GitHub repo(s)

First, clone the GitHub repo cf-k8s-prometheus which has the config files needed for the installation and deployment. This is in addition to cloning the cf-for-k8s repository. Switch into the cf-for-k8s repository.

git clone https://github.com/cloudfoundry/cf-k8s-prometheus.gitgit clone https://github.com/cloudfoundry/cf-for-k8s.gitcd cf-for-k8s

Note: Inside the cf-k8s-prometheus repo you will find the required config files.

cf-k8s-prometheus/
|-- LICENSE
|-- NOTICE
|-- README.md
|-- config
|   |-- add-prometheus-node-exporter.yml
|   `-- add-prometheus-server.yml
|-- experimental
    |-- add-grafana.yml
    `-- add-prometheus-pushgateway.yml

Generate a YAML template

Make use of the auto-generate script included in the cf-for-k8s repo to generate a YAML template to be used.

./hack/generate-values.sh -d 35.192.46.62.xip.io > ~/tempdir/cf-values.yml

Use the Carvel tool ytt to generate the final YAML template that will be used for deployment. When generating the declarative syntax for this template, remember to include the configs of Prometheus and Grafana.

ytt -f ~/cf-for-k8s/config \
    -f ~/cf-k8s-prometheus/config \
    -f ~/cf-k8s-prometheus/experimental/add-grafana.yml \
    -f ~/tempdir/cf-values.yml \
     > ~/tempdir/cf-for-k8s-rendered.yml

Deploy the application and tools

We will then deploy cf-for-k8s, along with Prometheus and Grafana, using the kapp tool. This will install cf-for-k8s, along with Prometheus on the Kubernetes cluster.

kapp deploy -a cf -f /home/ram/tempdir/cf-for-k8s-rendered.yml -y

K9s is a tool that will allow you to watch the progress of your installation from the perspective of pods being created. Here’s a screenshot with the pods for Prometheus and Grafana highlighted.

Typically, users are required to patch the Istio Ingress gateway with a domain or a static IP, so it can be accessed from external networks. In addition, there are a couple of additional steps that you need to take to verify the installation by logging in, creating spaces, and deploying a test app. That is slightly out of scope for this article, but refer to the article with complete installation steps.

Clone and deploy

Next, clone the repo of a sample Golang application which will be used for testing the installed Prometheus and Grafana tools. Deploy this app using the cf push command.

git clone https://github.com/cloudfoundry/cf-for-k8s-metric-examples.git
cf push

Note: A manifest.yml file with the following metadata will be included in the top-level directory of the application.

metadata:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "2112"
      prometheus.io/path: "/metrics"

Set up port forwarding

Once deployed, you can get the pod names using the kubectl command. With these pod names, set up a port forwarding to a local port as specified in the manifest.yml file.

kubectl get pods -n cf-workloads
NAME                                       READY    STATUS      RESTARTS    AGE
go-app-with-metrics-devel-cf3200ae74-0   2/2     Running     0          4h38m
go-app-with-metrics-devel-cf3200ae74-1   2/2     Running     0          4h38m
restart-workloads-for-istio1-7-3-r6cxp   0/1     Completed       0          9hkubectl port-forward -n cf-workloads go-app-with-metrics-devel-cf3200ae74-0 2112
Forwarding from 127.0.0.1:2112 -> 2112
Forwarding from [::1]:2112 -> 2112
Handling connection for 2112

Opening the route in a browser (in this case localhost:2112/metrics) will display all the metrics that are being collected from the application running on the pod.

Repeat a similar port forwarding exercise with the Grafana pod name. This will allow you to access the Grafana server from your machine.

kubectl port-forward grafana-dc6787967-qkvnf -n cf-system 3000

This is the set of basic steps that you need to take to get Prometheus and Grafana tooling up and running on your Kubernetes infrastructure when working with cf-for-k8s.

Give us your feedback

As with any open source project, support for Prometheus and Grafana is a work in (perpetual) progress. Cloud Foundry contributors and community members are hard at work in enabling complete support for these application monitoring tools within the platform, while also resolving any performance issues that come along.

We would love to hear feedback from you! To join the conversation, use the CloudFoundry Slack, specifically, the #logging-and-metrics channel. We welcome all levels of participation from our users. You can also reach us via our Twitter handle — @cloudfoundry.