Monitoring your Cloud Foundry application with mozzle

I’m a heavy Cloud Foundry user. I deploy applications daily. Sometimes I even use the CF system for developing purposes. But this wasn’t the case some time ago. Before becoming a CF user, I was a BOSH user.

BOSH is one level higher (or lower, depends where are you looking from) in the technical stack. What that means is one of the ways to install and operate a CF system is by using BOSH. And there are some things that are different on the distinct levels. Today I’m going to talk about one thing that was much different from my perspective as a BOSH user, and as a Cloud Foundry user — monitoring.

BOSH has one thing called bosh agent, which is installed on each BOSH-spinned VM. This bosh agent, among other things, periodically samples the VM’s OS for certain infrastructure characteristics, simply said — metrics, collects them and forwards them to a configured destination. This enables one to inspect all infrastructure metrics such as CPU consumption, memory usage, disk usage, etc. for a particular BOSH deployment, without actually doing anything. This is great because of many reasons — it helps you with troubleshooting your system, it helps you understand the behavior of your system, it allows you to define alerts, and so on.

But this is not the case with Cloud Foundry. When you deploy an application, the same infrastructure metrics are available for all of your application instances, but there is no tool (at least I’m not aware of) that collects them and allows you to forward them to a 3rd party system, as with the BOSH case. In the CF case it is possible to do even more — since applications are first-class objects, it is possible to track the interaction between the CF system and the application (or between the end user and the CF system) and report those events as well.

Realizing that, an idea come to my mind — why not create such a thing? And I did. It is called mozzle and it is available on github.

It pulls infrastructure metrics for your applications, without the need of doing any instrumentation in your application code. There are three types of infra metrics that are available for use:

• Machinery — CPU consumption, disk usage, memory usage
• HTTP — Request/Responses, status codes, bandwidth, …
• Events — Application crashes, updates, scales, ssh-es, …

It then allows you to forward those metrics to a 3rd party monitoring system. It also provides an example of such an implementation, that forwards all of the metrics mentioned above into a RIG (Riemann, InfluxDB, Grafana) stack. The github repo comes with a vagrant automation that will bootstrap the stack for you in a local VM, so you can play around.

Demo time

Enough talking, let’s see it in action. I’ve already cloned the github repo on my machine. Let’s setup the RIG stack before using mozzle .

$ cd mozzle/demo/mib
$ vagrant up --provision

This should run for a while. When it finishes, you can open http://localhost:3000 to verify the result. You should see Grafana’s login page. The default credentials are admin:admin .

After login, you see a list of predefined dashboards — namely Overview, Events and HTTP Statistics. Your application metrics will be visible there.

The last step is to instruct mozzle to start pulling metrics. It operates on CF space level and has option to derive the monitored target from the CF CLI. Let’s try it.

$ cf target
API endpoint:   https://api.run.pivotal.io (API version: 2.69.0)
User:           [my-email]@gmail.com
Org:            NASA
Space:          rocket
$ mozzle -use-cf-cli-target

The execution should block and metrics should start to appear in the Grafana dashboards. This is how it should look like after running it for a while.

Extensions

But you’ll say — what if I do not have RIG stack available, but something else. And here comes the best part — it is easily extensible. You need to implement a single method interface and you have your custom monitoring system plugged-in. (Maybe here’s the point when I should mention that it is implemented in Go)

type Emitter interface {
    Emit(m Metric)
}

The Emitter interfaces represents an object that is capable of emitting application metrics. And the Metric structure contains a whole lot of information regarding what it is actually measuring.

// Metric is a metric regarding an application.
type Metric struct {
	// Application is the name of the application.
	Application string
	// ApplicationID is the GUID of the application.
	ApplicationID string
	// Organization is the name of the application's organization.
	Organization string
	// Space is the name of the application's space.
	Space string

	// Time is the time when the event occurred.
	Time int64
	// Service for which the metric is relevant.
	Service string
	// Metric value. Could be int64, float32 or float64.
	Metric interface{}
	// State is a text description of the service's state - e.g. 'ok', 'warn'.
	State string
	// Attributes are key-value pairs describing the metric.
	Attributes map[string]string
}

What you need to do is juts translate the Metric into some format that your monitoring stack knows to read — and that’s it!

To recap — if you’re developing or operating a Cloud Foundry applications and you’re looking for a way to monitor them (and you definitely should be, if not doing it already) mozzle can be of great help.

You can find more details about it on its github page. Any feedback is welcome! Same holds for PRs :)