15 Steps to Write an Application Prometheus Exporter in GO
tl;dr: Full example code: https://github.com/teamzerolabs/mirth_channel_exporter, read on to see step by step instructions
Background
Exporters are the heart and soul of Prometheus Monitoring Pipelines. If you run into a situation where an exporter does not exist yet, you are encouraged to write your own. We will cover the steps required to make the exporter. No worries, it is quick.
Goal
- Write an exporter in GO.
- Exporter will call REST API against an application (Mirth Connect in this example) when scraped.
- Exporter will convert the result payload into Metrics.
1. Setup GO and Package Dependencies
- Follow the steps on https://golang.org/doc/install
- Create a folder “my_first_exporter”
go mod init my_first_exporter
go get github.com/prometheus/client_golang
go get github.com/joho/godotenv--> creates go.mod file
--> Installs dependency into the go.mod file
2. Create Entry-point and Import Dependencies
Create main.go file, and paste in the following:
package main
import (
"github.com/joho/godotenv"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)3. Put in Entry-point function main()
func main() {
}4. Add prometheus metrics endpoint and listen on the server port
func main() {
http.Handle("/metrics", promhttp.Handler())
log.Fatal(http.ListenAndServe(":9101", nil))
}5. Explore External Services API with curl
The application I am monitoring is MirthConnect. I will be making two API calls:
a. get channel statistics
b. get channel id and name mapping
curl -k --location --request GET 'https://apihost/api/channels/statistics' \
--user admin:admin
curl -k --location --request GET 'https://apihost/api/channels/idsAndNames' \
--user admin:admin6. Convert the curl call into go http calls, and handle the result parsing
This is the most difficult step in the entire process if you are new to Go. For my example, my endpoints return XML payload. This means I had to deserialize XML with the “encoding/xml” package.
A successful conversion means my GO program can perform the same API calls as the curl commands. Here are some GO packages to read up about:
- “crypto/tls" = specify TLS connection options.
“io/ioutil” = reading the result payload from buffer into strings
“net/http” = create transport and clients
“strconv” = converting string to numbers like floating point/integer
You can iterate through the payload and print out their values with the “log” package.
7. Declare Prometheus metric descriptions
In Prometheus, each metric is made of the following: metric name/metric label values/metric help text/metric type/measurement.
Example:
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code=”200"} 1.829159e+06
promhttp_metric_handler_requests_total{code=”500"} 0
promhttp_metric_handler_requests_total{code=”503"} 0
For application scrapers, we will define Prometheus metric descriptions, which includes metric name/metric labels/metric help text.
messagesReceived = prometheus.NewDesc(
prometheus.BuildFQName(namespace, "", "messages_received_total"),
"How many messages have been received (per channel).",
[]string{"channel"}, nil,
)8. Declare the Prometheus Exporter with interface stubs
Custom exporters requires 4 stubs:
a. A structure with member variables
b. A factory method that returns the structure
c. Describe function
d. Collect function
type Exporter struct {
mirthEndpoint, mirthUsername, mirthPassword string
}
func NewExporter(mirthEndpoint string, mirthUsername string, mirthPassword string) *Exporter {
return &Exporter{
mirthEndpoint: mirthEndpoint,
mirthUsername: mirthUsername,
mirthPassword: mirthPassword,
}
}func (e *Exporter) Describe(ch chan<- *prometheus.Desc) {
}func (e *Exporter) Collect(ch chan<- prometheus.Metric) {
}
9. Inside describe function, send it the metric descriptions from step 7.
func (e *Exporter) Describe(ch chan<- *prometheus.Desc) {
ch <- up
ch <- messagesReceived
ch <- messagesFiltered
ch <- messagesQueued
ch <- messagesSent
ch <- messagesErrored
}10. Move the logic of api calls from 6 into the collect function
Here we want to take the logic from step 6, and instead of printing things out to the screen, sending it into prometheus.Metric channel:
func (e *Exporter) Collect(ch chan<- prometheus.Metric) {
channelIdNameMap, err := e.LoadChannelIdNameMap()
if err != nil {
ch <- prometheus.MustNewConstMetric(
up, prometheus.GaugeValue, 0,
)
log.Println(err)
return
}
ch <- prometheus.MustNewConstMetric(
up, prometheus.GaugeValue, 1,
)
e.HitMirthRestApisAndUpdateMetrics(channelIdNameMap, ch)
}As you perform the api call, be sure to send in the measurements using prometheus.MustNewConstMetric(prometheus.Desc, metric type, measurement)
For cases where you need to pass in extra labels, include them at the end of the argument list like this:
channelError, _ := strconv.ParseFloat(channelStatsList.Channels[i].Error, 64)ch <- prometheus.MustNewConstMetric(
messagesErrored, prometheus.GaugeValue, channelError, channelName,
)
11. Declare the exporter in main function, and register it
exporter := NewExporter(mirthEndpoint, mirthUsername, mirthPassword)
prometheus.MustRegister(exporter)Your exporter is now ready to use!
Every time you hit the metrics route, it will perform the api call, and return the result in Prometheus Text file format.
The rest of the steps deal with tidying things up for easier deployment.
12. Move hard coded api paths into flags
So far, we are hard coding a couple of things like application base url, metric route url, and exporter port. We can make the program more flexible by parsing these values from command line flags:
var (
listenAddress = flag.String("web.listen-address", ":9141",
"Address to listen on for telemetry")
metricsPath = flag.String("web.telemetry-path", "/metrics",
"Path under which to expose metrics")
)func main() {
flag.Parse()
...
http.Handle(*metricsPath, promhttp.Handler())
log.Fatal(http.ListenAndServe(*listenAddress, nil))
}
13. Move credentials into Environment variables
What if application endpoint changes place, or login credentials change? We can load these from environment variables. In this example, I am using the godotenv package to help with storing variable values locally in the same folder:
import (
"os"
)func main() {
err := godotenv.Load()
if err != nil {
log.Println("Error loading .env file, assume env variables are set.")
} mirthEndpoint := os.Getenv("MIRTH_ENDPOINT")
mirthUsername := os.Getenv("MIRTH_USERNAME")
mirthPassword := os.Getenv("MIRTH_PASSWORD")
}
14. Include a Makefile for quick building on different platforms
Makefiles can save you a lot of typing during development. For exporters that needs to build to multiple platforms (testing on windows/mac, running in Linux), you can start with the following:
linux:
GOOS=linux GOARCH=amd64 go build
mac:
GOOS=darwin GOARCH=amd64 go buildSimply invoke make mac or make linux to see different executable file show up.
15. Writing a service file to run this go program as a daemon
Depending on where this exporter will run, you can either write a service file, or a Dockerfile.
A simple Centos 7 Service file looks like the following:
[Unit]
Description=mirth channel exporter
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=1
WorkingDirectory=/mirth/mirthconnect
EnvironmentFile=/etc/sysconfig/mirth_channel_exporter
ExecStart=/mirth/mirthconnect/mirth_channel_exporter
[Install]
WantedBy=multi-user.targetThat’s all!
Throughout the flow, the only difficult step is 6. If you know what calls are available and how to parse them, the rest come naturally. Come create more exporters and get more metrics flowing today!
Thanks for reading, if you need to grab some application metrics quickly, you can contact us at info@teamzerolabs.com!
