Write a Prometheus API Exporter in Record Time

Jack Yeh

Published in

TeamZeroLabs

6 min readNov 8, 2020

Got an API you want to monitor the trend of? You can do it really fast with the right sets of tools.

Data. Is. Everywhere.

With the collective effort of the entire connected world, more and more data sets are available to the public on the net. With the right keyword, you can find all kinds of real time updates. In this article, I will show you how you can take one such data endpoint and create a Prometheus Metrics Exporter in front. But first, a few key terms:

Metric — A measurement of something, examples include Temperature, Mileage, Power percentage, etc.
Sample — An instance of a metric, associated with a timestamp and other accessory information.
Time Series — A bunch of samples, often presented as a graph.
Prometheus — A monitoring program that visits targets to collect metrics and store them in local time series database. It also exposes both an UI to plot graphs and an API to query time series data.
Exporter — A program that display Metric samples for Prometheus to collect.
Grafana — A Data visualization program that can display data in Graphs/Pie Charts/Tables from various data sources; one of which is Prometheus.

If we want to visualize measurements taken at different time, we can do the following:

Setup exporters
Setup Prometheus to scrape these exporters for samples. Over time, Prometheus will build up Time Series for samples collected.
Setup Grafana to visualize the data within Prometheus.
… And more! (Alerting on thresholds, and refining types of measurement as time goes on)

Next, we will go over the steps for creating an API exporter. The API I have chosen to use is from disease.sh. We will pull metrics from there every 5 minutes.

You can find the completed exporter here: https://github.com/teamzerolabs/covid_exporter

A minimum Typescript API exporter only needs 3 Library:

Express
Prom-Client
Axios

and 4 files:

main.ts
metrics.ts
response.type.ts
data-fetcher.ts

Step 1: Create the base project

Follow these steps to create the folder and install dependencies:

mkdir covid_export && cd covid_export
yarn init
npm i typescript --save-dev
npx tsc --init
yarn add --dev @types/node @types/express ts-node-dev ts-node ts-loader tsconfig-paths
yarn add typescript prom-client axios express
mkdir src

Our exporter will use AXIOS to call into the APIs available at disease.sh every 5 minutes, update metrics internally, and allow Prometheus to collect samples via Express.

Step 2: Setup the Prometheus endpoint

import express from 'express'
import promClient from 'prom-client'

promClient.collectDefaultMetrics()

console.log(
  `Hello folks. We will setup a process that hits an api every 5 minutes, and update prometheus metrics.`
)

const metricServer = express()
metricServer.get('/metrics', (req, res) => {
  console.log('Scraped')
  res.send(promClient.register.metrics())
})

metricServer.listen(9991, () =>
  console.log(`🚨 Prometheus listening on port 9991 /metrics`)
)

In the script above, we create an express instance, and declare a route to return Prometheus metrics. We also listen on port 9991. You can run the program with yarn start and visit it in the browser.

Make sure you see the default metrics exported before continuing!

Step 3: Look at Data returned by the API, and write functions to fetch them

More api documentations can be found at : https://disease.sh/docs/
This endpoint https://disease.sh/v3/covid-19/all returns data in this format:

{
  updated: number
  cases: number
  todayCases: number
  deaths: number
  todayDeaths: number
  recovered: number
  todayRecovered: number
  active: number
  critical: number
  casesPerOneMillion: number
  deathsPerOneMillion: number
  tests: number
  testsPerOneMillion: number
  population: number
  oneCasePerPeople: number
  oneDeathPerPeople: number
  oneTestPerPeople: number
  activePerOneMillion: number
  recoveredPerOneMillion: number
  criticalPerOneMillion: number
  affectedCountries: number
}

We will declare the types for the APIs that we are interested in response.type.ts .

export type CovidStateResponse = {
  state: string
  updated: number
  cases: number
  todayCases: number
  deaths: number
  todayDeaths: number
  active: number
  casesPerOneMillion: number
  deathsPerOneMillion: number
  tests: number
  testsPerOneMillion: number
  population: number
}
export type CovidStatesResponse = CovidStateResponse[]
...
Do the same for World and Country responses.

Once the types are defined, we can use them with Axios calls in data-fetcher.ts:

export async function fetchCovidAll(): Promise<CovidAllResponse | undefined> {
  const result = await Axios({
    url: 'https://disease.sh/v3/covid-19/all',
  })
  if (result.status === 200) {
    return result.data as CovidAllResponse
  } else {
    return undefined
  }
}

export async function fetchCovidStates(): Promise<
  CovidStatesResponse | undefined
> {
  const result = await Axios({
    url: 'https://disease.sh/v3/covid-19/states',
  })
  if (result.status === 200) {
    return result.data as CovidStatesResponse
  } else {
    return undefined
  }
}

Each function performs the simple job of getting the json data.

Step 4: Declare Prometheus Metrics

Now that we have functions to retrieve the data, we are two steps away from completing the exporter.

Make a metrics.ts file and start declaring Prometheus metric variables in there:

import { Gauge } from 'prom-client'

export const covid_cases_total = new Gauge({
  name: 'covid_cases_total',
  help: 'Total number of covid cases; type = [world, us_states, country]',
  labelNames: ['type', 'name'],
})

export const covid_active_total = new Gauge({
  name: 'covid_active_total',
  help:
    'Total number of covid active cases; type = [world, us_states, country]',
  labelNames: ['type', 'name'],
})

export const covid_recovered_total = new Gauge({
  name: 'covid_recovered_total',
  help:
    'Total number of covid recovered cases; type = [world, us_states, country]',
  labelNames: ['type', 'name'],
})
...
Etc

Each metric has 3 properties:

name: Metric name, also needs to be a valid variable identifier
help doc: a quick explanation on what this metric measures.
label names: We store additional information in these fields such as categories, types, and names.

Last Step: Calling API and storing results into Prometheus Metric Variables

Go back to data-fetcher.ts, and add the collectMetrics method:

export async function collectMetrics() {
  const allResponse = await fetchCovidAll()
  const stateResponse = await fetchCovidStates()
  const countryResponse = await fetchCovidCountries()

  if (allResponse) {
    covid_cases_total.labels('world', 'all').set(allResponse.cases)
    covid_active_total.labels('world', 'all').set(allResponse.active)
    covid_recovered_total.labels('world', 'all').set(allResponse.recovered)
    covid_deaths_total.labels('world', 'all').set(allResponse.deaths)
    covid_tests_total.labels('world', 'all').set(allResponse.tests)
    covid_population_total.labels('world', 'all').set(allResponse.population)
  }

  if (stateResponse) {
    stateResponse.forEach((state) => {
      covid_cases_total.labels('us_state', state.state).set(state.cases)
      covid_active_total.labels('us_state', state.state).set(state.active)
      covid_deaths_total.labels('us_state', state.state).set(state.deaths)
      covid_tests_total.labels('us_state', state.state).set(state.tests)
      covid_population_total
        .labels('us_state', state.state)
        .set(state.population)
    })
  }

  if (countryResponse) {
    countryResponse.forEach((country) => {
      covid_cases_total.labels('country', country.country).set(country.cases)
      covid_active_total.labels('country', country.country).set(country.active)
      covid_recovered_total
        .labels('country', country.country)
        .set(country.recovered)
      covid_deaths_total.labels('country', country.country).set(country.deaths)
      covid_tests_total.labels('country', country.country).set(country.tests)
      covid_population_total
        .labels('country', country.country)
        .set(country.population)
    })
  }

  console.log(`Metrics refreshed!`)
}

We first issue requests into the endpoint, and on successful responses, we loop through the data payload and store each metric into the appropriate prometheus metrics variable with the right label values.

Next, add these lines into main.ts :

setInterval(() => {
  collectMetrics()
}, 5 * 60 * 1000)

collectMetrics()

Which runs the collection once on program startup, and once every 5 minutes.

That’s all, the exporter is now ready for Prometheus to scrape. Try running it and you should see these metrics exported in the browser:

# HELP covid_cases_total Total number of covid cases; type = [world, us_states, country]
# TYPE covid_cases_total gauge
covid_cases_total{type="world",name="all"} 50263885
covid_cases_total{type="us_state",name="Texas"} 1012428
covid_cases_total{type="us_state",name="California"} 970392
covid_cases_total{type="us_state",name="Florida"} 837077
covid_cases_total{type="us_state",name="New York"} 562365
covid_cases_total{type="us_state",name="Illinois"} 477978
covid_cases_total{type="us_state",name="Georgia"} 404622
...# HELP covid_active_total Total number of covid active cases; type = [world, us_states, country]
# TYPE covid_active_total gauge
covid_active_total{type="world",name="all"} 13457625
covid_active_total{type="us_state",name="Texas"} 163723
covid_active_total{type="us_state",name="California"} 458159
covid_active_total{type="us_state",name="Florida"} 226127
...covid_active_total{type="country",name="Ukraine"} 238393
covid_active_total{type="country",name="Uruguay"} 477
covid_active_total{type="country",name="Uzbekistan"} 2236
covid_active_total{type="country",name="Venezuela"} 4372
covid_active_total{type="country",name="Vietnam"} 108
covid_active_total{type="country",name="Wallis and Futuna"} 0
covid_active_total{type="country",name="Western Sahara"} 1
covid_active_total{type="country",name="Yemen"} 93
covid_active_total{type="country",name="Zambia"} 686
covid_active_total{type="country",name="Zimbabwe"} 252
...# HELP covid_tests_total Total number of covid tests; type = [world, us_states, country]
# TYPE covid_tests_total gauge
covid_tests_total{type="world",name="all"} 864527553
covid_tests_total{type="us_state",name="Texas"} 9547430
covid_tests_total{type="us_state",name="California"} 19565151
covid_tests_total{type="us_state",name="Florida"} 10453650
covid_tests_total{type="us_state",name="New York"} 15519172

Believe it or not, it actually took me more time to write this article than in writing the code itself!

Measure More Metrics and Write More Exporters

Got an API to measure and visualize? Team Zero Labs can help, contact us at info@teamzerolabs.com