Subtract two different metrics in Prometheus

Photo by William Warby on Unsplash

Let’s skip the fact that this should be waaaaaay easier to find out than combing dozens of stack overflow articles and where ever else you were before you landed here…

The answer is below, but, it’s best if you try to understand WHY PromQL was written like this, and hopefully you’ll agree with their intention. Labels are EVERYTHING in prometheus… They affect soooo many aspects of gathering metrics and more importantly they completely affect comparing metrics. Sure, you looked at two metrics and they have some resulting numbers… To you it makes total sense to just tap in:

METRIC1 - METRIC2 = ???

That WOULD make sense if you were making a SQL query and choosing “select and where etc. etc.”, but with PromQL you need to remember that every metric comes with a label payload, usually labels like:

• instance
• job
• group
• and any other custom labels you might find

Chant this mantra:

In order to compare two different metrics, you must write the query so that the same labels are also being compared…

For example there are 3 labels shown in this metric:

• group
• instance
• job

node_time_seconds{group="production",instance="myhost",job="production"} 1586282218.7363448

I have a custom metric that calculates epoch time of a file for comparison, the labels it contains are very different:

time_epoch{name="XXX",file_path="/home/path/path/path",file_size="XXXXXX",group="production",instance="myhost123",job="production"} 1586291547.245

My goal is to subtract one result from the other… But you can’t directly do that unless you reduce the number of labels you are going to compare against in the query. This can be done two ways:

• use “ignoring (job,instance,group)”
• use “on (instance,job)”
• or, better yet, don’t ignore any of them, and match none of them with “on ()”

The quickest way to do it is to set one of the metrics “on ()” like this:

(node_time_seconds{instance="myhost"} - on () time_epoch{name="XXX"})
value = -8323.487774133682

Hopefully this helps you out understanding this little nuance with PromQL and making comparisons.