Replace and remove a label in a Prometheus Query

Photo by Michael Dziedzic on Unsplash

Your team has a fixed format of data streaming in from your nodes and going into prometheus. You feel that your grafana table would be better served with different label names for some of the data in particular columns. You can’t make any changes to that format on the backend, but you can make some label name changes via a query — sound familiar?!!?.

What if you just want to remove a label?

sum without (old_job_id) (your_metric)

This will remove the label “old_job_id” from the results of “your_metric”.

What if you want to replace it?

Your goal is to simply replace the old label name “old_job_id” with a new label name “new_task_id”. Prometheus label_replace will really “add” the new label name. It will preserve the old label name as well… So, that could be a problem, it’s not a true “replace in place”.

So if you want to “add” your new label name and “remove” the old label name, you need to do this:

sum without (old_job_id) (label_replace(metric, "new_task_id", "$1", "old_job_id", "(.*)"))

Here’s how this reads:

• sum without (old_job_id) will remove the old label name from the query output
• metric is your metric, like “node_filesystem_avail_bytes”
• “new_task_id” is where you would put your new label name
• “$1” is regex for using the string in new label name, don’t change this
• “old_job_id” is where you’ll put your old label, the one you want to get rid of
• (.*……. that mess is regex that will replace the whole label name

Now that you have your new query, you can feed this into your grafana query and voila!