How to derive other metrics of Performance Counter in Azure Monitor

[As of June 10, 2021]

The original article is here (in Japanese).

Azure Monitor パフォーマンスカウンターの算出ロジック

Recently I wrote the following entry.

Why isn’t “Used memory MBytes” in performance counter of Azure Monitor equal to “used” got via…

When I answered to the person, he/she asked me again about the related question.

Could you please share how to derive the following metrics with me?

Memory usage ratio (% Used Memory)
Disk usage ratio (% Used Space)
Free size of disks (Free Megabytes)
CPU usage ratio (% Processor Time)

How to collect metrics

Azure Monitor agent and Log Analytics agent use PAL (Platform Abstraction Layer) to collect and calculate metrics and send to Azure Monitor. Source codes for PAL is published on GitHub.

microsoft/pal

Memory, CPU, and Disk related metrics are collected and calculated on SCXSystemLib, specifically on *instance.cpp.

Memory

Target source code is scxsystemlib/memory/memoryinstance.cpp. You can see that /proc/meminfo and /proc/vmstat are used as datasources.

Memory usage (Used Memory MBytes)

As I wrote in the previous entry, the following equation is used to derive the metric. In case of Linux, you can find this equation on this line of the source code.

Used Memory MBytes = MemTotal - MemAvailable

Memory usage ratio (% Used Memory)

We can calculate the ratio with the following equation.

% Used Memory = (MemTotal - MemAvailable) / MemTotal * 100

Disk

There are five source codes whose name contains the string of “instance”. As you can see performance counter collects metrics of logical disk in the following screenshot, you can focus on scxsystemlib/disk/staticlogicaldiskinstance.cpp.

Free size of disks (Free Megabytes)

The system call of statvfs() is executed to collect base info for calculating free size of disks. The equation is found in this line of staticlogicaldiskinstance.cpp.

// file system block size * # of free blocks
m_availableSpace = f_bfree * f_frsize

Disk usage ratio (% Used Space)

The following equation is used to derive the metric, which is the same value via df -m.

% Used Space = (m_sizeInBytes - m_availableSpace) / m_sizeInBytes * 100

CPU

CPU related metrics are collected and calculated in scxsystemlib/cpu/cpuinstance.cpp.

CPU usage (% Processor Time)

This line reveals that several delta tics are used to derive CPU usage.

m_processorTime
= (user_delta_tics + nice_delta_tics + system_delta_tics + irq_delta_tics + softirq_delta_tics) / 
total_delta_tics * 100 

Since outputs from mpstat are already rounded, each metric from performance counter is not equal to one got via mpstat, though nearly equal to.