Finding Java Hotspots with Satoris Hotspot
Getting the most out of the Satoris hotspot metering extension
Satoris, now freely available to download, comes packed with numerous metering extensions, but with only two enabled by default. One is the console metering extension for capturing snapshots of the metering model from the client monitoring console. The other is the hotspot metering extension, which dynamically disables the metering of particular named probes after a period of metering by the metering engine. The decision to disable a probe is largely done to control measurement overhead and to improve the accuracy and relevancy of the resulting performance model.
The Hotspot Scorecard
There are 2 primary set points for the hotspot metering extension.
Alternatively, the above can be rewritten in the following short form.
Assuming the default hotspot meter is clock.time then the above configuration instructs the hotspot metering extension to eventually disable those probes consistently having an inclusive wall clock time below 10 microseconds and an exclusive wall clock time below 2 microseconds. The inclusive clock time being the time spent executing the method and exclusive being that part of the inclusive time pertaining solely to the probe, the method, and not other probes metered within (nested) the executing scope of the probe. Another name for exclusive is inherent-time or self-time.
If method a takes 20 microseconds to execute and calls method b, which takes 10 microseconds to execute, then a’s inclusive time is 20 and the exclusive is 10.
The eventuality of a probe being disabled is controlled by a number of properties. The first being the initial property, which represents the opening balance on a scorecard created and maintained by the hotpot metering extension for a particular probe name. A hotspot scorecard balance is shared and updated across multiple threads firing the same named probes.
In general, a probe name is the name of the instrumented method prefaced with the method’s fully qualified class name. The parameter signature isn’t included.
When a metered probe has completed the delta between the paired meter readings is compared with the threshold. If the value is equal to or above the threshold then a credit is added to the running balance for the probe name. If it’s below then a debit value is subtracted from the balance. When a balance reaches zero for a particular probe name the “disabled” label is added to it. Once labeled the engine and extension(s) will vote not to meter any further.
A similar set of properties exist for exclusive (inherent) meter readings.
If method a takes 20 μs to execute and only 1 μs is exclusive (inherent) then the total adjustment to the scorecard balance will be minus 1 — 1 credit and 2 debit.
A good way to think about the two thresholds is in terms of call pruning. The threshold prunes away the leaf paths that repeatedly execute below the threshold. The inherent threshold, on the other hand, eliminates nodes and segments within a path acting merely as pass-thru call points of little cost.
When measuring the performance of a low latency execution code path it is good practice to make both thresholds near the same value. In such cases, it is best to employ iterative adaptive performance benchmarking alongside.
The Probes Open API can be used to test at runtime if a name was disabled.
When the console metering extension takes a snapshot of the metering model maintained by the metering engine it by default excludes metering data collected for those probes labeled as being disabled. To have disabled probes present within in a snapshot and viewable within the client monitoring tool add the following system property to the configuration file.
The hotspot metering extension does more than just disable the metering of firing probes deemed non-hotspots. It also classifies, labels, probe names as it intercepts, interprets, and tallies the balance on the hotspot scorecard. When the balance goes above a lower property value a probe name is labeled as a hotspot. If the balance subsequently falls below this value then the metering extension will remove the hotspot label from the probe’s name.
The hotspot metering extension will continue to reassess and relabel a probe name, a metering group, until the scorecard balance reaches zero, and becomes labeled disabled, or when the balance goes above an upper value, after which a probe name is labeled as being unmanaged. Once unmanaged the hotspot metering extension will remove all augmentation and future interception of a probe with that name to further reduce metering overhead.