Meaningful Metrics:
Analyzing Ratios
The Do’s and Don’ts of Developing Measurements That Matter
One of the most common issues I’ve come across in my career is a failure to thoughtfully consider the development and construction of ratios for use in KPI and general reporting. If you get only one insight from this article, know that as a general rule:
You should try to pair nominal values with meaningful ratios in all reports and visualizations.
Personally, I am a big fan of a trifecta that includes both a nominal sum, a meaningful ratio, and a simple segmentation. If you wrap it in a donut, all the sweeter!
Extra points for insightful color coding. In this example, I would assume that the products are as distinct as their colors and that green = good.
But Let’s Focus On The Ratios
Ratios most common fall into two buckets — percentages or averages. Feel free to add a third bucket of All The Rest for the MECE crowd.
A strong argument can be made the percentages are both the easier and more powerful of the two. In our example, we use a year over year growth rate. Easy enough, right?
Unfortunately, even a simple year over year calculation starts to show some of the intricacies of ratios. This is most simply labeled as — what is the denominator?
Now you may be thinking — the same period from last year… duh! Only that is really over simplifying things. If our example is a month-end number, perhaps you have a point. Well, you have a point as long as neither the numerator nor denominator is a February affected by a leap year. What if the number is weekly? There are several standards for calculating week of the year. To further the complexity, many trends are affecting by holidays, monthly cycles (like month end), and other factors. These factors often fall in different weeks during any given year.
While our Y/Y example illustrates a percentage used to measure comparison to a prior period, we may also what to compare a subset to a whole. Note — as a rule:
Numerators and denominator in any useful ratio must have meaningful logical connection, not just a history of correlation.
If your numerator and denominator do not share a meaningful logical connection (and you still want to use this ratio), do not put a percentage sign on it!
Unfortunately, even assuming you have strong logical discipline in the creation of your percentages, you must remember that these measures are binary. In other words, the percentage may change because of movement in either the denominator or the numerator. Your percentage is not so insightful, if that is not apparent.
Averages have all their own trials and tribulations. First, means, medians, and modes are all often called averages. Averages are often filtered and/or weighted with little done to call that out.
Averages have a secondary issue as well. I have written extensively about the myth of the average. Averages are meant to simplify the quantification of normally distributed populations. Populations that are not normally distributed have averages that are often quite meaningless and misleading.
Finally, averages calculated in time series are also susceptible to changing denominators. As a rule:
If you are going to calculate an average over time, you should do your best to lock the denominator.
Many an analyst has pronounced improving or deteriorating performance based on the movement of an average, only to realize that the underlying population had changed along the way. In other words, performance hadn’t improved, the population had simply changed. If you are going to quantify population changes there are better measures than averages.
At this point, we could look at the third bucket. This would include interesting ratios for certain, but they often stretch, if not break, the logical connection rule. They are typically chosen for loose connections and high correlation. But in general, they fall into the same risks as percentages and averages (with perhaps slightly higher likelihood).
So in summary, try to pair nominals and rates. Do not overlook the trend of denominator and numerator when monitoring a ratio. Ratios work best when their populations are held steady. Know the details when a ratio is used as a point of comparison, they matter greatly! And finally, try to keep the highest level of logical connection between your two numbers.
Oh, and thanks for reading!
