Vanity Metrics
Errors of Value, not Statistics
by Ethar Alali
In what appears to be becoming a good regular conversation most mornings, I was debating the unmeritorious position of vanity metrics with Gregor Wikstrand. The conversation started from my meme response to a tweet of his yesterday.
Definition: Vanity Metrics
For readers unfamiliar with the term, a vanity metric is a measure of a single, often independent variable. The kind of variables you learn in school that are [mis]used to make ourselves feel great, but are pretty lousy prediction and decision making variables. They almost always fail to consider the other side of trade-offs and are generally pretty useless.
Yet they are used in abundance throughout procurement processes and have been for some time. It’s only now that organisations are starting to realise the problems with vanity metrics. Indeed, I stopped applying for tenders where Lowest Cost is used and have even rejected some Most Economically Advantageous Tenders (MEAT) processes because they are mathematically disadvantageous.
The term is probably most readily attributed to Eric Reis in the Lean Startup but can be difficult to spot with traditional business models, mainly because people are focused on their own little world.
Here’s a hypothetical example. A person walks into a shop and wants to buy a box of 10 rubber ducks priced in 8 units of your favourite currency (I’ll use pounds). They have to make a decision on whether or not the cost of £8 gives them the value they want out of it. i.e. are they getting value for money from each investment they make?
Questions force you into thinking critically. Thinking “Why?”. Otherwise it’s just another drunk, eBay impulse buy you’ll regret.
However, the explicit, advertised position is £8. That isn’t enough information for them to make a decision on the value of the rubber duck. The other side is contextually dependent. There are several questions the customers needs to ask themselves, including but not limited to:
- Do I have that many children?
- Do I have a bath or a shower?
- Why do I want to buy this anyway?
Another, more realistic example, is visitors to a website. SEO companies often sell, what I will unashamedly call the “snake oil” of website visits. That is a variable. “We are optimising x”.
As a business, why do website visits matter to you? What does that give you that you can’t get another way with more certainty? You’ll notice that these questions force you into thinking critically. Thinking “Why?”. Otherwise it’s just another drunk, eBay impulse buy you’ll regret later.
Fix it! Step 1: Building a Value Proposition
You want more sales through a digital channel? Great! That is heading in the right direction. Give yourself a “value hypothesis”. What do you want? Why do you want it? What constraints does it place?
“I want to increase my sales by 30%…”
Consider if you are able to meet the value proposition consistently.
“…So that I can keep my business busy…”
And crucially, place your constraints around it. If you have 10 staff only, and 30% would fully utilise them, then state that as part of your value hypothesis.
“…yet my 10 folk can manage.”
Then read it all out:
“I want to increase my sales by 30%, So I can keep my business busy, yet my 10 folk can manage”
Now, even this isn’t the final answer. What you really mean here is you want to maximise profit. Profit as we know, is a simple equation:
profit = income - expenditure
There are two variables here, one maps to staff wages, the other to your sales revenue. So when you see a sales graph like this:
The first question you should ask yourself is “OK, what were my costs?”, because I can almost guarantee that your graphs will not look the same:
Making the Match
The reality is these days, Apple is pretty good at correlating their profit graphs to revenue graphs. There are graphs which are so much worse! Indeed, many of the DotCom era never made their costs back, so were running on nothing but a loss. In order to get value from your efforts, you have to monitor all the necessary signals, but no more and for no longer than is necessary.
In order to be at scale, you would also have scaled your sales revenue too. To concentrate on the need to fix, without augmentation the cost of the fix, is another example of a vanity metric, albeit an ugly one.
This brings me on to the topic that kicked off this blog, the difference between statistical significance and vanity metrics. First, I want to be clear about the ideal use of statistics in businesses in general.
Risk
Risk is a measure of the probability of an event and crucially the sensitivity the business has to that risk. If something happens once a week, but costs you £10 each week, whilst it would cost £10 million to fix, then you would have to be alive for 1 million weeks for it to make any difference.
Granted this will increase with scale, but in order to be at scale, you would also have scaled your sales revenue too. To concentrate on the need to fix, without augmentation the cost of the fix, and the realisation that your sales revenue will have to have scaled, is another example of a vanity metric, albeit an ugly one. This gives you what financiers call “Value at Risk” or VAR, which is simply:
VAR = “Probability of event” multiplied by “Cost at each event”
Opportunity
By the same token, statistics can and should be used for opportunities. The opportunity is the size of the market (statistical sampling can be used here and then the population extrapolated from it)and crucially, the value of the sale. Simple enough. This has a similar value opportunity of:
VOP = “Probability of sale” multiplied by “Value at each event”
Seem familiar? Right! Numbers are just numbers. It is the interpretation of those numbers that gives them human meaning. Both equations have two variables, of which probability is only one. Indeed, when you approach some Venture Capital, you’ll notice they bring you down to earth with the statement that “you won’t capture all of the market. You might get 3%, tops!” This is because they are aware of the need to factor in the probability of the event, which the pitchers probably were not.
Statistical Unsoundness v Vanity Metrics
Now, to answer Gregor’s question of where they fit within statistical unsoundness. The key word is “probability”. After all, that is what statistics studies. You can model the value as the dependent statistic you’re measuring, but in truth, you do have full, causal control of one of them. In the above examples, the cost or the sales value. The other is the statistic. Simplez!
When studying that statistic, the normal rules of solid research and experimentation apply. Try to use the mainstay of scientific research:
- Double-blind
- Randomised
- Controlled
Trials run in the form of A/B-tests, or multivariate testing models. There are crucial aspects to reliably tell if the result is significantly better than chance and/or the other variate(s) and so state something is better is a simply false assertion. The chi-square test is a typical frequentist method of determining of a statistical event is significant or not and this means there are really three types of result:
- Confirm the null hypothesis
- Accept the Alternate Hypothesis
- Inconclusive results
The third of these often accompanies a statistically insignificant change in the occurrence of an event, making you unable to disprove the null hypothesis, or a failure to consider a covariate that could also have lead to the result.
Conclusion
It’s not within the scope of this article to introduce statistics or experimentation. However, it is important to understand the difference between statistical [in]significance and vanity metrics. Like SEO, it is possible to be statistically significant in the results of experimenting with organic search. However, if we don’t factor in the whole picture, the result is a vanity metric.
Ethar is Director of Axelisys. He works with large and small companies alike, to help them work more effectively with their IT estates.
