The High Cost of Web Analytics
Insights cost more than you think, but just how much?
Talk of web analytics tools can conjure up thoughts of visits, bounce rates, pageviews, referrals, KPIs, tags, and so much analytics jargon. What’s your analytics “stack” look like? The list of analytics platforms and services is long — Google Analytics, Adobe, Mixpanel, Quantcast, KISSmetrics, Similarweb, Amplitude, FullStory, and on and on.
Decision-makers rely on web analytics to see inside the black box of online user behavior. Analytics tools provide developers, product managers, designers, marketers, and whoever needs information the power to uncover insights to drive decisions.
Even though analytics tools promise “actionable insights,” getting to those answers takes work. It can be expensive. Have you ever stopped and considered the cost?
Estimating the cost of web analytics.
Estimating the cost of your analytics stack is more complicated than it might seem at first blush — it’s not as simple as adding up monthly subscription fees.
Consider these six factors when estimating the cost of your web analytics stack.
1. The cost of the tool (or tools), itself.
The “sticker price” of a particular tool is the most easily quantifiable cost of analysis — and it’s also where discussion around analytics tools can get hung up.
Many analytics tools are free — i.e. there is no monthly fee to use them. Google Analytics is free. FullStory offers a free version, too, as do many other analytics platforms.
Outside of free tools, there are, of course, paid, subscription-based analytics options. These paid services can vary dramatically in price depending on lots of factors.
When estimating the cost of web analytics, “free” or “paid” should only be the beginning of your estimation. Don’t get stuck on the sticker price of a tool — particularly if that price is “free.”
“There ain’t no such thing as a free lunch” or “TANSTAAFL” is an expression popular in economics¹ for the general truth that it’s impossible to get something for nothing. When it comes to analytics tools, it’s easy to focus on “free;” but remember: TANSTAAFL.
Estimating the cost of web analytics has as much to do with the sticker price as it does with the non-trivial — yet not obvious — costs that have nothing to do with paying money to a service. It’s these less discussed analytics costs that will be the focus of discussion for the majority of this post.
Start with the price (even if it’s free), but then keep going.
2. The cost to implement the tool.
No magic wand exists that will install a given analytics tool², and implementing a tracking code across your web site or app can get downright complicated.
Depending on how your site or app is built, it can require any number of code injections — from putting code into proprietary, one-off pages to including it in the headers of templates, into widgets in Content Management Systems, etc.
Managing implementation never really stops — as your site or app grows and changes, it has to be maintained. The more complexity baked into your analytics tool, the more expensive maintenance can be.
Estimating the cost of implementation requires identifying what team or individual will own the process and factoring in the cost of their time. The cost to install and maintain analytics code can be measured by in terms of both salary and the work they aren’t doing — time spent maintaining analytics code is time not spent fixing bugs, improving the product, etc.
3. The cost of the tool on performance.
Every bit of code on your site or app has the potential to effect performance. Performance — site speed — can have a subtle, yet profound effect on the online experience for your users.
Much has been written on this subject³ and it’s common knowledge that performance matters, but consider one fact: Google has found that a mere 400ms delay in search results can result in a 0.44 percent drop in search volume⁴. Given Google is seeing some 2 Trillion searches per year, that 0.44 drop could mean a 9 Billion decrease in searches.
Site performance matters to your bottom line, so the impact of a given analytics tool (or a number of tools) on the speed and performance of your site or app can be measured in terms of bouncing users, missed sales, and lower KPIs.
4. The cost of low quality data.
This is a big one.
What you pay in terms of data quality is a hard to measure — data quality pivots on many factors from basic things like the relevance and reliability of the data to more complex factors that can be effected by our own cognitive hard-wiring. Let’s take a closer look.
Relevance of the data. Is the data relevant to the problems you’re looking to solve? Is it reliable? Relevance depends on whether the data received helps answer the question at hand. It also depends on timeliness — by the time you get the data, is it still useful for answering your question.
Google is a relevance machine: you ask it a question and it instantly gives you highly relevant search results. It also depends on timeliness — by the time you get the data out of the tool, will it still make a difference to your decisions?
Reliability of the data. How quickly can you get the answers you need? What additional setup is required to run a specific analysis? If the analytics tool provides the raw data you require, how long does it take to process the data into a meaningful insight?
The actual metrics provided. The metrics provided by a given analytics tool can have a strong-yet-subtle effect on the quality of data because metrics have built-in assumptions that frame your understanding. Let’s unpack this.
» What you see is all there is. In Daniel Kahneman’s book Thinking Fast and Slow, Kahneman shares research that supports the idea that people make decisions using only the information available to them — without considering what’s not explicitly known. “What you see is all there is” or “WYSIATI” is a way to remember this source of overconfidence in behavior. As it applies to web analytics, the tendency is to focus on what you know. In that way, the data that analytics tools present becomes authoritative — the singular basis for decisions — and can lead to “jumping to conclusions” without looking for gaps in your information.
» Framing effects. Framing effects take into consideration how the presentation of information effects how that information is interpreted. Consider the following example of a framing effect — ask yourself, which phrasing sounds more positive? More negative? 90% of users ignored the new feature on your website over the first week after launch -VS- 10% engaged with the new feature in the first week.
Framing effects are a major focus of behavioral psychology⁵ because how information is framed matters greatly to decisions. On the web, the effect of framing is far-reaching — consider the default “opt-in/out” switch on a sign-up form being flipped “on” instead of “off.” If it’s toggled to the “on,” position, you are all-but-guaranteed to have more people opt-in by default.
» Assumptions. Anyone who spends time with data understands that the pretty graph or clean metric comes with a list of assumptions and caveats. As the saying goes, “The devil is in the details.” Meanwhile, given that analysis is conducted to “figure something out,” the person crunching the numbers has the ability to bias the results based on their perspective. In the academic community, Nobel prize winning economist Ronald Coase may have summed up this phenomenon best in saying:
“If you torture the data long enough, it will confess.”
— Ronald Coase
If you’ve worked around analysts for any length of time, you will inevitably hear the sarcastic question raised, “What do you want the numbers to be?”
Anytime you have decided on a given metric, you have asserted it’s validity as a basis for decisions. Depending on the metric, this can be problematic. Metrics that aggregate data come with their own set of assumptions. For example, using averages without consideration for the nuances of the underlying data can lead to some strange outcomes. An excellent example of this is how by using average body measurements of pilots to design cockpits the Air Force put pilots in potentially deadly airplanes.
An example to understand how the metrics provided affect your data quality. Consider bounce rate. Bounce rate captures what percentage of users land on a page only to say “Bye, Felicia!” and leave the site.
Bounce rate, which is simple in theory, is complicated in application. For one, the “framing effect” built-in to bounce rate is that a high bounce rate is “bad.” However, given the number of single-page apps, dynamic pages with infinite scroll, and other common nuances within any given web site or application, bounce rate can be downright irrelevant as a standalone metric. By focusing on bounce rate, you have accepted the assumption that bounce rate as a useful metric on which to base decisions (WYSIATI). But in actuality, “it depends” — as is explicitly discussed in Google Analytic’s support documentation on Bounce Rate⁶.
» Sampling. If your web analytics tool uses sampling, the cost on data quality is in certainty — how confident can you be the sampled data reflects the underlying reality? Variations in what’s being measured can affect the veracity and significance of a given sample size. Many analytics tools rely on sampling to reduce the data processing requirements of their service. How reliable are those sampling decisions?
Meanwhile, edge cases when it comes to online behavior are less the exception than the rule. Differences in devices, browsers, and operating systems, and other factors all combine to make results highly specific to individual users. How does a given analytics tool gather it’s data? If it uses sampling, the reliability of the sampling procedures can be an important consideration.
Data quality matters. The cost of low quality data is making poor decisions that cost you time and money. “What you see is all there is” means that your analytics tool stack has a pile of answers ready for you, regardless of whether the answers are any good. If those answers are low quality (or unclear in suggesting what you should do), you find you have no choice but to do additional research through that tool or through looking to other analytics tools.
5. The cost of multiple tools.
The perfect, omniscient web analytics tool that has all the answers doesn’t exist (yet!). That means teams must stack together disparate tools in order to fill gaps in their knowledge.
Each tool will be susceptible to the costs detailed above — maintenance, data quality, effects on performance, etc., but in combination, you get the synergistic bonus cost of friction in having to shift across tools, manage multiple subscriptions, understand how to use each tool including knowing which tool is useful and for what, integrate where possible, translate metrics, and on and on.
Take the problems of a single analytical tool and compound them.
The cost of managing multiple tools can add up. The extra layer of complexity created through managing multiple tools must be factored in when estimating the total cost of your web analytics tools.
6. The cost of analysis, itself.
Last, but certainly not least, is the cost of actually running an analysis using the tool. This cost breaks down into how hard it is to use the tool, whether or not it can be used by decision-makers directly or if you need to hire analysts to do the number crunching. And both of these factors will cost you most in terms of time.
» The ease-of-use of the analytics tool. Can a decision-maker pick up the tool and immediately extract data — get at an insight? How much training is required to make the tool useful? The more complex the tool, the more friction it puts on getting to data.
» Hiring analysts. Ease-of-use directly affects whether or not decision-makers will need to rely on analysts to crunch the numbers. If the value in a tool requires analysts, you have the obvious cost of extra headcount but the less obvious cost of time.
» Time. “Time is money” — so the saying goes. Time has come up before in our analysis as it is a primary driver of data quality. Timeliness is required for data to be relevant. How do you measure the value of time? Answering that would require it’s own analysis; however, a simple heuristic works. The value of information is inversely proportional to to the time it takes to get that information. Applied to web analytics, you can apply this heuristic in two ways:
- Time-to-data. Measured as how long it takes to get at the raw data, “time-to-data” is affected by the complexity of the tool — setup requirements, ease-of-use, how easy it is to get data out of the tool, etc. If you’ve ever hunted for some obscure information only to find a very useful graph that can’t be exported into a table, you know the pain of time-to-data.
- Time-to-insights. Unlike time-t0-data, time-to-insights is how long it takes to go from raw data to meaningful, which is to say actionable, information. Analytics tools may have incredible data that is easy to extract, hence low time-to-data, but painfully expensive to turn that data into actionable insights.
Let’s take an example. It takes you 20 minutes on average to extract data for a report — that’s your time-to-data. Then, data-in-hand, it takes you two hours to process that data and tease out an insight—that’s your time-t0-insight. Now, repeat that process a few hundred times and if you’re an analyst, factor in the friction of passing on the report only to find that it needs to be re-run for a different time period or some other variable. Time-to-data and time-to-insights add up in a hurry.
Adding up all the costs of your analytics.
If you need a machine and don’t buy it, then you will ultimately find that you have paid for it and don’t have it.
— Henry Ford⁷
Here’s a quick recap of the six buckets to consider when estimating the cost of your web analytics:
Why go into such detail to estimate the cost of an analytics tool?
Because analytical tools are expensive — even if they are “free,” and it’s not at all obvious just how expensive they are. By analyzing the cost of a given web analytics tool (or stack of tools), you can articulate not only how expensive a given tool is, but also why it’s expensive.
If you’re just adding into your mix yet-another-free-tool — hey it’s free, why not? — have you remembered TANSTAAFL? Will the insights you get out of that free tool actually be worth the effort of installing, maintaining, and paying attention to that tool?
Your analytics tools may be costing you.
Whether it’s through developers spending months implementing and maintaining the tools, stakeholders struggling through aggregated reports that lack clarity or obvious actions, poor decisions made from assumption-loaded data, framing effects that limit how you even understand the information, or any number of other problems examined in this analysis, perhaps the juice isn’t worth the squeeze.
Analytics tools only work when they bring quality data and insights that is relevant to the stakeholders who need that data to make good decisions. The best analytics tools will be clear and actionable, having low time-to-data and time-to-insights.
Go forth and estimate the cost of your tools.
Keep asking the hard questions about your tools. Run an analysis and estimate the fully-baked cost of your web analytics stack.
And remember: the enlightenment you seek may come at a high price.
NOTES.
¹ While this expression has been around for a long time, it was popularized by Robert Heinlein in his book The Moon is a Harsh Mistress (1966).
² Some tools make it easier to install code across your site or app — such as Google Tag Manager.
³ For an excellent read, check out Paul Bakaus’ The Illusion of Speed (Bakaus is behind Google Amp)
⁴ Read Think with Google’s article The Google Gospel of Speed.
⁵ Wikipedia article on Framing Effect (psychology). Also see Daniel Kahneman’s book Thinking Fast and Slow.
⁶ Read the Google Analytics support article on Bounce Rate.
⁷ Quote via Clayton Christensen in his book How Will You Measure Your Life? Read the excellent synopsis on marginal vs full-cost thinking that relates to this quote and that book here.
