Analytics & The Quest to Make the Right Decision
by Otis Anderson
The Office Analytics team is, primarily, a product analytics team.
… what does that mean?
It means that we analyze user behavior.
We look for good leading indicators of business success.
We build dashboards to automate the answers to common questions.
We maintain an a/b testing platform.
We analyze experiments.
And we do all this for enterprise products, which means it’s slightly harder than it would be otherwise.
That’s what we do. But why?
Our overall point of view is that an analytics team should improve a company’s decisions. Which means that, ultimately, we have to make making good decisions easier. But as easy as we make them, unfortunately, sometimes it’s still cheaper to make bad decisions. Because, in the short run, it’s cheaper to be less informed and to do lazy reasoning than it is to be more informed and conscientious.
So:
An analytics team’s job is to make the right decision the cheapest one.
In general, data scientists don’t spend a lot of time talking about the costs involved in making a better decision. The following illustration shows roughly the process of making a decision that is, at least, data influenced.
Every step of this process involves sacrificing something: engineering time, money, storage, analyst time, users (if you acquire data from them in ways they find obnoxious) and, most importantly, the time and attention of decision-makers.
That last one is key. People don’t have to listen to an analysis. It takes some effort for decision-makers to even decide that they need data before making a decision. So, if there is any hope of improving decision making, the time and attention it takes to incorporate data should be as little as possible.
That’s why I’ve always been firmly in agreement with Drew Conway that data science in practice is a cross section of social science. In terms of the usage of data science at a technology company, the subject is the humans using your product, and the audience is the humans making it. So you had best spend most of your cleverness on thinking about human behavior.
Product analytics as we do it is essentially studying human behavior in the least expensive and most interesting way possible, while maintaining an acceptable level of accuracy. We produce decisions, but what makes people want to use those decisions is really what our business is.
Otis Anderson is a Data Scientist Manager at Yammer. This is his brain on stats.
