How NOT to become a Data-Driven Organization
Six Soft Pitfalls to Avoid on the Way to a Data Driven Organization
Starting a blog post by writing about the data revolution is a cliché (but I am going to start with this statement anyway). The potential of data for organizations is an undoubted and widely-agreed fact. As geeks of algorithms and technology, we at YellowRoad enjoy learning about the many advancements surrounding data. It is a crucial part of our professionalism. However, as we meet and serve more and more organizations, the one thing that becomes apparent to us is that the technical data aspects (aka algorithms and technology) are only second in importance to data-driven initiatives. It is the soft organizational aspects that eventually lead to success or failure.Now, please don’t get us wrong.
We provide technical services (cutting edge, if I may add). Doing things right, technically wise, is a must. It is challenging, and we see many pitfalls in that arena, as well. The thing is that even with the best technical practices, no success will be obtained, without the firm foundations of strong (yet soft) business aspects.
In this post, we will describe 6 soft business pitfalls that lead to failure for many organizations on their way to become data driven.
1. Lack of data vision (or, vision is a leadership responsibility, not a technical aspect)
Information systems emerged mainly for transaction processing. In transaction processing, the design of data structures and data processing procedures is merely technical. Consequently, IT divisions (mainly in enterprises, in those days) often took a technocratic perspective on systems, and management in these organizations learned that data was something technical that needed no management guidance.
Being a data-driven organization means using data to tackle business challenges of a higher level. For managers who got used to think that data is unimportant in the tactic and strategic level, the change in perspective is sometimes hard. These managers might acknowledge the potential importance of data, but they often fail in showing the right way to their organization, data wise. It is very hard to make real progress in materializing the potential of data, without a clear and sharp data vision.
We have seen C-level executives who show lots of enthusiasm towards data many times. They acknowledge the potential and say that their organizations must compete on data. Trying to coin the enthusiasm into a somewhat more practical level (i.e., how are you going to compete on data?) will sometimes reveal a surprising gap. We even hear: “Why don’t you tell me, you are the experts”. So yes, we see ourselves as trusted consultants in data strategy. But, when data has the potential to significantly affect and even transform businesses, we truly believe that it is the role of the organization’s leadership to take the call.
2. Inability to act according to the outcomes of the data
A weak data vision often creates situations in which data-driven business objectives are not planned all the way through. For example, a very common data-driven task is churn prediction (to be honest, it was more common a decade or two ago than it is today). The intuition makes sense. Recruiting a new customer costs way more than retaining the existing customers. If we can predict churn before it actually happens, and somewhat proactively retain the customers who are expected to churn, then we have an economic way of increasing the number of active customers. In many cases, however, the thing is that there is a very low certainty in the propensity of a specific customer to churn. For example, let us assume that two percent of the customers churn every month. And let us further assume that we could allocate a segment of customers who are 10 times more probable to churn (10 times! That’s a lot). Still, in this segment, 4 out of 5 customers will not churn, which actually multiplies the costs of the proactive actions by 5 (you have to invest in customers who are not really churns), and often makes retention through churn prediction way less attractive than it seems at first. Not being able to act according to the outcomes may be caused by many reasons: we don’t have such actions in mind, law or regulations do not allow such actions, costs are too high, actions are unethical or immoral, and more. While serving our customers, and learning about a new data driven idea we often ask: “And then what? How will you act given this and that outcome?” We believe that having concrete answers to these questions is a must, and in most cases, the answers must be clear before you start.
3. A Voodoo-Based Organizational Culture
Organizations that are not driven by data actually operate in an open loop. Employees in these organizations make decisions, which make an impact that they cannot measure. Without measuring the outcomes of different decisions, exploring and concluding, no learning occurs. When organizations operate in an open loop for too long, it is often the case that voodoos replace the real feedback.
In such organizations, there are often a wide set of beliefs regarding causes and effects. These beliefs are sometimes true, but most of the time they are either inaccurate, describe situations of the past, or are complete nonsense.
A typical example of a voodoo-based culture is referring to a specific segment of customers as VIPs, due to a small number (or sometimes even a single observation) of highly profitable customers in this segment being observed. Another type of voodoo belief results from inappropriate statistical analysis. For example, in funnel analysis (which characterizes many online marketing operations), you sometimes find companies running A/B testing that merely take averages, with no statistical significance, whatsoever. Clearly the power of these A/B tests is merely slightly better than random. Similarly, you often find organizations that mix correlation and causation (having two correlated variables doesn’t mean that one of them affects the other).
When the voodoo-based culture is strong, even hard evidence is not always convincing. In an adaptive organization, consistent efforts to act based on hard evidence will eventually help in changing the culture, but in a less adaptive organization, we often see that the voodoos win data.
4. Data is not available or is in pure quality
Being data driven clearly requires data. Sounds trivial, right? Well, as surprising as it may seem, we have been asked so many times to design a data-driven system when there is no relevant data at all. When we ask about that negligible aspect, the typical reply is: “design the algorithm for us, and we will start running it when the data arrives”. Planning working algorithms requires evaluating and testing them. Simulated data is nice, clean, and easy to handle. When the real data arrives, it always, always, proves to be totally different from the plans. It’s poor in quality, has lots of missing value, different formats, unstable sampling rates and more.
It is good practice to think and plan out a system before you have the data. In such planning, you might gain a better understanding of the nature of the data that you need for your system. But, no working system which is based on data can be designed and launched without real data.
5. Under-trained data team
With all the advancements that are being made around data, many data analysis tools have become available to the public. Some of these tools are really easy to work with. You can do magic with some BI tools, and with basically no programming skills you can build a machine learning model within a single afternoon. Consequently, many organizations try to analyze data in-house, without having the proper trained team (it is somewhat like a non-handyman using a hammer — finding the tool and holding it is easy, but fixing a new set of shelves with it is a completely different aspiration). In many organizations, we see under-trained teams working with data. This post is on soft pitfalls. The post on technical pitfalls will be longer in at least two orders of magnitudes. With data analysis, you should be careful, knowledgeable, and experience each and every step that you make.
6. Wrong KPIs
Measurability is a key aspect of being data driven. Leadership and management have soft areas, and some things can be managed beautifully based on soft skills, without being measured. However, data processes should be optimized to measurable KPIs. A common mistake is mixing the set of business KPIs with the measures that are used for optimizing algorithms.
Leadership and management have soft areas, and some things can be managed beautifully based on soft skills, without being measured. However, data processes should be optimized to measurable KPIs. A common mistake is mixing the set of business KPIs with the measures that are used for optimizing algorithms.
The set of business KPIs indicate the wellbeing of the organization. It is very like a blood test that reflects something about the proper functionality of several key organs, it contains indications about the different aspects of the business. However, when you optimize a process or algorithm you typically need to decide on a single measure to optimize it to.
