Anti-patterns in Enterprise Data Sciences
What is the market not telling you?
Nassim Nicholas Taleb, in his book “The Bed of Procrustes” noted that “people focus on role models; it is more effective to find anti-models — people you don’t want to resemble when you grow up.” While Taleb’s quote is commonly referenced across journalism and social media today, it isn’t as applicable to real life as we might think — at least not to real business life. Rather than seeking out contrarian voices, industry leaders tend to be overly influenced by mainstream thinking. Consider all the searches for those SEO-optimized webpages that perennially comprise page-one search results, reinforcing the same timeworn, made-for-PowerPoint sets of ideas over and over again. “Establish a data lake,” they instruct business users. “Invest in data sciences.” “Compute and experiment at scale.” And on and on. It is staggering to consider the number of companies fundamentally built on such “best practices”, their leaders devoutly believing that in doing so they will roll out inevitably trend-setting products and features and win the battle for customers.
Thinking in the fields of Data Sciences and Artificial Intelligence (AI) has been no different. A quick search for “best practices in machine learning” returns more than over 200 million Google results. In fact, “machine learning” has been the poster child of enterprise success stories for the last decade. It has led to the astronomical rise in data scientist salaries and demand. With data being the new rocket fuel, data science has become the hottest career path of the new century. Fueled by the astronomical amounts of money raised by start-ups, these corporate stars boldly leverage Machine Learning to drive AI-based insights to the heart of the enterprise. In the 2019 MIT SMR and BCG Research Report, fully 90% of respondents agreed that AI represents a business opportunity for their company, while the lack of it was deemed a strategic risk. “What if competitors, particularly unencumbered new entrants, figure out AI before we do?” quoted one of the respondents.
There have been multiple forces pushing AI’s growth, including its ability to provide greater visibility into consumer behavior and demand, an increase in the accessibility of information, technological ease and progress in the AI field — along with a much better understanding of how to integrate Data Sciences and Analytics into everyday operations. Enterprises are also increasingly focusing on a lifetime-value-centric approach, which represents a departure from purely retrospective approach of understanding and rewarding customers. Instead, companies are embracing a more forward-looking approach that leverages predictive and prescriptive components of the analytics spectrum.
But as encouraging as this all sounds for AI enthusiasts, many statistics offer a very different perspective. For example, the MIT SMR and BCG research report also highlights that seven out of 10 companies surveyed have reported minimal or no impact from AI. Of all the companies that have made some investment into ML and AI, less than 40% can correlate business gains in the past three years to these same investments. If these tools are so transformational, if market dynamics are more or less equally exposed to every company, and if everyone has access to the best practices in processes, technology, and workforce management, then one has to ask: Why are there only a handful have great AI stories to tell the world?
A quick scan of industry reports points to hundreds of factors causing these poor returns on investment. With respect to AI, companies are said to be “not doing the right things” or “doing things wrong.” When one digs into these vague charges — when we put on our data scientist hats and boots and start digging into the nuances of how enterprise analytics run on the ground — a clearer picture emerges. Upon this closer inspection, some very interesting patterns emerge that are consistent across industry verticals, geographies, and business functions. These patterns are repetitive and easily reproducible across functions. Avoiding these patterns would seem to be an almost “natural” solution to persistent AI-related problems. Together, they comprise what might be called industry “worst practices” and emerge across different organizations as, in effect, “anti-patterns.”
Seen from the 30,000-foot level, these anti-patterns include:
· Misplaced reliance on “averages”
· Failure to pay attention to customer behavior across the buying journey
· Enterprise-data priorities that fail to develop feature-based strategies
· Lack of a “North Star” — a best-possible-outcome approach to experiment design
· A focus on evolving first instead of evolving fast
· An overemphasis on model building over model optimization
· A failure of inclusivity in AI and ML team building
Rather than stumbling on the nitty gritty mechanics of implementing AI and ML, companies more often fall victim to program-level design flaws and inefficiencies. These failures prevent many companies from ever realizing the full value of these costly and otherwise quite powerful technologies. But the good news is that these anti-patterns are, in the main, organizational problems that, once addressed, can unlock computing power and generate significant return on investment. Let us take a closer look at each of these worst-practice anti-patterns and how to avoid or correct them.
1. Using Outdated “Averages” to Guide Marketing Science
The etymology of “average” goes back to 700 BC, but the adoption of its more “action-oriented” definition began in medieval and early modern Western merchant-marine law contracts. These contracts had to do with cases in which a merchant sailing ship encountered stormy weather and had to jettison some cargo to make the ship lighter and safer. In that circumstance, the contract specified, merchants with goods onboard the ship had to pay an equal share of the value of all the goods lost overboard — rather than being libel only for the cost of their own goods thrown overboard. From that point on, this concept of “averaging” was adopted by British insurers, creditors, and merchants and popularized the idea of spreading losses across whole portfolios of assets. Soon thereafter, the concept of averaging applied to all manner of things both quantitative and otherwise.
In marketing sciences, averages have been used not only to ease consumption of quantitative details, but also to solve for technology constraints. Since organizations did not yet have the technological and operational capabilities to analyze customers and their behaviors at an individual level, they used the concept of averages to simplify their understanding of the marketplace, and then build and execute marketing plans on that understanding.
However, the difference between theoretical and practical implications of averages have diverged extensively in the past decade. Averages have very high usefulness across domains, but they do not do a good job of telling us the “why” and the “so-what” of what companies do. With the advent of varying dynamics in individual preferences and behaviors, the traditional approach of creating marketing segments and reporting average performance metrics has become passé. Open-source advancements in technology, operations, and interpretation provide rich mechanisms to understand and act on the basis of individual — not average — preferences. This ability to mass customize has becomes one of the core differentiators for companies that build scalable, real-time product and personalization capabilities. Meanwhile, those companies and their PowerPoint slides depicting segments and averages are quickly slipping into irrelevance.
A decade ago, it would have been unheard of to base business planning on individual-level insights or personalization. It is now standard procedure among leading enterprises. With appropriate data modelling and amid a plethora of algorithms, the ability to cluster the elements of individual journeys can empower us so much more that the use of “averages” to drill down into behaviors and understand the “why” behind that customer’s actions, interests and intent. Understanding buyer motivation reveals those critical intervention points that enable us to build personalized offers and experiences.
2. Not Differentiating Between Customers and Their Journeys
A lot has been written about The Golden Circle, a theory that places great value on an organization’s ability to understanding the “why” behind what they do. The concept’s originator, Simon Sinek, encourages leaders to look at each problem with the same lens — that of the fundamental purpose of the organization. Once that lens has been defined, strategic objective can be formulated, followed by tactical planning to build and then execute that strategy.
This progression seems simple. The problem is that the simpler things get, the more difficult they often are to correctly execute. With respect to marketing sciences, the adoption of the Golden Circle gets even muddier as we cut across the circles. This is not only because of nuances created by complex technology landscapes, or because the existence of so many algorithms can paralyze enterprises with indecision. It is also because of our inability to clearly define exactly what it is we want to understand about each customer.
In the myriad of digital customer interactions and brand expositions, behaviors metamorphose into highly dynamic facets, leading to different personas. A customer making a purchase on a mobile handset will go through different exploration trajectories, interact with different layers of the business ecosystem and exhibit altogether different behaviors than another customer who buys via their television. The characteristics of these journeys continue to diverge as the product categories do the same. For example, the path to purchasing a mobile handset will vary significantly from that of securing a home loan. This variation is only magnified by the use of different platforms.
Standard marketing-segmentation approaches look at the “average behavior” of a customer. In doing so, they fail to capture the subtleties around journeys, instead yielding segments that look good on PowerPoint charts but are rarely actually used to design marketing programs. Undaunted, many organizations keep working with these “average” segments, shaping their marketing strategies and building new offerings around them. When the campaigns and experiments are run, the results may seem good. But as we drill down into the data, the improvements and the benefits begin to fade.
Since business KPIs are easier to interpret in terms of customers (for example, new customers acquired in the last 3 months or increase in average spend per customer), the problem definition and measurement have to be done at the customer level. But there is no singular “level” because, as customers exhibit different behaviors, these varied personas emerge. Since targeting is done to alter a specific behavior, the development and execution must be done at a granular level of behavior per persona.
As an example, for a use case that deals with predicting the propensity of a customer to buy a new product, one of the objectives could be to move the customer to make a purchase, thereby increasing the measurable KPIs of product holdings and revenue. The development of the predictive model to achieve this must be designed around the observed persona-behavior level. For instance, all customers who are sensitive to price and promos and have low product awareness but a very high digital affinity will be a part of one behavioral segment. The targeting must also be done with respect to the customer journeys. If the customers in a behavioral segment spend more than 20 minutes online and browse a new product category and have a potential lifetime value exceeding $20,000, then the best model might trigger an EDM that offers a 10% discount during weekend mornings when they are most likely to be browsing.
Theoretically, the three separate tasks of defining, building and executing on a predictive model may seem easy. But the missing linkage between these stages is not only one of the most common anti-patterns observed across enterprises, but also the one that can have the biggest impact on topline.
3. Not Moving from “Data strategy” to “Feature strategy”
The notion of “Big Data” surged around the beginning of last decade, with major publishers and influencers speaking about the need for organizations to get on this wave early and strong. Those who failed to do so, it was said, faced the prospect of long-term failure. This message was incessantly pushed across all touchpoints (albeit in varied conventions and formats) and led to many new analytics offerings including Hadoop, Spark, and Deep Learning. Analytics and technology services companies consulted to help companies assemble the right technology and analytics stacks, and to understand how to develop the “right” data strategy. This resulted in many companies dumping their data assets into a pool, creating never-ending block diagrams and connecting processes to showcase all the rich information they had. The effort could take anywhere from 6 months to several years as new roles were created, governance became a highly regulated process, and everyone became so excited about all the benefits to come. But they weren’t excited for long.
All the major companies today have already adopted Big Data analytics in some form or the other and set up a Data Lake as a part of their initial, foundational exercise. Few knew what to do with these large bodies of data, with a very limited portion of these enterprise data assets being used in any significant way. In the process, several roles were terminated, people either transferred to different divisions or simply became lost amid burgeoning roles and hierarchies.
And yet, even amid all this turmoil, small teams of data scientists at some companies kept working on their standard use cases, leveraging data from the data lake to generate “features” and “signals” to help them understand the “why” behind what they were doing. These companies that increasingly focused and built capabilities around a “feature strategy” saw their data lakes continually updated and leveraged, while other companies saw the value of their data assets dwindle away.
This concept of a “feature strategy” includes thinking about the “why” before dumping assets into the pool. This is a process of developing use cases that drive growth, and of analyzing the capabilities required to build engines to support these use cases. This, in turn, leads to the creation of specific features and data assets to fuel these engines. It also includes building and inferring customer journey properties and augmenting use cases with information beyond traditionally used averages and derivatives (rations and deltas). Over time, the top-down approach of feature strategy also provides a way for businesses to make the case for increases in “better data,” as opposed to simply more “big data.” In the process, it becomes possible to measure the benefit of individual data sources and fields and to track their adoption by different teams and functions.
For anyone seeking advancements in their analytics journeys, it is crucial to move away from data lakes to what might be referred to as continuously evolving “data rivers.” At the same time, now is the moment to move from a data strategy to a feature strategy that sheds new light on not just creation, but consumption.
4. Missing a “North Star” in Experiment Design
Significant advancements have been made in experiment design and Omni-orchestration that reach across multiple journeys, devices and touchpoints. But very little has been done to optimize what to measure or what “implicit feedback” means in the design process. There are two key factors adding to the confusion. First is observer bias and self-serving bias in experiment design, wherein companies shape their experiments to align with an expected outcome. An example of this might be the way an automobile company chooses features, build models and designs audiences to sell a familiar design. The second factor is the mindset of data science teams that prompts them to maximize the lift over baseline but not necessarily minimize the difference between the best that is possible and where they are at that moment in time.
Traditional experiment-design methods have been devised and leveraged solely as “comparative frameworks” to understand the incremental value (A versus B), and not differential value (“what is the most I can make from an experiment”). For example, if a campaign shows 300% improvement over baseline, does it mean the campaign has done well? Definitely. But does it also mean that the business done as well as it could? Maybe. We are better off maximizing the benefit over what we can see versus what we cannot see. In a healthy experiment design, getting a lift over baseline (or, control or holdout) is the basic measure of success. But for the purposes of acquiring feedback and enabling continuous improvement, it is critical to have a gold standard — a North Star that guides your experiment design.
The fact is that, when it comes to possible improvement, a lot is often left on the table. Two related thought experiments are gaining traction in the industry. Both tend to create a paradigm shift in how experiments are approached. The first experiment is the idea of reinforcement learning. This is a more technical idea that has an established process and strives to balance exploration and exploitation to minimize, over time, the differential between “the best a design can do” and the “current state of the design.”
The second thought experiment requires a change in mindset, pushing teams to think of the performance of their experiments with respect to this clearly defined, aggressive notion of the “North Star” goal. In this framework, even if a predictive model yields good market results (that 300% over the baseline model), the experiment will be deemed an approximate failure if the delta relative to the North Star is too large.
This approach to measuring success is, of course, highly debatable. But it does serve to enforce appropriate focus on the design and rationale for experimentation, thereby increasing the integration of business and sciences in large-scale marketing campaigns. If organizations need a good experiment-design framework, the mandate should be to build a strong scientific foundation, put rigor around the rationale for the definition of “North Star” within the organization, create a set of measurement levers around that North Star, and use cutting-edge practices such as reinforcement learning to achieve that gold standard.
5. Not Considering That a Fast Mover Often Trumps a First Mover
The ability to establish a trend within a product landscape has always been a much sought-after skill among both established and startup companies. To help companies grab the high ground, there are more than 7,000 established platforms and vendors in the Martech space alone (as indicated by chiefmartec in their 2019 report), and the list is expected to grow anywhere from 5–7% annually. Even though these service-provider offerings are growing and increasingly focused on gaining first-mover advantage, it is a different playground altogether for those seeking this service.
Marketing science is a process involving several components and some companies have set the foundation for designing and scaling them. But the key is not to aspire to be first mover, but to get the process right and learn to evolve quickly. Each trend in the market is like a living organism: It changes every day, adapting to its ecosystem and modifying itself to stay relevant.
Customer 360, for example, was on the receiving end of the hype in the mid-2000s and, to this day, remains for many a must-have. Even so, companies both big and small continue to struggle with the tremendous breadth of this concept — and with correctly implementing it. Building, using, and continuously evolving a customer 360 is comprised of three separate, organically improving processes. Companies that stop once their 360 is built and in use — but neglect to continue evolving it to match an ever-changing customer landscape — soon likely find themselves lagging behind. More than building a data product itself, it is important to focus on speeding up its usability, reproducibility and usage. Emphasis must be on demystifying the offerings for all those involved, and on building a common vocabulary within the organization.
6. Not Predicting and Investing in Winners
Ask data science teams across organizations about the validation approaches they take when building machine learning products and solutions and they’ll probably start talking about techniques like cross-validation, RMSE, loss metrics and so on. Very rarely will they emphasize business metrics such as profit, reduction in cost to acquire new customers or digital transition rates. This focus on technology is in not ubiquitous, nor is it technically wrong. It does, however, raise the issue of optimization-driven development. Companies are dedicating a great deal of resources to automate the process of model development and insight generation, achieving this by parallelizing the crude effort required for steps such as model selection and feature engineering. But once the model is built, very rarely do they put additional effort into optimizing the “consumption” of the model.
To illustrate this point, let’s say that a company builds five different propensity models, one for each of the five products they sell. The goal is to identify the next best product for their customers, and to end up with similar propensities across all products. In most of our observations, people tend to use plain heuristics and rule-based mechanisms to identify the “best 1 out of 5” products that should be recommended. Rarely do data sciences team apply the lens of business optimization over these raw propensities to identify this “best” offering. If, instead, they took the raw likelihood of purchase and leveraged information about operational costs and margin associated with each product, they could recommend the product that would best optimize lifetime value or profitability.
Similar conditioning is observed in the construction of marketing campaigns. Since the cost of marketing is very low, the thinking goes, there is no need to bother with optimizing the targeting mechanism. Companies either target the masses through a series of drip-marketing tactics, or use a model heuristic (top 1 or 2 deciles) to run experiments and measure performance and benefits. Frankly, such approaches are lazy and inefficient, leading to content fatigue (too many touchpoints), poor benchmarks (too many or too few people) and the rewarding of inappropriate audience segments. Propensities without optimization (for model, audience, offerings, etc.) is a grave mistake for organizations that envision building large-scale data products.
7. Getting Inclusivity Wrong
In India, there is a concept of Vasudhaiva Kutumbakam (originally from Sanskrit: वसुधैव कुटुम्बकम्). This phrase appears in Maha Upanishad, a foundational Hindu text. It consists of several words: “vasudhā,” or earth; “ēva,” or indeed; and “kutumbakam,” or family. The phrase literally means “the world is one family” and has several interpretations across Indian heritage and beyond. It also has relevance in the world of business, in the context of enterprise-analytics setups that host several teams, functions and roles. In this context, it applies to the entire process from creation to consumption of analytics.
Over the last decade, there has been a sharp rise in the amount of literature and products expounding on the importance of the collaborative, automated and cross-functional development of analytics. This approach was first popularized as “agile development.” Now we witness the rise of automated machine learning. At the same time, self-service BI/analytics platforms have come a long way. And everyone is debating, misusing, or confusing everyone else about the differences (or, lack of) between Machine Learning and Artificial Intelligence. Technologists have started strengthening their claims (and businesses have started believing them) that large-scale problems, whether well-defined or not, can be solved simply by incorporating the latest and hottest solution or platform.
The kernel of value, however, resides with enabling the consumption — and not just the delivery — of the insights derived from these tools. A team’s ability to consume this information is only as good as its ability to facilitate the exchange of information. From an external point of view, this seems like a minor problem. Internally, people tend to debate that the gap between a fantastic creation and mediocre consumption is simply due to an interstice within the organization. What these same people fail to realize is how deep this interstice really is, believing that it is the result of technology gaps rather than design fallacies.
This all leads us back to Vasudhaiva Kutumbakam and the topic of inclusivity. Companies that excel at innovation inculcate the practice of “analytics inclusion,” defined as the inclusion of processes, people, and collaboration that drives better information visibility across all stages of development. In such a paradigm, the data scientist is kept in loop about the broader vision and business outcomes, and senior management is kept in the loop about the data and processes. One might doubt that senior executives having anything significant to add to questions of data or models. But the purpose of inclusivity is not necessarily to problem solve, but to extend visibility across all layers. It doesn’t always mean more people must be involved across processes, but it does always tend to open new communication bridges and create environments in which people are able to ask more and better questions, identify the owners and the hits and misfits, and lead to designs that represent the objectives of key people across the organization.
Solving Anti-Patterns May Not be Sexy, But It Delivers Results
Building data products and large-scale experimentation workbenches is a difficult but clearly solvable task — if people are equipped with the correct toolkits and aware of worst practices and how to avoid or correct them. They must learn to step away from the concept of averages, and leverage new data capabilities to work at the level of individual customers. They must learn to focus on the behavior of customers as they move through the buying journey. They must learn to acquire not just more data, but better data — the right data. They must hold their experiment designs to a higher standard, learn to evolve quickly, optimize for long-term value and profitability, and think clearly how proper team composition is intrinsic to overall success.
Moving away from these anti-patterns requires a change in mind-set and culture, and it can take a lot of effort to alter long-standing corporate cultures. The changes may not be as “sexy” as some of the things that come up in Google searches or are touted in the current analytics-space market and, as a result, may be met with resistance by people who are swayed by the latest shiny object. Nonetheless, with appropriate, consistent top-down messaging within the organization, the successful implementation of these changes invariably becomes the core differentiating element. Fixing these anti-patterns enables large-scale data and analytics integration in the company. In this day and age, that must be a guiding bottom line.
