How to Science the Product — Part 1
In this two-part series (See Part 2), I want to explain what data science means in the business context, why it matters, and how it should be practiced. In the first part, I make the case for “How to Science the Product,” an approach that uses science as a way to understand a domain and make decisions about product development. I discuss how this mentality works as an organization’s broader data-driven product culture and how to achieve it. In the second part, I discuss practical strategies for implementing product data science in organizations, including models of operation, recruitment, and training.¹
Introduction
In a world where “Data Science” and “AI” have become buzzwords, they’re often heralded as the cure-all for organizational woes. However, the reality is far more nuanced. While data science isn’t a silver bullet, its effective application can drastically enhance any organization’s outcomes.
Here, I redefine the scope of “science” by using it as a verb, an action — an approach to elevate products and services within an organization. When I say “science the product,” I’m not referring to science as a noun but rather as a dynamic, ever-evolving practice. It’s about integrating the scientific method into the fabric of product development, design, and launch processes.
Science is the method of reliably and systematically acquiring actionable information about the world. Data science’s function in business is to guide decision-making by providing actionable information that reduces uncertainty. Science is the guarantee that such information is systematically acquired based on observed evidence; therefore, it is reliable and valid.
Why Do We Need Data Science?
In the realm of decision-making, the need for data science stems from a critical observation about how we approach choices within organizations. Broadly speaking, there are two primary approaches: the HiPPO method and the Data-Driven Decision Making approach.
Firstly, let’s tackle the HiPPO approach — the Highest Paid Person’s Opinion — a perilous mindset prevalent in many organizations. This approach relies heavily on the opinions and instincts of top-ranking individuals, often glorifying the decision-making prowess of visionaries like Steve Jobs or Elon Musk. However, what’s often overlooked is the survivorship bias inherent in this perspective. For every successful visionary, there are countless failures that remain unseen. As it turns out, human intuition isn’t a reliable predictor of successful ideas when put into practice. Studies in large tech companies consistently reveal a stark reality: only a fraction of ideas — roughly 1 in 3 — show a positive and statistically significant impact. This success rate dwindles further to 1 in 5 or 6 for well-optimized products like Google Search (Source). These statistics highlight the glaringly high chances of the HiPPO being wrong.
Contrasting this is the second approach: Data-Driven Decision Making. The difficulty of decision-making depends on the proportion of known to unknowns; data-driven decision-making aims to increase the proportion of known (to reduce uncertainty) by applying scientific tools and methods. Science aims to achieve that by decomposing the uncertainty into knowable and unknowable. This method revolves around using evidence — data — and employing statistical tools to draw conclusions based on real-world observations. These conclusions constitute what we reliably know and can ground our decisions on.
How Do We Bind Science and Product?
I use the term “product” expansively — encompassing not just physical entities but also the broader outcomes of an organization, whether tangible products or intangible services.
The core objective of applying the scientific method to the product is to enhance an organization’s outcomes in a sustainable manner. Embracing science as an actionable strategy offers a framework for studying, observing, experimenting, and testing ideas within the realm of your product — that leads to concrete actions to optimize product development and leveraging data-driven insights to inform decision-making. A data-driven scientific method isn’t just beneficial; it’s fundamental for any product’s long-term, sustainable success. This approach only works when it focuses on cultivating a broader data-driven product culture within an organization, empowering individuals to derive conclusions from observed data, thus fortifying their decisions with evidence-based reasoning. The real secret lies in infusing a scientific mindset into the very fabric of the organization’s DNA.
This is where the role of the data science team becomes pivotal. Their responsibility goes beyond crunching numbers; it’s about instilling this culture, enabling and empowering individuals across the organization to leverage data-driven insights in their decision-making processes. The objective is to create an environment where data isn’t just a tool for a select few but an integral part of every decision-making discussion, leading to a more robust, informed, and successful organization.
How To Create A Science-Oriented Data-Driven Product Culture
Creating a science-oriented, data-driven product culture isn’t just about adopting tools or hiring data scientists; it’s about embracing a fundamental shift in acquiring knowledge and making decisions. It’s an epistemological stance — a commitment to forming beliefs based on evidence and rigorous testing. This process happens in five stages²:
1. Humility: This culture values ideas based on evidence, not titles or personal biases. Testing ideas and capturing relevant evidence to substantiate claims becomes imperative. This requires humility in leadership to accept science offers more reliable answers than their gut feelings.
2. Capture and Accessibility of Relevant Data: Building infrastructure to capture and access pertinent data is critical for any evidence-based analysis. This data serves as the bedrock for informed decision-making.
3. Measurement and Control: Without measurement, improvement becomes elusive. Using captured evidence to measure what truly matters for the organization and controlling these measures through experimentation is pivotal.
4. Acceptance and Open-Mindedness: Cultivating a culture where the organization embraces the results of well-designed experiments and analyses, even if they challenge established norms or beliefs, especially among leadership, fosters growth.
5. Internalization and Knowledge Accumulation: Documenting insights, sharing knowledge, and fostering habits of accumulating deep insights about the product area are crucial. This leads to a fundamental understanding of the product or “the science of your product.”
Implementing these phases leads to a culture anchoring decisions in data and evidence, and demanding data to support claims or solutions becomes the norm.
In conclusion, while the principles of a science-oriented, data-driven product culture seem promising, the true challenge lies in data science leaders making this process seamless and accessible for the entire organization. The goal isn’t to create hurdles or obstacles that impede progress but to streamline these practices to accelerate decision-making and drive measurable returns without burdening the organization’s momentum. Leadership in data science plays a pivotal role in crafting an environment where the scientific approach becomes ingrained in the organization’s DNA. The true success of a data-driven culture doesn’t just rest on adopting these principles — it hinges on the ability of data science leaders to make this journey painless and fruitful for the entire organization.
Notes:
[1] I used an LLM for copyediting to improve the grammar and readability of the post.
[2] This view on creating a data-driven organizational culture is shaped and influenced by Ronny Kahavi’s work and presentations, including Trustworthy Online Controlled Experiments.