Applying the Scientific Method in Product
How experimentation and data-backed decisions can help our product deliver value.
Our good friend John Doe has been continuously improving his product, processes, and work relationships with each iteration at ProductNation. The whole team has gained significant knowledge and skills that help deliver value, boosting morale to its highest level since John joined the company! Stakeholders and management both agree that the team has matured and are comfortable with its current performance.
As a result, the Scrum Team is more engaged with their product. Everybody seeks new opportunities to collaborate and participate in ceremonies, with John taking the lead as new ideas are discussed and presented to Stakeholders.
Developers feel that the time is right to implement the newest technology trends to keep the product competitive and their skills up to date. John spends more time talking with customers, asking them about their experience and suggestions for improvement. Stakeholders are also very active, investigating competitors and reading industry forecasts.
As with any new endeavor, there is a learning curve. Stakeholders hold back on ideas they find risky while developers take a biased stance about what would deliver more value. As Product Owner, John intervenes by asking probing questions that will allow him to craft a Sprint Goal that maximizes value:
- How would our customers benefit from this idea?
- What risks are we running by developing or not developing this idea?
Surprisingly, nobody knows! The team agrees to investigate and reconvene to make an informed decision, which leaves the next Sprint shorthanded with spikes and maintenance tasks. It’s time to go to the drawing board!
Science
Based on his recent experience engaging with customers, John has developed a theory. Customers usually don’t know what they want until they receive something functional to try out. With this in mind, it’s up to the team to develop a value proposition and ask customers for feedback.
John is ready to take new challenges as a Product Owner. This time he’ll need to discover what to build! He is aware that Product Management is not just about launching great products. It’s about delivering a great experience that addresses a particular need in a feasible and viable way. He will have to perform some tests to determine if the proposed idea does deliver the expected value.
Merrian-Webster defines the scientific method as:
Principles and procedures for the systematic pursuit of knowledge involving the recognition and formulation of a problem, the collection of data through observation and experiment, and the formulation and testing of hypotheses
To begin experimenting, the team will have to make an assumption and elaborate a hypothesis that will then be proven or disproven via testing. The results can elaborate the idea via another hypothesis or change direction towards a different one.
Hypothesis
Merrian-Webster defines a hypothesis as:
An assumption or concession made for the sake of argument.
An interpretation of a practical situation or condition taken as the ground for action.
A tentative assumption made in order to draw out and test its logical or empirical consequences.
The antecedent clause of a conditional statement.
Jeff Gothelf and Josh Seiden elaborated a framework in Lean UX: Applying Lean Principles to Improve User Experience:
1. Declare assumptions
Start by declaring assumptions. This process will help to identify risks and prioritize them.
2. Create MVP
Create a Minimum Viable Product that addresses a customer need. This process converts the initial hypothesis into a functional product.
3. Run an experiment
Test the validity of the initial assumption via the MVP. Release it to customers and ask for feedback. Run in a controlled environment.
4. Feedback and research
Gather feedback, confirm or deny assumption, inspect and adapt, reiterate.
John champions the idea of a safe environment where the team can fail fast to gain the necessary experience to iterate towards success. In this scenario, concepts like MVPs, Prototypes, A/B tests, and Mockups are helpful to confirm or disregard hypotheses with no customer impact.
John realizes that with every prototype, mockup, or test, his theory continues to prove itself. Customers usually don’t know what they want until they receive something functional to try out; At this point is when the magic happens. Behaviors are studied, use cases arise, and feedback begins to flow.
Way to go Mr. Doe! Keep on eliciting, iterating, and delighting customers!
Our other good friend Mary Jane is also excelling at WeAreProducts. After the engineering team’s Definition of Ready update and continuous improvement policy, she started reviewing all her processes, looking for optimization opportunities.
As Product Manager, she continuously speaks with internal departments, customers, and stakeholders, which sometimes becomes overwhelming. All the information, requests, and tasks she receives aim to influence her to make product decisions.
Mary knows that, even though there might be strong opinions or “gut feelings” towards a decision, she must always keep an objective mindset that puts customers first. If customer expectations are met, by consequence, all other OKRs will do as well.
- What could be the best way to remain unbiased and centered towards customer expectations without leaving out business viability and technical feasibility?
- Can it be used as a guide towards meeting our product vision?
- Can it also be used to assess complex situations and justify product decisions?
It’s time to put on your lab coat. We’re going to science the hell out of data!
Data oriented decisions
WeAreProducts requires all documentation to be available to team members and stakeholders. Mary has access to several documents and databases related to her products and services. Business-critical and customer-sensitive data is safeguarded by encryption and controlled access. Authorized personnel has limited access to certain sections only.
The engineering team proposed to store information in a data lake so that business intelligence could access and query it on demand. Mary relies on them and other departments when she needs data analysis. The majority of her requests are statistical reports about customer usage and infrastructure health.
This time Mary wants to dig a little deeper. She knows that proper analytical tools can deliver valuable insights. Customer behavior models, personalized marketing, and intelligent suggestions are defined using data. All of these can help her make appropriate product decisions.
Data science
Wikipedia defines data science as:
An inter-disciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from many structural and unstructured data.
The OSEMN framework
Dr. Cher Han Lau describes the OSEMN framework in his post 5 Steps of a Data Science Project Lifecycle
1. Obtain Data
Gather data from relevant data sources. Mary has a data lake ready to query!
2. Scrub Data
Pre-process data to a clean format that machines understand. Data lakes are repositories for storing data in its natural “raw” form. A pattern or data format will facilitate the easy processing of data.
3. Explore Data
Find significant patterns and trends using statistical methods. Roll up those lab coat arms; it’s hypothesis formulation time! Mary can ask a scientific question starting with an assumption. Do customers with canceled subscriptions renew more when offered a promotion than when new features are launched?
4. Model Data
Construct and train models to predict and forecast. Analyze how you answered your exploratory questions. If that particular context is interesting enough to explore further, then optimize your data to generate insights.
5. Interpret Data
Put the results into good use. Actionable insights are key outcomes in prescriptive and predictive analytics. With them, Mary can now more confidently attempt to repeat good outcomes and prevent negative ones.
Justified decisions
Mary is a great Product Manager. She can speak to customers, business stakeholders, and engineers with the same ease; This allows her to navigate through the three dimensions of product management, reach agreements and maximize value delivery. With data-backed decisions and constant customer feedback, she has positioned WeAreProducts in a privileged spot. We’re proud to have you onboard Madam!
Final thoughts
John and Mary are growing with their products! They apply the same empirical practices used in development to their careers, inspecting and adapting through every situation.
John is constantly learning and evolving. In recent Sprint Reviews, stakeholders openly share their ideas and ask for opinions before making a request. Requirement elicitation and prioritization keep John busy as he fosters everybody’s collective knowledge. With all of this input, he creates Sprint Goals that help the team focus on value delivery.
Mary takes new challenges and promotes a healthy environment for the team to thrive. Her customers receive value regularly, the business has steadily grown for the past year, and engineering is satisfied with its product’s maintainability, robustness, and performance.
Don’t stop now my friends, as there will always be more to achieve. I am confident new challenges and opportunities will come shortly. Keep it up!
