Xpanse Inc
Published in

Xpanse Inc

Graig Nettles and his stats

What Baseball and Search Algorithms Can Teach Us About Product Research at Startups

Questions to Answer About Research at Startups

  1. How do you incentivize your leadership team to split focus on the immediate and discover the potential future?
  2. How do you measure the success of something so fuzzy and indeterminate?
  3. How do you make product research a relevant contributor to business growth?

The Prime Directive: Focus

A Necessary Luxury

Big Research Prepared Us for the Future

Under the transistor-inspired water tower at Bell Labs in my hometown
Under the transistor-inspired water tower at Bell Labs in my hometown

Research Can Teach Us Whether to Swing or Not

  1. Swing at good pitches — best probability to get on base
  2. Don’t swing at bad pitches — good probability to get on base
  3. Swing at bad pitches — low probability to get on base
  4. Don’t swing at good pitches — zero probability to get on base

Measuring Product Research Results Like Web Search Results

  1. Swing at good pitch: true positive, good precision
  2. Don’t swing at bad pitch: true negative, good recall
  3. Swing at a bad pitch: false positive, bad precision
  4. Don’t swing at a good pitch: false negative, bad recall

The Goal: Be Relevant to the Roadmap

A Product Research Framework For Startups

A product research framework for startups
  1. Create the incentive structure. Our leaders have goals to identify good pitches and bad ones — and to swing at the right ones. Check out my recommended metrics, below.
  2. Harvest ideas backlog. Hold Big Ideas Sessions with your team members every two weeks. Gather, ideate, and build your backlog with a rough idea of priority based on cost to payoff potential. Creating a forum for this allows people to get things out of their heads into a holding place for later. Parking your ideas allows you to get back and focus on your MVP.
  3. Investigate based on priority. Ask a business analyst and researcher or product manager to spend one day to look at the opportunity of the idea in light of what the market can teach us. I call this “delegating research to the market.” The economics of the broader market can inform you where to invest bigger in long-term bets. This investigation is scheduled in a sprint. It has acceptance criteria, success measures, and a deliverable. The investigating duo makes a recommendation. Leadership decides whether to advance or kill the idea.
  4. The research backlog is where deeper research investment decisions are made. Each research project is swagged and prioritized based on cost and potential for payoff.
  5. A research team of three is assembled to evaluate a core hypothesis related to the idea. Typically, this is a researcher, a design technologist, and an engineer. The researcher does the core job of evaluating the big idea and works with their team to develop a product prototype. Like the investigation effort, this is done in sprint fashion. They plan, execute, deliver, and iterate for a pre-committed number of sprints, based on budget and progress made.
  6. The pitch — the results of the research is the moment when the idea becomes a pitch, or not. The research team should clearly state if this is worthy of the team taking a swing at, including a recommendation of where it might land on the roadmap. Leadership reviews and approves.
  7. Roadmap! Or not.

Product Research Metrics that Connect to the Incentives

Drive these ↗

Drive these ↘



Xpanse Inc is building the modern OS for the greater mortgage industry

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Dave Thomas

Engineering and Consumer Platform Leader @Xpanse; Product design, technology platform strategy, and ground fighting geek in Seattle