What Baseball and Search Algorithms Can Teach Us About Product Research at Startups

“We are trying to prove ourselves wrong as quickly as possible, because only in that way can we find progress.”

― Richard P. Feynman

Questions to Answer About Research at Startups

Long-term research has great potential to improve the success of startups. But it can be defocusing and an inefficient use of scarce resources.

1. How do you incentivize your leadership team to split focus on the immediate and discover the potential future?
2. How do you measure the success of something so fuzzy and indeterminate?
3. How do you make product research a relevant contributor to business growth?

The Prime Directive: Focus

A startup must focus on going from zero to one, finding product market fit, and funding growth through revenue. But without looking downfield, where are they heading next? This takes more than visioneering. It takes allocating your scarce resources to researching, prototyping, and testing possible next generation follow-ons. As any startup veteran knows (or ask Astro Teller of Google X), most of these ideas will fail.

A Necessary Luxury

Harnessing the creative potential of your startup team requires an ability to discover, filter, and go long on those ideas that might reap benefits over time. Many startups consider long-term thinking a luxury. If you plan to be in the game three years out, you should have a product research strategy that dovetails with your product roadmap.

Successful product research should drive relevant contributions to the roadmap by introducing work that is meaningfully connected to market growth or preventing work that is not likely to yield growth opportunities.

Big Research Prepared Us for the Future

I grew up about a half a mile from the iconic Bell Labs research facility in Holmdel, New Jersey. Nine Nobel Prizes and five Turing Awards went to Bell Labs scientists. The transistor, laser, solar cells, information theory, the C and C++ programming languages, UNIX (that’s the predecessor of Linux for the youngsters out there), source code control, fiber optic cable, proof of the Big Bang theory, and cellular networks were invented there. My dad led a research team developing semiconductors and microprocessors at Bell Labs 45 years ago.

Under the transistor-inspired water tower at Bell Labs in my hometown

Around the same time that Intel was working on the seminal 8008 processor, Bell Labs was developing their own microprocessors to help drive global communication networks. My dad’s team built the world’s first single-chip 32 bit microprocessor. They knew 8 bits would only take them so far in connecting the world’s telephone networks.

Bell Telephone had to fund big research at Bell Labs. Humans hadn’t figured out how to network at scale yet. To grow their commercial business to the future state of the world, they had to research how to build their next generation of products.

When we think of research, we think of monster budget programs like Xerox PARC, Google moonshot/X, or Bell Laboratories. These massive research organizations were funded by their parent companies with hopes of driving innovation that would directly contribute to product opportunity.

How do we right-size this idea to work at startups?

“Between the wish and the thing the world lies waiting.”

― Cormac McCarthy

Research Can Teach Us Whether to Swing or Not

Think of a baseball player standing at bat. He is measured by his ability to get on base: the more bases, the better. Assuming he has sound judgment on what a good pitch looks like, he has these options, from the best decision to the worst decision:

1. Swing at good pitches — best probability to get on base
2. Don’t swing at bad pitches — good probability to get on base
3. Swing at bad pitches — low probability to get on base
4. Don’t swing at good pitches — zero probability to get on base

His performance is best if he selects the right pitches to swing at and let the bad pitches pass him by. His long-term career will be defined by all the things he swung for as well as all the things he didn’t take a swing at. Life is like that. As people, we are defined by the result of the things we decided not to do as much as what we decided to do.

Similarly, the shape of your startup is significantly influenced by what you learned not to build. You don’t measure progress based on just what you did, because what you did was to the exclusion of all the things you decided not to do. Research can teach you both.

When your research team contributes something to your product roadmap that becomes commercially successful, that’s obviously a win. Your product team took a swing at a good pitch. More frequently though, you’ll discover a dead end in research, informing you to exclude that idea from your roadmap. Your product team shouldn’t swing at a bad pitch.

Measuring Product Research Results Like Web Search Results

Taking a page out of search algorithms, we want to shoot for relevance. In search, a result that has little to do with the intent is a failure (a false positive). Similarly, when you miss a relevant item in your results, that’s a failure too (a false negative). In machine learning, these are measured as precision and recall.

Connecting it back to our baseball player:

1. Swing at good pitch: true positive, good precision
2. Don’t swing at bad pitch: true negative, good recall
3. Swing at a bad pitch: false positive, bad precision
4. Don’t swing at a good pitch: false negative, bad recall

Successful product research should drive relevant contributions to the roadmap by introducing work that is meaningfully connected to market growth or preventing work that is not likely to yield growth opportunities.

Unsuccessful product research is just the opposite. It contributes product ideas that have insignificant chances for success (low precision) or misses opportunities entirely (low recall).

The Goal: Be Relevant to the Roadmap

By creating success metrics tied to precision and recall, you won’t make the classic mistake of only rewarding positives. We know from prospect theory and loss aversion that teams will avoid failure as defined by the incentive structures of your org’s performance goals. The incentive structure must define success like a search engine: be relevant. Reward both true positives and true negatives. They are the accumulated knowledge that builds the long term potential of your startup.

A Product Research Framework For Startups

A product research framework for startups

1. Create the incentive structure. Our leaders have goals to identify good pitches and bad ones — and to swing at the right ones. Check out my recommended metrics, below.
2. Harvest ideas backlog. Hold Big Ideas Sessions with your team members every two weeks. Gather, ideate, and build your backlog with a rough idea of priority based on cost to payoff potential. Creating a forum for this allows people to get things out of their heads into a holding place for later. Parking your ideas allows you to get back and focus on your MVP.
3. Investigate based on priority. Ask a business analyst and researcher or product manager to spend one day to look at the opportunity of the idea in light of what the market can teach us. I call this “delegating research to the market.” The economics of the broader market can inform you where to invest bigger in long-term bets. This investigation is scheduled in a sprint. It has acceptance criteria, success measures, and a deliverable. The investigating duo makes a recommendation. Leadership decides whether to advance or kill the idea.
4. The research backlog is where deeper research investment decisions are made. Each research project is swagged and prioritized based on cost and potential for payoff.
5. A research team of three is assembled to evaluate a core hypothesis related to the idea. Typically, this is a researcher, a design technologist, and an engineer. The researcher does the core job of evaluating the big idea and works with their team to develop a product prototype. Like the investigation effort, this is done in sprint fashion. They plan, execute, deliver, and iterate for a pre-committed number of sprints, based on budget and progress made.
6. The pitch — the results of the research is the moment when the idea becomes a pitch, or not. The research team should clearly state if this is worthy of the team taking a swing at, including a recommendation of where it might land on the roadmap. Leadership reviews and approves.
7. Roadmap! Or not.

Product Research Metrics that Connect to the Incentives

Drive these ↗

True positives : Relevant ideas rate = roadmap additions / all research topics / time

True negatives: Low relevance ideas killed rate = killed ideas / all research topics / time

Drive these ↘

False positives: Rate of low relevance ideas added to roadmap = unsuccessful roadmap additions / all research topics / time

False negatives: Rate of missed relevant ideas = (there’s a formula here, but it is borderline quixotic to measure)

Xpanse is building the modern OS for the greater mortgage industry that will connect and create value to borrowers, lenders, servicers, and providers. Conquering the last frontier of Fintech #BuiltInSeattle

We’re Hiring! Explore open roles via xpanse.com/careers