Chief Question Officers

“Chief Question Officer” is the unofficial role of many great product, design, and engineering leaders. 5 tips for asking better questions.

The best questions foster rigor, encourage focus, and teach instincts. Below are some favorites to try using when reviewing proposals for product plans or business strategy.

Adapted from this Tweet storm:

“What is the fastest path to learning?”

The biggest determinant to long-term product velocity is the pace of learning.
Learning is broader than just A/B experiments. How quickly are you developing new insights about customer needs and pain points?

Of course you need some mix of “earning” launches, ones you have high confidence will be a quick win for customers. But the “learning” launches are the ones that unlock future trajectory bending.

Judge PMs in the short-term on their pace of learning, and long-term on their impact.

Organizations that do the reverse perverse incentives towards short-term hill climbing.

“Are there any cheat codes?”

Put another way, in the classic scope vs. time vs. quality tradeoff: are there scope cuts we can make to speed up time while preserving quality?

There are many great types of product shipping cheats:

• Avoiding premature v1.1 polish
• Making scalable for only a subset of customers
• Smart defaults with fewer advanced settings
• Heuristics instead of complex algorithms
• Human curation to bootstrap data

In the early days of a product, the best framing on shortcuts is Paul Graham’s “Do things that don’t scale”.

Don’t view launch cheat codes just as “lean startup” pragmatism. Reducing feature scope to optimize for learning is an act of product humility. Even the best experiment driven teams only generate wins on <40% of launches, after-all.

“Is there a 2-way door?”

If the decision is, optimize for decision making speed; if it’s not, optimize for decision making quality. Borrowing from @JeffBezos:

The reality is the vast majority of decisions are two-way doors — as long as your org has the right post-launch sensors and is great at error recovery.
Error prevention is costly and time consuming. Use it only when absolutely necessary.

The ladder up move: figure out how to decompose a monolithic, seemingly 1-way door product decision into a sequence of reversible, stackable launches.

“Is this the right investment mix?”

Great product leads often take an investment approach and look at feature funding as portfolio allocation problem. It’s important to make sure teams are balancing the right mix of feature maturity bets.

• Seed = speculative bet with outsized upside. These could be entirely new product capabilities, swings at step-function changes in the growth funnel, features that can create new network effects, etc.
• Series A = validated customer demand ready to scale. These are often experiments or limited releases that looked promising; now it’s time to scale up to 100% and see what the full impact can be.
• Series B = proven, scaled feature needing 1.x features. The customer-feature fit is fully validated and a meaningful driver of the business. Now it’s time to relentlessly refine and improve the quality of the experience.
• … Series G = big company working on v10 of a plateaued growth product line that still has meaningful revenue. This is unlikely a place to do great product work.

A related great read on product investment types is Adam Nash’s post on 3 feature buckets: metric movers, customer requests, and and customer delight:

“What are the anti-goals?”

The goals are often boringly obvious; what you deliberately chose to not prioritize is more interesting and can force harder debates.

Good anti-goals range from metrics to use cases to customer types that you want to intentionally avoid prioritizing/moving.

An example: when improving Twitter’s timeline ranking, obvious goals might be tweet engagement rates and WAUs. A non-obvious anti-goal might be sessions/DAUs, which very well could decrease.

Questions > Answers

Recap of helpful questions to ask when reviewing product proposals:

• What is the fastest path to learning?
• Are there any cheat codes?
• Is there a 2-way door?
• Is this the right investment mix?
• What are the anti-goals?

Bonus: it’s easier to scale yourself with questions, and more fun for teams when they create the answers.

More where this came from

This story is published in Noteworthy, where thousands come every day to learn about the people & ideas shaping the products we love.

Follow our publication to see more product & design stories featured by the Journal team.