Symbiotic Prototyping: Minimizing Value Risk While the Engineering Team Is on Vacation

If we asked a room full of product managers what was keeping them up at night, the most common answer would be a version of “building product that doesn’t get used,” more concisely known as “Value Risk.”

In The Four Big Risks, Marty Cagan calls out the major risk categories products face:

1. value risk (whether customers will buy it or users will chose it)

2. usability risk (whether users can figure out how to use it)

3. feasibility risk (whether our engineers can build out what we need with the time, skills, and technology we have)

4. business viability risk (whether this solution also works for the various aspects of our business)

Of the four risks, “value risk” is the most fundamental: does this product effectively solve a meaningful pain point for users? Usability, feasibility, and business viability risks are only relevant if we’re confident in the product’s value.

With value risk looming largest, identifying strategies to quickly assess and mitigate value risk is one of the most important jobs of product teams.

B2C Painted Door Tests

B2C brands often use Painted Door Tests to evaluate interest in a new capability. Painted Door Tests are designed to measure a user’s appetite for a feature without requiring that the feature be built.

For example, let’s imagine that a social network is considering adding a feature that shows which of your friends are discussing a topic a user recently posted about. The social network might add a “See Who’s Talking About This” button and measure the number of clicks the button receives. By adding a placeholder version of the button, the team can begin to assess whether there’s enough interest or value (proxied by clicks) to consider building the new feature without actually having to build out the functionality.

Once a feature passes the initial Painted Door Test, there’s still more work to do in evaluating the feature’s impact. For example, we would probably want to measure whether sessions including the “See Who’s Talking About This” button lead to long-term engagement. Still, the Painted Door Test provides a quick read on user interest.

Challenges of B2B Painted Door Tests

For B2B companies, Painted Door Tests are tricky to execute for a couple of reasons:

• Smaller user base: while it’s not uncommon for B2C products to have thousands or millions of active users, B2B brands typically have fewer users. The smaller user base can make it hard to see clear directional results from Painted Door Tests.
• Disruptive to users: users often depend on B2B software to do their jobs and might be annoyed if they click on an exciting new feature that turns out to be a mirage.
• Draws attention to functionality gaps in the product: Painted Door Tests in B2B products can draw attention to missing functionality in a product that a user might be paying a high price tag for. This can be especially problematic if the feature offered by a Painted Door Test is not ultimately integrated into the product.
• Greater complexity in measuring business critical outcomes: In a B2C setting, clicking on a Painted Door feature is a signal that a feature is desired, but it still might not translate into a business critical result, like increased time in an app. The gap between expressing interest and desired long-term behavior is often larger and more complex for B2B products. For example, desired B2B outcomes might include behaviors like setting up a tracking event on a website, increasing ad spend, or setting up a new data feed. Because many of these B2B outcomes are preceded by a series of events that occur within the four walls of the client (budgeting changes, internal approvals, etc), the link between a Painted Door Test and its desired result can be challenging to measure.

With a little creativity, we can take the spirit of rapid validation from Painted Door Tests and apply it to a B2B setting. We’ll quickly create a prototype to get direct customer feedback and measure whether there’s interest in the feature and whether that interest translates to a business critical action.

Symbiotic Prototyping

Since Painted Door Tests can draw attention to functionality gaps, potentially create friction for B2B users, and suffer from small sample sizes, any attempt to run these tests in a B2B context must mitigate these obstacles.

Enter symbiotic prototyping, a validation technique designed for insights-focused B2B products.

All good relationships need an element of symbiosis, and the relationship between product managers and users is no different. Symbiotic prototyping creates value for the user in the form of business insight and creates value for the product manager by enabling the direct observation of product usage in a complex B2B setting.

During symbiotic prototyping, we take a new product idea, and prototype that idea with client data, resulting in an insight about the client’s business. We share that insight with a user in the course of normal business communication and work with them as they take the insight and turn it into an action.

During this process we’ll receive implicit and explicit feedback from the user. Since the feedback isn’t tied to an existing product feature, and instead is framed as a helpful insight shared in an email, responses are likely to be more candid and the product manager has the opportunity to change directions based on the feedback.

After working through the symbiotic prototyping process, we’ll have a read on whether the insights our potential product surfaces, provide value and drive impact for users.

Here are the steps to get a symbiotic prototype up and running:

1. Define the target outcome: what specific metric or action is the potential new product intended to drive?
2. Prototype: the “prototype” is a few sentences sharing the key insight on the client’s business and maybe a graph or two for illustrative purposes. The idea is to move quickly and clearly communicate information to the user, not to think about interactions in the product or to perfect the design. The design will become critical only if the potential product passes the value test.
3. Test: share the prototype with the user, keeping an ear to the ground for questions and feedback, and observe whether the user drives to the target outcome.
4. Iterate: based on both explicit and implicit feedback from the customer, iterate on the prototype and share with a few more users.

Let’s walk through a case study illustrating how we’d apply these four steps in a real world setting.

The Scenario

Let’s imagine that our product is an on-site analytics tool. We’re considering adding a feature that highlights the demographics of users contributing to the growth or decline of key actions on the website. We are prototyping the feature for a robo-advisor client.

Target Outcome

We’ve observed that the number of A/B tests set up on our on-site analytics platform is a strong indicator of client retention. The key outcome we’re optimizing for is whether the client sets up an A/B test based on the information we provide.

Prototype

Looking at the client’s account, we see that the “Increase Deposit” action on the website is an event that they are consistently analyzing and that the event was triggered 15% less frequently this week relative to last week. Furthermore, we notice that 75% of the decline resulted from users aged 25–35.

Test

Iterate

During the exchange with Carlos, it’s clear that we’ve hit on a meaningful customer pain point (he too is concerned that the “Increase Deposit” event is down) and our product hypothesis of highlighting which customer segments are responsible for the decline in events seems promising. However, Carlos alluded to a couple of potential product improvements:

• Highlighting whether the engagement decline from 25–35 year olds was due to fewer users in that age range coming to the website or whether they visited the website but were not engaging with the “Increase Deposit” action.
• Identifying website changes that coincided with the decline in the key event.

Based on Carlos’s feedback, we would reply with the data on session decline for the 25–35 age range and information on other changes that Carlos’ team had recently made to the website.

Most importantly, we would partner closely with Carlos as he root causes and addresses the decline in the “Increase Deposit” event by observing and asking questions to inform our product direction. Here are a couple areas we should pay special attention to:

• Does Carlos launch the A/B test? Why or why not? How could we have accelerated Carlos’ ability to launch the test?
• What information is Carlos hoping to understand after launching the A/B test?
• How does Carlos share information on the decline of the “Increase Deposit” event with his colleagues at work? How might we make sharing that information easier?
• What questions does Carlos ask us along the way?

After walking through the symbiotic prototyping process with Carlos, we would take our new and improved prototype based on Carlos’s feedback and repeat this same exercise with a few more customers, integrating their feedback with each cycle.

Once we’ve reached a place where clients consistently launch A/B tests based on the insights we’ve provided, we would move into the product specification and design process. During this phase, we’d determine how to surface the key customer segments driving changes in event frequency within our product, and go through another validation process focused on product usability.

If we aren’t ultimately able to consistently drive towards the A/B outcome, that’s also a success. The ability to quickly toss out product ideas that don’t provide value to the user is the mark of a great product manager.

Taking a step back, there are several benefits to the symbiotic prototyping process:

• Unbiased feedback: During the symbiotic prototyping process, the user receives a helpful insight from a product manager, but isn’t aware that it will impact product development. Breaking away from the constraints of particular product specifications, the user is able to provide feedback that is broader and more candid. Once a wireframe or specific product requirements are shared with the user, the conversation has a tendency to focus on the merits of the proposed solution.
• Encourages the product manager to develop products with client data in mind: Product managers often go down the path of building detailed product requirements without seeing how the product behaves with actual client data. Developing the broad strokes of a product solution and then quickly diving into the data helps the product manager move away from solutions that might not be informative or useful with real world data (due to data quality issues, sub-optimal granularity, data completeness challenges, etc).
• Opportunity for co-innovation: Symbiotic prototyping provides an opportunity for a product manager to embed themselves with a user and observe the process of identifying an insight and taking action on that insight. The product manager can experiment with new functionality based on feedback from the user and observe obstacles they encounter along the way. As a bonus, this hands-on collaboration almost always leads to a stronger relationship between the client and the vendor.
• Value-add for clients: Symbiotic prototyping provides the user with an insight that helps them succeed at their job. In the scenario with Carlos, the decline in the “Increase Deposit” event was a critical issue for the company. With our help, Carlos has a hypothesis that he can share with his team on which customer segments are contributing to that decline.
• Conserves engineering resources: Symbiotic prototyping doesn’t require any investment from the engineering team, allowing engineers to focus on building functionality that has been successfully validated across the four stages of product risk.

Next time you’re determining whether a product idea will deliver value for clients I’d encourage giving symbiotic prototyping a try. The symbiotic prototyping process provides high-quality feedback to product managers while strengthening client relationships and saving engineering time for product work where both the customer value and company value are clear.

Originally published at http://joycegordon.me on September 30, 2020.