The Power of Outcomes

Develop effective and maybe surprising solutions by focusing on measurable behavior changes

In product organizations, like in any other, there are as many opinions as people. Especially when facing strategic decisions, managing and aligning those opinions and ideas, and keeping people from jumping to conclusions and solutions prematurely quickly becomes a challenge of its own.

In this article, you’ll learn how shifting your focus to outcomes enables you and your organization to approach challenges in new ways, align on what matters, and discover solutions that might surprise you.

Key Takeaways

• The terminology of outputs, outcomes, and impact was widely popularized in the context of product development by Josh Seiden and Jeff Gothelf.
• It emerged from the concept of program logic models utilized in the social impact sector for developing and evaluating programs of interventions to address complex systemic problems.
• Outcomes are defined as desired measurable behavior changes of system actors that lead up to the desired impact.
• Mapping backward from the impact or the system changes you aim to produce helps you to discover the outcomes relevant to solving your challenge.
• Such a map, or theory of change, allows you to structure your challenge and align your activities and outputs produced toward the results that matter most.

This article builds upon and reiterates ideas and concepts discussed in the previous piece:

Thinking In Circles
Why Stephen R. Covey’s concept of concern, influence, and control serves greatly as a fundament for a product mindset

Action Aiming For Influence

Lately, I rather accidentally stumbled upon the interesting story of how the Dutch e-bike brand Van Moof smartly tackled an unexpected business challenge: When they started delivering their products to the US market in 2015, a lot of their electric bicycles got damaged during transport to their customers. This resulted in high costs due to return shipments and replacements, and, of course, a bad customer experience.

Confronted with a problem that significantly impacted its business in a new market — its current circle of concern — the company quickly realized that the problem was rooted way outside of its circle of control and, based on its knowledge, identified the most probable point of failure.

We couldn’t say for certain, but US handlers didn’t seem to take as much care as we’d hoped.

Their first approaches to solving the problem were rather reactive. They considered the causal behavior as given and unchangeable and tried to deal with the symptoms instead of addressing the issue. So, what do you do when your shipping company is not treating your delivery with the necessary care?

Leaving delivery to others seemed like a fair deal. But now we had to think of ways to influence that final step. Tougher boxes? Better packaging? Different shipping partners? Nothing worked.

But in their ongoing search for a solution to their business problem, they finally made a small observation that led to a critical change in perspective, shifting their focus from their circle of control to their circle of influence.

Bikes obviously didn’t have the kind of priority flat-screen TVs have for example… And that was it. The lightbulb moment.

It became clear to them that the root cause of their business problem actually wasn’t a static and unchangeable property of the system. It was simply the behavior of humans who, in a stressful environment, adapted the amount of care with which they handled a delivery according to a superficial judgment of its value and fragility. With this improved understanding of the causal behavior, the question to be answered changed from ”How do we protect our products from being damaged?” to “How do we make our logistic partners perceive our packages as more valuable and fragile, for example, like flat-screen TVs?”

Van Moof shifted their focus from their circle of control to their circle of influence by starting to think about how to change the behaviors that actually cause their issue.

As a result, they were able to derive an astoundingly simple and cheap solution:

Our boxes are about the same size as a really big, expensive, flat-screen television. So we put an image of one on every box. We assumed handlers would care a little more about that. And we were right.

The changed packaging design changed the behavior of parcel carriers and drove down return costs. Source: https://www.vanmoof.com/blog/en/tv-bike-box

With their solution, they hooked right into the mechanics that controlled the behavior of the people dealing with their packages on their way to the customer. A simple change of the perceived value and fragility by adapting the print on their packaging led to a behavioral change of the relevant system actors and directly and drastically affected their business results of concern.

That small tweak had an outsized impact. Overnight our shipping damages dropped by 70–80%.

On Outputs, Outcomes, And Impact

The story of Van Moof’s ingeniously simple solution is not only a great example of creative problem-solving. It is also a beautiful case study that practically illustrates the meaning of three terms that have in recent years become an inherent part of product management and design lingo: output, outcome, and impact.

The terminology was popularized in the product world by Josh Seiden and Jeff Gothelf thanks to their books “Lean UX” (co-authored by both, first published in 2012) and “Outcomes Over Outputs” (Seiden, published in 2019). With the intention to support team communication and alignment they introduced and defined the three terms as follows:

• Outputs: the things we build and deliver like products, features, or services.
• Outcomes: the measurable changes in human behavior which are caused by our outputs, and which drive business results.
• Impact: the measurable change of business results driven by our outcomes.

To better understand their individual meaning, let’s locate them in our mental model of Covey’s circles. Outputs are the things we do or create, our activities, products, and services. Because we are the acting entity producing those outputs, we can confidently consider them part of our circle of control. In contrast, impact as a change in business results reflects changes at the system level, in this case: the business. Creating these changes lies outside of our spheres of direct control and influence. For product people and most other roles, it’s the business that eventually forms their circle of concern. Consequently, this is where they’d want the impact of their work to manifest, e.g. as an increase in revenue. Finally, let’s look at outcomes: As some kind of in-between, Seiden and Gothelf define them as “changes in human behavior”. Those humans, whose behavior we aim to change, are system actors we aim to influence through our outputs in order to realize the desired impact. In the context of a business, such system actors could be, for example, customers, clients, suppliers, or regulators. The changes in behavior we aim to produce as direct results of our actions, the outcomes, reside in our circle of influence. They are the causal link between what we do and the results we want to ultimately achieve. While Covey’s circles map out the terrain, you can consider impacts, outcomes, and outputs as specific locations or destinations on this map along which you plan your path forward toward your intended results.

Looking back at the Van Moof story, let’s look at the impact, or measurable business results, the company was striving for: As the damaged bikes caused costs for returns and a lot of customer complaints, we can quite confidently assume that the measurable change they were after was a reduction of returns and customer complaints due to damaged deliveries. How did they achieve their goal? They changed “human behavior”, namely the behavior of the people handling their deliveries who they got to handle Van Moof deliveries with more care. This outcome was a direct result of Van Moof adapting its packaging design. The new design is a direct output controlled and created by the company.

Theories Of Change

In Seiden and Gothelf’s definitions of outcome and impact, one word plays a central role: change. The emphasis on — ideally measurable — change in those definitions came along with the terminology the two authors borrowed from a context where change is at the center of everything: social interventions and welfare programs. Seiden refers in “Outcomes Over Output” (page 15) to Program Logic Models applied in the social impact sector. Such Program Logic Models as described and utilized by the W.K. Kellog Foundation serve the purpose of developing a systemic view of a problem and its often diverse reasons, identifying target areas for interventions, planning those, and evaluating their results. The elements such a model is composed of are:

• Inputs: the resources a program has available to direct toward performing their activities like people, funding, organizational or community resources.
• Activities: the different intentional activities the program performs utilizing its resources like actions, processes, or events.
• Outputs: the direct products of the program’s activities like offerings and services delivered.
• Outcomes: the changes in behavior, attitudes, knowledge, or skills in the program’s target audience, short-term and medium-term.
• Impact: the intended or unintended fundamental and long-term changes occurring in the community, organization, or system the target audience is a part of as a result of program activities.

The structure of a Program Logic Model through the lens of Covey’s circles of concern, influence, and control.

Program Logic Models build upon ideas and concepts put forward by the Roundtable on Comprehensive Community Initiatives for Families and Children at the Aspen Insitute in the 1990s, and are a tool to help organizations to develop their program theories. Similar to a logic tree, a program theory or “theory of change” is a structured formulation of the logical hierarchy of all the causes leading to a problem within a system and how interventions through their intermediate outcomes are expected to create a long-lasting impact on the issue. Such an issue could be, for example, a high rate of school dropouts in a local community. Designing a program to decrease the number of dropouts would require the program designers to theorize about the influencing factors and behaviors within this community that are causal to students dropping out and, from those, derive the changes that need to be realized in order to reach the program goals. This approach not only surfaces the program designers’ explicit and implicit assumptions about how the program will produce the expected results but additionally suggests already concrete points of measurement, so-called outcome measures or indicators, which will allow monitoring progress toward the program’s objectives along the chain of outcomes over time.

A key person behind the concept of program theories was Carol H. Weiss who had been researching ways to scientifically evaluate social interventions and community programs since the 1970s. Because of the complexity of social systems, deciding on the right interventions and evaluating their effects (and potential side-effects) is inherently difficult but at the same time crucial given limited resources like funding. She introduced the idea of “theory-based evaluation,” and in her contribution “Nothing as Practical as Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families” to the publication New Approaches To Evaluation Community Initiatives, published 1995 by the Aspen Institute stated (page 66):

[…] I suggest an alternative mode of evaluation, theory-based evaluation. In lieu of standard evaluation methods, I advance the idea of basing evaluation on the “theories of change” that underlie the initiatives.

[…] The concept of grounding evaluation in theories of change takes for granted that social programs are based on explicit or implicit theories about how and why the program will work.

Weiss correctly points out that any design of a program initially is just a “theory” of how a system works and how the desired change in this system is meant to be achieved. While Seiden and Gothelf don’t speak about theories of change explicitly, they cover the theorizing in a process of surfacing and collecting assumptions about influencing factors for a business result, and forming narrowly scoped testable hypotheses. An example hypothesis based on our Van Moof example could look like this:

We believe that if we change our logistic partners’ employees’ first sight perception of the value and fragility of the content of our packages,
this will lead to an increase in care they handle our deliveries with.

We’ll know this is true when we see the ratios of complaints and returns due to damaged deliveries to the total number of deliveries go down significantly.

As the example illustrates, in such a hypothesis as you might already be familiar with from the Lean Startup method or hypothesis-driven development, you’ll find the two key aspects of Weiss’ suggestion combined:

• The explicit formulation of the “theory”, also called assumption or belief, of how an action (output) is expected to cause a specific change in behavior (outcome) that will create the desired change in a business result (impact)
• The basis for “evaluation” or, as Seiden calls it, evidence that will confirm or falsify the hypothesis

While at first look it might seem a little counter-intuitive to lean on something taken from the world of non-profits to guide business decisions, it actually makes a lot of sense: Social or environmental impact organizations usually need to operate with very limited resources in areas of inherent complexity, and, due to their permanent dependency on external funding, continuously need to provide proof of their effectiveness. It doesn’t come as a surprise that we saw such practices and tools emerge from this sector that allows consciously constructing a systemic view of a problem space, theorizing about its structure, influencing factors, and interdependencies, and developing measurable solution hypotheses as theories of change to guide the effective allocation of resources. As Peter F. Drucker, one of the pioneers of modern management, already noticed in his 1989 Harvard Business Review article “What Business Can Learn from Non-profits”:

Our nonprofit organizations — are becoming America’s management leaders. In two areas, strategy and the effectiveness of the board, they are practicing what most American businesses only preach. […]

Starting with the mission and its requirements may be the first lesson business can learn from successful nonprofits. It focuses the organization on action. It defines the specific strategies needed to attain the crucial goals. It creates a disciplined organization.

Outcomes In Product And Beyond

The Van Moof example demonstrates that making a product successful requires a holistic perspective that takes system actor behavior all along the value creation and delivery process into account. Adopting an outcome-oriented mindset in product development, or better business in general, and starting to think in theories of change allows you to approach and solve challenges from such a systemic point of view. Working backward from the results, the impact on the system you want to realize, with the focus on your circle of influence rather than your circle of control, will lead you to discover sometimes surprising solutions. By postponing the question of what is to be done in favor of reaching a clear understanding of what change is to be achieved, you stay open-minded while you explore possible solutions to your problem. Because yet another product feature might not be the best response to your next challenge.

Outputs, outcomes, and impact as concepts and shared vocabulary can help an organization to better communicate and align on goals and guide its members’ actions. To put some more emphasis on the systemic view and make it easier to use the terms in other contexts throughout and beyond an organization, I settled for myself on slightly adapted definitions of the three terms:

• Outputs: the behaviors, actions, and products performed, created, and controlled by us as an actor in a given system context.
• Outcomes: the measurable changes in other system actors’ behavior in the given system context as a result of our outputs.
• Impact: the measurable change to the system behavior in the given context induced by our outcomes.

In an ideal setting, as a product manager, your circle of concern is clearly outlined by a set of relevant business results you are meant to drive or support with your team. Goals here are usually set in the form of explicit and measurable changes in one or more of these business results, for example in the form of organization-level OKRs. These desired changes in system behavior are what we got to know as the impact.

Examples:

• Increase product revenue in the next 6 months (+ x %).
• Double user numbers till the end of the year (+ 100 %).

To derive meaningful actions in support of the expected impact, we map backward from it to our circle of influence and construct our theory of change by thinking about what behaviors of which system actors, like product users, buyers, regulators, or partners, affect the business results we want to see change. Customer feedback or analytics is a great source here for discovering such behaviors. In this process, we explore, evaluate, and prioritize the possible outcomes our team could focus on.

Examples:

• Have fewer users drop out in the sign-up flow (completion rate + y %).
• Have more users convert to a paid plan (conversion rate + z %).

Finally, the product team moves its focus to its circle of control, generates ideas of things it could build, and formulates assumptions about how these outputs would produce the desired outcomes. Rephrasing the desired outcomes as How Might We (HMW) questions (e.g. How might we have more users convert to a paid plan?) can help here to keep the ideation process directed toward the outcomes throughout the process.

Example:

How might we reduce the number of users who drop out of the sign-up flow?

• Simplify the sign-up process by reducing the mandatory data to input.
• Introduce a free trial of the paid plan if users enter their payment data.

From those ideas and the data backing them, you can form a prioritized list of testable hypotheses that you can then start acting on.

Example:

Because we see ~20% of users drop out of the sign-up flow when filling out the quite extensive sign-up form,
we believe that if we reduce the amound of mandatory fields and move the non-mandatory fields to a separate and skippable page
this will lead to an increase in users successfully completing the sign-up.

We’ll know this is true when we see at least 10% more users complete the sign-up successfully.

Whether the action you take based on your hypothesis is user research to increase your confidence in a hypothesis, an experiment to cheaply confirm your assumption, or the actual implementation of the intervention you lined out, there is a permanent clear link to a defined behavior change as the desired outcome of your actions as well as a defined way to measure and confirm your action’s effectiveness.

This continuous thread connecting each initiative to high-level business results is what establishes an outcome-focused organization and ideally is not limited to product. It is also this close connection that makes the effectiveness of the organization heavily dependent on the quality of goals set by its management, as well as the management’s capability to facilitate a cross-department process of developing a shared theory of change based on these goals. Once established such a theory can serve as a powerful tool for aligning, orchestrating, and monitoring initiatives across the whole organization.

Like program designers have logic models, the product manager’s toolbox also offers a set of methods to help teams to construct their own theories of change in structured and testable ways, and quickly get from theory to action. We’ll take a look at some of these methods separately in one of the upcoming articles.

Measuring Outcomes

Carol H. Weiss came up with the concept of program theories to provide a way of evaluating the effectiveness of social impact-focused initiatives like community programs. That’s why measurability is a crucial element of theorizing about change as it makes the theory testable and allows us to observe the influence of interventions on the system over time. In the high-level examples above, impact and outcomes both rely on quantitative measures to make the expected change explicit. Thinking change in numbers from the start when developing your theory of change that will guide your next initiatives is extremely helpful. Measurable behaviors ensure that your theory builds on an observable reality, and force you to be specific about your assumptions on how your outcomes are meant to produce the desired impact. In the next article, we will have a closer look at how metrics and indicators are critical for spinning the thread between impacts and outcomes and serve as a starting point for ideation and solution discovery.

(This article was originally published in Decoding Product.)