Pattern Language for Experiment Writing

A hypothesis for supporting an organizational culture of experimentation

Carl J. Rogers, July 2022

Introduction

As an organizational change agent, much of our work can be considered through the lens of conducting experiments. So it is helpful to have a way to help organizations structure their experiments. However not all organizations will have the readiness for experimentation, nor see change and transformation as experimentation. Unfortunately for those that do, there is a prevalent practice to pilot transformation experiments locally and then roll out widely across the organization without recognizing that each new wave of change is an experiment in itself. Such an approach has a high chance of failure. Yet even for organizations that understand that each group of people needs to craft their own experiments, it does not mean that each part of the organization needs to do so in isolation and absence of understanding of what has gone before.

This article explores the hypothesis that applying pattern language to experiments will result in greater appreciation of which past experiments are likely to be appropriate from across different groups within an organization, i.e. from one department to the next. This includes appreciation of the problem to be solved, the context they operate in, and the factors of a previous experiment that might be appropriately adapted. We will have confidence to proceed in considering this hypothesis as true when we have evidence of positive outcomes from this format being applied. This would be both within the author’s own organizational context, as well as examples from others who have read and applied this experiment. This is the basis of a future, evidence based case study of this approach being applied. This article presents the concepts, supporting theory, and illustrative examples.

Hypothesis Driven Development

The paragraph above is written in the style of Hypothesis Driven Development (HDD), which is set out by Barry O’Reilly in his article How to Implement Hypothesis Driven Development [1]:

We believe <this capability>
// applying pattern language to HDD

Will result in <this outcome>
// The context and factors of a prior experiment is considered before being repeated

We will have confidence to proceed when <we see a measurable signal>
// The pattern is being applied within my organization plus positive external feedback

Readers with familiarity in how agile teams articulate much of their work will recognize this structure as a progression from both User Stories and Behavior Driven Development. As with both these formats, we can extend the 3Cs template [2] drawn from Extreme Programming onto HDD. There are many other template formats for writing out an experiment, and a few common ways to amplify HDD are with additional questions, such as:

• Who will do what
• Who needs to change?
• How will we know if our hypothesis is true?
• How will we know if our hypothesis is false?
• When will we test?
• How can we amplify positive outcomes?
• How can we dampen negative outcomes?

Applying Pattern Language

Pattern Language is a term coined by Christopher Alexander in his 1977 book A Pattern Language[3]. This can be defined as a coherent set of patterns organized around a knowledge domain, in such a way as to capture many best practices of an approach, with the purpose of communicating wisdom and insight. Meszaros and Doble’s article A Pattern Language for Pattern Writing[4] provides in itself a running example of these best practices of pattern language writing. Their solution to the pattern Mandatory Elements Present illustrates the relationship of elements within a pattern. These elements can be minimally adapted to support the communication of an experiment within the wider context for which it was designed, as shown below.

This helps provide a useful framing for determining if there is any likely value in repeating an experiment:

• Are we collectively experiencing a similar problem as the one described in this hypothesis pattern?
• Are we operating within a similar context?
• Are there similar forces at play?

From this point onwards this article becomes its own running example as to applying a pattern language approach to HDD. This is summarized in figure 2 below.

As we are <this group of people>

The collective wisdom on change management is that everybody needs to be involved in the change for it to have any real chance of lasting or meaningful change. The beliefs and teachings of the Sociocracy movement supports this viewpoint. The principle of Equivalence[5] is to involve people in making and evolving decisions that affect them. The people who write the experiment are those that are affected by it.

This first section of the pattern then is for a collective group to be involved in writing the experiment to define who they are. This cannot be done by a select few on behalf of others. From an Integral theory[6] perspective this is operating from the lower left ‘We space’ of a collective internal perspective, not the projection of an ‘I’ individual onto a wider group of individuals on their behalf. Any attempt of this would likely be perceived collectively as an external policy or decree, in the ‘Its’ space, for which a barrier through resistance to change would emerge.

The Integral Model

As an organization builds a library of past experiments built as experiments, or indeed a public library of shared experiments across many organizations, it is valuable to understand and consider first who was motivated to run each experiment.

Experiencing <this problem>

This section describes the problem that is being experienced by the collective who are writing their experiment. There exists a wealth of writing around identifying the root cause over addressing symptoms. The 5 whys and fishbone exercises are two widely adopted approaches. The process of establishing the true root cause may in itself be an iterative, experiment based approach. Good facilitation may be required to support a consent-based approach of establishing what the root cause is. Sociocracy 3.0 has patterns to support groups to achieve this through the Describe Organizational Drivers[7] pattern, which helps provide structured questions to consider when the problem is articulated through an organizational motivation for change.

The Consent Decision-Making[8] pattern provides a constructive process to achieve consent that a Driver truly reflects a motivation for change, and describes conditions that the group collectively believes to be true. The resulting agreement of how this Driver is articulated can form the problem statement. Many other viable approaches exist.

The problem being expressed in this article, is that we see a high number of failed experiments and low engagement in change efforts. The rationale for this is built through each section of the article.

Operating within <this context>

The context describes the wider environment in which the experiment writers are operating. From an integral model perspective, context often relates to the lower right quadrant of the exterior collective, the ‘Its’ quadrant. This is the social behaviors, environment systems, structures, laws, and norms of social interactions which determine how the internal collective, the ‘we’ space, operates.

The hypothesis of this article is that organizations most likely to attempt to exclude people affected by an experiment in its design come predominantly from Amber and Orange organizations. These organizations are defined in Frederic Laloux’s book Reinventing Organizations[9]. In Amber, such as most government organizations, the future is a repetition of the past. In Orange, most large organizations today, management is by objectives, with freedom to determine how these are achieved (see Single Loop Learning below).

By definition, true green and teal organizations would be involving their people in experimentation design as their focus is to build a culture of empowerment and inclusion to boost employee motivation. As the problem and context statements are likely to be different between amber and orange vs green and teal organizations, the latter group likely needs to craft their own experiment about experiment knowledge sharing.

Where <these forces> are at play

As presented in the introduction, many organizations will pilot transformation experiments locally and then roll out widely across the organization. Sadly, this often occurs without establishing an effective feedback loop to see the efficacy of such repeated application of the experiment. Such organizational experiments lack empirical process control. Given that much of organizational change is firmly within the realm of complex decision making in an unordered domain[10], these pilots are often doomed to fail.

This reality exists when an experiment relies on the philosophy that an absolute truth exists, that it can be rationally understood through the scientific method, and therefore that the experiment’s method can be repeated to yield the same outcome. This is experimentation for complicated sense making within an ordered domain. Simply put, the unknowable variable that cannot be understood except in retrospect, is people. The conditions that exist that make a successful experiment with a pilot group cannot be known upfront. Individual attitudes, subjective views, action logic; group dynamics; local policies; and the relation to this to the organizational culture is but a very thin slice of the conditions that demonstrate the complex variables at play from one group of people to the next.

As a result an experiment cannot be repeated, only recreated and recontextualized by the next group of people. Techniques such as co-creating causal loops as a coaching tool, supported by coaching questions can help a group understand what their current reality is.

Assessing Pattern Fit

Patterns are designed to be one-pass readable. If what you are reading is not resonating then stop and move onto an alternative pattern. To illustrate, consider that an experiment crafted by a group struggling to manage its dependencies is unlikely to be applicable to a group that has none. Neither group has the same problem so why would the same solution be of value?

As a second example, imagine that two groups of software engineers do share a similar problem around managing dependencies. However their contexts are very different. The first group represents an interconnected set of software engineering teams with centralized testing and release processes. They have minimal continuous integration and deployment capabilities. Therefore their dependencies are managed through the interconnected relationships of people within this system. The second group has advanced DevOps practices that allow them to manage dependencies within the code itself. Therefore while the problem may be expressed as the same, their context is very different. The experiment’s proposed solution is unlikely to be viable.

When there is a case that the problem and context matches from one group to the next, say in a wider enterprise that is rolling transformation across in waves, recognizing the forces at play can by the revealing factor of whether an experiment can be successfully replicated (still with the explicit consent of following groups choosing to repeat an experiment). Imagine two groups within an organization that has advanced DevOps practices. The first group has developed cohesive team dynamics that are exemplified by high trust, shared commitment and openness. The other group demonstrates a distinct absence of trust. While the problem and context remain the same in limiting dependencies through code, the people dynamics may very likely inhibit a successful outcome for the second group. The second group may therefore choose to adapt the experiment to suit the forces in play for them, or indeed decide to run other experiments first. I.e. experiments to build a collaborative whole team mindset.

Single, Double and Triple-Loop Learning

In addition to contextualizing an experiment, pattern language for HDD can be used to consider Single, Double, and Triple-Loop learning and change[11].

Trying — and failing in part or whole — with multiple experiments to resolve the same problem may be indicative of Single Loop learning. This is about taking action to change our behaviors. The experiment writers are trying every idea to solve a problem within the prevailing constraints of the existing forces and context in play. As an illustrative example, a group of software teams working across three time zones experiments with many different meeting formats, meeting times, and working hours to solve the problem of limited collaboration in the face of few overlapping hours.

Once the collective whole in this example have exhausted their options, and haven’t resigned themselves to this situation as unresolvable, they may begin to reassess their beliefs, views and opinions that led to this current reality in the first place. This is Second Loop learning, changing our thinking. Perhaps the hidden costs of inhibited collaboration, and meeting overload at certain times of the day, does not outweigh the very visible savings on a balance sheet from outsourcing teams from a distant timezone? Such questions may lead the group to experiment with altering the forces they understand, and so shift the context that they are operating within. Continuing our example, the group decides to invest in DevOps to push dependencies into the code, and to create cross functional teams in each time zone in order to again diminish inter-team dependencies.

Third Loop learning is harder again, it is about changing our perceptions. From an integral model perspective, this is about altering our collective identity in the lower left ‘we space’ quadrant. The experiment writers would at this point be asking much more powerful questions, such as how we might shift ourselves to a different form in order to create new options? To conclude our example, The collective whole recognizes that scaling around capacity without focus to distinct customer missions has created unnecessary complexity within their organization. And so the organization decides to create three whole teams of teams that have distinct missions, one in each time zone. They need to build new capabilities in each area, but it allows three new groups that can write their own independent experiments for future change and growth. These are difficult questions, often resulting in difficult decisions. There is no guarantee that an organization will have the desire or ability to engage in either Third or Second Loop learning. What can we do as change agents to support them in this?

Conclusion

Hypothesis Driven Development and experiment writing templates in general have been valuable tools in well over a decade of organizational transformation. For coaches, they provide a powerful medium through which we can help people express change in a structured way, to challenge limiting beliefs, and dive deeper towards solving a root cause problem versus resolving just the symptoms.

The pattern template extends this further, by helping writers consider where they are now through expressing their context and understanding the forces at play. Each element of the pattern structure could be considered to contain a toolkit of options to help craft better experiments. A few tools were presented in this article, there are many more that you may already know of, or may wish to seek out for yourself. Perhaps the most powerful elements of this approach can be revealed in the first box, As we are <this group of people>, and amplifying the first line of HDD, we believe. What kind of group are we? Are all voices present? How could we have this conversation so it empowers everyone? Do we need to reassess our beliefs? Who do we need to become to succeed here?

Sources

1. How to Implement Hypothesis Driven Development by Barry O’Reilly
2. Essential XP: Card, Conversation, Confirmation by Ron Jeffries
3. A Pattern Language by Christopher Alexander, ISBN 9780195019193
4. A Pattern Language for Pattern Writing by Gerard Meszaros and Jim Doble
5. The Principle of Equivalence, Sociocracy 3.0
6. Integral Theory (summarized article) by Ken Wilber
7. Describe Organizational Drivers, Sociocracy 3.0
8. Consent Decision-Making, Sociocracy 3.0
9. Reinventing Organizations by Frederic Laloux, ISBN 978–2960133509
10. Cynefin Framework by Dave Snowden
11. Masterful Coaching by Robert Hargrove, page 28, ISBN 978–0787960841