Fostering a mindset and method for deliberate change
How to apply the test, learn, refine principle in your role, team and organisation
Our principles for creating beautiful software
Test, learn, refine is one of our product and technology principles at Xero. Applying this principle includes behaviours that product professionals know well.
However, it means much more than these basics. Test, learn, refine can be applied usefully at multiple levels and scales, touching everyone in a modern organisation, and impacting how we collectively improve over time.
Individual contributors
Individual contributors have development plans with goals and strategies for progression. Test, learn, refine is a way to reflect on what’s working, how we are embedding learning into the doing, and how we might make adjustments.
Teams
Exactly the same thing. There are ways we want to improve how we work together. Test, learn, refine is the method and mindset for doing this.
Product creators
Solutions are ultimately hypotheses. We use test, learn, refine to gather insights and improve these solutions.
Business leaders
Reviews and governance rituals, at their best, apply the test, learn, refine method to strategy. For example, Xero applies it to the full cycle of how we create beautiful products. It’s our approach to innovation and operational excellence.
How rigorously the principle is applied is important for all organisations (and also for all of us outside of work). By considering the roots of this principle in continual improvement and reflective practice, we can take advantage of its transformative power. We can begin embedding test, learn, refine into our rhythms and rituals at work, and in life.
What is test, learn, refine?
Deliberate change
Test, learn, refine is a method and mindset for deliberate change. It’s what human communities do — we learn and change. Our ability to improve outcomes is fundamental to our species. Other animals also improve capabilities with practice, but humans take this to another level.
The emergence of collective learning — our ability to pass along learning across generations — may in fact be uniquely human, and a major threshold in the history of the universe (check out The Big History Project for more details on this).
Continual improvement as a core competency
Modern approaches for continual improvement that we use today were developed following World War II. Methods like Statistical Process Control, Six Sigma, the Toyota Product System, Total Quality Management and Kaizen helped Japan rebuild.
These worked so well that Japan captured a large portion of the automobile market. In a nutshell, early improvement models like Six Sigma, use statistical methods to reduce variations (or defects) in a process, while Lean (which came a bit later) reduces waste. These were later combined and popularised as Lean Six Sigma, which focuses on both goals simultaneously.
The earliest implementations of these methods yielded an essential insight:
Continual improvement is an organisational culture issue, not simply a method.
Success, it was learned, requires a deep understanding of continual improvement, which has been considered an essential leadership competency ever since. For example, Jack Welch at General Electric insisted that all leaders become Six Sigma Greenbelt Certified, by completing projects that either grew revenue or reduced costs.
Reflective practice
Methods for reducing defects were fit-for-purpose in industrial settings, but variations on this approach were needed for knowledge organisations and IT product organisations. Here, continual improvement requires reflective practice — that is, mindful and systematic consideration of recent experience to find actionable insights that can improve outcomes.
Reflective practice is more than a mindset — its power resides in embedding related behaviours at all levels into the rhythms and rituals of the organisation.
The essence of reflective practice can be conceived as a cycle: plan, do, reflect. At any given time, we are either planning what we will do next, doing it, or reflecting on what worked and what we could do better the next time. This cycle is common among product teams in the form of sprint planning, sprint activity and sprint retrospectives.
A variation on this cycle is: plan, do, study, act. This is called the Deming Cycle, named after W. Edwards Deming, who is sometimes referred to as the father of modern quality control. Deming went to Japan after the war and introduced statistical process control. This had a huge impact on the regrowth of Japanese manufacturing. USA automakers did not apply these methods rigorously for a few decades. Once they did, they quickly began regaining ground in the automobile market.
Reflective practice applies rigour to the ‘reflect’ portion of the cycle — which is sometimes given too little attention. This idea is captured in one of the principles mentioned in the Agile Manifesto:
“At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly.”
Getting started with test, learn, refine
If you want to start at the very beginning — how might you begin embedding test, learn, refine in your role, team or organisation? Here are some tips.
Who
Identify your champions. At Xero, many people across the organisation fit this description. However, we also have coaches and facilitators as internal experts who, by their nature and training, live and enable reflective practice and continual improvement. Organisations need leaders who understand both the cultural and methodological sides of continual improvement.
What
Consider how test, learn, refine can be applied at all levels of an organisation, from individual contributors to teams and business leaders. Understanding these opportunities is a great place to start. Gather your champions and experts, and prioritise where to begin based on what feels most critical.
How
For each of those areas, ask some ‘how might we’ questions. How might we most usefully reflect on recent experience? How often might we do this? How might we capture insights and feed them forward into support actions? How might we track what we have learned and test related hypotheses? How might we foster both the mindset and methods of continual improvement, treating this as a core competency?
When
Lastly, determine when exactly these questions will be asked and answered. What are the existing rituals that might serve as the opportunities for such discussion and collaboration? Alternatively, might new rituals be required?
In summary, we are asking: how might we embed test, learn, refine (a continual improvement and reflective practice) into the rhythms and rituals of our work? The answers will differ across contexts.
The questions can also be used to audit current practices: how do we currently do these things, and how well are they working? These questions are appropriate for all levels of an organisation, equally powerful and practical for seniors leaders and development teams. Leaders should ideally role model both the mindsets and the methods.
Getting better at getting better
If you want to get better at something — anything — then reflective practice is the meta-strategy. Like the Golden Rule (treat others as you want to be treated), it is almost always the right thing to do. However, there is no perfect approach. Therefore, we are left with the popular notion of getting better at getting better. This is not abstract or theoretical — you really can operationalise this goal within your life and your business by asking:
What might it look like to apply test, learn, refine to our current methods of reflective practice?
A simple example of this would be a product team conducting a retrospective on their current retrospective methods, and asking: how well are we identifying actionable insights and feeding them forward into our behaviours? How might we do that better? This comes later, though, once the basic rituals for reflective practice have been established.
One last, essential question: how might we apply test, learn, refine to our core purpose and vision? Xero’s vision is to be the most insightful and trusted platform, and our purpose is to make life better for people in small business, their advisors and communities around the world. Test, learn, refine asks: how are we doing with these goals? How might we do better?
Why spend our energy and resources here?
All of this talk about reflective practice and continual improvement has one common foundation: desire. We want some part of the world to be different — and maybe even some part of us to be different. Those are our goals. It is human nature to move toward what we desire. We do this unconsciously — trial and error is happening all the time. But that is slow and inefficient. Instead, we can bring method to our goals. That is what strategy ultimately means.
Test, learn, refine is a mindset and method for deliberate change — a strategy for striving toward whatever we desire.
