Retracing Dr. W. Edwards Deming’s Work Through Scrum in Software Development
It is amazing that some of the old theory would, later on, transform our perception about some concept. Something like Euclid 5th postulate was being studied for centuries and gave us a breakthrough in the field of Math. Many industry leaders, academics, field practitioners and students dissect and participate in intellectual discourse about the topic. Sadly, that may not be the case with Dr. W. Edwards Deming works and its relation with Scrum. A quick search of the discussion about the relation of those two is surprisingly scarce.
William Edwards Deming (1900–1993) was, among other things, a statistician and management consultant. Deming and several other advocates in the field helped popularize the use of statistics in industrial transformation and management in particular. Deming’s work advocated that management should seek a systemic approach to solve organizational problems and relegate from the solution driven by short term gain. An organization is composed of a system of interrelated and interdependent people and processes. Therefore, the success of a system depends on having a deep understanding of the systemic optimization and orchestration of the delicate relationship between its component. From his background in statistics, Deming was able to teach the management to find improvement in a place where they didn’t look before. His work, later on, pioneered on what is known today as Quality Management.
Scrum is a process framework for developing and sustaining complex products. Authored by Jeff Sutherland and Ken Schwaber, Scrum principles and essences is contained in the Scrum Guides.
Scrum Guides serves as a document that describes the roles, artifacts, events, uses and principles in Scrum. Even though Scrum can be used in multiple fields of work, it is most popular in the software development field. The fact is a given due to the close relation of the authors with the software development field. Scrum emphasizes empiricism to face one of the common problem domain in software development, a complex domain. According to the Cynefin framework, one of the characteristics of a complex problem is where the unknown factors are yet to be known (unknown-unknown). In this problem domain, probe and act offer advantages over analyze and act, Scrum is a great framework for this approach.
At first glance, the two separate work may not share any correlation. However, upon closer inspection, Scrum by the extension of its components support and complement nicely with Deming’s philosophy of management.
To dive deeper into the topic, Deming gave us one of his most useful tools for Quality leaders and practitioners to better understand the organization they worked for. Throughout multiple seminars during his time, Deming composed a holistic philosophy that was named as System of Profound Knowledge (SoPK). Deming’s SoPK is a thought framework that anyone can use to inspect the reality of the system they work in. The SoPK main components are not to be taken individually and be inspected only within its context. It contains a structured approach of thinking to address (the negligence) quality issues prevalent during his time. Some of his work may need a closer look and taken in with context, however, his SoPK still serves as the general framework to gain insight. SoPK consists of 4 parts, where we will look into it individually and summarize all of the parts in the end. This writing series will attempt to draw a meaningful connection between the two ideas through a closer look using Deming’s lenses. It is not intended to be an exhaustive list of correlations, rather it will serve as a personal observation to how Deming work is related to Scrum in software development.
Part 1: Appreciation for a System
When we have an appreciation for the system, we will start to understand and clearly perceive its behaviors, natures, and values. Equipped with this knowledge, we will then have more clarity on how to improve it. Deming argued that the people who have the authority to influence and shape the system ultimately responsible for whatever outcome the system produces. It is irresponsible to put the burden of accountability to the people inside the system that have no direct influence on the system’s operation structure. In the context of Scrum, the responsibility and the accountability of the success and result of a system is now recommended to be entrusted to Scrum Team. In order to cope with the uncertainty of a complex problem domain, the system should be given a degree of flexibility to better respond uncertainty.
Part 2: Knowledge of Variation
The behavior that stems from not understanding the system outlook and simply hoped the people will do their best to complete the work is the main criticism of not having knowledge about variation in a system. Deming showed that a stable system will yield a stable output within the acceptable variation. We should seek to improve the system not by over-focusing on the special cause but aim to improve the system by systematically addressing the common cause. This over-obsession with the special cause is not recommended either in Scrum, most of the practices are geared to achieve consistency by regularly focusing on the areas that causing common problems to occur. Our ability to reduce complexity in the regular development cycle is the main outlet to nimbly maneuver in an uncertain problem area.
Part 3: Theory of Knowledge
How we are able to reduce complexity is mainly related to our ability to decompose and abstract a domain problem. In a complex problem, where unknowns are unknown, a big front analysis may outlive its usefulness. A regular small experimental cycle to address this complex problem is the recommended approach. This is to enhance the learning needed to gradually clear out the unknowns by inspecting our working artifact against a hypothesis. That is why it is practically advantageous to have a collaborative learning environment where the artifacts are transparent and the people work in it are focusing solely to gain knowledge on solving this problem.
Part 4: Psychology
It becomes the main reason why people at the center of this system should work in an optimal environment where psychological safety is present. The dynamics between the people working inside the system will be influenced by its psychological structure. The shape of the psychological structure is closely related to organizational values and cultures. When we define what constitutes as a culture we don’t just examine the stated organization’s visions, missions, and values. We also need to consider the actual sum of the behavior that is exercised consciously and unconsciously within the organization. The actual behavior that is being exhibited will be reflected as the byproduct of how the system is allowing it to happen in the first place. The psychological structure will impact the ability of a system to collaborate and address complexity in a problem area.
Closing up this writing series, Deming’s SoPK is a timeless thought framework. It teaches us to have an outside holistic view and look beyond our limited and bias perspective when we serve as part of the system. Deming composed his SoPK to be interrelated and interdependent. His thought framework was composed as a system of thinking to inspect and improve the system that we operate on. Deming’s scholar viewed the SoPK as a meta-knowledge, which means a knowledge to gain other knowledge. In this case, SoPK can be leveraged nicely when we are inspecting what Scrum looks within our context.
From personal observation, Scrum strengths stem from its incompleteness. Its nature, as an additive framework, requires its practitioners to add their own knowledge and experience to complete the use of Scrum. In this case, SoPK can be leveraged nicely to be used to inspect the reality of our implementation.
The SoPK has “Profound Knowledge” in part of its name. It can create the “aha” moment when we apply it to inspect our reality. Hopefully, it’ll create more magical moments of our self-understanding when we implement Scrum in our daily basis. To enable us to continue our software development journey guided by what Deming referred to as profound knowledge.
I’d like to give a shout to the wonderful Serious Scrum community. My personal thanks to Max Heiliger, Sjoerd Nijland and Willem-Jan Ageling to make this writing series happened.
Like this article? Go and check our community out.
Do you want to publish in Serious Scrum? Connect with us on Slack to make it happen!
We run a Serious Scrum channel on Slack. You’re all invited. Feel free to reach out and connect with us on Slack to share your thoughts.
