Theories X and Y and Agile failure in organizations
And no one pours new wine into old wineskins. If he does, the wine will burst the skins, and both the wine and the wineskins will be ruined. Instead, new wine is poured into new wine-skins. (Jesus Christ)
Although Agile approach to software development has become very popular in the last decades, many organizations perceive it to fail to meet their expectations. In this article, my intention is simply to dwell on what I consider to be one of the main causes of this problem, starting with a reminder of a concept developed by Douglas MacGregor in his book “The Human Side of Enterprise”. Even though this book gained a lot of momentum in the 1960s, it can help us to understand the reason why organizations often fail in implementing Agile frameworks and achieving the expected outcomes that have motivated them to implement those practices.
An interesting fact about the history of Organizational Theory is that the more you hear about new trends in management and work design, the more you are convinced that many of them are simply new terms to describe old ideas, or attempts to apply some of those ideas to solve current problems. In other words: very often, the new is not really that new. However, this is not a bad thing. It is a good thing that our generation is not simply ignoring the contribution from those brilliant people from the past. It means we are building steps over stairs that were built before us, instead of start building new stairs from the ground.
Douglas McGregor was a very influential figure in management and organizations studies. His studies were mainly focused on how our beliefs shape our behaviour and how our behaviour affects other people’s behaviour around us. He articulated the Theory X, as what he saw as the dominant belief system about employees. Theory X presumes that the average human being is indolent, averse to work, not very bright and indifferent to organizational needs. Managers that share Theory X beliefs are convinced that without an active intervention by management people would be passive — even resistant — to organizational needs. Employees must therefore be persuaded, rewarded, punished, and controlled — their activities must be directed.
According to him, this traditional managerial view of employees is what leads to a command-control style of management, based on tight controls, strict policies, and a punishment and reward system. However, this system does not necessarily lead to better results. Since “Force breeds to counter-forces”, in many cases people will resist to this autocratic management style. This resistance ends up leading to poor results, which reinforces Theory X beliefs that employees are less intelligent and need constant direction. That’s why he stated that Theory X is a “self-fulfilling prophecy”.
However, external control and the threat of punishment are not the only means for bringing about effort towards organizational objectives. McGregor articulated Theory Y, as what he saw as a different system of beliefs about employees. As opposed to Theory X, Theory Y assumes that creativity is widely distributed in the society, and that, under proper conditions, the average human being learns to seek responsibility. People will exercise self-control and self-direction in the service of the objectives to which they are committed, because this way they are pursuing the rewards of satisfaction of ego and self-actualization.
Influenced by those beliefs, Theory Y management style will try to arrange things differently, so that individuals can achieve their own goals and happily accomplish the organization’s goals at the same time. Theory Y managers give to their employees more freedom and trust, and their approach gravitates towards relating to the worker on a more personal level, as opposed to a teaching-based relationship.
So far some of you may be asking: “What does all of these has to do with Agile approach to software development?”. The answer is “everything”. Agile frameworks have been developed under the premise of the Agile principles, expressed in the Manifesto for Agile Software Development. Therefore, if the principles are not present, the frameworks cannot work properly.
Agile comprises various approaches to software development under which requirements and solutions evolve through the collaborative effort of self-organizing and cross-functional teams and their customers. It also advocates adaptive planning, evolutionary development, early delivery, and continuous improvement, and encourages rapid and flexible response to change. In other words, Agile mindset embraces trust.
Agile methodologies are built on the pillars of self-organization, self-reflection, and self-adjustment. They are based on creating an environment in which people can agree and commit to the organizational goals, and they are given trust to make their own decisions about how they organize the work to be done. These frameworks cannot fit in an organizational environment in which Theory X beliefs are predominant. Therefore, when an organization decides to start an Agile transformation, their managers need first to be aware of the leadership mindset that it requires.
Theory Y management style is not a new Management trend. It was observed and studied by Douglas McGregor in the 1950s and gained great popularity in the Management academy in the 1960s. Theory Y principles have been studied by manager candidates and implemented in many organizations for decades. Nevertheless, the Theory X management style is still dominant in many organizations. This leads us to believe that changing people’s beliefs is not a simple assignment. It is fair to say that it can be compared to converting to a new religion. Agile transformation is this much of a challenge, and this can be one good reason why many organizations have been failing to implement it.
