Agile: Transactional and Transformational Leadership
As part of my ad-hoc series on agile and organisations, I started by arguing that Agile = Culture, then went on to explore the concept of Culture for Agile and looked in to models organisational of change and transformation. Now, I want to turn my attention to leadership. In the 2017 State of DevOps Report, the authors pay particular attention to Transformational Leadership. Measured against five dimensions, transformational leadership was positively correlated with IT performance and business performance. The concept of transformational leadership is very familiar to anyone who has studied organisational leadership, popularised by the political scientist James MacGregor Burns. Burns studied US Presidents and contrasted two ideas of leadership: transactional and transformational. The transactional leader focused on the use of exchange as the core mechanism of leadership; the leader uses rewards and punishments to guide the behaviour of the led. In contrast, transformational leadership inspires change by shifting the thinking of the led as a form of charismatic leadership; extensions of the theory link transformational leadership to the leader’s personality.
The idea of transformational leadership and it’s dichotomy with transactional leadership triggered two lines of thought and I was struggling to bring them together — until I realised the right answer was not to. The first line of thought is the contrast between transformational and transactional approaches to leadership, people, and organisations. The second is to delve into charisma and charismatic leadership. The latter topic, I’ll leave for the future; this post will focus on transactional vs transformational dichotomy and understand how that can inform agile change.
Setting up two ideas in opposition as a dichotomy is fundamental to the way our brains operate: we define things by what they are not as well as by what they are. The particular ‘transactional’ vs ‘transactional’ dichotomy echoes throughout the management literature and is much deeper than leadership. Given the previous arguments that culture change is pivotal to successful agile transformation, the fact that transformational leadership is linked to successful IT business is expected. It is worth exploring this essential dichotomy in more depth. To do that, we need to go back to the early days of management theory — starting with Frederick Taylor.
Taylorism, Fordism, and The Machine Organisation
Well, if you are a high-priced man, you will do exactly as this man tells you to-morrow, from morning till night. When he tells you to pick up a pig and walk, you pick it up and you walk, and when he tells you to sit down and rest, you sit down. You do that right straight through the day. And what’s more, no back talk. Now a high-priced man does just what he’s told to do, and no back talk. Do you understand that? When this man tells you to walk, you walk; when he tells you to sit down, you sit down, and you don’t talk back at him.
In 1911, the engineer Frederick Taylor published The Principles of Scientific Management based on the view that tasks could be scientifically analysed and optimised, illustrated by the supposed conversation with ‘Schmidt’ — his exemplar of the worker — quoted above. Taylor’s approach is to separate the analysis of the work from the performance of the tasks and applying scientific discipline to the analysis which would be significantly more efficient and effective. Henry Ford adopted a similar approach: standardisation and specialisation allowed relatively unskilled workers to complete a complex task through the use of a sophisticated process. The role of management was to direct the work: to perform the analysis and to direct the worker.
These ideas are still pervasive, and seen throughout the way our organisations are run: project management techniques involve a detailed break down of the project into tasks; managing each task to deliver the project. It is no co-incidence that Henry Gantt — he of the eponymous charts — was an advocate of Scientific Management. Separating analysis from delivery, breaking work into tasks, then tracking progress against each task is a reasonable description of the waterfall software development. The “high-priced man” Taylor referred to points to a strictly transactional model of leadership.
The Hawthorne Effect
In the 1920s, a group of researchers started a set of time and motion studies in the Western Electric’s Hawthorne Plants; these studies included experimentally changing the working conditions so they could find the ideal conditions to maximise productivity. Following good experimental design, they separated the workers into two groups: a control and an experimental group. The control group were largely left alone, to work as they had always worked. With the experimental group, the research team started changing a wide range of working conditions: most famously adjusting the illumination levels.
The results were unexpected. With every change, the productivity of the experimental group increased compared to the control. In their book, Rothlisberger and Dickson attributed the increase in productivity to the mere fact of paying attention to the staff. The fact that people change their behaviour — usually positively — when engaged in an experiment has become known as the “Hawthorne Effect” and launched the Human Relations school of management which holds that by paying attention to people and caring for them as people, rather than doers of tasks, results in significant improvements in operational performance. This approach aligns to the transformational model of leadership.
A Persistent Dichotomy
I’ve taken the time to dig into this history, partly because it’s interesting but mostly because these two ideas together form one of the most essential dichotomies that lie at the heart of organisational theory: on one side, we have the transactional, technical focus on the work to be done; on the other, the focus on the people and relationships that constitute the organisation — a transformational model.
This dichotomy has persisted since the foundations of organisational theory nearly 100 years ago; findings, such as that reported from the State of DevOps report, have been consistently and reliably reported for at least as long. But the scientific management disciplines persist: organisations are rife with GANTT charts, with work breakdown structures, with bureaucratic hierarchies, and with the separation of management (and analysis) from work (and delivery). To help understand this, it is worth a quick scan of some ways this dichotomy has manifest in studies of organisations and the people within them — summarised the table below.
Lessons from Opposites
From all of these dichotomies starting with transactional and transformational leadership, there are lessons for agile management and transformation.
The most important lesson is the general one: agile is tapping into ideas and themes that have a long history in the various disciplines that make up management and organisational theory. Rather than reinventing or rediscovering these ideas, it’s worthwhile learning from those that have gone before us in trying to improve our agile organisations and leadership. The seeds of agile can be traced back to the Hawthorne Studies: this reaction against the scientific management is almost identical to agile as a reaction against waterfall software management; the same dichotomy lies between transactional and transformational leadership. The rest of the lessons come from dipping into these disciplines and scratching the surface of the ideas and wisdom they can bring to agile leadership.
Starting at the level of the organisation, Burns and Stalker found that smaller, organic organisations made up of multi-talented individuals led to teamwork within the organisation. This approach was more effective in responding to changing environments, compared to the mechanistic organisations. Trist’s Tavistock Institute found that the objectively less efficient new method of short-wall mining achieved better results than the old method long-wall mining, based on mass production techniques. The reason for this was that short-wall mining encouraged teamwork, which was more important for productivity than the structured planning of the task.
The direct impact of the social dimension of work on productivity, which exceeds the value of structured analysis and work breakdown.
Organisational theory draws attention to smaller units: groups and teams, which have consistently been shown to be pivotal to successful innovation. Katzenbach and Smith contrasted ‘working groups’ and ‘teams’. For them, a working group is defined by individual output (the performance of each person’s tasks); teams are defined by collaborative output. The team, in this model, takes on particular social-psychological dimensions that enable high performance and is built on a series of characteristics including small size, complementary skills and all contributing to the goal, a common purpose with demanding goals, agreed ways of working. All characteristics of a good agile team and pivotal to the success of agile models and to innovative organisations. In a very real sense, the goal of an agile coach is to build teams and create an environment where those teams are allowed to succeed — the transformational approach builds stronger teams. Transactional leadership produces working groups.
To understand the behaviour of individuals within teams and organisations we can look to human motivation. McGregor described two models of motivation: Theory X assumes people are motivated by individual goals and rewards; Theory Y assumes that people want to do a good job and improve themselves. The latter view aligns to the findings from the Hawthorne Studies, however McGregor found these were assumptions made by managers — rather than inherent in the people they were managing. Victor Vroom found that managers adopt managerial styles that align with their assumptions about human nature — autocratic or participative. Combining these ideas gives us a further lesson: how you behave as a leader is shaped by the assumptions you make about those you are coaching.
That, in turn, becomes a self-fulfilling prophecy: how you behave towards those you are coaching shapes their behaviour and their attitudes.
Coordinating efforts of people and teams within organisations is the realm of management and leadership. Blake and Mouton identified two primary dimensions — the transactional concern for task and the transformational concern for people — that classified different managerial “styles”; each individual manager should identify their natural style. Hershey and Blanchard also looked at managerial styles, but assumed that the individual manager could adapt their style to the most appropriate for their situation, with the essential message that no single way of working is necessarily better. Agile teams need to focus on both relationships and on outcomes — the correct focus will depend on the team and their current propensity for self-management. Similarly, each agile leader will have different strengths and weaknesses: they should play to their strengths and work on developing themselves in areas of weakness. This message needs to be tempered with the findings from motivation: the approach adopted will influence the way the team behaves. Overly transactional management focused on outcomes will inhibit the development of self-management within the team.
The final domain is decision making, Simon’s work on organisational decision making characterised decisions as ‘programmed’ or ‘non-programmed’. Programmed decisions were appropriate in predictable, stable situations and are amenable to analysis, while non-programmed decisions require more work to structure and define the problem as part of decision making. Simon advocated adopting an approach suited to the nature of the decision. In many ways this takes us back to the start: for well known, stable domains, structured analysis is more valuable. However, the vast majority of problems and decisions in the technology world are not well structured. So approaches that are fast-moving and geared to structuring as well making decisions are more effective.
This post has been a small indulgence into a field that I’ve worked in for almost 30 years. By leaping off from the dichotomy of transactional and transformational leadership, we looked into some of the history of organisational theory through the lens of other, similar dichotomies. This has shown that agile is another manifestation of well-established ideas on how to lead and manage organisations. The problems that software development teams are facing have been problems tackled by organisational theory since it’s inception; the solution proposed by agile is, similarly, not new: it is a particular implementation of well-established ideas. For those working in agile, being exposed to this history to give a broader tool-kit of ideas to tap into to guide and improve agile practice, to learn from others rather than re-treading the same old ground. It can also increase the adoption of agile by recognising that it’s an approach that aligns to similar ideas under different names in different domains.
But this lesson also comes with a warning: as powerful as agile is, it won’t answer every question or fit every situation, much like the organic organisational model hasn’t been adopted by every organisation. The barriers to the adoption of the human relations school of management will likely be barriers to the adoption of agile. An agile leader will need to tackle these barriers. Even where agile does fit, it will need to be adapted and adjusted to suit the organisation and the problems it encounters. Finding and creating that adaption is another pivotal part of the role of the agile leader.
I have followed this discussion of contrasting transformational and transactional leadership with some thoughts on charisma and transformational leadership.
