The Science behind Scrum (part 2): Improving productivity. Scientific management and sociotechnical systems
This post is the second in a series of posts that I will use to explore the science behind Scrum and Agile methodologies, and specifically its history. For those interested, you can also check my previous post about Scrum and Learning organizations.
In this post I want to explore the historical and scientific roots of how Scrum attempts to improve the productivity and quality of the software development process. I will discuss two important approaches from the field of organizational development (OD) and how they influenced Scrum. My primary goal is to place Scrum in a historical and scientific context and open up a broader perspective. I will assume basic knowledge of Scrum from the reader. Also, I will not present the usual practical tips as I think they don’t add to the overall goal of this post. If you are interested in reading more, check the references below.
Where it began; The Industrial Revolution and Scientific Management
First, we have to go way, way back to the industrial revolution. Somewhere near the end of the eighteenth century the society, culture and economy of (mostly) Western countries was deeply transformed by the rise of industrial technologies in mining, manufacturing, logistics and agriculture. Before the industrial revolution, industry was mostly a matter of farmers with their kettle, a local bakery, blacksmith or perhaps a small factory with horse-powered machinery. Industrialization started with the advent of steam powered machinery. This allowed small factories to dramatically increase their output, even further augmented by the discovery of electricity, combustion engines, chemicals and metallurgy. Most of all, these technological advances allowed factories to grow well beyond their original sizes, drawing hundreds of thousands of people from the poor farmlands to the cities. The growing size and complexity of organizations during the end of the eighteenth century, driven ever onwards by new technologies, introduced new problems as management practices became wholly insufficient. This caused a number of people to explore new ways of managing these organizations and their employees.
During this time, a young man called Frederick Taylor (1856–1915) was working in a steel factory. He noticed that employees strongly differed in their productivity. He reasoned that the best way to increase productivity was to standardize the process of manufacturing based on empirical (scientific) measurement. He did this quite literally, by using a stopwatch or motion analyses to measure the time or effort a task took and by exploring means to increase the output. Taylor also addressed the problem of motivation, by arguing that employees are not internally motivated to do their work as fast as possible but need to rewarded externally based on their performance (piece-rate). Setting up reward systems, measuring and fine-tuning tasks and selecting the right worker for the job was something that Taylor considered to be the responsibility of more intelligent and mentally alert employees; the administrators (Taylor, 1911). This distinction between management and the people on the ‘work floor’ is something we still see today in the management tiers of modern hierarchical organizations. This approach to management was called Scientific Management and it’s very influential and widespread (even today).
Scientific Management greatly improved productivity and helped organizations grow by focusing on rational control, empirical task analysis, optimization and selecting the right employee for the job. But it was also the target of many criticisms. Although it would be a mischaracterization to state — as some do — that scientific management actively de-humanized organizations, it is true that it considered employees to be little more than parts of a bigger machine. Its focus on task productivity and external rewards ignored important human factors and ignored the complexities of human relations within organizations.
Human Relations: The ‘I’ in organization
The most important criticism was raised by a number of studies conducted at the Hawthorne Power Plants during the first quarter of the nineteenth century. For five years, the effect of a number of human factors on productivity was researched:
- Light intensity: Researchers noticed that simply changing the intensity of light in a room of workers increased productivity, which ran counter to the assumption that only external rewards could do this;
- Relay assembly experiments: Researchers monitored a number of women working on a relay assembly line. A number of variables were changed during the experiment (like duration of lunch, shortening of work hours and the number of breaks), but all based on the suggestions of the women. The researchers found that any change, even changes back to the original state, improved productivity. This study showed that being taken seriously and having input in decisions (autonomy) is an important motivator;
- Bank wiring experiments: Researchers monitored a group of men that received payment incentives based on their individual productivity. They noticed that productivity actually decreased. Further analysis showed that the men formed peer groups that actively suppressed members that outperformed the rest, fearing that others might be punished. This study showed that employees are more strongly influenced by social pressure than by external rewards by themselves;
Together, these studies showed the strong influence of human factors on productivity. Employees have internal motivations, desires, dreams and fears and are influenced by peer groups, social, cultural and psychological factors. They increase or decrease productivity to an often larger degree than external rewards or task optimization in themselves. The Hawthorne studies resulted in the Human Relations school of thought and are often considered to be the birth of Human Resource Management and even modern Organizational Psychology. It is also the starting point for many of the insights that are prevalent in Scrum and Agile Software Development.
The legacy of Scientific Management
Scientific management is still one of the most influential schools of thought within management circles. It offers a very tempting view of organizational reality by promising dramatic increases in productivity by empirically measuring and optimizing tasks and processes and ignoring human complexities. Its influence can be easily identified in Total Quality Management (TQM), Business Process Redesign (BPR), Six Sigma, Lean Manufacturing and many other methodologies. But the same goes for how organizations are structured and how work is designed. Scientific management favors jobs that are highly specified, fragmented and repetitive and diffuses the overall responsibility across many employees, all doing their part. This is something will still see today in most organizations.
Principles of scientific management can also be identified in software development. Most traditional approaches to software development (like the waterfall model) either implicitly or explicitly assume that the development process is a very predictable and controllable process and ignore many complexities, including human factors. In these cases, the role of management is to measure and control the process and the tasks performed by employees. External rewards for delivering on-time are not uncommon. Furthermore, the total responsibility of work is distributed across a great number of specialized jobs (analysts, developers, testers, managers, etc.) which results in ‘silo-thinking’.
Although Scrum actively seeks to address the complexities of software development, it still shares some of its basic assumptions with Scientific Management. After all, the empirical control so important to Taylor is also one of the pillars of Scrum. But don’t worry. I’m not going to argue that Agile is, in fact, a modern incarnation of Scientific Management. Although it was clearly influenced by Taylor’s ideas, Scrum is more strongly influenced by more modern approaches to organizational development. This, hopefully, I will now show.
Addressing the missing (human) piece: Sociotechnical Systems
After the onset of the Human Relations school of thought, many sociologists (like Weber and Mayo), cognitive and organizational psychologists became interested in human factors. I can’t do justice do all the work done in the following decades, but I would like to address one of the theories that most strongly influenced Scrum.
During the mid-twentieth century, several organizational psychologists at the London Tavistock Institute developed the Sociotechnical Systems (STS) approach to organizational development. This was mostly in response to growing problems in modern organizations with absenteeism, turnover, quality and productivity (Cummings, 2004). As it turned out, the work design principles influenced by Scientific Management resulted in boring and meaningless work. Even though technology and pay was improving, task productivity was decreasing. Eric Trist and his colleagues from the Tavistock institute (1963) identified two principles for work design:
- Interaction of technical and social factors: Work is influenced by two systems of variables. The technical system concerns the tasks involved, any tools, the location, etc. while the social system concerns the social and psychological needs of the employees. Both systems always influence the work, whether designed or not;
- Joint optimization: When designing work, both systems need to be addressed equally as they continuously interact and influence each other. Failure to design both, will certainly result in loss of productivity;
Following an important case study in production-oriented environments, Trist (1951) identified a number of principals that improve the fit between social and technical systems:
- Teams as ‘unit of work’: The focus should be on teams that perform work rather than on individuals;
- Whole tasks in small teams: Small teams should take on whole tasks and be made responsible to complete them. They should all the required expertise to do so;
- Increase autonomy: Teams are responsible for organizing and controlling their work. Managers and supervisors become advisors rather than overseers. They control the process, not the tasks;
- Self-organize and adapt continuously to deal with complexity: Teams self-organize to adapt to the continuously changing complexities of the organizational reality and the work they are performing;
Socio-technical systems at work at Volvo’s Kalmar Plant
These principles have been implemented in a variety of ways. One of the most well-known examples is in a Volvo factory in the mid-1970s (Rollinson & Broadfield, 2002). Using assembly lines, Volvo had long managed to stay competitive. But quality, productivity and commitment had been falling for a while. As an experiment, Volvo decided to use their new Kalmar plant to introduce a work method based on the sociotechnical systems approach. Instead of having each employee perform one small task on the assembly line (as inspired by Scientific Management), Volvo formed a number of work cells that became responsible for the completion of entire sub-systems of a car (like the brakes, electronics, etc.) while a computer-controlled trolley moved cars from cell to cell when the work was finished. Teams were made responsible for their own planning, scheduling and task allocation and — in newer plants — also for recruitment and training.
Although there was some initial resistance from management — they weren’t used to ‘stand by and watch’ — the experiment was a phenomenal success. Job satisfaction, productivity and quality greatly improved (Bailey, 1983) and inspired many companies to perform similar experiments and extend on them. There are many more examples from the car manufacturing industry, like Toyota’s Production System (TPS). They share a strong emphasis on teamwork and the social aspects of the work.
From socio-technical systems to Scrum
It isn’t hard to identify the influences of the sociotechnical systems approach in Scrum. In their seminal article for the Harvard Business Review, Hirotaka Takeuchi and Ikujiro Nonaka (1986) laid the groundwork for a holistic approach to work design that would lead to the development Scrum. They interviewed a number of employees from several successful manufacturing organizations (like Honda, HP and Xerox) and identified six key aspects of the approaches these companies used to optimize their production process:
- Built-in instability: management sets challenging goals and gives the team the freedom to achieve these goals;
- Self-organizing project teams: A team is allowed to self-organize to achieve the goal in the most optimal way. This requires teams to have a lot of autonomy, a very strong desire to continuously improve (to ‘self-transcend’) and to cross-fertilize knowledge within the team;
- Overlapping development phases: Development requires certain phases, but these should overlap to absorb vibration or changing requirements;
- Multi-learning: Members of a team learn in multiple ways. They learn from each other and by rotating roles (cross functional learning) and by learning across multiple organizational levels (multilevel learning);
- Subtle control: Rather than the traditional command-and-control authoritarian style of management, managers should subtly control the work process of the team. They can do this by creating the right climate for teams to work effectively, select the right people for the jobs, managing the rhythm of the process and tolerating mistakes;
- Transfer of learning: To avoid losing valuable knowledge gained by teams throughout the process, learning should be facilitated throughout the organization in many ways.
The six aspects closely mirror the principles of sociotechnical systems. Even the use of ‘holistic’ matches the idea that technical and social systems of work design should be aligned. Interestingly, Takeuchi & Nonaka (1986) make no mention of sociotechnical systems or any other precursor model or theory. Although the authors are sometimes credited with the invention of a new approach for commercial product development, their work is clearly based (or describes) on existing models and approaches.
Although Takeuchi and Nonaka use the term ‘Scrum’ in their article, they use it mostly as an analogy. The term ‘Scrum’ as the name for a method for software development was first coined by Ken Schwaber and Jeff Sutherland. They formalized and presented Scrum as a methodology for software development during the annual OOPSLA congres (Object-Oriented Programs, Systems, Languages & Applications) (1995). The rest of the story is known to those familiar with Scrum.
In this post, I set out to explore the historical and scientific precursors of how Scrum attempts to improve productivity and quality. First, I explored scientific management and how this influenced Scrum. I then discussed criticisms of scientific management and how this resulted in a larger focus on the influence of human factors on productivity. These criticisms have been addressed by many modern theories of organizational development, but I specifically discussed sociotechnical systems theory because of its resemblance of Scrum. Finally, I discussed the seminal paper by Takeuchi and Nonaka that laid the groundwork for what we later be formalized into Scrum.
I hope that by writing this blog, you learned a bit about some of the precursors of Scrum. By placing Scrum in its historical and scientific context, I hope to open up to a broader perspective and inspire those working with Scrum. As with any method, evolution will take place. Scrum has already been adjusted slightly over the years by its inventors. The best thing we can do is to keep learning, to keep optimizing and to keep reflecting on how we can use insights from the past for the present.
Note: I will investigate other roots of Scrum in future posts. There are many, many more. I also intend to add more practical pointers for those interested.
Bailey, J. (1983). Job Design and Work Organization, London: Prentice Hall
Cummings, T. (2002). Organizational Development and Change, in J. J. Boonstra, Dynamics of Organizational Change and Learning, West Sussex: John Wiley & Sons Ltd;
Rollinson, D. & A. Broadfield (2002), Organisational Behaviour and Analysis, Harlow: Prentice Hall;
Sutherland, J. V. & K. Schwaber (1995), Business object design and implementation: OOPSLA ’95 Workshop Proceedings, Michigan: University of Michigan;
Takeuchi, H. & I. Nonaka (1986), The New Product Development Game, Harvard Business Review, 1986 (January-February);
Taylor, F. W. (1911), The Principles of Scientific Management, New York: Harper Bros;
Trist, E. & W. Bamforth (1951), Some Social and Psychological Consequences of the Long Wall Method of Coal-Getting, Human Relations, Vol. 4, 3–38;
Originally published at blog.agilistic.nl on February 19, 2012.