The Agile Crisis — a Primer
This article briefly summarizes what the author identifies as the current Agile Crisis. Since 2016 more and more prominent voices raise their concerns regarding the state of Agile. Those voices not only include lots and lots of Developers in the frontlines of software production but also prominent authors and forefathers of the Agile Manifesto. Leading figures like Martin Fowler, Andy Hunt, Ron Jeffries and Robert “Uncle Bob” C. Martin put forward some serious concerns with common determinator. Based on that I will argue that the currently perceived shortcomings of Agile and Scrum are not just ”faux scrum” but inherent to the agile approach as practiced today and a superficial understanding of the nature of Agile. A comprehensive introduction into the Agile Philosophy and Methodology is beyond the scope of this article. It is intended to be read by people already familiar with Agile who want to catch up on the latest discussion around Agile since 2016 to which I refer as Agile Crisis.
It will back up the call for a post-scrum-area in software development, address some myths in the current perception of Agile and will try to build a theoretical foundation for developers not happy with the current state of our industry.
“Dissatisfaction is a sign of wanting to improve”
— M. Fowler
Which Crisis?
Robert C. Martin recently pointed out, the Agile approach itself was a reaction to a crisis in software development¹: Since the birth of the software industry, the number of programmers doubled every five years. This means half of the programmers hold less than 5 years of experience. Constantly a lot of positions need to be filled quickly. Martin stresses that, where Alan Turing was convinced programming had to done by disciplined mathematicians, today it is predominantly “undisciplined young males” who were brought into these positions. What they lack in experience and professional discipline they are willing to make up in effort, willing to working ‘crazy hours’. They can be hyper-focused, but sometimes on the wrong things. Calmititious and most important for business: those kind of developers are available and they tend to be inexpensive. But fatally “an efficient business discipline coupled to an undisciplined engineering team will very rapidly make a mess”.
“an efficient business discipline coupled to an undisciplined engineering team will very rapidly make a mess”
— Robert C. “Uncle Bob” Martin
When the industry faced the fact that a vast amount of projects are failing the trade was alarmed. And history confirmed those concerns again and again: even with computers becoming more and more powerful, software tools and languages more and more sophisticated an alarming number of software projects experience failure. Finished products became more and more cumbersome to handle and erroneous too. While management required more and more complex systems, a growing number of developers lacked knowledge and expertise to produce useful results.
Out of this situation, the Agile movement arose: It tried to identify common patterns applied in software development, which actually worked. And it looked into the greatest common factors as guidance and how the values linking them together relate. At the core they proclaimed:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
— The Agile Manifesto
This Manifesto is said to be the foundation for the rise of Agile methods. The Processes associated with Agile today are such processes as Scrum, Kanban, Extreme Programming and alike. Defacto Scrum became almost synonymous with Agile. With roughly 85% of projects who are self-identified as agile are also self-identified as Scrum².
The Mainstream: Flaccid Agile and the Agile Industrial Complex
According to Martin Fowler³ up to 2016 Agile has become mainstream and is not any longer exotic or frowned upon on, as in the early days. But at the same time where Agile has become established, a growing number of people, including prominent signatories of the original Agile Manifesto are dissatisfied with the state of affairs. Fowler denounces a lot of modes of operations, labeled Agile as, Flacid Agile or Dark Scrum. According to him, agile values have been corrupted mostly for three reasons: First, an ever-growing agile-industrial complex⁴. This agile-industrial complex just briefly trains people, provides shiny few-day-certificates and pushes those people into consulting or managing positions. Secondly, a focus on methodology. This has led to over-emphasizing formal rules at the cost of the lack of recognition of technical excellence by able and experienced engineers. And thirdly focusing on projects instead of products. Instead of connecting developers with clients and focusing on quality, the deadline and finishing of the project is prized.
The Agile Split: Technical vs. Business Practices
Robert C. Martin puts forward a slightly different focus⁵: While he agrees with most of Fowler’s observations, he comes to the conclusion that there is a divide in the agile community. Claiming only the Software Craftsman- and women stayed the course and true to the original Agile values. Being one of the leading figures in the Software Craftsmanship school, he emphasizes the individual responsibility of the Craftsperson, to uphold the values of Agile and well-crafted software against any opposing forces. But he also acknowledges that the agile movement was overtaken by project managers who misunderstood it as a new methodology for managing software projects and not caring much about the guiding values it was founded upon. Where Kent Beck’s goal in 2001, for the Agile movement, was to “heal the divide between programmers and management” this divide prolongs today. Those Hoards of products managers, having certificates but no understanding of how developing reliable software works, took over the Agile movement and created the Agile Split: Practitioners of Software Craftsmanship following Technical Practices versus project managers pushing forward Business Practices⁶. Which led to the Programmers fleeing from Agile.
The Failure of Agile
The very first thoughts on this were brought forward by one of the original authors of the Agile Manifesto: In 2015 Andy Hunt posted: ”The Failure of Agile”⁷. And probably it was Hunt who originally coined the term “flaccid agile”, within his very own and sophisticated style of using the English language. According to him, flacid agile is just a half-hearted attempt following a few proven practices.
“We have large swaths of people doing “flacid agile”, a half-hearted attempt at following a few selected software development practices, poorly”
— Andy Hunt
This is because beginners of a new skill tend to follow context-free rules and are likely to become Agile zealots overemphasizing some formal methods linked to agile. Doing just stuff in which they already feel comfortable, instead of thinking for themselves. Rather than following agile methodology blindly it needs to be adjusted to one’s needs⁸.
Can Agile be saved?
As a freelancer in mobile computing, I have worked with a lot of companies of all different sizes. From small start-ups, fancy digital agencies and big companies like major internet providers, media companies and insurances based in Germany and Asia. Doing some kind of Scrum has been the defacto standard mode of operations for most of them since 2011. But I also worked with companies just during their transition period towards agile. And some companies I have visited before and after they claimed to be Agile. According to my experiences, all of the observations made by Hunt, Fowler and Martin are true.
And they all the critical practitioner s of Agile came to a similar conclusion: People are just not doing it right. But they disagree on the deeper reasons why, and how to fix that. So the question arises? Is Agile broken? Do we need to fix it? If so, how? Or is something downreaching going on and Agile won’t help us with that, no matter what?
In my experience, most companies doing good working with claiming to be agile also did very well without. At least Martin seems to be already aware of this.
“An Agile team will be Agile no matter how the project is managed. On the other hand, a team that is not Agile will not become Agile simply by virtue of a new and fancy project management strategy.”
— Robert “Uncle Bob” C. Martin
Having spoken to a couple of developers and colleagues, there seems to be a huge amount of frustration regarding Agile Software Development going on under the surface. It is good that prominent spokesmen of the Agile movement are starting to speak out. And it is not only them, but also a growing number of developers and technical blogs who carefully and hesitantly start to raise their concerns. On the other hand, there are still those members of the Agile Industry telling us we are just ‘doing it wrong’. And then there is the HR department of our next job, expecting you to have experience in working agile and to love doing so.
“It’s all too common these days to see arguments on Twitter or mailing lists with these rules-bound zealots arguing that ”you’re not agile” because you aren’t following the rules to their satisfaction.”
— Andy Hunt
So who is right? Let us investigate this by looking at the predominat defensive⁹ mechanism heard today. And into some misconceptions of the Agile Lore.
Dark Scrum: The true Scotsman
The true Scotsman is a logical fallacy often used as an “ad hoc rescue” and rhetorical figure described by Bradley Dowden¹⁰ and it is exactly what the apologists of the agile crowd are doing:
Person A: “No Scotsman puts sugar on his porridge.”
Person B: “But my uncle Angus is a Scotsman and he puts sugar on his porridge.”
Person A: “But no *true* Scotsman puts sugar on his porridge.”
Ron Jeffries article on Dark Scrum¹¹ is a good example of that. He realizes that Scrum regularly oppresses developers and describes the negative dynamics which typically unfold during the Sprint Cycle, exactly pinned down to Sprint Planning, Review and Retrospective. He explains the dynamics how the typical positions in a Scrum Team and the power holders within a company contribute to Scrum becoming dark and how oppression, mistrust and blame-casting rise where trust, self-organization and a long term perspective is needed. I am sure everybody having a fair amount of experience with Scrum can relate to this analysis.
Jeffries also points us to the hard and methodical answers which should be used:
- Incremental Development
- Acceptance Testing
- Incremental Design
- Refactoring
- Programmer Testing (typically done by Test Driven Development)
- Continuous Integration
Those are all well-known practices, that for certain will stay in the industry for good, even once it moves past Agile. And yes: They are useful and make us more efficient. But everything Jeffries describes also happens in teams which follow all those practices. But much unlike many others, he realizes that there is an imbalance of power between developers and power holders. Good software can only be delivered when the team follows their professional practices. But within Scrum it still is the responsibility of the developers to hold the management in check:
“Keep these people’s [the power holders] nose to the grindstone and their feet to the fire. That has always been what it takes to get software done. Scrum doesn’t change that.”
Jeffries is convinced Dark Scrum can be moved to *real* Scrum. By an effort of the developers and by improving education on Scrum in general.
It is a true Scotsman.
Tanner has written an article about “How To Spot an Agile Imposter [shop] at Your Next Interview”¹² worth reading. Ironically in my experience a lot of companies these days, are to some degree self-aware they are *not* doing real Agile, and will rather sell you something like: Well, we are “kind of scrumish”, we do “agile but not real agile”, we do “something like ‘Scrumban”¹³. But what is the point of telling us that? If Agile is about self-organization and change, there can’t be Real Agile or Wrong Agile.
It doesn’t make sense to fall back to finding the Real Agile™. There are Agile Values as described in the Manifesto. There are processes linked and co-related to Agile or Scrum. They evolved together. Saying “this and this isn’t agile” will get us nowhere.
The old Waterfall model: a modern myth
The last barrier defending Agile usually is the mythical waterfall. As soon as somebody criticizes Agile this person usually is reminded that at least is better than the old waterfall model of software development. Because before we were rescued by Agile everybody was using the waterfall model, which was totally wrong, Right?
Wrong! Ironically “the waterfall” never existed as such. The iconic slides illustrating the waterfall model first appeared in an article by Winston Roye in 1970 about problems in IT Project management of large software systems¹⁴. However, he never used the word waterfall as there was no such thing as a waterfall method at that time. Ogura Toshihide traced the term waterfall back to the article ”Software requirements: Are they really a problem?” by T.E. Bell and T.A. Thyler in 1976¹⁵.
Not only that, but already in 1970, Roye explained that good software cannot be created without iterations, testing thoroughly and gathering feedback from the users as soon as possible¹⁶. Maybe the waterfall is just Agile Lore used to scare the Kids who don’t want to listen.
This illustration by Winston Rocye became infamous as the waterfall model. However, he did not use the term “waterfall” in his paper and used it to show how the development process is *not* working.
The imposition of Agile Processes
One striking symptom of the Agile Crisis is the impositions of Agile on teams, which seems to be a common practice today. If Agile is so great and really gives more power and autonomy to the developers, why is it commonly imposed by upper management? In my experience, this is quite common and this impression is backed up by an increasing number of Agile Thought Leaders¹⁷.
“Imposing a process on a team is completely opposed to the principles of agile software, and has been since its inception.”
— Daniel Mezick
In his article on the Agile Industrial Complex¹⁸ in late 2016, which described these phenomena first, Daniel Mezick issues a clear warning on this kind of imposition from above and an #InviteNotImpose Movement is starting to get traction. At its core, this is once again the conflict between Agile Values and a focus on (imposed) business Processes related to Agile.
The truths of the agile manifesto seem to hold self-evident. But the shortcomings in analyzing why exactly they do not work out as planned is baffling. Are we all too soft on agile? Is the emperor naked and nobody dares to openly speak the truth?
Agile and the rationalization of software production
A big misconception, blocking our understanding of the nature of Agile, is the idea Agile is overcoming Taylorism as Fowler suggested¹⁹. Like the waterfall-myth, this does not hold true, upon closer inspection:
Writing and engineering software is (maybe soon: used to be?) highly qualified Knowledge work. It is inherently different from manufacturing goods and can not easily be rationalized. But working in the common Agile Processes as suggested by Scrum or Kanban is as close to working in an assembly line as writing software can be:
- Work is broken down into the smallest and easiest steps possible
- The pace of the work is controlled, measured and managed
The ultimate goal is to make the software-worker disposable by the process and even the gap between highly experienced engineers and less skilled members of the team. This is done for the benefit of productivity and predictable quality of the resulting product, in a way that aims to be as reproducible as possible. However, by doing this, Pioneers and Geniuses on the one end and Spaghetti-Script-Cowboys on the other end of the spectrum are not longer indispensable and are clearly out of date.
Slicing work and eliminating required skill as much as possible by leveraging pre-defined processes is the heart of Taylorism. But the software industry does not mass assemble the same model of a car over and over again as 20th century Fordism did and can not just copy the assembly line. And even manufacturing moved away from this approach. If we look at some other aspects of Agile Production processes, we see more aspects which also emerged in modern mass production such as:
- Controlling and maintaining high quality is in the center of attention.
- total control of productivity, with the goal of eliminating failures and maximizing resource usage
- integration of the staff into production by an emphasis on teamwork
- more autonomy in decision making on the site of actual production
- flat hierarchies and de-emphasizing of status symbols
These all are also elementary components of modern management methods, developed in Japan, and often labeled as ”Toyotismus”. Some Sociologists identify Toyotism as post-Fordism. Some as neither pre- nor post-Fordism²⁰. As with Taylorism, Toyotism is also a highly rationalized process and they have much in common.
The closeness of Agile Processes to Toyotism can also be exemplified by examining Kanban, which is the second most popular Agile Methodology today. “Kanban” is the Japanese word for Signboard, a central element in Japanese style just-in-time production and the Kanban applied in Software production is a direct descendant of the Kanban which emerged in Toyotism. It was first applied to Software in Seattle 2007²¹. But also Scrum shows a high degree of cousinhood with this management style.
Those lean and modern approaches in Management are the keys to the rise of Japans modern Industry and its high competitiveness in the Electronics and Automobile Industry. The Agile approach did *not* overcome Taylorism, it just implemented something resembling its modern form: Lean manufacturing — The Toyota Way.
The over-idealization of Agile needs to stop
The Agile Movement has been over-idealized. In part due to the original protagonist which were enthusiastic about the Agile Values. This enthusiasm was amplified by the Agile Industrial Complex, which quickly grew into a business. Considered sober, the shift towards Agile changed the ways of the software industry. But at the end of the day, it is just a rationalization in software production comparable to other rationalization processes in other industries. It modernized how we are doing our work, but it is more driven by the quest for efficiency than by values. Ironically the efficiency cannot fully unfold when the values are disregarded.
Agile seems to have reached a plateau. While in many places it increased efficiency and productivity²², the split between management and software workers seems to continue. A lot of people intuitively feel that the promise is broken, but it is hard to argue against rationalization in a competitive world. The old Hierarchies and power structures often hide behind flat Hierarchies and teamwork, but now they are harder to pin down and criticize.
Some problems of Agile might be due to a poor understanding of some practitioners and disregarding its values. But assuming that Agile is the end of history in Software Development is incredibly short-sighted. Assuming that dissatisfaction with Agile only arises from “not doing it right” keeps us from asking about its inherent shortcomings.
Going post-agile is not just looking for a new name for marketing and reinventing it or a semantic Diffusion²³ as Fowler argued. Postmodernism is not a semantic Diffusion of modernism. It’s something that followed and evolved from modernism, by better understanding the nature of modernism and its inherent limitations.
Agile started as kind of (project) management crisis and management still seems to be a hard problem in IT. We need to look deeper into the management crisis and the software crisis and revisit it. At the moment it seems, the ideals of the Agile Community are difficult to uphold in the harsh reality of the Business world. Calling upon the individual responsibility of every single IT professional just isn’t enough. Martin showed the industry is rapidly expanding and developers have less and less experience or flush in from other fields in vast numbers. In many cases they may lack the maturity to do so.
The distrust in Developers, which supposedly need to be disciplined and controlled by management, is another issue we need to address. It is inhumane, toxic and prolongs the nerd-cliché and won’t help in finding people to work in IT, which have a broader perspective and are not only good with computers, their languages and some algorithms and formal logic.
Trust is the basis for any good communication. But Trust cannot be demanded. It needs to be earned. This Problem is highly related to Agile as trust is essential for any Agile team. But it can never be imposed.
And the issue of trust cannot be addressed without looking at the problem of power. Agile, especially Scrum, is more about efficiency than about empowering developers and it is not a shift away from Taylorism. On closer inspection, this will be visible in every single conflict within companies trying to transform towards Agile. Quite the opposite is true: it makes people more replaceable and controllable and is a modern and competitive form of Management. But when the power holders become invincible and teams start to impose control on themselves, conflicts are avoided and responsibility becomes concealed. The humane promise of Agile is broken.
Within the scope of this article, I can only sketch out which questions need to be raised and where the Agile community has some blind spots. But I think the points, which we only reluctantly want to tackle, are probably most worthy of our attention. Keeping the software industry a great place to work requires a deep reflection on Agile and where we as developers and human beings want to go. The most important question still is how we want to unfold the power of the machines for the better of humankind. The digital Industry is already shaping the future of work, but how can we take better control of it?
This Article is a primer and invitation for discussion and a call for looking deeper into the Crisis. Please contribute and let me know your thoughts.
Discussion
I am very interested to hear your thoughts and experiences regarding Agile (especially beyond the “you are just doing it wrong” case.
- What is the core essence of Agile: values, processes anything other? And how does this show in real-world Agile? Does your understanding hold up with empiricism?
- Does Agile help teams communicate and grow?
- Rationalization through agile: Curse or blessing?
If you are interested in connecting with me, please do so and please drop me a short line where you are coming from.
[1] Robert C. Martin, 2016: The Future of Programming
[2] Empirical Evidence of Agile Methods, presented by Grigori Melnik, at Microsoft Summit 2017, see also Stack Overflow Developer Survey 2018
[3] Martin Fowler, 2018: The State of Agile Software in 2018.
[4] The term agile industrial complex was originally coined by Daniel Mezick in late 2016. Daniel Mezick, 12.12.2016: The Agile Industrial Complex
[5] The Future of Programming see Footnote 1, 1:07 and following.
[6] see Footnote 1 and Robert C. Martin, 28.8.2018: The Tragedy of Craftsmanship.
[7] Andy Hunt 5.6.2015: The Failure of Agile
[8] Andy Hunt 7.11.2016: Stop Practicing and Start Growing
[9] I assume the reader agrees with the defensive stance of agile advocates by now. A deeper discussion is beyond the scope of this article.
[10] see Wikipedia on No true Scotsman
[11] Ron Jeffries, 8.9.2016: Dark Scrum
[12] Tanner Worthham, 3.11.2018: How To Spot an Agile Imposter at Your Next Interview
[13] A mix of Scrum and Kanban
[14] Winston Royce, 1970: Managing the development of large Software Systems
[15] A recommended introduction on this misunderstanding was published by Tobias Pfeiffer, 2.3.2012: Why Waterfall was a big misunderstanding from the beginning — reading the original paper. Ogura Toshihide points to his own research (which is only published in Japanese) in the comments.
[16] See Footnote 14. Some of his thoughts, like his obsession with relying on excessive documentation, are not agile at all and would be considered clearly out of date today.
[17] see Footnote 5. A comprehensive and continuously growing list of Agile Thought Leaders who speak out against any imposition of agile processes (the term is a contradiction in itself) is compiled in the concluding part of this article.
[18] see Footnote 4.
[19] See Footnote 3.
[20] Manuel Castells, Das Informatiomszeitalter, 2nd Ed. 2001. Cf “Toyotismus” Kooperation zwischen Management und Belegschaft, multifunktionale Arbeitskraft, totale Qualitätskontrolle und Reduktion von Ungewissheit. The original trilogy on “The Information Age” was first published in English. Part 1 as “The Information Age: Economy, Society, and Culture, Volume 1: The Rise of the Network Society” in 1996.
[21] Rasmus Kaae, 19.12.2016: History of Kanban
[22] Empirical Evidence of Agile Methods, presented by Grigori Melnik, at Microsoft Summit 2017
[23] Martin Fowler, 14.12.2006: SemanticDiffusion
📝 Read this story later in Journal.
🗞 Wake up every Sunday morning to the week’s most noteworthy Tech stories, opinions, and news waiting in your inbox: Get the noteworthy newsletter >
