Keep Your Agile off My Lawn, Please
When seventeen men met at a ski resort over twenty years ago to talk about how to create working software faster, they could not know the impact of their Manifesto for Agile Software Development on the industry. Now, two decades later, there are dozens of Agile books, conferences, podcasts, and certifications. Everyone and their grandmother is doing Agile. If you don’t, you will perish. Agile is the only way to stay competitive, they say. If it doesn’t work for you, then you’re doing it wrong.
In this article I argue against the notion of Agile being a silver bullet for all kinds of software development. Specifically, I believe it is actually harmful to apply Agile to projects with a clear roadmap in an enterprise context.
Where Agile shines — and why our team doesn’t need it
Before I badmouth Agile, I want to point out that it has its merits in the area it was actually designed for: delivering software as an ever evolving product with frequently changing requirements and direct customer feedback. If this is your use case, it totally makes sense to ship as often as possible, keep the team small, and document as little as necessary. You want quick feedback from your end users. New features need to be sent out continuously. In a competitive market, you cannot rest on your laurels. Time to market is money.
The software I work on can barely be called a ‘product’. My company builds commissioned ‘bread-and-butter’ software, often for the government. As a citizen, you will never see its UI or interact knowingly with it, and it has only a couple hundred simultaneous users. We get contracted for a few years to deliver a piece of software, with a clear picture in mind what it should be able to do at the end of the project lifetime. The users (sadly) don’t really have a saying in how the application should look and feel. Sure, they would benefit from keyboard shortcuts (the prevalence of web apps has apparently erased this feature from collective memory) or less time consuming animations, but that’s not what we’re being paid for to build. The customer is king, and in this case, the customer is the representative of a company or governmental body. If the software is horrible to use, but exactly as the client requested, it is considered a win.
Here’s the catch: we pretty much know all the requirements in advance. For example, if there’s a new EU law to implement by 2025 (and not a day before!), then we know we need these forms and send data to those entities after submission. It would be entirely reasonable to realize this kind of project in a waterfall model. There would still be some release cycle; you don’t want to wait three years only to learn you built the wrong thing. But there is no real feedback loop with end users as you are not allowed go live with these features before the big red button has been pushed. The only feedback you get is from the managers (or ‘product owners’) on the other side of the table.
So, if everything is known in advance down to the specs of the machines the software will be running on, would it really make sense to just work from release to release, only ever having the short sighted view of next sprint’s user stories? Would you hold sprint reviews to demonstrate API implementations for non-technical business people, which in a prime example of the law of triviality only object if they dislike a button’s color? Because that’s what we did.
Unfortunately, my company jumped on the Agile bandwagon and wants to cram every project into this format. The customers get told that Agile makes software development faster and allows them to get exactly what they want. I suspect many companies actually do agile development to mask budget overshoots and restrict liability; it is harder for a customer to complain about the end product if they were complicit in shaping it.
So, our project had to become Agile, and as hoards of consultants will tell you, Agile is essentially Scrum.
The cult of Scrum
My dislike of Agile was originally founded in my exposure to tacked-on Scrum.
Let me make one thing clear: Scrum is not Agile. In fact, it is actually anti-agile. Agility is about adapting your processes to your needs, whereas the Scrum guide tells you
Changing the core design or ideas of Scrum, leaving out elements, or not following the rules of Scrum, covers up problems and limits the benefits of Scrum, potentially even rendering it useless.
The message is: “Here’s a bunch of rules, please self-organize into following them to a T.”
There’s a whole industry behind Scrum. I am officially a certified Professional Scrum Master I, which means I memorized the alarmingly short Scrum guide and answered a multiple choice quiz online for which my company paid 150 bucks.
Reading the guide in preparation for the certification made me question whether I was in some kind of reality show. The guide looks like someone wrote it in their lunch break and didn’t bother to change Microsoft Word’s default font. There is lots of terminology, ‘product owner’, ‘sprint’, ‘backlog’, with only short, somewhat esoteric explanations and no practical examples. “Scrum is founded on empiricism”, the guide says, without any part of it showing how they came to all these principles. Why exactly should a Scrum team be 10 or fewer people, and a daily Scrum be 15 minutes long? Why not 12 people and 10 minutes? Why can only the Product Owner cancel the sprint? Why do I have to make up a ‘sprint goal’ when in my reality the goal is always to deliver the selected stories? Where’s the discussion of the pros and cons of these rules? Has this ever worked in practice?
Scrum has basically become a cult, with the Scrum guide as its scripture. Scrum masters with little to no experience in real projects swarm out and preach the good word. Scrum is a great method because the guide says so, and if you dare to do things differently, you’re not doing Scrum™. Your failure is your own inadequacy in applying the method.
Everything has to fit into a sprint. If it doesn’t, then your user story is too big, says the helpful Scrum master. Well, as it turns out, some work does not align nicely with the size of a sprint, and cutting it up would mean delivering a non-feature with half of the data missing or wrong. Meeting this self-imposed deadline results in a decrease in quality. Implementations are rushed and messy, features not thoroughly tested, non-functional requirements swept under the rug. Did the customer ask to get a piece of software every two weeks? No, we were the ones voluntarily pushing the method onto us.
There are no project managers in Scrum, no business analysts or tech leads. So you expect either the product owner (remember: ThEre cAN OnLy bE oNe) to translate all requirements into developer language, or all members of the team have to deal with the details of the domain as well. Don’t get me wrong, not understanding what you are coding at all is abysmal and I’ve seen more than one bug surface because developers were just blindly following data mapping sheets without knowing what they mean. But do you really want developers to read pages of EU legislation or technical specifications? Punish the entire team to partake in endless client meetings, or alternatively, rotate the member responsible so the client never has a reference person?
Documentation considered harmful
Okay, so we established Scrum is not agile software development, or at least the latter can do fine without the former. Should we then turn to the manifesto itself for enlightenment?
There are four principles in the Agile manifesto formulated as “prefer x over y”, followed by the statement
That is, while there is value in the items on the right, we value the items on the left more.
Unfortunately, this part is often ignored. The sentiment “Working software over comprehensive documentation” doesn’t mean you should throw all your documentation out of the window, rather, make sure to deliver value to your customers instead of wasting time writing technical handbooks no one but the author reads. Developers like to misinterpret this statement and use it as a “get rid of documentation free card”.
In the domain where I work in, documentation (more precise: specification documents) is paramount. I don’t believe in self-documenting software as the single source of truth. Sure, I can look at the code base, click through layers of method calls in my IDE or read tests scattered around, but a) it doesn’t give me the bigger picture how the requirements interact, and b) it tells me what the software does, but not what it should do. No automated test in the world can save you from implementing a story wrong if you’re also the one writing the test for it. A unit test only confirms the developer’s mental picture, but may not align with what the story author had in mind. How often do you encounter “oh I thought you meant…” and “I should have been more clear” in everyday communication?
In one of our projects the client requires us to write a detailed specification of the software as a Word document, amassing hundreds of pages in semi-pseudocode natural language. Everyone hates working with that document: the business analysts who spend weeks to keep it up to date with the user stories delivered in the last release, the architects who need to reflect refactorings in there, and the developers who get lost in domain speak.
However, it has value that cannot be disputed by anyone: it is a comprehensive document of what the software is supposed to do. If you find something strange in the code, you can look it up in the specification and tell if it was really supposed to run only on every full moon. It’s also a great starting point for new team members, giving them a bird’s eye view of the software. Screenshots of the user interface make it easier to correlate dialogues with implementations in the core layer.
Plans and processes
The next statement I don’t agree with is “Individuals and interactions over processes and tools”.
The most efficient and effective method conveying information to and within a team is face-to-face conversation.
says one of the twelve Principles behind the Agile Manifesto. This definitely hasn’t aged well, but even before the pandemic it was not always feasible to meet in person (we are a team of 40 in multiple cities across Europe). It’s also dangerous to convey information only verbally. If that person leaves your project, or just gets sick for a week, all knowledge leaves with them as well. Quick chats at the desk are great to clarify requirements or decide upon a technical solution, but make sure to document the what and why afterwards rigorously. Three months later, someone may desperately look for an explanation why the hell you did what you did. More often than less, that person is you. Tools like Jira or Mattermost are a great way to record discussions and have them remotely.
Developers hate processes because they feel boxed in. But this only proves that the processes they experience are bad, not that processes are inherently wrong. Processes informed by past events are safeguards against chaos. If everyone wonders why they do the stuff they have to do, there are two possibilities: either the process really is useless (cf. Scrum), or you did a poor job explaining its value (cf. Scrum). Most of the time it’s the latter. For example, instead of just saying “after the end of the business day following code freeze, no new features may be merged into the release branch”, clarify the purpose with “this helps to give the experts ample time to test all features, increases the stability of the build, and prevents you from being pressured into finishing your work this week at all costs”.
Last but not least, there’s “Responding to change over following a plan”. Once again, this makes total sense when shaping a product like a streaming service. You really do not know what you need to build a year ahead. It would be crazy to develop a five-year plan and stick to it no matter what happens. However, when realizing a project with known parameters (so and so many users, x thousand records per day, reports should update hourly), it would be equally crazy not to take advantage of this knowledge and plan for the future. Agile scorns the big design up front (BDUF, no it’s not a Pokémon), but in my experience, taking your time thinking about all the functional (and non-functional) requirements and corner cases really pays off.
A couple of years ago, there was a new big EU regulation coming up. Although we didn’t have access to the API yet, we knew all the use cases in advance, and how important it was to have a fallback in case anything goes wrong, otherwise a lot of people would have a very bad day. We shut ourselves in and discussed all the different ways how the system would have to act. We designed the data model, API contract and error handling up front. The software we built based on this design was resilient against a multitude of failure scenarios, performant, and most importantly, there was always a clear road to follow. There is truth in this proverb of unknown origin:
Weeks of coding can save you hours of planning.
There is one lesson from the agile software development manifesto which we strive to apply to our projects, and I would summarize it as:
Allow change, and reflect on what you did.
Yes, I didn’t say “welcome changing requirements” like the manifesto. Nobody likes changing requirements. How would you feel knowing the work of weeks was for naught? But you should at least be prepared for it to happen. It’s hard to code with change in mind, and sometimes it may lead to overengineering a generalization that will never be needed. My rule of thumb is: Listen to your customers; if they’re unsure about a particular part of a feature, make room for it to be replaced later. Also, some code duplication is fine if it means it is easier to change the order of things later, but never duplicate business rules like calculating a success flag.
Reflecting upon the past is essential in producing better software more effectively. If there are roadblocks slowing you down (“*sigh* a full CI build takes 90 minutes“), think about ways to remove them instead of treating them like universal constants. Likewise, mistakes and bugs happen, but when they do, don’t blame the people (and use git blame only to find the ticket, not the person). Instead, find out how it could slip through your safety net and adapt your processes to prevent the same mistake in the future. That’s called ‘just culture’, and the aviation industry is a great role model for it. The checklists and procedures which make flying one of the safest modes of transportation today are constantly updated after incidents. For example, after the Tenerife disaster in 1977 when two jets collided on the same runway due to miscommunication, the phraseology (the wording pilots and controllers use to communicate) was standardized and a readback of all instructions became mandatory. This probably helped safe many lives in the years following the accident.
Where do we go from here?
Is Agile really dead? Hardly so. (Albeit, if you use it as a noun like I did, Dave Thomas will be very sad.) However, Agile evangelists want to make you believe it is the only way to collaborate on writing software today, and they usually have a recipe in their briefcase you should follow right now to be successful. In reality, truth lies between those two axes. If an Agile approach doesn’t suit you, scrap it instead of forcing your work into a process. As said above, the core principle of agile software development is adapting your process to the problem, not vice versa.
I understand why it became so popular. On one hand, developers don’t like suits to tell them what to do. The prospect of organizing your work yourself seemed tempting. On the other hand, there is an inexplicable urge in every developer to try out the shiny new thing because some FAANG company used it successfully, be it a new framework, programming language, or management style. (On a side rant, I think that’s what’s keeping us from being worthy to call ourselves an ‘industry’. After having worked five years with the same faremwork and finally have mastered all of its quirks (something you’re expected to do in other industries before you get contracted), we are bored and believe the tech stack is outdated. The sheer hubris! Have you ever heard a plumber complain about installing kitchen appliances his whole life? A builder lament over the mundane work of casting cement walls?)
So, what to do? In the projects I work in, we still have to find a process that empowers us to write high-quality software without missing deadlines. Personally, I like the Manifesto for Async Software Development, a tongue-in-cheek version of the Agile manifesto. It’s by no means a framework for project management, but highlights how tools and documentation make it easier to work together remotely.
Eventually, we will have to create our own management style we find ourselves comfortable to work in instead of staying in the treadmill. In the end, that’s what Agile is all about: Change.