Agility — Theory and Reality
I have seen many companies doing „agile transformations“. Compared to my understanding of agility, they are normally not very successful. I often have the feeling that going agile is just throwing in some new terminology and rituals, while keeping the organization inflexible and focused on the wrong things.
This article is a collection of personal experiences and assessments of agile transformations in organizations. There is no single truth. I have experienced how agile principles make sense on a smaller scale and cannot always match these experiences with how agility is lived in larger organizations. I am mostly interested in agility within the context of software engineering. But I would argue that most points I make also apply to other disciplines.
In this article I compare the principles behind the Agile Manifesto with their executions in the name of agility. I give examples where agile transformations fail to adhere to the principles and I give possible reasons for that.
Continuous Delivery
“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”
“Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.”
What it should look like
There are good books about how to achieve Continuous Delivery in software engineering. You need to automate your build, test and delivery processes to a high degree to be able to deliver your software quickly. Today we have lots of teams around the globe that deliver or even deploy their software changes within hours to production. The virtualization of infrastructure enables continuous delivery even on lower levels than software. Ideally one team delivers all layers needed to run their software at once.
How “Continuous Delivery” is perverted
I have seen agile transformations where teams were renamed to „Continuous Delivery teams“ without changing the team composition or any processes within the team. They did not introduce anything needed to deliver faster and more reliable. They defined the term „Continuous Delivery“ for themselves as having fixed deployment dates like once in a month. So customers who wait for an important bugfix have to wait for one month, even though the bugfix is simple. This is neither agile nor early delivery.
Late changes
“Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.”
How it should look like
How a team can process changing requirements depends on their path from requirement to production. If the path is long, the difference between development and production gets too big to react to changes that are stuck in development. Because you will need to resolve conflicts with other pending changes. Then you will need to test all the pending changes again, not just the changed requirement. In the meantime you will get another change request, making you repeat all the steps again. And it takes longer and longer, the more the development state diverges from the production state.
It is better to bring an experimental feature to production and maybe never switch on its feature toggle than waiting for its perfect implementation and blocking other important changes in the process. And for APIs it is easier and less error-prone to design your changes backward compatible instead of waiting for all clients to do the required changes.
A short way to production requires an efficient requirement process and appropriate code quality. With “appropriate” I mean a quality that minimizes the time your code changes take and will take. Bad quality will make quick changes either impossible or error-prone, while perfect quality requires too much time, sometimes for parts of the software that will never change.
“We need a fixed scope”
Agile methods like Scrum insist on having fixed scopes for a given time period like a Sprint. A fixed scope for two weeks is way better than a waterfall project. And way better than agile frameworks that want to fix scopes for a “product increment” that could easily be two or three months. But even in two weeks, how can you reject bugfixes for critical productive issues? Of course you can plan bug buffers, but how can you say whether the buffer has the correct size? And also, why do you introduce something like a buffer for bugs to stick to your agile process instead of changing your process. Agility is exactly that: Adapt your process to get the best results for your customer. Do not adapt your product by not fixing all bugs or wasting your time on discussions about bug buffers to satisfy your process (Scrum). Your customers will be more satisfied with a better product and a bad burndown chart than with the opposite.
Don’t get me wrong, it is very important that developers can keep their concentration on one non-changing task at a time. So your business people need to understand that. It is good to visualize the time you lose if they don’t.
Another strange effect of these fixed scopes is that teams that develop complex systems with many dependencies split their stories into a conception and multiple development tasks spread over multiple Sprints. They are doing that because the conception and development of all system components do not fit into one Sprint. So they plan to write down the requirements in Sprint 1, the solution design in Sprint 2, the development task for system component A in Sprint 3 and component B in Sprint 4. Finally the development of one Story lasts months and they are unable to react to requirement changes.
Agile teams
“Business people and developers must work together daily throughout the project.”
All required skills in one team
What do you need to get from an idea to a running system? You will need more than just developers. You will need business people who understand the business domain and the problems to solve. You may also require software architects, requirement engineers, testers, help desks and system operators (or devops). I would go even further than the manifesto’s principle: You should have all these required persons and skills in your team or at least you work with them on a weekly basis. The Agile Manifesto tries to mitigate the issue here, that developers that are detached from business do not get quick feedback such that the implementation gets into the wrong direction. Similarly, I believe that in a software lifecycle there should not be multiple teams divided by disciplines that deal with the same software. You should have all skills in one team that can cover the whole lifecycle. This is the only way that ensures continuous delivery and appropriate runtime aspects like customer support.
Agile transformations without organization restructurings
In larger organizations I have never seen that agile transformations were successful in this point. The agile transformation is often driven by the IT department or the development teams. They introduced Scrum in their development teams or Kanban in their IT support teams. But this is not really what makes the whole organization agile. Even if companies of middle size introduced agility over the whole company, the strong division by role was never really weakened. The way from a business idea to a final running software went always from a business department to a software architect or business engineering group, to a development team, to a testing team, to the system operators, and finally to the support team. The feedback loop between the teams was likely better than before the agile transformation but still not as effective as having a team with all the required skills.
Silicon valleys everywhere
„Build projects around motivated individuals.“
„Give them the environment and support they need, and trust them to get the job done.“
I think the most issues I described so far have to do with these principles. Most of the success stories of agile ways of working come from high-tech companies where all employees are highly motivated, educated and earn good salaries. However, not all companies are successful Silicon Valley startups. An average company has much more diverse staff, some of them are ready for an agile transformation and some are not. Thus, I think it is better for large companies to create their own small startups with motivated employees than transforming the whole organization. They can introduce agile rituals, terminology and motivate more employees to overtake more responsibility in their large organization too, but in the end the organization cannot change enough to take full advantage of agility. The larger the organization, the harder it seems to get agile at full scale.
Agility vs. Globalization
„The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.“
Remote meetings
In my experience, there is no other way to discuss problems and get a common understanding than in personal meetings. As a speaker, I will need to see facial expressions of all participants to see whether they can follow, whether there are questions or who is actually interested in the topic. It is easier to draw together a chart on paper than doing this in remote tools. You can just turn around and compare your drawing with another outcome of the session. Also haptic interactions and direct eye contact inspire more and interactions like commitments can be taken more seriously.
Have you ever seen developer xyz in person?
The reality often looks much more different. At least for me. My company and all my customers are placed in Switzerland. Here, there are many companies that do not have the expertise for Software development. So they need external companies to implement their software. Often they also need help for describing their requirements, planning the implementation and finally running the software. Most external companies have expertise in only one of these disciplines. And some disciplines like software engineering cannot be covered by Swiss companies only, because we do not have enough qualified engineers. And of course, it is tempting for managers to hire externals from countries with lower wages, so they might save costs. In my opinion, often they don‘t, but on Excel sheets it looks like.
Thus, my reality often looks like I am in a project team that involves multiple companies around the globe and project members have different mother tongues. This is quite the opposite of efficient information exchange. There is so much loss of information not only because you miss the advantage of face-to-face meetings, but also due to language and cultural barriers, translation work, technical issues and often indirect communication. With the latter I mean for example situations where the business expert talks to the architect, the architect to the developers, the developers to the testers, but there is no common understanding of the actual business problem.
We need more measurements
„Working software is the primary measure of progress.“
In recent years, software development has evolved fast. There are tons of new tools that support software processes. They support code reviews, different source code branching models, different kinds of static code measurements and also build and deployment automatization. This also exposed a number of metrics that were welcomed by managers to finally have numbers to assess teams and software. I have often seen teams that needed to fulfill measurements like test coverage, code style violations, number of reviews per code change and many others. This may be a good practice dependent on the type of software architecture, team maturity and organization structures. But it must always be accepted and introduced by the team itself instead of demanded from managers. Otherwise, the team lies too much focus on fulfilling these requirements instead of getting actual business value, because they are assessed more on these metrics than on business value.
Sustainability
„Agile processes promote sustainable development.“
„The sponsors, developers, and users should be able to maintain a constant pace indefinitely.“
„Continuous attention to technical excellence and good design enhances agility.“
Continuous Delivery requires sustainable development
In the chapter about Continuous Delivery we discussed the need of late changes. The ability to accept late changes might be tempting for business to neglect their duty of planning and bringing in important requirements in time. Thus, it is important that the developers explain how this behavior affects the development process in a bad way. To be able to keep delivering features at a constant pace, developers need to keep the pressure for new business value low enough to still be able to keep the code base in appropriate quality.
Agility does not mean changing requirements as often as possible
Developers that do not reject high pressure and cannot explain the problems that come with it are often used to organizations far away from agile, where the marketing team for example orders software from the development team, while there is not much interaction during the development process.
High pressure also often comes from development teams that try to work at a constant pace while new high priority requirements come at a rather inconstant rate. This unveils the break between the teams. It does not work well when just one team works agile while the other does not.
Simplicity can be achieved with good communication only
„Simplicity — the art of maximizing the amount of work not done — is essential.“
„The best architectures, requirements, and designs emerge from self-organizing teams.“
Complex architectures often come from bad requirements
I have seen completely different implementations of the same problem, where one implementation was much more expensive than the other while not providing any more business benefits. The difference was how the requirement was brought to the team. In one case, where the given business process was just brought to the development team, the implementation on top of the process was very complex. In the other case, the business process was designed together with the developers, who could already bring in their expertise that helped to maximize business value versus costs.
The problem with legacy systems and processes
One can argue that the difference between these two cases was just that one process was new and another one legacy. This „legacy excuse“ is common but also cheap. Everything we do now becomes legacy one day. And good legacy software does not have the same issues as bad legacy systems. The difference between the cases above could also be explained by one of the following points:
- Building software on top of an already existing (software or offline) system always accepts all the flaws of the existing system. If the development team could have adapted the business process before starting with the implementation, the legacy aspect would not be as dramatic. I have seen web applications that were overly complex, because the underlying process was optimized for the offline world. If the process would have been improved before, the system would not have been complex at all.
- Issues with legacy systems come from bad interface design. It is hard to change something when you do not know how this will affect the consumers of your interfaces. This can be mitigated by clear boundaries with well defined contracts, monitoring and tracing capabilities such that we know who is using our interfaces and how they are used. Deploy your changes and run automated tests on the consumer side to see the effects.
- If your legacy system consists of multiple modules with clear interface contracts between each other, it is easy to exchange just single modules when needed. This has a much lower impact and is less costly.
Retrospection
„At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.“
I think this is the easiest point to implement when introducing agile methods. Just do retrospectives on a regular basis. The only fallacy I often see is that only things on top of the agile method are adapted, not the method itself. This is wrong. You should be able to change everything that improves your work. Agility is just a tool, not a religion that needs to guard itself against other ideas.
Summary
I feel like agile transformations often improve an organization only a little bit, but most chances agility gives are left out. This is mainly because not every kind of organization can be made agile, agility is most successful in new, highly motivated and smaller teams. The team must be able to transform themselves and adapt to new needs. Instead, agile transformations are often based on a rigid framework that focuses on some fixed terminology and practices.
