Making Agile At Scale A Reality
Applying sustainable principles in your organisation through trunk-based development, feature flags, a fully automated CI/CD pipeline and unified observability & control
Agile delivery methodologies and their benefits have been discussed, written, and debated at length till the proverbial cows have long come home. In here I have instead focused on an aspect of Agile that I have not frequently encountered, be it in discussion forums or related literature.
While a lot of literature on Agile centers on its practices, rituals and how its benefits make for more efficient realisation of an organisation’s goals, the studies do not often delve into the end-to-end changes required for Agile to be successfully implemented. Fundamental changes are required in the way software is developed throughout the Software Development Life Cycle (SDLC), ensuring delivery of working software at the end of every sprint.
When we consider implementing Agile at scale, it becomes an even tougher nut to crack as changes need to be delivered on a working software, and with multiple squads/teams engaged on it at the same time. Imagine, for example, developers across 12 squads working concurrently on a set of different but connected business outcomes.
This brings to mind the expression, ‘building a plane while flying it’ that captures the complexity accompanying rapid evolution of a platform, while operating it without a hitch.
The key points discussed here are the development and delivery practices that bring the benefits of Agile, ensuring early and continuous delivery; receiving frequent and regular feedback provides opportunities to make continuous improvements, resulting in a productive work cycle with frequent software updates.
One needs to look at the context of stringing complex business outcomes together, while straddling multiple stakeholders, user groups, and geographies. It is also important to bear in mind that that a massive change in the organisation’s landscape entails intensive discovery, redefining people’s roles & responsibilities, way of working, along with re-imagining processes and policies that unfold progressively. Last but not the least, the need for robust change management.
Trunk-Based Development
A key consideration for an Agile at-scale implementation with an underlying microservice based design is to adopt Trunk-Based Development. Traditionally, developers work on separate feature branches. This, however, creates at least two problems.
First comes the inability for the developers in a squad to refer to the same and the latest code base and have an appropriately detailed understanding of the changes being made by other developers. This also prevents developers from getting timely feedback on the code they have delivered. The second would be what can be called ‘code-merge hell.’ And imagine if hellfire has to be endured in every sprint!
Salvation in such cases can be delivered with Trunk-Based Development, where all developers in and across the squads work together on a single code trunk, committing code snippets to the trunk as soon as a piece of code is compilable without needing or waiting for the whole feature to be complete. The key need for the code trunk is that it maintains integrity and is always successfully compiled/ready to be compiled i.e., always in a release ready state.
Feature Flags
A practical challenge faced while delivering working software at every sprint and having a code trunk that is always ready for release, is that the full feature functionality may not be complete and ready for use while some parts and underlying user stories have been accepted and pushed into production. Hence, there is a need to introduce feature flags in the code being written and delivered. This allows for a switch-on, switch-off capability for the working code to be pushed into the Continuous Integration/Continuous Deployment (CI/CD) pipeline and delivered to production with a blue/green deployment approach.
One might argue that this increases coding complexity. However, if feature flags are designed at the onset, there would be minimal increment to coding complexity and effort. The dark deployments enabled wherein code is pushed into production but not immediately used, serves another critical need that I will delve into further below. Needless to say, feature flags are a significant help in switching off parts of the software, without impacting the larger functionality or platform should the need arise.
CI/CD Pipeline
As one visualises a consistent albeit fast-paced evolution of the software, a few analogies spring to mind. One of which is the most complex yet perfect of all systems — the human body. As the various parts of the human body work in harmony, changes essential for growth and healing need to be introduced at speed, but in a subtle manner so as to not cause trauma or disruption.
The same applies to any complex, fluid and working software. Software changes are best introduced in small, frequent doses, without user disruption. This is enabled by having a highly automated CI/CD pipeline that brings together practices such as automated code scans, automated testing, blue-green deployments, automated deployments and reversion capabilities.
Given the high frequency at which change is being introduced - multiple times in a day, it is important to eliminate manual intervention in the CI/CD pipeline. For example, apart from automated code scan utilities, there could be a need to introduce scripts that cater to specific aspects of code quality. The need to complement change at speed with stability cannot be overemphasised — it is critical to ensure a software is reliable; hence the need to introduce changes in small doses, that does not rock the boat.
Another imperative to ensure stability is to make sure that the difference in code and functionality, between the test and production environments, is kept to a minimum. This ensures that a large quantum of change is never introduced at once.
Unified Observability and Control
My last point extends beyond the software code itself. While we build software that is constantly evolving, there is a need to complement it with a few other constructs that provide unified observability and control, such as:
a) A robust set of automated test routines that can be executed regularly and at will. It is important for these to be constantly upgraded when there are new functionalities added to the system, or when an issue is raised;
b) A set of measures and metrics that effectively monitor user feedback and tickets. Measures include application rating score and verbatims, while metrics include mean time to acknowledge, mean time to resolution and mean time between failures (MTTA, MTTR and MTBF); and
c) Diagnostic utilities that effectively monitor the performance of each component, services of the software in real-time, hotspots and anomaly identification, and pre-empting issues and impact.
These need to be enabled via a set of fit-for-purpose tools, be it for embedding Test Driven Development or Behavior Driven Development practices, performance testing, or enabling the visibility of application metrics or diagnostics.
The above constructs help the software platform balance the speed of changes, flexibility, cost efficiency, resilience, stability - achieving the holy grail of “Faster, Better, Cheaper.” The above also provide essential inputs to the error budget, a key measure for calibrating application stability and for having the optimal balance on platform stabilisation vis-à-vis speed of change.
In summary, the Agile rituals of sprint planning, scrums, and retrospectives are one part of the picture. There are many key aspects, such as the way we design, develop, and deploy software that are critical in making Agile at scale a sustainable reality. Experiences differ across products and organisations, and I look forward to your observations and insights from your own journeys.
Soumya Ghoshal is a managing director, Group Technology & Operations at DBS.
