Agile or Waterfall? No, it’s all about good engineering and good business
Agile rigidity can be bad for projects, teams, and business
The Agile Manifesto was released in 2001. Since then, the Agile methodology has taken the software development world by storm and truly disrupted the industry.
Most companies in the business of software development follow agile methodologies in their organizations to varying degrees.
The growth in agile adoption is powered by an expanding ecosystem of advocacy groups, consultants, training and certifying bodies, tools vendors, and new process ideas.
Agile Rigidity
Agile approaches have produced spectacular results in certain contexts. At the same time, they have failed to deliver the expectations in many others.
Most of the published results and experiences with Agile have emerged from the Agile ecosystem. They focus on what and where it worked well. They study scenarios and experiences from the perspective of “hindrances” to agile success.
They provide recommendations on how others need to change and adapt for agile to be successful but hardly offer any perspective on how agile (or the specific agile process like Scrum, Kanban, or XP, to be accurate) may need to adapt to deliver value for the project, business, and team.
Agile coaches and consultants typically don’t acknowledge that “traditional” methodologies may work better for many types of projects or certain phases of projects. Or, that traditional methods enhanced with good practices inspired by Agile can be the best option for many projects and teams.
The 16th State of Agile Report reports that 20% of the respondents (pulled from the agile community) are “very satisfied” with Agile, 51% (the biggest chunk) are “somewhat satisfied”, and 28% are “not satisfied”.
According to the same report, half of the agile community uses a combination of methodologies — agile, waterfall, iterative waterfall, and others — for their needs.
The Agile Manifesto and its Principles
The Agile Manifesto and principles highlight the mindset and thinking that can address some genuine needs and realities of software development teams and consumers. However, they did not define processes and set compliance criteria for running Agile projects.
The root cause for most unsuccessful attempts seems to be the flawed understanding of Agile as a defined process aided by industry-leading tools as against embracing its principles and the value and insight that they offer.
Agile methodologies don’t work well in all contexts — projects, industries, and cultures — when understood and enforced strictly as a specific process (understood as a set of rituals).
In fact, this flies in the face of the Agile Manifesto that values “individuals and interactions over processes and tools” and “responding to change over following a plan”.
Jim Highsmith, representing the signatories of The Agile Manifesto, notes that they are against “pushing process for process’ sake versus trying to do the best for the “customer” and deliver something timely and tangible and “as promised””.
Good engineering and good business
I want to make the case that the choice of methodology or process should reflect good engineering and good business. The process need not be identical to any available standard prescription but should be tailored to serve the best interests of the project goals, team, and business.
Good engineering means coming up with the best technical solution when evaluated by a few critical parameters like efficiency, quality, usability, robustness, technical debt, scalability, and maintainability. Good business informs the project’s requirements and features, cost, resources, iterations, and timeline.
Interests of good engineering and good business tend to keep a check on each other, with business having the edge.
Where does Agile work very well?
Agile methodologies work particularly well for “pure” software development (i.e., not tied to any hardware or anything in the physical world) projects like web and mobile application development. A working prototype with a minimal set of features can be developed in a few weeks, which can be continually improved based on feedback from the user base.
It is apt for small and medium-sized teams with mature engineers capable of working independently.
Costs are likely to be more due to the active involvement of testers and business teams throughout the product duration. However, the business value may justify the cost for SaaS and mobile apps product companies. Testing automation can be pivotal for success in terms of both quality and cost management.
Agile teams will still benefit a lot by not ignoring proven steps like planning, design, and even documentation.
Ignoring planning and design can result in a lot of technical debt.
Clear, accurate, and useful documentation (not a synonym for “writing tomes”) helps improve the user experience in many cases. Lack of documentation impedes maintenance, learning, and knowledge sharing resulting in high training costs and heartburn.
Where do “traditional” methodologies work very well?
Agile methodologies are often compared with waterfall. After working more than a decade and a half in the software industry (though not the “pure” genre), I’d say that textbook waterfall belongs only in the “old world” industries where products are realized in the physical world.
In a software setting, the stakeholders almost always want something somewhat working at least every two months. No business (which really means business) pumping in hundreds of thousand dollars will wait year(s) to discover its fate.
“Waterfall” software projects have always been incremental (unless they are tiny, lasting just a few weeks). But the entire project is planned out well at the start which is a great advantage when the requirements and expectations are clear. So is the design phase which lays a solid foundation.
Waterfall can work wonders for delivering quality and features “as promised” within the planned budget and timeline when the team has prior experience executing similar projects along with clear static requirements and expectations.
This approach will not work well when the requirements are not clear or dynamic.
Teams working with waterfall should try to implement beneficial agile practices like continuous integration and testing, testing automation, and daily stand-ups (time-boxed). These can help them significantly in uncovering bugs early, avoiding surprises, and improving productivity.
Teams need to study their context well to choose the right methodology that will work best for them. Context includes the project’s nature and requirements, customer expectations, and the team’s training and culture. They should be open to adopting best practices from everywhere to enhance their methodology.
They should worry less about adhering to prescriptions from gurus. Instead, they should focus on delivering the best results for business, end users, and themselves.
