Contrasting the Waterfall Model, Agile, Lean and DevOps
Continuous Delivery, Continuous Integration and Culture
Introduction
A lot of discussions in the software development and IT project management communities are often around software development methodologies. The waterfall model, a traditional plan-driven approach to software development has been around for decades. Critics argue that the waterfall model lacks flexibility to accommodate customer changes and that its linear stages to software development are not people-centered, do not encourage customer collaboration and leave no room for creativity nor innovation. To improve software development, individuals have adopted methodologies that focus on customer collaboration, continuous delivery, constant feedback and communication between developers, customers and users while delivering software incrementally in small releases. This has led many individuals to becoming advocates of agile way of thinking.
However, in order to improve IT service performance and to give organizations competitive advantages, we need more than just simple agile methodologies. Gartner declared that “DevOps movement was born of the need to improve the agility of IT service delivery and emphasizes people and culture and seeks to improve collaboration between development and operations teams while seeking to remove the unnecessary impediments to service and application delivery by making use of agile and lean concepts” (Wurster, et al., 2013). For this reason, DevOps can be considered as the integration and application of different software development methodologies, operational processes and social psychological beliefs for transforming IT service delivery. In this perspective, DevOps is a new way of thinking, a spirit and a philosophy for transforming organizations.
We compare agile methodologies to lean concepts and practices so as to understand the relationships between these two ways of working and see how they encourage continuous integration and continuous delivery of IT services and how the combination of these two methodologies create a new culture, philosophy and spirit for development and operations teams. When it comes to telling which software development methodology is good, better or the best, it all depends on the view of the proponents of the methodology, the type of system to be developed and the culture of the organization.
The waterfall model vs the agile way of thinking
The waterfall model, also referred to as the software life cycle is an example of a plan-driven approach to software development. Ian Sommerville in his book entitled “Software Engineering” supports that the waterfall model is a linear approach to software development where the results of each phase is one or more documents that are signed-off and that the following phase should not start until the previous phase has finished. He also supports that in the waterfall process model, you must plan and schedule all the process activities before starting work on them (Sommerville, 2011).
While the waterfall process model focuses on following well-defined plans and detailed documentations, agile is considered by many IT professionals as a way of thinking and not a methodology. Is this true? Well, a group of software practitioners advocated a set of principles based on best practices and their previous success and failure experiences with many software development projects regarding what works and does not work. Each one of these highly-experienced professionals had their own different philosophies or way of thinking about how they approached software development. They developed the manifesto for agile software development which states the values and principles of agile software development philosophy (Misra, et al., 2012).
As we read from the above quote, the individuals who created agile practices and principles termed the output of their thinking as a philosophy (a way of thinking) and not a methodology. Philosophy informs everything we do in a method (Roberts, 2012).
So why is agile a philosophy and not a methodology?
In my humble opinion, it is because agile principles denote the methods used in agile practices and these principles or methods result from critical empirical thinking which is a philosophy subjective to individuals. Another reason why agile is not a methodology but a philosophy is that agile does not refer to a single method but encompasses different software development methodologies and IT project management practices such as scrum, extreme programming practices, pair programming, etc.
Agile manifesto and principles vs the waterfall process model
You cannot compare agile software development to the waterfall process model without mentioning the agile manifesto and principles. The agile manifesto states the values and principles of agile software development while broadly comparing them to what is believed to be the primary focus of a plan driven approach to software development. The following is the agile manifesto as literally stated by its 17 authors (Beck, et al., 2001):
“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.”
Thus, the waterfall model is based on the spirit of strict conformance to executing software development stages in a linear fashion from A to Z due to the assumption made that requirements can be clearly specified prior to developing the system. This had worked with the development of operating systems, embedded control systems and large and complex systems where a detailed plan is required. In contrast, the spirit of agile philosophy relies on an incremental approach to software specification, development and delivery (Sommerville, 2011). Agile is the driver for success and innovation and emphasizes individuals, working software, customer collaboration and rapid responses to customers’ needs by accommodating their later changes to requirements throughout the development processes.
When can I choose the waterfall process model over agile methods?
It is proven that in principle, one should choose to use the waterfall model only when the requirements are well understood and unlikely to change radically during system development (Sommerville, 2011).
While agile methods are being widely used by individuals, not all projects are suitable for agile methodologies. The reason why proponents of the waterfall model argue that agile methodologies degrade the structure of many systems is because some agile practitioners do not want to admit that they can also make use of the waterfall model in software development. Safety critical systems and embedded control systems require prior adequate and consistent planning of what have to be included in the release of the system and how the system will operate. For instance, making changes to the initial design of an aircraft control system in the later phases of the development process may result in the introduction of bugs that can comprise the reliability and safety of the system under development.
Despite the fact that agile methods such as scrum which leaves the software project scope flexible to accommodating later changes to software requirements, there is still a little bit of planning. The waterfall model depicts the traditional SDLC. Every software development methodology makes use of the SDLC but in a different way. While the waterfall model follows a linear approach to executing the SDLC, agile methods rely on an incremental approach to software specification, development and delivery.
Why choose agile way of working over the waterfall model?
The twelve principles of agile software development stated in the agile manifesto describe in broader details the reasons why one may choose to follow an agile approach rather than the waterfall plan-driven approach to software development. Agile practices create an organizational culture that gives importance to customers. In order to explain the main reason why one may choose agile over the waterfall model, we compare agile principles to the waterfall model as follow:
- While software delivery in the waterfall process model is toward the end of the contract, agile methods focus on satisfying the customer through early and frequent releases of valuable software integrating the customer’s prioritized functionalities.
- While later changes to the system specifications in the waterfall model is difficult to integrate as this may involve redesigning the all system, agile methods welcome changing requirements even late in the development of the product.
- While the waterfall model places emphasis on documentation, agile methods focus on frequently delivering working software to customers.
- While development teams work in isolation away from business people, agile methods encourage collaboration between development teams, business people and customers.
- While the waterfall model places importance on processes and tools, agile methods focus on motivating individuals by building a culture of trust to get the work done and providing all the supports needed by the team.
- While the waterfall model uses documentation to communicate the plan of what has to be done in a software project, the advocates of agile methods believe that face-to-face conversation is the most efficient way of conveying information.
- While documentation is used as a metric to assess the progress of the project, agile advocates believe that working software is the primary measure of progress.
- While processes of the waterfall model lead to ineffective communication between development teams and customers, which is considered as one of the main problems that lead customers to being dissatisfied, agile processes create a culture that promote sustainable development by encouraging collaboration between sponsors, developers and users throughout the development life cycle.
- While the waterfall model emphasizes on following a well-established plan which does not keep in mind potential later changes to the system requirements, agile methods advocates believe that best designs emerge from self-organizing teams.
- Finally, while the waterfall model put emphasize on improving development processes and tools, agile advocates believe that success is achieved when individuals assess their performance regularly and adjust their behavior accordingly.
Agile or lean for improvement? Maybe both for success and innovation?
Phil Abernathy states in a video that the four most common workplace challenges are:
- Poor quality of product
- Low morale and employee engagement
- Unsuccessful projects
- and high cost of operations
All these result in unhappy customers and low profitability. These are the problems that lean and agile solve. These ways of working solve these workplace problems by developing a culture of collaboration and continuous improvement with the help of simple practices that change behaviors. They improve the culture and the fabric of the way for ensuring fast delivery of tangible results from day one. Agile started off as a software development way of working and lean started as a production and operation system that helped Toyota overtake GM and become the number auto company in the world (Abernathy, 2013).
Moreover, Phil Abernathy supports that agile and lean are both based on the same organizational values of trust, respect, accountability and honesty. Above this, agile and lean are based on a common set of principles of collaboration, iterative delivery, continuous improvement, empowerment and transparency. Agile and lean have a set of practices that are focused at all times on delivering customer values. This creates the fabric leadership behavior and determines the culture and success. Agile and lean are ways of working as opposed to methods. Together they give the necessary understanding to continuously improve. When we combine agile and lean we get operational efficiency, project excellence, product and service quality and a great working culture that attracts the best people and gives a competitive advantage that is tough to imitate or beat (Abernathy, 2013).
Application of lean and agile
Rod Bray explains in a video that agile practices can be used to solve problems that fall in the quadrant of complex and that lean principles can be applied to solve complicated problems where expertise is required by pulling the waste out (Bray, 2013). The term agile and lean are often used interchangeably by some IT professionals. However they are different in their application. Agile was born of the need to improve software development process models which means agile is applied on projects. In contrast, lean was born of the need to improve production system operations and this means that lean practices are not directly applied to projects but to operations and processes involved in managing projects and delivering products and services.
After reading about the origin of the application of lean and agile, it is time to know why individuals are no longer just making use of agile methods but combining them with lean practices. In the following section, we analyze the transition from agile to lean and how lean practices are used to improve agile methods.
From agile methods to lean principles and practices
Agile methods have been used for years. Scrum approach to project management and extreme programming technique in software engineering have been used a lot. However, many individuals who have lot of years of experience in agile way of working are now seeing that agile methods have to be improved. Others argue that “lean is a necessary progression for organizations planning to scale up agility from the project or team level to the organizational level, which agile methods fail to address satisfyingly” (Smits, 2007). For this reason, the agile community is moving toward using lean software development.
What is lean thinking? Womack and Jones define lean as “a way of thinking that allows companies to specify value, line up value creating actions in the best sequence, conduct these activities without interruption whenever someone requests them, and perform them more and more effectively” (Womack and Jones, 1996). As with agile methods, lean thinking comes into practice through the integration of a set of principles and practices. Mary and Tom Poppendieck realized that the lean principles that was used to improve manufacturing and production could be adapted to fit software development. The lean principles relevant to software development are (Poppendieck and Poppendieck, 2003):
- Eliminate Waste
- Build Quality In
- Create Knowledge
- Defer Commitment
- Deliver Fast
- Respect People
- Optimize the Whole
Identification and elimination of waste from the process with respect to customer value is the primary and guiding principle of lean way of working (Wang, et al., 2012). For this reason, while agile methods have improved software development with scrum and extreme programming practices, lean principles improve agile practices. As the primary principle of the lean way of working is to eliminate waste, it is supported that “Lean thinking forces teams to eliminate anything that is not adding value to the product or service development process. Lean organizations try to avoid unnecessary meetings, tasks and documentation. They also try to eliminate building things that are not needed, which can be challenging because nowadays, market needs change more rapidly than ever” (Chan, 2013).
Thus, when we say from agile methods to lean principles and practices, what we actually mean is not that agile methods are no longer being used by individuals but what we mean is that agile methods are integrating lean practices and principles so that these methods can scale to organizational level by improving all the development and production system operations. In the following section, we explain how lean principles are used in agile.
Lean within agile
As declared by Chris Sims in an article entitled “Fowler: Agile Vs Lean Misses the Point”, “many of the people who developed the current crop of agile methodologies were strongly influenced by lean manufacturing and the ideas behind it. This can be seen in the many commonalities between lean and agile, including (Sims, 2008):”
- People centric approach
- Empowered teams
- Adaptive planning
- Continuous improvement
On top of the commonalities between agile and lean ways of working as identified by Chris Sims, we can also add collaboration, continuous integration, simplicity and fast delivery of product and services. However, if we read from the evolution of agile practices, we will see that agile methods were influenced right at their beginnings by lean principles.
Thus, we use Kim Chan’s statement to conclude this section by saying that “lean and agile share many of the same principles and many agile principles are borrowed from lean thinking” (Sims, 2008).
Difference between lean and agile
Régis Medina in a video from the “Lean IT Summit 2013” contrasts these two methodologies by defining agile as a “process for building the right products with the minimum overhead in an unclear environment and lean being a set of practices for developing the right skills to deliver more value with the less waste.”(Medina, 2013). Thus, while lean and agile practices are based on some common principles, they are different approaches. Agile methods are practiced to improve software development processes whereas lean way of working is used to improve production system operations which lead us to concluding that lean principles are not only used in project management processes but in the entire organization from project or team level to the organizational level.
Transforming IT service delivery with DevOps by using agile and lean practices
Agile way of working such as scrum and extreme programming have improved software development. Lean software development principles are not only improving agile methods but also the entire production system operations. However, when it comes to improving IT performance in order to give organizations competitive advantages, we need a new way of thinking, a new way of working that improve all the production and management processes and operations from the team or project level to the organizational level while encouraging collaboration between all the individuals involved for fast delivery of valuable products and services.
For this reason, a new culture, corporate philosophy and way of working is emerging. This way of working integrates agile methods, lean principles and practices, social psychological beliefs for motivating workers, systems thinking for building complex systems, continuous integration and continuous improvement of IT products and services for satisfying both customers and production and development teams. This new way of working is DevOps.
What is DevOps?
There are several definitions given by different individuals practicing the DevOps culture. To quote a few of them we give the following definitions for DevOps:
- Adam Jacobs in a presentation defined DevOps as “a cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners”. This guru of DevOps also states that DevOps is reinventing the way we run our businesses. Moreover, he argue that DevOps is not the same but unique to the people who have practiced it (Jacobs, 2015).
- Gartner analysts declare that DevOps “… is a culture shift designed to improve quality of solutions that are business-oriented and rapidly evolving and can be easily molded to today’s needs” (Wurster, et al., 2013).
Thus, DevOps is a movement that integrates different ways of thinking and different ways of working for transforming organizations by improving IT services and products delivery.
How to successfully integrate DevOps culture in an organization?
We cannot talk about DevOps in a corporate environment without integrating a set of principles and practices that make development and operations teams work together. For this reason, Garter analysts support that DevOps takes into account several commonly agreed practices which form the fundamentals of DevOps practices. These practices are (Wurster, et al., 2013):
- Cross-functional teams and skills
- Continuous delivery
- Continuous assessment
- Optimum utilization of toolsets
- Automated deployment pipeline
We are DevOps when we are:
- Agile (continuous integration and continuous delivery)
- Lean (waste elimination and value stream mapping)
- Making use of the best social psychological believes to motivate people and manage our organizations.
- Making continuous improvement of products, services and processes
- Creating a culture that invites and encourages development and operations teams to work together.
- You can add on this list based on your own professional experience.
Continuous integration and continuous improvement are one of the drivers of the DevOps culture. In order to understand how DevOps as an organizational culture is practiced through agile and lean principles, we discuss in the following sections, continuous integration and continuous delivery.
Continuous integration
Martin Fowler defines continuous integration as “a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early” (Fowler, 2006). Continuous integration is one the drivers of agile practices. In fact, we can just say that DevOps makes use of agile best practices to encourage continuous integration through iterative development. As stated in one of the principles of agile way of working, “software is delivered frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale” (Beck, et al., 2001). The main advantage of continuous integration is that it makes it easier to detect bugs earlier during development instead of waiting to integrate everything in the end which makes it expensive to fix if there are any bugs.
The scrum approach to software project management encourages continuous integration throughout a sprint. As developers are working on a product backlog, they continuously integrate their codes in a shared repository every day. This integration is facilitated by the use of source and version control software which allows developers to work in different machines even locations while still integrating their codes into the same repository that can be used later to deploy the entire system. Continuous integration is a must for fast and frequent delivery of valuable software.
DevOps and continuous delivery
As stated in the fundamental principles of agile practices, “the highest priority is given to satisfy customers through early and continuous delivery of valuable software” (Beck, et al., 2001) . However, DevOps is all about continuous delivery as this way of working aims to transform IT products and services delivery within organizations. Jez Humble argue that “implementing continuous delivery means making sure your software is always production ready throughout its entire lifecycle — that any build could potentially be released to users at the touch of a button using a fully automated process in a matter of seconds or minutes” (Humble, 2010).
However, continuous delivery demands continuous integration for valuable software to be delivered to customers anytime as it may be requested. In order to make this happen, DevOps put emphasis on automation. Automating the deployment pipeline is a must in DevOps as continuous delivery depends on it. Thus, DevOps is all about being agile and eliminating waste by lining up value adding activities and executing them in a sequence. As DevOps aims to transform the way organizations develop and deliver IT products and services, it needs agile principles to do so. Continuous integration and continuous delivery of valuable software are the keys to delivering products and services to customers more frequently and satisfying the customer with valuable software at a click of a button.
Difference between DevOps and Agile
Agile methodologies and the waterfall model are approaches to software development. In contrast, DevOps is a corporate way of working that strive to promote collaboration between development and production teams by making use of different methodologies. This means that DevOps is not just about being agile or lean. It is more than that. DevOps includes agile and lean principles, systems thinking, the ITIL framework, social psychological beliefs that help organizations to motivate people and many more things.
Relationship between DevOps, agile and lean
Professor Josef Langerman in a video states that “there are three ways of working which are: DevOps, lean and automation. DevOps for us is agile, what I mean by agile is continuous delivery, continuous integration. It is all of these modern engineering practices” (Langerman, 2015). Thus, DevOps as a spirit, way of thinking and way of working would not be possible without agile and lean principles. Another thing to mention is automation. As supported by Gartner analysts, automated deployment pipeline is one of the fundamentals of DevOps (Wurster, et al., 2013). These analysts also support that automating deployment pipeline enables a frictionless deployment. Thus, you are DevOps when you are agile and lean.
Conclusion
A new way of working that encourages collaboration between development and operations teams to work together is emerging. This is DevOps. It is the next big thing in software development and IT products and services delivery. Through the use of lean and agile principles, systems thinking, ITIL and social psychological beliefs for motivating people, the spirit of DevOps is transforming the way IT products and services are delivered. DevOps is all about people and automation. It integrates the best parts of different software development methodologies and best practices in combination with some of the best social psychological beliefs for motivating people and encouraging the entire organization to work together in order to deliver valuable products and services that satisfy not only the customer but also the people involved in the development and delivery of these IT products and services.
Waterfall or agile? this depends on the type of system to be developed and the culture of the organization. Agile and lean are based on some commonly shared principles and values that emphasize people by encouraging customer collaboration and fast delivery of valuable software through continuous integration and continuous delivery. However, while agile methods are widely used as an approach to managing software development (scrum) and as a software engineering practice (extreme programming), lean principles were originally designed for improving production systems operations. Despite the difference in terms of application, agile methods use lean principles.
