A contrast and comparison of the Waterfall, Agile, DevOps and Lean Software Development Methodologies.

Top 10 Reading List of the best books/resources recommended by leading theorists

The following reading list of books are recommended by some of the world’s leading theorists through my engagements with them on Twitter, namely, @courtneynash, @adamhjk, @anders_wallgren, @AndiMann, @andrewsmhay, @beerops, @Snipeyhead:

1) Effective DevOps (Jennifer Davis, Katherine Daniels)

2) What is DevOps? (By Mike Loukides)

3) Building a DevOps Culture (Mandi Walls)

4) The Goal (Dr. Eliyahu Goldratt)

5) The Theory of Constraints and its Thinking Process

6) Lean Enterprise (Jez Humble, Joanne Molesky, Barry O’Reilly)

7) Designing Delivery (Jeff Sussna)

8) DevOps in practice (J. Paul Reed)

9) The secret of DevOps: It’s always been about People, Not Technology, http://readwrite.com/2015/07/29/devops-people-not-technology/, (Adam Jacob, 2014)

10) Building a DevOps Culture (Mandi Wallis, 2015)

Introduction (DevOps — a culture shift)

Before the year 2009, developers and IT operations used to operate in silo’s. The common way of working between business and IT was typically kept very seperate which made it harder for delivery to take place more frequently with a common knowledge and understanding of the goal.

Dr. William E. Demming once said, “The secret is cooperation between components toward the aim of the organization.” (W.E Demming, 2014). This statement by one of the founders of the DevOps movement illustrates the importance that was realised for communication, collaboration and integration between software developers and IT operations (newrelic.com, 2015).

The DevOps movement is exactly that — a movement. There is currently not one official definition that can be pinned down, however, it is an evolution that keeps changing with the times. The movement was born out of the need to improve IT service agility and is found to underpin the philosophy found in the Agile manifesto which has specific emphasis on people and culture. This explains why DevOps aims to improve the collaboration between operations and the development teams (Gartner, 2013).

Patrick DeBois is said to have coined the phrase “DevOps“ in 2009, with the term deeply rooted in methodologies like Agile and Lean. DevOps is not a single methodology and is made up of different aspects that stress the need to have both efficiency and effectiveness (Gartner, 2013).

Rouan Wilsenach correctly captures what the DevOps culture is about in his article. Sure, Agile has broken down silos between requirements analysis, testing and development, but there activities like deployment, maintenance and operations have also suffered some separation which has indeed been addressed by the DevOps culture.

The main aim of DevOps has been to remove these existing silos and improve collaboration between development and operations. There have in fact been some important cultural shifts that have happened at both an organizational level and at a team level. DevOps influences Team culture by introducing an attitude of shared responsibility through feedback, automation and building quality in. At an organizational level, culture is influenced through having autonomous and no silos (Wilsenach R, 2015).

This assignment aims to provide a background on the DevOps movement and the culture that has emerged from this movement. Bearing in mind the fact that DevOps is not a single methodology; the background will highlight the different aspects of DevOps and provide a view of the footprint of the movement on a global scale.

The background of this assignment also aims to contrast and compare the Waterfall, Agile, DevOps and Lean Software Development Methodologies. Thereafter, an overview of the continuous improvements branch of DevOps will be discussed, with specific mention of continuous integration and continuous delivery. Thereafter, an evaluation of the contrast and comparisons will be discussed and a conclusion will be reached.

Background

The goal of DevOps is to deliver software into production. That is the single most important reason for having this movement. With it comes striving for continuous improvement and applying known principles to the SDLC and delivery pipeline (Sharma S, 2014).

DevOps Origins

In her article, Sanjeev Sharma of IBM explains that DevOps started as a movement in companies that existed mainly on the internet. With these companies existing on the web, it became very important that the environment was stable and always available as more and more customers where coming on to these online stores (Sharma S, 2014).

The development team would create the new software and required products and services and deploy them out to the live environment and they would have performed what is expected of them. The operations team on the other hand would be measured purely of supporting these functionalities that are put out by the development team and would therefore need to ensure that the website is available and it is stable.

Sharma explains that these goals were conflicting and it therefore become increasingly important to for development and operation to balance their measures in order to bring about success. In fact by becoming Lean, success was achieved by collaborating and discussing operational concerns much earlier in the development lifecycle and becoming more efficient by eliminating waste, duplications and having to do work again (Sharma S, 2014).

DevOps and Waterfall

The waterfall methodology was the first process model to be used. It is from the Waterfall model that an evolution occurred and led to DevOps cultural shift that is happening today.

Waterfall typically has phases that do not overlap but are well defined to determine what next is in the process. The rigid nature of this model ensures that there are specific deliverables after each phase and therefore makes it easier for control purposes. This has certainly been a baseline for the subsequent methodologies that have emerged later on. Waterfall works really well when there are small projects and certainly does help to have requirements crystal clear.

Typical disadvantages of Waterfall have in my opinion created an opportunity for growth into newer methodologies like Agile, Lean and now DevOps. Disadvantages include:

- Not suitable for larger or complex projects

- Only working software is delivered later on in the life cycle

- Changes are not welcome in later phases of the model

- Rigid in nature which created a block to integration and delivery

In contrast to Waterfall, DevOps offers and encourages continuous integration which really has evolved according to Margaret Rouse (Rouse M, 2016) from daily build as a standard to submission of work on a daily basis for a build to be conducted with every change. It allows for constant feedback on the status of software and that helps with the detection of defects early on in the process. Continuous integration came about later in the Agile methodology through Extreme Programming and focuses on concepts like committing code frequently, categorizing developer tests, staging builds, etc. which all can be used to benefit the Waterfall process (Rouse M, 2016).

DevOps and Lean

Sharma points out that it is only when teams are working together that they can work toward a common goal of becoming Lean. Just like the word suggests, lean is really about shedding the “fat” that causes the team to be inefficient and unproductive. It has been found that it this “fat” that causes bottlenecks in the delivery process, going against productivity which is the key goal of Lean and DevOps. Furthermore, these bottlenecks cause over-production and rework (Sharma S, 2014).

Also part of improving the delivery process is eliminating waste in the delivery process and reducing wait times. Some typical examples of bottlenecks within a typical digital team are:

- Environments randomly going down at different points of the day

- Developers, testers and product owners waiting for the environments to be configured and test data to be provisioned

- Meetings which cause teams to not be productive at the end of the day

- Environment cloning at go live usually breaking functionality which requires more testing to be done

- Wait time due to mandatory approval gates and go/no-go meetings

The lean methodology is in fact consistent with the DevOps culture and really aims to improve the team culture for the better. It is really important to identify and then shed off the “fat” that causes a team to be ineffective. Being lean is indeed the best way to get to the goal of delivering software into production.

DevOps and Agile

The manifesto for Agile Software Development aims to cover better ways of developing software by doing it and helping others do it. The values that have emerged from this work are (Pressman R, 2009):

- Individuals and interactions over processes and tools

- Working software over comprehensive documentation

- Customer collaboration over contract negotiations

- Responding to change over following a plan

Pressman defines agility as effective response to change, effective communication, drawing the customer to the team and organizing the team to give better control of the work performed. DevOps favours agility as it yields rapid and incremental delivery of software because the main goal of DevOps is to get software to the customer (Pressman R, 2009).

DevOps adheres to the general processes of agile that encourage the frequent releases of software, accommodate the customer as part of the development cycles and ensure better practices. The agile branch of DevOps will focus more on techniques that can be used when going agile as a transformation mean, like XP, Scrum, continuous delivery etc.

Continuous delivery is very effective in helping organizations to become lean while using the agile methodology. Because Agile is purely an evolution of the Waterfall methodology, it still keeps some elements from waterfall like documentation but introduces flexibility through the SDLC. Therefore, by using this concept it then becomes easier to weave in reliable and low risk releases and continually adapt software with real time customer feedback which both the development and the operations team can work on earlier on in the process before releasing a final batch of software (Thoughtworks Inc, 2016).

Benefits of DevOps

According to the 2013 Puppet Labs State of DevOps survey, companies that practice DevOps get more done and have been found to deploy up to 30 times more than their competitors. In addition to these incredible stats, less than 50% of their deployments fail (Newrelic.com, 2015).

One of the major benefits which stem from this movement is the shift in thinking and therefore doing things. The movement has challenged the traditional way of working and even challenged traditional roles that people in organizations have become familiarized with. The output therefore is a cross-functional team that is constantly collaborating to deliver both technical and business benefits. These benefits include (Newrelic.com, 2015):

- Continuous software delivery

- Simpler and less complex problems to fix

- Leading to a faster resolution of problems and shorter turnaround times

- Faster delivery of features into production (e.g. through frequent releases)

- Stability of operating environments will be improved (more uptime of backend services)

As illustrated in figure 1, the DevOps movement is ever evolving to ensure continuous operations.

Overview (Continuous Operations)

Continuous improvement is on the branches of the DevOps movement. Many companies use this method as a way to identify cost saving opportunities as it stream lines work and reduce costs at the same time (leankit.com, 2015).

Continuous improvement is viewed as both a formal practice (like the kaizen manufacturing) and just an informal set of guidelines. Companies can practice continuous improvement in their daily activities and project managers can in fact incorporate this way of working in project where there is usually an opportunity for improvement or events can be set up to achieve improvements. These events are known as Rapid Improvement and Value stream mapping. They can take up to five day and require the team members to drive the process by bringing to-do lists which will help focus the process (leankit.com, 2015).

Typically, the process for continuous improvement will start off with an understanding of what needs to be improved and conducting an in-depth analysis of the opportunities for change. Thereafter questions will follow in order to vote and choose the correct opportunities. There are techniques like the Kanban board that can be used to map out what already exists and possibly map the plan forward (leankit.com, 2015).

Evaluation

This section will evaluate a case study in the telecommunications sector, focusing on the pain points with their legacy system which were experienced by Vodafone Germany, the second largest telco in Germany. Vodafone Germany then partnered with Amdocs and HP on a successful nine month project to deploy a mediation solution that brought about benefits like exceptional performance, cost savings in hardware, an unmatched customer experience and a speeded up bill processing and fraud detection (Amdocs.com). This is real life example of how the contrasts of the different methodologies in fact works together to bring to life a new culture that both the organization and team adopt in order to successfully deliver working software quicker to the customer.

According to an article by Amdocs.com, Vodafone Germany has experienced an increase in the number of post-paid customers, (in the data sector specifically), bringing the number of customers to more than four million. This according to the report by Amdocs drives 100 percent a year increase in call detail records. The estimation that was incorrectly made however indicated that call detail records would grow by about fifty percent a year. The reality however is driven by a huge increase in mobile data brought about by a demand both in the past-paid and the pre-paid sector, mainly because of high speed mobile internet like the LTE mobile high speed broadband for fourth generation and the possible forth coming 5G (Amdocs.com).

The biggest pain point for Vodafone Germany was their legacy systems which were struggling to cope with the demand. The systems were designed and implemented at a time when voice was still the main focus and all that the system was meant to do was to handle voice data. This rigid legacy system then proved to be costly to maintain even though it could not handle internet and other application data.

Vodafone therefore partnered with the consulting house Amdocs to implement a mediation solution (Amdocs Mediation v8.1) that would complement their move to the Linux platform (low cost environment) and to integrate Amdocs Mediation with Amdocs Billing. They partnered with HP to assist with the systems integrations efforts, an important part of managing the project as per the PMBOK methodology. The project took nine months to implement from the concept phase up until go-live.

Contrary to typical projects of this nature within the Vodafone environment that take up to 3 years to implement, I am of the opinion that this specific project was such a success (and only implemented in nine months) because of their alignment to the DevOps movement.

For the first time, we witnessed different companies (Vodafone, HP and Amdocs) forming one cross-collaborative team working towards a common goal. This shift in the mindsets of these project participants bears testament to a Lean way of working. The project was also managed in such a way that Goldratt’s principles were implemented from the beginning. Firstly an opportunity identified to be changed was the way of working, which had a lot of potential waste to be eliminated (where duplicate work would have been produced within Vodafone, HP and Amdocs).

The plan to implement the work could have been drawn up using scrum boards, following the scrum process. The main aim of this way of working is to maximize efficiency and effectiveness within the team and all members of the team having a responsibility to contribute suggesting changes that are rolled out incrementally (as per Kaizen) even beyond the go-live date.

Conclusions

This research assignment has shown that indeed DevOps is a culture that is based on the evolution of the methodologies from Waterfall to Agile to Lean, rooted in Agile and Lean methodologies and that the DevOps movement is made up of different branches which all allude to the same principles; efficiency, effectiveness and getting software out to the customer.

This assignment provided a background on the DevOps movement and the culture that has emerged from this movement. Above all, it highlighted the fact that DevOps is not a single methodology with the different aspects of DevOps discussed and a view of the footprint of the movement on a global scale was also included.

Following the background, the research assignment provided an overview of the continuous improvement branch of DevOps. This paper also evaluated the case of Vodafone Germany in the telecommunications industry with a view of the Amdocs mediation solution project that they undertook.

With all the contrasts between the different methodologies, this case has effectively illustrated how the differences all work together in order to shift culturally and take on a way of working that ensures continuous delivery and continuous integration. Thereafter, the continuous improvement theories were used to evaluate how Vodafone Germany can use continuous improvement even after the successful development of their mediation service.

Finally, in the word of Laurie F et al, DevOps requires people, process and tools to promote a seamless collaboration among diverse but simultaneous users, all conversing in “different” languages; it will necessitate not just technology adoption but also a cultural shift (Gartner, 2013). In my opinion, DevOps has proven to be a cultural shift and organizations need to understand this and be willing and ready to change — and change more frequently as that is what will give a competitive advantage over the rest.