Don’t go chasin’ waterfall (development)

With all of the tools and solutions centered around agile best practices and DevOps adoption, it wouldn’t be wrong to assume that almost all companies developing software or shipping a digital service have fully transformed and transitioned away from old methodologies.

During the research phase for this, I figured it would be easy to find a company that still uses the “tried-and-true” method of waterfall for developing software. Plenty of folks reached out to me to tell me why companies are dumping waterfall, and yet I couldn’t get someone to delve into why they are currently using waterfall. While you won’t get an “inside look” at a company “doing waterfall,” I can leave you with this: Chase waterfall if you must, but don’t expect to become an innovator or market-game changer with this old methodology.

The way of waterfall

The definition of waterfall development comes from the way it steadily flows downward through the discrete phases of design, testing, production, implementation, and maintaining the software. Teams would figure out what they are going to build and what the requirements are going to be — it’s a specific and rigid process with requirements-gathering and is completely different than that of agile development.

According to Robert Reeves, CTO for Datical, there used to be designated people that would be in charge of gathering requirements. Then the design stage would be next, where a team would spend time designing the software, using pictures, a flowchart, and perhaps what the screen or user interface would look like, he said. Then came implementation, which is where developers wrote code, and then finally, verification, which is testing and then maintenance.

“The whole point and reason this is [a bad way to develop] is it assumes you have the perfect information,” said Reeves. “ When you gather those requirements, it assumes that nothing is going to change. What we learned with agile, if change was the only constant, then we need to make sure we can accept and deal with change.”

Reeves describes waterfall development to be the same way a team would build a bridge, a road or a building. Gathering requirements is the first step, followed by design, and build. The last step would be to verify that things will go right with the construction. Later, the bridge or road is maintained. Software doesn’t need to be built like a bridge, said Reeves, and “you don’t have to have your design so specific.”

However, since the waterfall model organizes the software development lifecycle into these clear stages, projects can be designed with great detail and as a result, the project’s scope is “robust and easy to follow,” said Flint Brenton, CEO and president at CollabNet.

He said that work builds incrementally using this model, creating a strong foundation for the project by the time it reaches the last phases: testing and deployment. As a veteran in the industry, Brenton has held roles at organizations that use waterfall, which is considered a “very traditional methodology,” he said.

Software teams eventually discovered there are problems with this old approach, and although it was the default methodology for decades, it’s nowstill considered to be incredibly pervasive, said Mark Levy, director of strategy for Micro Focus, an enterprise software company. Like Reeves mentioned, requirements are not always clear. Also, the market changes, customer’s expectations change, and there is zero feedback once you lock down a project.

Who’s using waterfall?

Even though agile methodology has been introduced and adopted over the past 15 years, waterfall development still remains a commonly used method in IT projects. According to Gartner’s IT Key Metrics Data report, the waterfall method as employed on 56% of development efforts in 2015, with iterative methods used in 21% of projects and agile in 23%.

Today, waterfall is a more expensive way to develop software, and you tend to build things that customers don’t want, said Levy. He noticed the prevalence of waterfall mainly around companies like financial banks, state and local utilities, and other companies that may have been around for 30 to 40 years and are trying to figure out how to transition from waterfall to something more agile.

“The profile [for waterfall] is typically state, federal government, financial at certain levels, like local or regional banks,” said Levy. “The big financial players seem to be ahead of the curve and they are exploring new [areas] since it’s competitive.”

Waterfall is a trusted approach to development today, and it may make sense for companies that have a project where the team’s culture and skill set are more traditional and the requirements are unlikely to evolve, said Brenton. This reasoning resonates with organizations in the financial industry, healthcare industries, or other organizations with strict regulatory considerations, since “waterfall offers such a solid foundation for meeting requirements early on in the lifecycle,” said Brenton.

“However, today we are seeing companies using or exploring more hybrid approaches, with a number of different development environments throughout the organization,” said Brenton. “ It doesn’t always have to be one way or the other, in fact, imposing a single style of software development enterprise-wide won’t necessarily solve the challenges of having a fractured and disparate development environment.”

Other organizations lag behind when it comes to adopting continuous integration, agile, and DevOps tools and practices. Levy said he sees local banks, as well as utilities, slow to adopt agile, but it depends on the market’s place in time. Retail is one industry that is further along because their market has been disrupted for awhile, he said, and because of this business disruption they are implementing a lot of DevOps and agile methodology. If they’re not, Levy said, they are at a complete competitive disadvantage.

Enterprises are also at a disadvantage because they tend to make high-velocity decisions slowly, according to a 2016 letter to shareholders from CEO of Amazon, Jeff Bezos. In his letter, he said that most decisions should probably be made with somewhere around 70% of the information you wish you had. Companies that wait until they have 90% of the information end up falling behind. Levy said that companies need to know that it’s not only about making decisions fast, it’s about the road to success and how that includes making wrong decisions.

“You make a lot of bad decisions on your way to success, and what is key is being agile and quick enough to recognize that and correct those decisions,” said Levy. “[When] you have something like waterfall methodology, which is very sequential and locked-down and time based, you can’t course -correct quickly.”

Becoming a market leader

In some cases, if a company really knows what they want to deliver and the project is too expensive to iterate and make changes, then waterfall is a possible method to choose, said Nate McKie, CTO of application and mobile development company, Asynchrony Labs. There are a few situations in the world, he said, where this is the case, where a company knows exactly what they need from the start. But, companies doing waterfall are going to have a hard time becoming a disruptor or innovator in the market.

“You need the ability to get things out there, do experiments, figure out what works and what doesn’t — waterfall just doesn’t allow for that, that’s not how it’s set up,” said McKie. “So when it comes to business, when it comes to trying to get ahead of your competitors, it’s going to be very difficult to do that, to do waterfall and be able to maintain that market lead.”

When discussing agile and waterfall benefits in the same context, both methodologies are often pitted against each other, said Brenton, where the speed of agile is emphasized and the simplicity and control of waterfall is highlighted. But, waterfall is working in some important ways for companies because it ensures that development is on track and requirements are met entirely. If followed properly, said Brenton, there are few surprises.

While he didn’t name specifics, Brenton added that many of CollabNet’s enterprise customers find that waterfall development is working out fine for them, and with a good foundation in place (waterfall or agile) executing a DevOps strategy is the next big step.

“We encourage any organization that is purely using waterfall today to consider exploring the benefits of agile and Scrum alongside DevOps,” said Brenton.

On the other hand, Datical’s Reeves said that there is no reason why a company should be maintaining the use of waterfall today, and frankly the companies he sees still in waterfall are doing it because “they’ve always done it this way.” He said even the companies that are doing hardware, testing products like missiles and planes are “breaking up” with waterfall development and finding ways to do agile.

“We are moving faster and faster with software development because of agile,” said Reeves. “We are moving so fast with how we write code and get it out the door and into the customer’s hands, whether it’s a mobile device or a service, and this proves that agile won. There is no way we could have done this if we were still doing waterfall development.”