Effective Agile + Waterfall Hybrid Project Management
Project management is the practice of initiating, planning, executing, controlling, and closing the work of a team to achieve specific goals and meet specific success criteria at the specified time. (Wikipedia, April 2019)
This article is rather long! So here’s a table of contents to skip right ahead to the section that you’re interested in
A brief history of project management
The field and practice of software project management is one that has been relatively slow to evolve.
In 1917, Henry Gantt developed what we now know today as the Gantt Chart — an innovative scheduling diagram that helped visualise and manage the various tasks of a complex projects and their interdependencies.
In 1957, 40 years later, Dupont Corporation introduced the Critical Path Method (CPM), a technique that maps out the dependant sequence of activities have the least scheduling flexibility.
In 1962, the US Department of Defence formalised the concept of the Work Breakdown Structure (WBS), a means of breaking down projects into manageable hierarchical smaller scopes. This structure also provided the framework for detailed cost estimation and control.
In 1969, the Project Management Institute was founded. They published the Project Management Body Of Knowledge (PMBOK) in 1996 and have last updated is with its 6th edition in 2017.
In 1986, Takeuchi and Nonaka introduced the concept of Scrum to the world in their HBR paper. Designed specifically for software development, Scrum did away with the rigid structures of project management and replaced it instead with an evolving prioritised backlog and adaptable iterative scopes of work that are estimated solely by the engineering team.
In 1997, Goldratt introduced the concept of Critical Chain Project Management (CCPM) which looked at project schedules through the focus on resources instead of tasks to optimise on delivery times.
In 2001, the Agile Manifest was published with its 12 core principles
Project Management Today
In a somewhat simplified view of the project management landscape today — there exist 2 primary schools of thought. The more traditional enterprise project planning that we refer to as Waterfall Project Management versus the more recent Agile methodologies.
The PMI and its PMBOK does not particularly promote one methodology over another, but for the most part — perhaps for historical reasons — it’s processes are anchored in the more traditional waterfall approach, geared towards traditional large scale enterprise projects.
As of the latest 6th edition published in 2017, a new section has been added to recognise the newer adaptive project management processes of Agile, Scrum, Kanban, Lean, XP and more.
PMBOK
There are a combined 97 processes defined in the PMBOK. They are grouped into 5 Process Groups and 10 Knowledge Areas
The 5 Process Groups
- Initiating: Obtaining authorisation to start a project, defining high-level scope, understanding stakeholders, identifying risks/assumptions/constraints.
- Planning: Detailing requirements, constraints and assumptions; creating a project management plan, creating work breakdown structure, developing project schedule; determining budget, resources; establishing communication channels
- Executing: Obtaining/managing project resources; executing tasks in the project plan; implementing approved changes according to the change management plan; performing quality assurance; managing communications and stakeholder management.
- Monitoring & Controlling: Control schedule, cost, quality, communication, risk, procurements and stakeholder engagement. Measuring project performance, managing changes; ensuring that deliverables confirm to quality standards; updating the risk register and communicating project status.
- Closing: Finalising all project activities, archiving documents, obtaining acceptance on all deliverables and communicating project closure; Transferring ownership or deliverables; Obtaining financial, legal and administrative closure; distributing the final project report; collecting lessons learned; archiving project documents; measuring customer satisfaction.
The above processes are grouped into the following 10 knowledge areas.
- Integration Management
- Scope Management
- Schedule Management
- Cost Management
- Quality Management
- Resources Management
- Communications Management
- Risk Management
- Procurement Management
- Stakeholder Management
Waterfall Project Management
The term waterfall comes from the visual outline of a Gantt Chart. We start at the top left and move through a project progressively leftwards and downwards through the tasks that have been estimated, sized and laid out until we reach the end of the project.
The processes of the PMBOK detail strategies for how to go about managing this process effectively in the 10 knowledge areas that they have defined. At the top level, the 5 stages to the waterfall are Requirements > Design > Implementation > Verification >Maintenance which are analogous to the PMBOK’s Initiating > Planning > Executing > Monitoring > Closing.
The waterfall methodology (and its more enhanced variants) is more commonly practiced in enterprise teams because it lends itself well to projects where research and discovery are executed separately from the project itself. Once a project starts, the requirements are clearly defined along with the risks. The scope is divided into smaller tasks via a Work Breakdown Structure, then plotted out into a project plan that is communicated to management, product managers, engineers, marketing etc so that everyone knows when each deliverable will be met.
It is also often the case with enterprises that have strict financial and procurement controls that projects need to be properly budgeted for and cost controlled. Knowing the full project plan, resources and WBS lends itself well to cost projections.
The waterfall methodology has reputation for being inflexible. While change request management is an inherent part of the methodology (and doing it effectively is the mark of a solid project manager) the process is still heavily structured towards accountability and CYA. Any changes big or small to this plan needs be evaluated for its impact on the overall scope such that a new plan can be redrawn up and re-communicated
As a result of this inflexibility, many use the waterfall model as a case study of what not to do. Yet while few organisations do pure waterfall these days, slightly modified models are still very prevalent.
Agile Project Management
Agile is a set of guiding principles that were initially designed by 17 software engineers. These guiding principles have spawned a few methodologies, the most popular of which is Scrum. Scrum is an empirical process that relies on transparency, inspection and adaptation to deliver incremental scopes of work.
A scrum team consists of 3 roles
- Product Owner: Defines the work to be done and their priorities that is managed in a Product Backlog
- Development Team: Determines how much effort is required for each item in the product backlog and consequently how much work can be completed in each Sprint. Also determines how the work will be done.
- Scrum Master: Facilitates the big and small processes of Scrum
These 3 roles work in concert to deliver work through the following Scrum events
- A Sprint: A time-boxed container for all other Scrum events during which a done, releasable product Increment is created. Usually 2 weeks to 4 weeks long
- Sprint Planning Meeting: At the start of a sprint. A planning session for the Product Owner to brief what the next objectives and priorities are; and for the development team to determine what can be delivered during a sprint.
- Daily Scrum: a 15 minute event for the team to determine what work will be completed in the following 24 hours. Often takes the structure of discussing what was completed, what comes next and what blockers have been identified.
- Sprint Review: At the end of a sprint. A review session to inspect the work done and adapt the product backlog accordingly.
- Sprint Retrospective: At the end of a sprint. A review session focused on the people, relationships, processes and tools involved in the sprint. Identifies how to better improve the quality of work processes.
There are a few other aspects to the methods of Scrum, but this is the essential process. The product owner composes a backlog of work to be done and only details what the currently known requirements are for the upcoming sprint. The full project requirements are not planned out up front. The team works to deliver functional iterations with each sprint, independently deciding what they can manage to best deliver. At regular intervals the team gathers to inspect the work that is done and give feedback such that the backlog and following sprints can adapt accordingly.
The beauty and allure of doing things Agile is that no effort is wasted detailing future aspects of the product backlog that may change as roadblocks emerge, market conditions shift or management decisions change. The team is focused solely on working towards what has been agreed to be the work of the upcoming sprint — and that is the limit and extent of any wasted effort.
Scrum works well for projects where the full requirements are yet to be clear and discovery/experimentation are part of the project process. It works well when you are looking to get to market quickly because each sprint should result in a releasable product. It works well when you are geared to reap the benefits of getting to market quickly by incorporating user data and feedback into actionable items of your backlog.
But if you work in a corporate environment — this probably gives you pause. How will I know when a project will complete? How will I budget? More importantly, how will I communicate what we can deliver? Worse yet — if you are trying to contract an agency how do you formulate a contract if there isn’t a concrete project plan, deliverables and costing?
These questions are exactly why we have designed a hybrid approach that brings the benefits of Agile to enterprises.
Agile + Waterfall, the 2359 way
Agile Hybrid methodologies are not new, they even have a small dedicated section in the PMBOK 6th Edition, but they are poorly defined and there is no “handbook” equivalent for a hybrid process. Resultantly, many adaptations result in a process that cuts both corners. We’re here to share our method and what we’ve found to work well.
As a disclaimer, I don’t think this process is for everyone. In fact, as far as we have used it, it applies to a very specific use case: a working interaction model between enterprise product teams and agencies.
In designing a project process, we first need to be very clear on what our priorities and objectives are.
Priorities, explicitly in this order.
- People: The development team and stakeholders always come first
- Product: What we deliver must always be of the highest quality
- Process: The rules to how we go about it must lend itself to the above 2 priorities
Objectives, with equal weight.
- Adaptable to changing requirements (Agile)
- Frequent testable deliverables (Agile)
- Ability to communicate what the project will deliver (Waterfall)
- Budget-able, cost-controlled projects (Waterfall)
- Delivering a high quality product
- Simple to understand, easy to adopt
- Works well for small and large projects
The short and sweet of this process is that at the high level project and product management processes we adopt a mostly waterfall practice while on the day to day execution we adopt the agile principles. As such, we can use the 5 step waterfall framework as the overarching structure to the engagement.
1. Requirements
At this stage the objective and the high level scope is discussed. Ideas are brainstormed for different approaches to best achieve the stated objectives. Various assumptions and testable hypothesis are put forth that are to be explored through the project process. Unlike traditional waterfall, we do not presume to know the solution. We only agree on a potential solution to the problem statement.
As per Scrum, an empirical data driven approach to problem solving is advised. Data from previous phases (if any), stakeholders and end-users should be collected into a pool of knowledge that is used to formulate a solution. UX workshops, user surveys and the like are tools that would be well employed at this stage.
The processes that we put forth here are
- Understanding the stakeholder. Pain points, needs and objectives.
- Researching potential approaches and solutions from a user, business and technical perspective and the potential challenges to each
- Defining a mutually agreeable high level scope and approach to the problem
2. Design
Design here refers to solution design (not just visual design). The objective of this stage is to suss out the feasibility, scope and cost of the potential solutions. It is counter-intuitive to attempt to effectively determine effort and costing without a full product specification being drawn out — but it is possible.
In our approach, we list all of the expected functions as User Story titles to form the agreed scope of work. It is important at this stage for the product owner to make clear as many of the requirements as is known (such as the complexity of animations, required scalability of the system, etc). This skeleton of a product backlog is used by the development team to estimate the effort on each Story, listing the assumptions as they go.
This stage is the opportunity for both the product owner and technical team to decide which assumptions are and aren’t acceptable as mere assumptions and need to be fully explored before moving ahead — the approach to do so can vary from executing mockups for certain screens, drawing out certain user flows or even executing proof of concepts.
Remember, the top priority is people — so both the stakeholders and the technical team need to be satisfied and comfortable with the scope and assumptions before moving ahead.
We estimate the amount of time required to get to the first workable iteration and then arrange the remaining stories by priority. The resultant gantt chart will determine the projected completion timeframe. For most cases we recommend the first sprint/iteration to take no more than 1 month and the overall project to take no more than 4 months. If the project is larger than that — we recommend approaching it as a multi phase engagement where each project is a phase.
From an agency perspective — we now have the 3 artefacts that are sufficient to compose a contract.
- A communicable list of deliverables based on an agreed set of assumptions
- An estimated timeline
- The relevant costing
The product owner and the development team agrees that that the product backlog functions as a starting point of what the development team can commit to, but the product backlog still remains flexible and adaptable so long as the final deliverable is met within the timeframe set forth.
The processes that we put forth here are
- Exploring and detailing critical requirements that cannot be left as assumptions
- Composing an agreed product backlog that consists of User Story titles as the scope of work
- Composing a timeline of sprints that outlines where each user story is prioritised
- Determining the cost to execute the backlog
- Listing of remaining assumptions and risks that will accompany the backlog
3. Implementation
Here is where we go into a fully agile mode of operation. We adopt from both Scrum and Kanban methodologies.
Kickoff: A single sprint. To start, the product owner and project manager will populate the details of the User Stories involved in the first sprint. A sprint planning meeting is held where a feasible Sprint Backlog is negotiated.
The team will begin execution of the initial sprint and conduct daily scrums. During daily scrums the team shares what has been done, discusses what is to be done and raises flags on potential blockers.
At the end of the first sprint, the team conducts a sprint review where the first iteration — a fully functional iteration — is presented. This session allows for immediate inspection and feedback of the first iteration by stakeholders as well as identification from the technical team on whether any of the original assumptions no longer hold water.
This feedback in turn allows the product owner and project manager to then adapt and adjust the backlog accordingly with written agreement from both parties on which user stories might be removed and replaced moving forward.
Then: Kanban Mode. After the first sprint is complete, we no longer run sprints. Instead the product backlog serves as a Kanban board where prioritised items are drawn down by the development team such that a testable build is released as often as possible — usually on a daily basis. The process of collecting feedback and adopting change requests becomes an ongoing daily activity. We no longer live by the rigidity of sprints.
On a weekly basis, the project manager, product owner and other stakeholders meet up for a review session where the agenda is to collate feedback on the deliverables and process.
Ensuring that change requests are incorporated and communicated to all stakeholders is the responsibility of the project manager. Ensuring that any change requests that are adopted are able to meet the final deliverable deadline is also the responsibility of the project manager. Ensuring that the final deliverable solves the intended objective of the project is the responsibility of the entire team.
The processes that we put forth here are
- An initial sprint planning meeting
- A single sprint to reach the first iteration
- An initial sprint review meeting
- Iterative daily builds based off a Kanban backlog
- Weekly review sessions on deliverables and process
- Change Request management
4. Verification
Unlike the waterfall model — this stage runs both parallel to and after the Execution stage.
The first type of verification is user feedback. User feedback should focus whether the solution being delivered addresses the objective and problem statement. This should be constant and continuous through the execution process — not at the end.
The second type of monitoring is Quality Assurance. During execution a QA engineer should be involved to draft test plans, test each iteration and adapt test plans as change requests are accepted. The QA is the gatekeeper to ensure that each story is implemented according to the acceptance criteria.
Once the first iteration is released, the QA engineer then writes automation UI tests to keep up pace with regression testing once daily builds start to flow.
At the end of the execution stage, the QA engineer is responsible for executing Internal Acceptance Testing while the project manager is responsible for executing User Acceptance Testing.
The processes involved in this stage are
- User feedback towards resolving the objective and problem statement
- Draft initial test plan
- Running of test cases with each iterative build
- Execution of IAT and UAT
5. Maintenance
Formal sign-off and acceptance on the deliverables is relatively standard. We also focus on a couple key activities
- Maintaining the people relationships: Gathering and evaluating feedback on people satisfaction and process to determine action points that will refine the process
- Maintaining the product: Managing uptime and scalability of the product; Evaluating how effectively the delivered product resolved the original problem statement and potential areas of improvements
While the project execution process contains iterative loops, the 5 stages of the entire process can also function as one iteration of a larger loop. Post-release, real-world behavioural data should feed directly into the next requirements gathering stage that will kickstart the subsequent project cycle.
The 2359 way
This post only shines a spot light on how best to blend the high level processes involved in our Agile + Waterfall Hybrid practice. The detailed strategies to execute each of these individual processes draws from existing processes within PMBOK, Scrum and Kanban.
The benefits of running this hybrid model are
- The product owner and stakeholders are familiar and comfortable with the enterprise waterfall engagement model — avoiding internal corporate inefficiencies.
- The engineering team are familiar and comfortable with the agile development model — avoiding development productivity headaches.
- Consequently, running this model requires little education to either party on how the project will be managed.
The constraints of running such a process are
- You need a strong project manager who is able to work as a frictionless bridge between both the product owner and development team
- You need a strong development team that does accurate estimations that all parties can trust.
(Without an effective project manager or trust between parties, the engagement will quickly fall apart with either the product owner getting burned on the deliverables or the development team burned on their costing.)
I hope this model will help more enterprises understand how to adopt agile practices into their organisations in an effective and relatively painless manner.
Ben has been in the project management field for 9 years, with experience managing projects for an in-house digital team at Apple Daily and more recently with the digital consultancy and agency 2359. He has managed projects with global companies including FOX, Disney, Jardine Matheson and more.
