Waterfall and agile don’t mix, right?

I find myself thinking of a developer in the year 2099, frowning into her holo-screen, coding in whatever language people in 2099 code in. Maybe she’s my great-granddaughter. I’m wondering how she organises her work, how she fits into her team, how she knows the bigger picture of what she’s delivering (if she knows the bigger picture). Is agile a word that’s at the forefront of her mind? I’m guessing it’s not. Assuming it’s not, is there a word she takes value and encouragement from to contextualise, describe and inform the way she works? And is there another word she kicks against, another approach she rails against as outdated and fundamentally wrong-headed?

Maybe she’s even reading this as some kind of engineering history lesson.

Introduction

In our world (back in 2019) everyone wants to run agile deliveries, or at least deliveries they can call agile. Agile is certainly the current orthodoxy within software development: such approaches allow you to respond and adapt to change while delivering the minimum necessary functionality for a production release. Agile is the way we organise our software projects now, whereas waterfall was the way our parents organised their software projects (whether they gave it that name or not). Hence agile seems to have won some kind of victory over waterfall, at least on the surface.

However, I’ve come to realise that what people really want (or at least what many project managers and stakeholders want but don’t talk about openly), is the promised certainty of a waterfall project but delivered in an agile manner.

There’s a paradox at work here: we like certainty of outcome, agile is aimed at dealing with uncertainty and change and thus should enable a more certain outcome but, seen from the very start of a delivery, agile can bring its own uncertainty and inherent change. Often we’re not certain up front exactly what is going to be delivered at the end of an agile delivery due to variability of scope. Thus we end up with a mix of agile principles in waterfall structures, agile techniques being used in a waterfall context.

And this is a bad thing, right? Waterfall is old-fashioned, right? It’s ‘obsolete thinking’ and it’s a bad idea to mix the two disciplines. Right?

We’ve recently built a new product following what was on the face of it an agile methodology: we planned on a weekly basis, broke all of our work down into items that took less than a day to complete and provided releases at the end of each sprint. However, it was clear as we reached the end of the delivery that, despite many of the agile techniques we used, overall it was fundamentally a waterfall delivery — the scope turned out to be fixed. And, to be clear, the project was a success. We’ve got lots of happy users.

Nevertheless, I’ve now got a couple of developers on my team with eyebrows cocked, feeling like they’ve been tricked into a waterfall delivery, feeling short-changed somehow.

So, is there anything inherently wrong with combining agile principles with an overall waterfall delivery? Doesn’t this mean you have to compromise those agile principles? Great-granddaughter in 2099, do these questions mean anything to you from where you sit?

Here’s a good thing about agile: creativity thrives on constraint

For our recent delivery the constraint of working in weekly sprints was undoubtedly a key driver of creativity, forcing us to come up with solutions to problems quickly and, in certain situations, to improvise. We continually broke problems down into manageable chunks and tackled them bit by bit, together as a team.

To an extent this guarded against the tyranny of perfection that can occur in waterfall deliveries, the tyranny that can present significant challenges to a project going live or a product being delivered on time/budget; for example, doing too much design up front or repeatedly iterating on a design in an attempt to reach the perfect solution and thus forgetting about the bigger picture. Our planning and refinement sessions meant we had a good understanding of all functional and technical areas across the team.

This is in counterpoint to recent arguments that agile practices stifle creativity. Creativity of course thrives on constraint (great-granddaughter, do you still have sonnets in 2099? People still write them in 2019 but not so much as in olden times). As we approached the end of the delivery we had a product that was ready to go live safely in advance of the planned release date.

Here’s a good thing about waterfall: sometimes it’s necessary to do more detailed design up front

Thriving on constraint is all well and good but sometimes a thorny problem or significant product change requires you to sit down, engage with it and spend a decent chunk of time working on a detailed design up front without the distraction of sprint-planning, sprint backlog refinement and so on.

There are a number of core areas of any delivery which can end up proving difficult and which, in fact, form the foundation of any system we build (i.e. we’re aware of their existence from the outset). Those areas don’t necessarily respond well to being broken down into smaller chunks and not being rigorously thought through up front, particularly where we need to get sign-off from a client.

We’ve been lucky in that one of the teams we interact with works at a slightly lower cadence and is thus in a better position to pick up more detailed designs and longer-running developments. I’m trying to work out the most productive way for my team to do something similar whenever necessary and, crucially, to recognise when it might be necessary to do more design work up front and flag it up appropriately.

Here’s something to keep in mind with agile: don’t iterate too much

There’s a clear cost to repeatedly iterating on the same feature. Yes, it’s possible to take a different approach to deliver a feature, try an initial approach one sprint, a different approach the next sprint, pivot and pivot again if necessary, but this can come at a cost (sometimes that cost is another feature). Without clear requirements documented up front there can be a tendency to iterate too much and agile processes don’t immediately provide the controls needed to curb repeated iteration.

In our latest delivery I confess there were times when I did miss the relative luxury of having a lengthy design document which outlined exactly what the product would and would not do (ten years ago I never thought I’d consider one of these documents a luxury). This is the manager’s desire for clarity and certainty up front.

Great-granddaughter, I hope you remember to take a step back every now and again, look up from the holo-screen and take time to think.

Here’s something else to keep in mind with agile: don’t forget to brush your teeth

The tendency to rush headlong into sprint after sprint can occasionally blind us to basic team hygiene factors, i.e. those simple and sensible things we need to do in order to look after the team and make sure everyone is supported to do their job properly. This includes making sure we have adequate training material for new people joining the team, providing adequate visibility of what our processes are and so on. The agile emphasis on delivery can at times provide a distraction from these hygiene factors and put a pressure on team members to figure things out as they go, all under the banner of being a “self-starter” or “independent learner”. We need to remember that not all people learn in the same way and we need to be able to accommodate all styles of learning.

There are lessons we’ve learned from years of waterfall deliveries that shouldn’t be forgotten just because we’re working at a higher cadence or with a different team structure. One of those key lessons is that the on-boarding process for new team members is of critical importance.

It’s true that the learning curves can be steep and time pressured when building new products. During our recent delivery we took time to create a development initiation period on the team, effectively treated as training where new members can learn on the job and their time isn’t included in our total for planning purposes (i.e. we reduce their ‘speed’). We’ve got more work to do here.

Conclusion

So, great-granddaughter reading this in 2099 (I’m now thinking you’re definitely reading this) I hope it’s clear that this is still a work in progress — we’re still figuring out the best way to organise ourselves, the best way to structure our work and we’ll always be doing so. That’s the nature of craft and practice — you never stop learning. I imagine you might be doing the same.

Being pragmatic and non-dogmatic about the way we work has always seemed the best way for me. Great-granddaughter in 2099, I hope it might also be the best way for you.