Agile Software in the Real World: Responding vs Planning
The Agile Manifesto has four lines. This article focuses on the last:
Responding to change over following a plan
— Agile Manifesto
Preventing too much planning.
Planning a software project is difficult. We all want to start projects knowing they will succeed, but how can we be sure?
Do customers really want this product or feature? Can we build the product in the time we need it? Is there technical complexity in the product that we can foresee if we plan well? Will the market change by the time we finish?
Some businesses try to answer all these questions before starting a project, so they don’t waste development time on projects that are doomed to fail.
But many businesses fail to realise that how much planning to do should depend on the project:
- How quickly can your development team release something and get feedback from customers?
- How much existing feedback do you have from customers and products?
- How damaging will it be if you release a bad product?
- What is your ability to measure customer engagement after release?
Many companies follow the same pattern of planning for every project, despite the above changing for every project. This means inevitably some projects will have too much planning and others not enough.
Why is planning sometimes a bad thing?
- Planning delays delivery of your MVP and the market may move on — you may miss your window and competitor products may take your share before you can deliver (delayed time-to-market).
- Planning can waste time discussing things which may not be relevant by the time you actually implement them.
- Planning delays real feedback from real users which is much more important than guessing what they want. (Release early, release often)
- Planning can invoke the sunk cost fallacy. The more time you have spent creating a plan, the harder it will be to convince the business to change or abandon the plan.
- Planning distracts the business from being data driven. Data driven decisions involve shipping a feature then checking the data for engagement. Are users using this feature as intended? Do they like it? Are they asking for surprising things? If your decision-making process is heavily tied to pre-project planning, you will have less resources for mid-project analysis.
Some planning is still good:
- Market research is still a good idea. If you don’t check competitor products you might build something that is not unique and already behind. That said, plenty of businesses have built products that are behind the market and still end up on top eventually — so be careful that your planning isn’t removing your innovation.
- Sometimes it’s good to sketch an outline of the system you intend to build, and flesh out the components. However, development teams can often be left to do this themselves as part of starting the project. In addition, there is a danger of over-engineering this rather than trying to get an MVP out as quickly as possible. Its easy to think you need extra components and abstraction layers when you are thinking about a 12–18 month project, and this is exactly the kind of mindset we should be avoiding with agile delivery.
- There may be parts of your business that need to line up, for example preparing a marketing release, an advert campaign. This will depend a lot on your type of business. My recommendation would be to try as much as possible to decouple those departments so that software development can be agile and you don’t start agreeing to timelines and deadlines before the project has started — work with data based projections (not gut feelings) once the project has started and you have hard data on velocity and scope. Perform the experiment: Assume the project has gone horribly wrong and will take 4x as long as you think, or even abandoned halfway through — how does this force the rest of the business to change? Often there is something you can do to protect the business instead of tying departments together in years of roadmaps and joint delivery (which will inevitably fail).
See also this article for an alternative view on why some projects do not suit Agile: A Critical View of the Agile Manifesto
Embrace uncertainty and change
If you are going to do less planning, you must embrace uncertainty and change. This means whatever you think about the project today could be completely wrong tomorrow. Arguably, even if you do lots of planning this still applies.
In practice, embracing uncertainty means:
- Your business should be ready to change direction very quickly. If someone has a better idea in the middle of the project, the business should be able to adapt instantly and change to do that instead. Don’t ignore good ideas just because you had them after the project started! Have you ever thought ‘wait, wouldn’t it be simpler if we just …’? These realisations should not cause a sinking feeling ‘oh no, this project was a waste of time’. but instead ‘I have a great idea what we can do next!’
- Your business should be ready to cancel/abandon projects entirely. Does your business ever cancel projects? Or does it continue projects when everyone knows they are a waste of time? If you are being Agile, you should be humble and expect that you have no idea what the customer wants and whatever you build could be a waste of time. Whenever you get some feedback this is valuable information — what is the business now going to do with this information? Continue as before, or change? Truly agile companies should ship an MVP as quickly as possible, then immediately ask: Is the MVP successful? Do we have something better to work on? Should we cancel this project? The less planning you do and the quicker you build your MVP, the less money you’ve wasted when you cancel a project.
- Your business will need to invest in data collection on product usage so that once your MVP has been shipped, you can analyse customer usage to decide what to do next in the project and whether to cancel it. The culture switch is from up-front planning to mid-project data analysis.
- The product owners and designers will need to work more closely with the development team. In the waterfall world you may have a design / product team create design documents which may go back and forth with technical leadership to try to design something feasible and useful. In an Agile world, after releasing an MVP your development team may need designers to design a new feature very quickly after pivoting to a new direction. Some companies fully embrace this by having cross-functional teams and embedding product owners and designers inside the teams themselves.
Accept you will ship prototypes and technical debt
A danger of building a quick hacky MVP product is that it’s likely to contain shortcuts, low quality code and technical debt. It can be difficult for those outside the development team to understand that a hacky MVP may be a throwaway solution and may require some rewriting in order to be secure and stable in the long term.
Technical debt should generally be taken on deliberately and tactically in order to release quicker to the customer. For example, if there is a chance you will abandon the project and delete all the code, it may not be worth spending time making it perfect before release.
Stop giving orders by documentation
The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation. — Agile Principles
Sometimes up-front planning results in throwing design documentation ‘over the wall’ to the development team. This is an extremely slow and inefficient means of communication. Have you ever had a long running debate about something in document comments or worse, over email referencing the document? (over days or even weeks?)
The worst debates are those where the two parties involved don’t understand the mindset of the other party at all and seem to be talking on completely different wavelengths — for example, one person wants a low level requirement and the other person thinks the project is a waste of time. Stop this situation occurring by working together early in the process and regularly during the project:
Business people and developers must work
together daily throughout the project. — Agile Principles
These discussions are often resolved much faster with a face to face discussion, whilst building interpersonal relationships which are often foundational for successful projects.
Often what the development team needs to know to start a project is not usually what you’d find inside design documentation, and questions that need answering in the middle of the project are frequently too specific to be answered in the documentation, so the documentation is too speculative at the start, and too vague/outdated throughout the project.
The better approach is have face to face meetings to kick off the project, and then frequently during the project itself. Responding to change, not following a plan!
Motivate developers by involving them from the start
Build projects around motivated individuals.
The best architectures, requirements, and designs
emerge from self-organizing teams.
There is an easy way to demotivate a development team: Give them a large stack of design documents and say ‘Build this. See you in 12 months’.
Successful development teams do not work as an outsourced ‘code this’ team, doing the bidding of an all-seeing master of the future. Successful development teams are typically involved in the project plan from the start and will feel a sense of ownership about the entire project, not just the code. Development at the start of the project should be about exploring requirements, technical risk and finding out if customers want your product at all.
Don’t guess when a project will be finished
The start of a project is the worst time to guess when it will be finished. Why? Because that’s when you know the least! You have no data.
Exception: If you have a team which has completed multiple previous projects and the projects are similar enough, then you can sometimes use that data to do some predictions up-front. The key is (a) be data-driven, don’t guess (b) give a date range not a date (c) the date range should be wide enough so that data from the project narrows it, not widens it
Instead of guessing, you should use data to calculate a projection range you have after a few sprints (~4-6 weeks), then the projection should be updated regularly. If you are using project management tools this can often be automated for you.
NOTE: You must keep an eye on the backlog scope of the project. If your backlog is growing faster than you are completing work, then you cannot project when the project will end.
If you can see the end date slipping further and further into the distance this is a good time to stop, analyse the project, and discuss why. Is your velocity dropping? Are you taking on too much technical debt? Is your scope increasing? Are there too many features being added?
Predictability is often more important to businesses than speed.
Caveat: Every company is different
The field of software is extremely broad and growing. Not all companies and projects should be using the same practices. For example, a fresh startup with limited funds will likely focus more on experimenting quickly with different products before they run out of money. A mature company may be more focused on retaining existing customers and may be more reluctant to push out low quality or experimental code.
Some companies, especially those building large enterprise pieces of software, may be limited to how quickly they can get feedback from customers. If customers take 2 years to use your software it’s really hard to get quick feedback! In this case it can be useful to obtain a customer ‘sponsor’ to work with throughout the project, or organise remote user testing / demo sessions where you ask customers to use the software whilst it is still in development.
Business should not simply copy Agile practices from others. Instead they should try to understand the reasoning for why the Agile Manifesto was created, so they can relate it to their business and adjust accordingly. How much planning to do is up to you, decide wisely!
