Visible Progress Requirements Can Raise Costs on New or Complex System Tech Development
The rush to keep delivering visible software features and clock-like incremental progress sometime raise the cost of a project. Agile methods like SCRUM are heavily based on translating naturally bumpy progress into incremental digital service development. This smooth progress is often helpful and real. It can also be an illusion of for the audience — much like “smoothing” of financial returns to keep investors happy. Professional project management knows when to break the rules of agile ideology to avoid increasing the project cost or risk.
Think of software work as falling into two categories: “farming” and “exploring”. Farming in this sense is predictable. It is the same work as last year. Having done it before, we know roughly how long it will take and can sprinkle some interface work into each sprint to keep our audience entertained.
Exploring work is where digital services cause company internal stress. The tools and requirements have changed, team members don’t have track record of domain and product knowledge, and sometimes the work naturally requires a rocket ship to be built before the visible progress of lift-off. “Engine work” is more likely to fall into this category than “interface work”.
For example: Alpha team is updating the current product. Not much has changed technically. They are adding small features and routine interface refinements at a regular pace. This is farming. It can be broken down into predictable, incremental steps.
Company communication is more difficult for Omega team. They are working with new technology because the business world has changed with new market entrants and customers demanding something more. A lot of value is created since AI must be trained, outside data connections to new partners of unknown response time are critical to the plan, and the results of this analysis must be iterated through a completely new 3D interface. This is exploring. The classic SCRUM approach is to “spike” features by slapping something together to explore the new terrain. A second, unspoken purpose is to deliver something visible to management before going back and do it all again. I’ve consulted companies where this was also a chance for middle managers to stab a rival before the next round of promotions. Life is complicated at the cutting edge.
Spikes are a good idea. Making spikes take longer by adding in the extra steps of delivering visible results is the problem. As a manager, you can instead save time and money by trusting the team to do their spikes as unit tests rather than visible features. New technologies come with risks and unknowns, so this serves two purposes. It reduces the duplicated work of a feature spike, and it is a case with Test Driven Development (TDD) makes sense for clearing away doubt in more incremental chunks and increasing the reliability of the product down the road.
The most difficult cases that stress management is when exploring is if “engine work” involves the interaction of a lot of moving parts. That interaction and inherent complexity of the task requires system level innovation and system tests. These are not naturally decomposed into unit tests. At all works together, and sometimes that is the point. I call this “single stage to orbit”, meaning one big rocket, you light it after a lot of and find out if it blows up or does something amazing. These cases are more rare, but they do happen and require trust and understanding within the team. Everyone is stressed solving these problems, not just those observing the lack of a regular drum beat of features. You can find other markers of progress if you break the work down in a way which is natural to the problem domain and team. But understand that it is natural for the estimates to be wildly off when the interaction of parts and organisations is novel and inherently complex. The only solution is experience and spending a bit more time with the team rather than looking only at the summary or visible surface.
