Optimise the Whole
One of the fundamental principles of lean software development is that we should “Optimise the Whole”. This can manifest in many different ways, but I’d like to share the story of how I learned about one particular aspect of this principle.
Back in the late 1980s, I was the sole developer for the first derivatives trading and risk management application in an Australian bank. Derivatives’ volume and complexity were a lot lower in those days, so it was manageable, but I still had a lot to learn. I needed to learn the fundamentals of options, how to build yield curves from different interest rate instruments, and how to combine all this to handle pricing and risk management. Algorithm implementations from our parent bank were a big help, but I needed to translate these algorithms from the existing language to Fortran, and I needed to ensure they could handle the differences in conventions between the US and Australian markets (there were a lot of spreadsheets involved in Australia at the time). As I progressed, I also discovered the project was exposing corner cases in the existing algorithms that were covered in principle but hadn’t, in fact, been exercised by the US traders. All in all, it was a very steep learning curve — I spent my time reading a lot of books, analysing spreadsheets, and making lists of questions I needed to ask the traders.
Why lists of questions? One of my constraints was that I could only contact the traders while the markets were closed and they weren’t in risk management meetings. In practice, this meant talking to them only before 9am and after 4pm, or thereabouts.
In the end, the implementation went pretty well and the traders were happy with the outcome. In a post-implementation conversation, I mentioned to a trader that if I’d been able to talk to them during the day we probably could have finished 20% faster, and development would have cost 20% less. Many of you will probably be able to predict his response, but it was surprising to my mid-20’s self: he said something like “Yeah, probably”, and turned away to talk to someone else. He didn’t contradict me, he just didn’t seem to be interested. How could that possibly be? I’d just offered the company a way to reduce development costs by 20%, and they just weren’t interested. What was going on?
When it’s all laid out like this, of course, the reason is fairly obvious: the bank as a whole isn’t aiming to optimise software development, it’s optimising for overall profit (and possibly some other things). In this particular case, it was much more important for the traders to be able to give their full attention to participating in the market and managing their risks than it was to reduce software development costs. My decreased productivity was a reasonable price to pay in the overall scheme of things.
Over the following decades, I’ve met lots of people who fell into the same trap I did — the trap of thinking that software development is (or should be) the most important thing to their organisation. That may be true in some environments, but it’s certainly not true in general. As software developers, we need to be willing to accept that some apparent inefficiencies are inevitable; the big trick is to distinguish the inevitable inefficiencies from the accidental and correctable ones. Is that oversight on releases a reasonable burden that makes releases more expensive but protects the organisation from an existential threat, or is it needless bureaucracy? You need to dive into the details to make a decision, but you also need to make sure that you keep the bigger organisational picture in mind.
The same pattern applies at multiple levels. Software development as a whole may need to accept some inefficiencies to benefit the overall organisational effort. A single team may need to accept some inefficiencies to benefit the overall software development effort. An individual may need to accept some inefficiencies to benefit the overall team effort. Expecting my individual experience to be optimised is often unreasonable, even though it’s very appealing!
Of course, this doesn’t mean that we should accept the other extreme, where an individual is made utterly miserable by the requirements and compromises demanded by the team, or the organisation. This sort of situation is undesirable, unsustainable, and possibly ineffective even for the overall organisation. We should be looking to create teams and processes where people can make regular progress on meaningful work, with as much autonomy as possible. We simply shouldn’t expect them to be nirvana for each and every person on the team — to expect that only leads to disappointment on all fronts.
My experience has been that while at one level this is obvious, it’s still hard to internalise. It’s easy to be frustrated about things that make my own life harder, especially if they appear to be immovable. In these cases, I find value in pausing and thinking about whether these impediments are serving a larger purpose, and whether it’s me being unreasonable rather than the other party being intransigent. It’s difficult, but I’ve found it makes me calmer and makes my work more comfortable. I hope sharing this story will help you feel the same way at some time, even just a little.