Opportunity Cost for developers — or how I stopped worrying about the future and learned to code for the present
In this post I will discuss the importance of opportunity cost in development and how developers fall into the trap of developing for a future that will likely never exist.
Opportunity cost is a concept familiar to economists but, perhaps, not very familiar to most people outside the economics profession. The canonical definition is as follows, from Wikipedia:
In microeconomic theory, the opportunity cost, also known as alternative cost, is the value (not a benefit) of a choice, relative to an alternative. When an option is chosen from two mutually exclusive alternatives, the opportunity cost is the “cost” incurred by not enjoying the benefit associated with the alternative choice.[1]
A very simple example of what this actually means in real life is having to choose between two desserts, costing the same, in a restaurant. The opportunity cost of choosing the sticky toffee pudding is the loss of enjoyment from not eating, say the Eton mess. So, if you think you are going to enjoy the Eton mess more, then that is what you should go for, obvious stuff.
At this point you might be wondering how this relates to development, so I will provide a more relevant example.
Opportunity cost in development
In my previous role, a customer had a requirement to ensure that all employees/contractors in a project had BPSS. This was a little bit of an administrative nightmare for the onboarding team who would do the submission of the BPSS application form on behalf of the employee/contractor if needed. The BPSS application forms were regularly partially and/or poorly filled in, which caused a lot of unnecessary effort for the onboarding team, as there were a high number of rejected applications, which had to be resubmitted, if this was not done in a timely manner, the employee/contractor might not actually be able to start work.
The company was asked to create an online form that ensured that all necessary information was collected. The form required some complex validation rules, e.g. the employee/contractor had to provide home addresses and employment references that covered a period of three years. There were some requirements that were related to the company’s onboarding process, as it made sense to collect all the necessary onboarding information at this point in time. The form would be filled in by the employee/contractor from the comfort of their home with a paper fallback for the less IT savvy personnel.
The development team discussed the requirements with the sales team and it was agreed that this type of customisable forms with complex validation would be a good feature to add to the system so it was decided that it would be added as a generic feature rather than as a single form for a single customer. It is worth pointing out that this would be the third form type (arguably it was an enhancement of an existing form type) that was being added to the system.
The feature took a single developer (the lead developer in the company) just over eight weeks of development to complete, this did not include testing or actually creating the form itself, a rather time-consuming effort as it required several review cycles due to typos on both sides. The developer was suitability proud of his achievement, so much so that this feature would feature in quite a lot of conversations in the future.
Opportunity cost in time and development
Compare the development time of the two approaches
· 1 week for custom form with validation
· 8 weeks for generic reusable form
The opportunity cost of this feature is all the other features, bug fixes, etc, that the lead developer could’ve worked on in those seven weeks, this is regardless of whether the feature was used or not. Note, it’s seven and not eight as that’s difference with the alternative, namely the week it would’ve taken to do the single form, although arguably given the actual usage, more on this later, it could be argued that the opportunity cost was eight weeks. To be sure this makes the assumption that the requirement had to be delivered, but that’s a conversation for another post.
The interesting bit is that the form the customer wanted was delivered and by the time I left the company there were a grand total of nine completed forms in the production system, all of them were completed during acceptance testing, in other words the form was never used for a real employee/contractor but, how about the feature itself? After all the advantage of adding it as a feature was that it could have been used by other customers, right? Well, it was not. In the, roughly, year and half between the feature having been released and me leaving the company, not a single customer used the feature. I feel that this bears repeating: Not a single customer used the feature. You could ask whether it was sold at all, how was it sold, how was it marketed. There was never a requirement to use it by any customer new or existing.
A quote from The Phoenix Project feels appropriate here:
Features are always a gamble, if you are lucky 10% will get the desired benefits, so the faster you can get those features to market and test them, the better off you will be.
May the future be with you
You are probably thinking that this is an extreme case, and you are probably right, but I think that using an extreme example can sometimes help to illustrate behaviours that are hidden from view and that, while on their own, might not be too damaging, when repeated often, can create issues.
The main issue with the example above was that a single case was prematurely generalised to tackle all, well, let’s face it, some use cases that other customers may, in the future, have. A harsh but accurate description of the whole process would be that we tried to forecast the future and failed miserably, which is something that happens with depressing regularity as predicting the future is a notorious difficult business.
Unfortunately, confirmation bias and the availability heuristic work against us in these types of situations as we’re only ever likely to remember all those times when our efforts toward generalisation or extra features paid off. We have all undoubtedly thought something along the lines of :
Well, I couldn’t have predicted that
when talking about the shortcomings of a feature we have lovingly developed and yet we have undoubtedly thought how clever we have been to have added this or that generalisation or code for an extra use case that allows the new requirement to be satisfied with very little or indeed no development at all.
The point I’m trying to make is that by trying to tackle future use cases or generalising our code too early, we are ignoring the opportunity costs that this has and these costs can be massive. Furthermore, tackling the use case when there is an actual need, will likely result in a better feature as there will be a real need rather than an attempt to future proof the code, which unfortunately tends to be against the wrong future.
This doesn’t only applies to full features, this can happen when we try to generalise or future proof a small part of the codebase, after all an extra half a day here and there and suddenly you are talking about weeks of person days that have been effectively wasted.
Complexity and maintenance
Arguably, the time has been worse than wasted as there is the double whammy of the unneeded code and the effect to codebase of this unneeded code. As the codebase’s complexity will almost certainly have increased, with the corresponding impact on maintainability and cognitive load for developers, especially junior developers.
In certain companies the urge to future proof comes from a difficulty to obtain a budget for development but perhaps paradoxically this approach to try to future proof everything makes it even harder to get said budget. The reason being that some (a lot?) of that budget is wasted on features/use cases that nobody ends up needing, with the consequent waste in resources and the negative feedback loop that this creates.
New approach
So, what’s the solution? You need to develop for now and the current requirements and not for any requirements that may come in the future as they may never become actual requirements. This should not be taken as a hard and fast rule as there could be cases where new requirements will be coming in a couple of sprints time, but you should definitely be wary of developing any code for future requirements as the future has a nasty habit of being wrong. The acronym that most closely captures this sentiment, and it has to be said that we love acronyms in IT, it’s YAGNI.
It is possible that doing an MVP feature and then refactoring to provide all the desired functionality could lead to higher costs. In the previous example, an MVP approach could not have been turned into a Full Feature but this is could be OK.
The question as to which approach makes more sense, i.e Full Feature vs MVP and refactor when needed, depends on how accurate The Phoenix Project quote is. If 10% is truly the rate for successful features in your organisation, the MVP approach is undoubtedly the right approach as on average, you will only incur a penalty one in every ten features.
It is a waste of time to write code and then do a feature that would negate the need to write the original code in the first place, but the whole point of this article is that this is no different and in fact, less likely, than the alternative, which is to write code that isn’t used.
Doing what’s needed
It could turn out you’ve not considered all likely possibilities but the advantage of this approach is you can go back and spend more time on the feature, you cannot get back the time you’ve spent writing code that will end up not being used, which might increase the technical debt and reduce the maintainability of the codebase.
Next time you find yourself developing code to cover a use case that is not relevant now or you are generalising some code to better be able to handle the future, think about what else could you be doing of value, i.e. the opportunity cost, and you will, hopefully, restrain yourself. Remember YAGNI, you aren’t going to need it or are you?
