Software Development Practices Desperately Need Attention
Better decisions are key to successful tech products
“Ultimately, a company’s value is just the sum of the decisions it makes and executes.” — Marcia Blenko, Harvard Business Review
Across every industry, we spend an offensive amount of time, energy and money looking for the next big technology innovation to make our business big dollars. Its not “intentional”, it’s just human nature. But what about attention towards the methods, techniques and tools for building these software products? Personally, I believe we don’t invest enough time, energy and money in the latter and we’re starting to feel the pain:
“Its time to go, Agile…With this considered, arguably no firm is able to compete by delivering software using approaches from the peak of the dotcom boom. They would not have survived, and many agile focused firms are also facing difficulty. Agile can barely keep up.” — Kai Ansaari, Technology Professional
“Why Scrum doesn’t make you agile…The Scrum guide suggests that the sprint, for example, is time-boxed and should have consistent durations throughout development, valuing the plan over responding to change….Although the intention of this approach has merit, in real life, shit happens.” — Karin Dames
“Agile is Dead…‘Agile’ is now an industry. It’s a machine that attacks us on the basis of the fear and asks us to pay money on the basis of fear.” — Dave Thomas, Computer Programmer & Author
“SAFe Isn’t Really Safe…SAFe is good but not good enough” — Ron Jeffries
“Agile is the new waterfall…The reality of Agile is that you still have immutable decisions made by business people with no real understanding of technology. Those decisions are then forced on to developers. The end result is the same as Waterfall, only the names have changed.” — ayasin, Architect & Developer
“Why is Agile the most overused word in software business?…The problem hardly ever comes from the team as most of the agile violations are forced by the project or quality management. Pressuring teams to work with cumbersome processes and doing superfluous documentation is not the right way of getting things done.” — Heikki Hellgren, Scrum Master
The 8 Decade Wrestle
To my surprise, I could not find a timeline of the history of software development practices when I did a Google Search so instead I’ve shared the image above. I’d say this sums up the current conversation about software development practices. But let’s take a quick look back our almost 8 decade wrestle for a software development framework or practice that would help us deliver that software product that would make us millions.
The roots of Agile software development history did not begin with the Agile Manifesto. According to Wikipedia, “iterative and incremental development methods can be traced back as early as 1957, with evolutionary project management and adaptive software development emerging in the early 1970s”.
During the 1990s, software engineers and developers began seeking faster delivery cycles “in reaction to the prevailing heavyweight (software development) methods that critics described as overly regulated, planned, and micro-managed.”
In 2001, the Agile Manifesto was born and over the years has become widespread and accepted by a large number of organisations and it’s people.
In 2008, the Lean Startup was first proposed by Eric Ries. The motivation was based on the high startup failure rates and Eric’s personal experiences adapting lean management practices to tech startup companies. However, applying this in the enterprise world has proven difficult because of the different approaches and considerations taken when building a new software product from scratch compared to evolving an existing software product that has already established product-market fit. Also, enterprises are designed around efficiency, accounting for ROI and misunderstand the concept of innovation because they “have a reputation to protect”.
Now, it is 2018, almost 8 decades since Alan Turing created the theory of software, and we still need to make improvements to the way we develop software. One of the biggest aspects that I believe we’ve overlooked is the simple fact that developing software products is mostly done by humans. Next I’ll talk about why this has a huge influence over the way we develop successful software products.
We’re Just Human
It is human nature to crave more perfection, more speed, more certainty, more money, more information, more time and more satisfaction, especially instant gratification of expected results. In today’s world, the internet has significantly influenced the way we access, consume and share information. As such, we expect to access, consume and share information a whole lot quicker which has reduced our patience and in some ways reduced our attention span. Because of this, when we are uncertain about something or are under pressure to make a quick decision, our often untrained mind quickly jumps to conclusions through (attribute) substitution, anchoring, confirmation bias or another cognitive biases to find an answer. But this is just us being human. All humans naturally make decisions that are more often irrational than rational because we just don’t have the mental capacity to consistently make rational decisions. Interestingly, research has shown that we cannot outperform making better decisions on our own than making decisions with the help of a system.
Now that we are slightly more informed on the human psychology around decision making, when we apply our natural behaviours into the fast-paced, dynamic environments of software development, it is a recipe for a money-wasting machine. In these environments, mismatched understandings, alternate perceptions, waves of uncertainty and ad-hoc decisions are in fact the norm.
But on the contrary, we are constantly hungry for certainty. Certainty is reflected in the majority of our actions that we perform on a daily basis from getting our monthly pay checks to knowing we have money in our bank account or consistent cashflow to pay for the mortgage. Innately, we crave this certainty when developing software. But in fact, software development is the completely opposite. The ideas that we come up are often high risk until there
Software development is nothing more than a massive exercise uncertainty. As such, we must think critically about our actions and which credible evidence is required to support our actions/decisions rather than just using “gut feel” or vanity metrics. Making the hard decisions not only requires experience but rational thinking. This brings me to what I think can be improved in our current methods.
Evolving Our Methods
Uncertainty numbs our ability to make good decisions.
There is one main difficulty in which many people and organisations have repeatedly struggle with, from my observations working in the technology space over the past 11 years. That struggle is with uncertainty. Uncertainty is the culprit for the many symptoms (eg. technical debt, estimations, etc.) that we have all experienced in software development. It has also been the driver for many of the frameworks and methodologies which we’ve applied to our software development practices for over the past 8 decades. However, we have yet to succeed in creating a set of techniques and tools which help humans to truly reduce uncertainty. Without this, we will continue to be paralysed in our decision-making but we need to make better decisions when developing software to be be successful. Our methods desperately need to evolve to help us better manage uncertainty, quickly adapt for the constant changes that we face and the frequently learnings had.
This is obviously easier said than done because the problems that we currently face largely involve and revolve around humans. Some of challenges include (but are not limited to):
- Untrained people
- People don’t make time (because they don’t understand)
- People don’t want to change
- People are lazy — why take a few hours if it can be done in a few minutes?
- People are distracted from or do not understand the driving purpose behind their actions.
The cognitive bias of humans is a powerful tool that can work both against and with us. By default, it works against us. Kahneman’s WYSIATI concept explains our lack of mental effort and to make decisions as the problems arise (System 1 thinking). We need to be trained in order to get it to work with us.
To evolve our methods, we first need to recognise that humans are incapable of consistently making rational decisions. In some ways, we are aware of this because we have gravitated towards methodologies and frameworks like Agile, Scrum, etc. but this is not enough. Then a true improvement in the way we develop successful software products would require us to consistently make rational decisions with credible evidence. This can only be achieved if we are truly able to reduce the continuous uncertainty which we are faced with. Until we can do that, uncertainty numbs our ability to make good decisions.
Drive Decisions with Evidence
Harvard Business and Bain & Company conducted a study which surveyed “top executives worldwide from 760 companies, most with revenues exceeding $1 billion, to understand how effective those companies were at making and executing their critical decisions.” Their results showed that the “top-quintile companies score an average of 71 out of 100 in decision effectiveness”. So compared to the other four quintiles, there is potential for the other organisations to more than double its decision effectiveness.
When it comes to developing software, we should resist the temptation to focus on solutions to the problem. Instead, we should be continuously questioning the problem that we’re tying to solve, to ensure we’re solving the right problem, and be using proven methods to make sure the numbers stack up. Often we are irrational and the simple calculations of numbers stacking up is overlooked because our emotions get in the way. At the start of the article, I shared some thoughts about the current software development methods and frameworks used today. Whilst that was only a small sample, it is a reflection of a growing desire to find a better way to deliver software products. As we can see from the comments, it appears that many people are too focused on the process itself and easily lose sight in practice on why they’re using such process or framework to begin with. There also appears to be an expectation of the process to give us answers as to how we solve our problems. However, a process will not give us the answers to our problems. It can only guide us. But I question if the current processes are guiding us in the right direction or just in a circle of more uncertainty?
And so the frustrations around seemingly unproductive software development activities continue. Whilst we’ve improved, I believe there is a core skill that many developing software products are missing — the ability to manage uncertainty and thus thinking critically. Software development involves uncertainty on a daily basis and, without the right training and tools, we will continue focusing on process rather than applying the appropriate critical thinking and tools which enable us to solve the hard problems. A cookie cutter process will not help us answer the hard questions. The cookie cutter process only helps to bring organisation and rigger to the chaos.
Ultimately, we need to take a decision-driven approach that involves creating and utilising credible evidence to reduce the temptation of our cognitive bias, the paralysing effects of uncertainty and our hunger to jump straight into designing solutions.
Better Decisions Lead to Success
Whilst “Agile” is currently one of the most popular methodologies being used today to deliver software products, its 17 years old and struggling to keep up with the demands and fast-pace at which we are required to develop software today.
In a previous article, I talked about the 5 capabilities that technology businesses need to survive 2019. Building on that, I believe that we need to be more disciplined in order to better manage uncertainty, risk when developing software products. A big part of that discipline includes being patient and thinking critically. There are also proven scientific and practical methods of reducing risk and uncertainty but yet we’ve continued to dance around the problem. For example, the decision matrix is a common tool used in Kanban and Scrum practices to address which tech debt to tackle next. However, research has proven that using such matrices are ambiguous and provide inconsistent results.
Today, it is common for many people give up on the “hard stuff” if they cannot answer a question within a few seconds or complete a task in the expected amount of time. Any business developing software products will continually face these various levels of uncertainty. It’s simply the nature developing software products. But what if we could use tools to help give us credible, reliable evidence about that “hard stuff” so we can make better decisions? Thus, I would like to propose that we focus on embracing uncertainty with appropriate techniques and tools and, as a result, improve the viability of our software products. I’ve actually developed a few techniques and tools with two other co-founders. If you’re a C-level individual, investor, manager or engineer and curious about how you can improve the way you manage uncertainty? Get in touch at hello@criticide.com.
In the mean time, here are a few guiding principles that I’ve found useful along the way:
- Think critically because no process or tool will solve the problem for you.
- “Data is dumb”. Ensure you know what you want to achieve with the data you collect (eg. what decisions will you make?)
- Embrace ambiguity and complexity by “asking the right questions” to continuously reduce your uncertainty about a problem.
Learned something? Click the 👏 to say “thanks!” and help others find this article.
Hold down the clap button if you liked the content!
Clap 35 times!
