Why We Still Suck at Building Great Software
There seems to be no end to ideologies, methodologies, frameworks, and processes attempting to solve the ills that plague software development organizations. Software that’s buggy, slow, or solves its problem space poorly repeatedly finds its way into the status quo. Maintaining software can often be prohibitively expensive: solving quality problems or adding new features to maintain the competitive edge often costs much more than expected. It takes too long to build (what appears to be) a small, uncomplicated feature or to fix a bug.
These two issues, quality and maintenance cost, of course, expand into discussions (or battles) about scope creep, definitions of done, technical practice, technical debt, schedules, estimates, accountability, and so on. A new aspiring silver bullet solution for these problems appears every week. However, I find that most of the popular software development strategies still fail to resolve (or even address) what lies at the core of these very crucial topics.
“Self-organized”
The empowerment of self-organizing teams, while great for engagement, morale, and technical craftsmanship, can often be an enabler for anemic or schizophrenic product strategy. Programmers, in general, tend to be more excited about the technical details than most anything else. Many teams of programmers will self-organize so that they can end the product planning meetings as quickly as possible and get back to doing what they love: writing code. This leaves the roadmap for the product dictated directly by the stakeholders, who will likely have different needs and goals. The path to mediocrity and incoherent features is well-trodden by committee-driven product strategy. For instance:
Let’s look at this a different way. The committee is the union of different voices, attempting to steer the product in different directions. If you’ve ever been to a carnival, you know exactly what happens when two kids share a bumper car and have different opinions about where to go. The car goes nowhere, except in tight little circles.
When the committee dictates the product’s direction, invariably you’ll get conflicting goals and requirements of dubious value. You’ll get requirements that are unrelated to the rest of the product. A stakeholder applies pressure, the product changes direction, the next stakeholder applies pressure in a different direction, and the product, rather than reaching a goal, just accumulates features and spins in circles. Because these features are often unrelated, or reflective of conflicting goals, the product isn’t evolving, it’s rotting. The spiraling motion of the product at the center of product planning by committee is Charybdis in a porcelain bowl.
When you find something in a computer program that doesn’t work well, who do you grumble about? Who is the one responsible for fixing it (if it ever does get fixed)? What’s their role? Their job title? Where does the blame inevitably land in software projects when they go awry?
On Blaming the Programmer
There’s an idea that forms in the mind of those that employ software developers:
If I set good programmers to work solving <problem>, we can make money with the software they write.
I’ll refer to this sentiment as the sola artifex fallacy — the idea that all it takes is craftsmen to build a successful product. This doesn’t appear to be a completely ridiculous idea. It puts faith in the intelligence and conscientiousness of programmers to build high-quality software, to learn the relevant details of the problem domain, and to develop user empathy. Yet, many excellent programmers, despite their best efforts, build software that is alien to their users. Sure, the software works, but it’s not solving the problem very well. The statement above gets twisted into
Why would a smart programmer build it like this?
This, of course, confounds their employers, who will, naturally, begin dictating how to solve the problem, often to painstaking detail. The faith in the programmers’ motivation to learn the problem domain is abandoned in favor of command-and-control, either by a single person, or by committee. This is the beginning of the age-old adversarial rift between development and business. The business doesn’t trust the developers to solve the problems the right way, and dictates solutions. The developers, natural problem solvers, resent having solutions dictated to them instead of being given problems to solve.
When this mistrust rules, the software becomes an accretion of solutions to problems that are misunderstood. The product decays into a tangled web of half-measures and low-hanging fruit fixes. How little time it takes a codebase to rot!
Creating Equilibrium
To stave off the moldering of a software project over time, there are forces that must be brought into balance.
Soundness — the software should be a well-crafted solution to its problem domain. Users who understand the problem domain should easily understand the software’s value. Failing to properly invest in soundness creates opportunity for competitors to steal your users with better user experience, tighter integrations, cleverer conceptual models, and more sophisticated features.
Marketability — the software should be solving problems that need to be solved. In order to sell software, someone must be willing to buy it. This can sometimes pull very strongly against soundness: stopgap solutions can often be very lucrative. However, soundness and marketability will often align in the long term. Under-investment in marketability yields poor ROI on development costs. Over-investment can create create chaotic products that lack cohesion and suffer from complex quality problems.
Quality — The software should be free from defects in concept and implementation. Under-investment in quality, of course, erodes users’ trust, and, in extreme cases, invites public criticism that keeps new users away. Products with severe quality problems usually have a problem with the balance of the other two forces as well.
For some software projects, balancing these three forces happens easily, naturally. Sola artifex implies that the programmers themselves will balance these forces in order to build a successful product. However, the balancing of these forces takes time, effort, and skill. Some programmers have the skills, and will invest the time and effort; however, that’s time and effort spent not writing code. For some projects, this investment is a minor tax on a programmer. As projects get larger, develop more complexity, or acquire more users, the balance of these forces becomes a full-time job. When nobody does this job, the business-versus-dev rift appears, and the product spirals into the abyss.
The crucial question is: who will do this job? What kind of a person is responsible for balancing these forces?
A product manager.
Casting Vision
When you see predictable, repeated success in any kind of activity that leverages the collective efforts, talents, skills, and passions of a group of people, you will always find a leader who clearly articulates their view of the future.
History will be kind to me for I intend to write it.
— Sir Winston Churchill
I have a dream…
— Dr. Martin Luther King, Jr.
A product manager can guide a product to success only if they cast a compelling vision for the product. Product managers must be dreamers, teachers, inspirers, storytellers. PMs must be skilled at delivering an exciting narrative about their product’s future. This story serves as a rallying cry for everyone involved: developers, stakeholders, managers, executives, investors.
In order for this vision to be compelling, it absolutely must go beyond organizing and prioritizing user feedback. Product vision requires seeing beyond what the users say they want, and planning for what they need and what they’ll buy. A talented product manager’s vision will unify many of the conflicting voices that would steer the product in opposite directions. Effective product vision doesn’t stop at meeting a checklist of required features, either. The most successful product managers will refine and distill the vision constantly, and make sure everyone knows:
This is what we’re doing, and this is why.
If a product manager can’t confidently wield a strong, persuasive product vision, forces will fall out of balance. Marketability will rule over all, and the product “manager” will simply become a requirements secretary, powerless to steer the product out of the swirling vortex. Maintaining compelling product vision is a powerful tool for redirecting the chaos of the committee and harnessing the stakeholders’ feedback productively. Without vision, soundness will fall to the wayside, because soundness requires strategic, proactive investment. A product manager must have a convincing argument at the ready when someone asks, “why should we spend the next six months on this?”
Bridging the Rift
The relationship between the product manager and the developer is sacred. Its value cannot be understated. A product manager cannot build the product without the developer, and without the product manager’s vision the developer will build the wrong thing. The strength of this relationship is the only thing that can resolve the mistrust between developers and business people.
Why is this relationship so important? The developers who build the product are the best people to navigate it around technical obstacles. The developers know the relative effort required to achieve the various parts of the product vision. The developers understand the details. Therefore, the developers are the only people who can facilitate a positive feedback loop whereby product vision becomes product reality faster and faster over time.
A product manager who sells their vision to the developers and refines that vision based on the programmer’s expounding of its technical details is well-poised to maintain tension on the quality of the product. An effective product manager listens to the developers. A product manager without a clear understanding of their product’s current state of technical soundness is blind. In contrast, a shrewd product manager will use the areas of risk identified by the developers to refine their strategic objectives. Sometimes, this may be deferring risky work in favor of quick wins. Other times, accepting additional low-ROI effort early helps position the product for a series of high-value strategic wins later. Understanding the state of the software’s technical craftsmanship is vital in strategic planning and essential in maintaining product quality.
The developer’s centrality to a product’s quality goes far beyond the management of its technical details. To consider a product “high quality” simply by its defect count, cyclomatic complexity, class count, transitive class coupling, lines of code, or any other technical metric is to find that a product has drifted away from soundness. As I stated earlier, quality is the absence of defects in both concept and implementation. Technical metrics are primarily concerned with the quality of implementation. Defects in concept cannot be neglected, nor can they be prevented by the product manager alone.
A healthy relationship between product manager and developer fosters a codebase whose innermost constructs mirror — in both nomenclature and structure — the problem domain. Far too often, a myopic view of software quality as primarily a technical concern leads to a divergence between the concepts that are core to the product’s problem space and the details of the product’s implementation. This places extra burden on programmers, who must mentally translate from problem to solution, where the abstractions are often significantly different. Effective product managers recognize this mismatch as waste, and will cultivate a vernacular with programmers that reflects the problem domain.
This recruits the developer in the stewardship of the product’s quality of concept. The analytical minds of programmers will find the conceptual inconsistencies in even the most well-laid plans. Wise product managers will welcome this feedback and incorporate it into their plans. They will invest in teaching the language of the problem domain to the developers. They will embrace, with patience, how precise developers will insist that this language be. By doing so, this enables developers to craft precise abstractions into the product that reflect the problem domain rather than leaving them to invent their own jargon. By unifying the language of the business and the language of the code, a product manager minimizes the friction in maintaining the product’s conceptual purity, and by extension, the product’s soundness. A product manager harnesses the intelligence of the developers not only to refine the craftsmanship of the product’s implementation, but also its concepts and its vision.
This Isn’t A Unicorn
At its heart, successful product management requires the ability to captivate, to inspire, to articulate a vision and draw people to follow along in pursuit of that vision. A product manager is a leader, and must be both willing and empowered to lead. The best PMs aren’t bullies, but they’re skilled at persuasion. They perceive both the short and long term ROI of any given feature, and prioritize features to balance emphasis towards soundness or marketability as needed.
PMs filter and redirect the chaos of conflicting stakeholder feedback, preventing that chaos from driving the product towards schizophrenia. They enable developers to build software in pursuit of clear goals, and advocate for sufficient resources to maintain product quality. They communicate with investors and managers, always at the ready with clear vision, and, when necessary, statistics to support their plans. A product manager is always well-informed regarding the market, the stakeholders’ desires, and the current state of the code. They actively balance the forces that steer the product.
The sola artifex fallacy assumes that programmers will muster their faculties to address product management concerns in the marginal time when they’re not writing code. This is dangerously neglectful. Great software doesn’t happen as a byproduct of the latest technique for shepherding columns of sticky notes on a wall. If the soundness, marketability, and quality of a product aren’t aligned for success, there is no process, framework, or development methodology that will balance them for you. If the product is steered by the contradictory desires of a plurality of stakeholders, it will decay. Despite what the zealots may tell you, a programmer alone isn’t enough to build great software.