Ode to XP
Part 2 — Enabling Software Development Organization Transformation
“Extreme Programming Explained,” the seminal book on Extreme Programming (XP) written by Kent Beck, was released in 1999. XP was the basis for the first agile development process I ever worked with; as a result, it’s is still near and dear to my heart. And it turns out I’m not alone. In just the last few weeks, I’ve had conversations with several Slalom Builders about XP and the guidance it provided to early agile adopters who were seeking both a better software development process and, more importantly, a way to love their work and their life. Given the recent reminder of how foundational XP has become, and to celebrate the 20th anniversary of “Extreme Programming Explained,” I’ve written a multipart series on XP.
Part 1 of the series revisits the values, principles and practices outlined in “Extreme Programming Explained (2nd Edition)” to highlight the core ideas that have spurred a decades-long software development revolution. Part 2 of the series examines the software development challenges that many organizations still face and how XP’s guidance and vision, two decades later, is still applicable and relevant to solving their challenges. Part 3 of the series reflects on XP and outlines an updated set of values, principles and practices for the decades ahead.
“The goal of Extreme Programming (XP) is outstanding software development. Software can be developed at lower cost, with fewer defects, with higher productivity, and with much higher return on investment. The same teams that are struggling today can achieve these results by careful attention to and refinement of how they work…”
— Extreme Programming Explained (2nd Edition)
Twenty years after the first edition of “Extreme Programming Explained” was published, truly outstanding software development is still hard to fathom for many organizations. The State of DevOps Report 2018 notes that over half (52%) of organizations fall outside the “high performer” category and only a small group (7%) fall into the “elite performer” category. These categories are defined using metrics like: team throughput, software stability, and system availability to categorize the performance of software development organizations. The significant percentage of organizations that fall outside the top-tier groups raises a simple but challenging question: why do organizations continue to struggle to achieve outstanding software development two decades after XP was introduced?
“Accelerate,” a recently published book by Nicole Forsgren, Jez Humble, and Gene Kim, examines four years of State of DevOps survey results. Through rigorous analysis, the authors found that many XP practices (or modern extensions of XP practices) do contribute to improved software development performance and, in turn, more successful organizational performance. With the State of DevOps data and “Accelerate” analysis now available, the question of whether certain practices can have a measurable impact on software delivery performance and organizational performance can finally be answered scientifically.
XP provided a set of values, principles and practices that became the foundation for modern software development. Over the last two decades, XP has been extended, refined and supplemented to produce a comprehensive, scientifically proven path to software development excellence. The onus is now on software development organizations to adopt the XP practices, processes, and culture of the Elite Performers who are leading the way.
Practices
XP is different from other popular agile methodologies in that it doesn’t just focus on the process of development, it focuses on the execution of development. Like many who have used XP practices, such as Continuous Integration, Daily Deployment, and Test-First Programming, Kent Beck knew these practices worked but dismissed the need to scientifically demonstrate that they worked. As he said in “Extreme Programming Explained (2nd Edition)”, “I’m often asked for ‘scientific’ evidence for the practices in XP, as if science could somehow bear the responsibility for project success or failure.” While Beck dismissed the need for proof, struggling development organizations may need that scientific reinforcement to encourage the adoption of proven practices they currently lack. Luckily, almost two decades later, “Accelerate” provides just the science Kent was often asked for. By examining thousands of survey responses, the authors have identified a set of practices that, when combined, lead to software development organization excellence. The subsequent sections highlight some of the approaches examined in “Accelerate,” focusing on those that overlap with or extend XP practices.
Continuous Delivery
The “Accelerate” authors found that continuous delivery has a measurable impact on software delivery performance and organizational culture. Continuous Delivery is an approach defined by a collection of practices, many of which can be traced back to XP.
Continuous Integration
Continuous Delivery is a natural extension of Continuous Integration. Continuous Integration, as defined by XP, stops after code has been compiled, unit and component tested, scanned, and packaged. Continuous Delivery includes all those activities as well as deployment to one or more test environments so that system-level acceptance tests can be executed against the deployment. The concept of Continuous Integration is based on team members working in small chunks and getting fast feedback as their respective small chunks are (frequently!) integrated. “Accelerate” has bolstered the argument for that XP practice by highlighting the importance of Continuous Integration, as part of the broader Continuous Delivery concept.
Struggling organizations should adopt Continuous Integration to encourage small, frequent changes and should strive for Continuous Delivery.
Daily Deployment
Where XP advocated for a Daily Deployment, Continuous Delivery recommends deployment and (acceptance testing) after any change. To help make frequent deployments possible, “Accelerate” demonstrates that Deployment Automation and Test Automation are key practices to add to Continuous Integration.
Struggling organizations should make (at least) Daily Deployments their first deployment goal. By doing so, they’ll force themselves to automate deployment. Once automated deployment is complete, automated testing is a logical next step.
Test-First Programming
Test-first programming or Test-Driven Development (TDD) is often a contentious topic. In the purest definition, test code should be written before any application code is written. “Accelerate” does not examine whether TDD has a measurable impact on software delivery performance, but it does show that Automated Testing as part of a Continuous Delivery pipeline does. Whether the test code comes before, during or after application code is written, having an automated safety net that relies on tests at all levels (unit, component, acceptance) is imperative.
Struggling organizations that are not thinking about testing at all levels, or that are not automating as much of that testing as possible, would benefit from revisiting the motivations behind XP’s Test-First Programming practice.
MTTR and Lead Time
Kent Beck trusts “two metrics to measure the health of XP teams. The first is number of defects found after development….The second metric…is the time lag between the beginning of investment in an idea and when the idea first generates revenue.” The “Accelerate” authors use four metrics to measure Software Delivery Performance: Lead Time (time to get a new idea to users), Deployment Frequency (number of deployments per time period), Mean Time to Restore (time to resolve a production issue), and Change Fail Percentage (percentage of changes that don’t function as expected). Lead Time and Change Fail Percentage are nearly synonymous with the two measures Beck outlined.
Struggling organizations who aren’t capturing these four metrics for their own development teams should start immediately. Two decades after XP, these metrics are still at the forefront of assessing development team performance.
Lean Product Development
Another theme from “Accelerate” is that organizational culture has a direct impact on development performance and business outcomes and that Lean Product Development has a direct impact on organizational culture. Lean Product Development isn’t related to an XP practice, but it is extremely similar to the XP value of Simplicity. By adopting structures, processes, and approaches that are simple and value-focused, and by constantly assessing alternatives, an organization will become and stay lean. The Code and Tests practice from XP is a manifestation of applying a lean mentality. In the words of Beck, “Customers pay for what the system does today and what the team can make the system do tomorrow. Any artifacts contributing to these two sources of value are themselves valuable. Everything else is waste.”
No matter where an organization is on the performance spectrum, maintaining a focus on simplicity, value creation, and visibility is imperative.
Organizational Culture
“Accelerate” highlights the importance of culture within a software development organization, specifically the work of Rob Westrum, who defined a Generative Culture as one that instills “a focus on the mission.” Organizational culture has a direct impact on software delivery performance and can be shaped by continuous delivery practices. Not surprisingly, a generative culture is well-aligned with XP values and principles, as the mapping below demonstrates.
As the State of DevOps 2018 report emphasizes, while practices can drive cultural change, a supportive leadership that values a generative culture can accelerate transformations. Leadership supportive of a generative culture is very likely also a proponent of the related XP values and principles. For example, a leadership team that truly believes that “failure should lead to inquiry” likely also encourages frequent, meaningful feedback, respect for team members, and the courage to ask and answer challenging questions in order to learn from mistakes.
Development organizations should look beyond practices to ensure the organizational culture being promoted is one that fosters feedback and improvement.
Engaged Workforce
“Accelerate” also highlights the desire to keep team members engaged and happy by removing taxing, burdensome tasks. By doing so, software development organizations can expect their employees to produce excellent software more reliably and more effectively. XP has a similar philosophy, which is captured in the Energized Work and Slack practices. “Accelerate” helps to support this approach by asserting that an engaged team contributes to higher performance. Kent Beck captured this idea as well under the Whole Team concept, remarking that “People need a sense of ‘team.’ We Belong. We are in this together. We support each other’s work, growth, and learning.” XP was well ahead of its time in advocating for a more sustainable, more human approach to software development. A positive, team-based sentiment is also consistent with a generative culture.
Organizations who are struggling may find their employees are also struggling. By adopting practices that make work less stressful, more rewarding, and more conducive to building deeper connections can address those challenges.
Final Thought
XP outlined an approach to modern software development that covers development processes and engineering practices. The research in “Accelerate” proves that many of XP’s “extreme” ideas are now deeply woven into a collection of approaches that lead to outstanding software development. For organizations struggling to reach an elite level of performance, revisiting the ideas behind XP is still an excellent first step.
