Quest for the ultimate software development methodology
Scrum, Kanban, XP, Squads, Shape Up, SpaceUP, you name it. The quest for finding the ultimate software development methodology seems to be never-ending.
Recently, a friend of mine referred me to a book by 37signals about their newer development methodology called Shape Up. I enjoyed reading it and found it inspiring in a comparable way to their previous book called ReWork I read a couple of years back.
However, somewhere midway through I realized: It is a trap! If you come to read such a book with the intent to learn how to organize your software development process, then you are going to be disappointed at the end. Let me explain why I think so.
It hurts, let’s adopt a methodology!
The quest to find the solution for organizing software development process is usually triggered by some kind of pain. The problem can be the overal feature velocity of the team, or it can be the cycle time, or recurrent quality issues. Sometimes the biggest pain comes from cross-team communication and collaboration. The pain points vary by team.
When such difficulties arise, the team members start seeking a way out, ideally a quick and proven way out of the valley of pain. And that’s exactly the time when people start researching the publicly available methodologies and trying to pick a savior which can help them to get their ship back on course and alleviate their pains.
Once a particular methodology is chosen, then you start hearing things like “we are implementing Scrum” or “we are in agile transformation”. Sometimes professional agile coaches descend on site to expedite adoption of the ultimate new methodology (or mythology).
See, THEY figured it out!
Speaking of publicly available methodologies, besides the well-known ones such as Scrum, Kanban, extreme programming, and scaled agile framework (SAFe), from time to time a new methodology gets traction and mentions in the media, usually published by a well-known technology brand describing how they have mastered their software development process. One such example is the Spotify Squad Framework (more about it later) which was quite popular about five years ago. Another example is the above-mentioned Shape Up by 37signals or Ataccama’s SpaceUP from our friends at Ataccama.
What do these “proprietary” methodologies have in common? They are primarily a marketing tool. While they might have been created with the best intentions and addressing actual internal needs, their publication is primarily aimed at demonstrating thought leadership and attracting talent: A talented software developer frustrated by how work is organized in his current company might easily get tempted to join those who have resolved the pain points and mastered the software development process to the level they can teach others how to do it.
Dashed Hopes
Coming back to the Spotify Squad Framework, while it was all the buzz back then and seemingly resolved many issues of scaling agile teams to larger and more complex product development organizations, it eventually emerged, that first it did solve some issues while creating others and second it was aspirational, and even Spotify itself never implemented it fully:
Why Spotify Squads Are a Popular Failure for Product Teams | Chameleon — www.chameleon.io
Explore the Spotify Squad framework, how Spotify incorporated it, the product development methods it helped with, and why today the $50 billion company has abandoned this organizational model. Plus, what you can learn from its failure.
Spotify’s Failed #SquadGoals — www.jeremiahlee.com
“The Spotify model” got a bunch of companies talking like Taylor Swift about startup culture, but four former Spotify employees reveal the truth: its eponymous way of working failed before it scaled.
The lesson here is that one can’t just blindly adopt something published by a successful company hoping that it will solve all software development issues in his or her concrete context. It might not apply well in the context, or it might later turn out it does not work in general. Proceed with caution and apply critical thinking.
And this is how WE do it:
All right, now after we have dealt with competition and their software development mythologies, let’s share our secret recipe for success. Or better to be honest and speak upfront: this is how we do it at the Cognitive Intelligence team at Cisco, it is not perfect, and it might not work for you.
Principles
First and foremost, as a team, it is important to agree on key operating principles. These shall be relatively stable, because they should reflect your core values. For the purpose of this article, let’s use the famous Manifesto for Agile Software Development, which clearly outlines such a set of principles:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
(Source: https://agilemanifesto.org/)
Objectives
Keeping the operating principles in mind, the team needs to identify the main pain points in its current development methodology and derive key improvement areas — the objectives.
These objectives are likely to be different for each team but will also change over time (see the next section). You might want to optimize knowledge sharing between product and engineering, you might need to accommodate for differences between research and development workloads, you might need to reduce cycle time, improve quality and customer experience, or reduce ongoing effort spent on maintenance tasks.
Once you identify your objectives, try to outline the team activities and rituals which will help you to achieve those objectives and don’t forget to agree on how you are going to measure progress.
Adaptation
The only constant is change. Throughout my years with Cisco’s Cognitive Intelligence team, we have been through several iterations of what we call Cognitive Planning Framework — an internal normative document which describes our development methodology. I drafted the very first version shortly after joining the team in 2015 and then we made a major revision of the framework approximately every two years with more frequent minor changes as needed.
When creating the methodology, we followed the outline above: start with the principles, identify objectives, define the key activities and their owners, and codify the process in the normative document describing what we do.
Anytime we tried to be aspirational and attempted to codify things we did not actually do, we realized it no later than during the next revision of the document and we did our best to get rid of the aspirational ballast. As this normative document is used during regular compliance audits, it needs to be as close to reality as possible.
Those regular audits are also a good trigger to make us have a look and review the planning framework. Anytime we reviewed it after some time, we realized we drifted as our needs changed since the last revision. For example, at some point we really needed to deliver complex initiatives encompassing the whole organization composed of many R&D teams. At that time, our main focus was on improving cross-team communication and collaboration and we needed means to synchronize at regular intervals to be able to operate in a lockstep.
As our organization grew and we diverged into several loosely coupled projects, the emphasis on overall coordination decreased and we shifted our focus to optimizing performance on a smaller scale for individual groups of teams. As a result, each team is taking a slightly different approach to interpreting and refining the framework, but it works as long as the overall objectives are met.
The Secret Ingredient
As seen on the Cisco Cognitive Intelligence example, the key point is not to find and implement some ultimate software development methodology. Even if you find one which perfectly fits your needs today, I would be really surprised if the same process would match your needs two years down the road. In case you are happy with your development methodology for many years in a row without change, the case might be that it is not actually good news, because it means you stopped evolving and adapting. Software development is still a relatively young and dynamically evolving discipline, thus tools and methods of the best teams in the industry change and evolve quite rapidly as well and many times represent a competitive advantage.
That does not mean you should stop reading about how other teams and companies organize their software development, but always take it with a pinch of salt and look for an inspiration, not for some magical solution: