How We Improved The Zombie Scrum Survey
How we used data from 2.000 teams to improve the survey and deliver more accurate results.
You can also listen to this post in this episode of ongoing podcast “The Liberators Network”.
In the summer of 2019, we launched the Zombie Scrum Survey. With this free online survey, we wanted to give Scrum Teams all over the world a way to see how they are doing. And stop the spread of Zombie Scrum. Even though the metaphor is admittedly tongue-in-cheek, we designed the survey as a reliable and robust way to help Scrum Teams reflect and improve.
Since then, 2.000+ teams (and 2.600+ individuals) have participated. With so much data, we want to deliver on our second promise: to better understand what makes Scrum Teams more or less effective and use those insights to improve the survey.
In this post, we share how we improved the Zombie Scrum Survey based on the data we collected. We also share preliminary findings.
Give it a try
If you haven’t already done so, we highly recommend giving the survey a try at https://survey.zombiescrum.org. It doesn’t cost you anything, doesn’t require an account and takes about 10–12 minutes to complete. Afterward, you receive detailed feedback based on how your team is faring compared to other Scrum Teams and we offer suggestions on what you can do to improve.
After completing the survey once, you can invite your entire team to (anonymously) participate as well and see if you feel the same way.
Our Findings in a Nutshell
We’ve run detailed statistical analyses on the data collected to date to better understand the process of Scrum Teams. We will capture our findings in more detail in an upcoming post and a peer-reviewed scientific publication, but here are some preliminary patterns:
- When Scrum Teams release to production during or soon after a Sprint, they deliver more value and have more satisfied stakeholders.
- When Scrum Teams focus on value (e.g. a Product Owner who keeps focus and closely involves stakeholders), more value is delivered by the teams, their stakeholders are more satisfied and team morale goes up.
- The ability of teams to focus on value and to release frequently goes up as teams become more cross-functional, spend more time on refinement and use their Sprint Reviews purposefully to gather feedback from stakeholders.
- Management support is vital to the success of Scrum Teams. As support goes up, teams are more able to focus on value, release frequently, and become cross-functional.
- These patterns generalize across teams of different ages and sizes, as well as organizations of different sizes and ages.
With the topics we currently measure in the model, we can explain 52% of the variance in stakeholder happiness, 44% of the variance in team value, and 48% of the variance in team morale. This basically means that we can predict the scores on the outcomes solely based on everything else we measure with quite some certainty. Ranges from 44–52% are high, as values beyond 20% are generally considered “strong” in the social sciences for this kind of cross-sectional research.
How Does This Help Scrum Teams?
With the model behind our survey, we can develop a better understanding of what makes Scrum Teams successful (or not). In this case, we define success as happy stakeholders, more value delivered every Sprint and higher team morale.
Our model gives direction to teams and organizations as to what to put effort into. Our data tells us clearly that Scrum Teams require two parallel processes: they need to release frequently to stakeholders and they need a Product Owner who keeps them focused on what is valuable to stakeholders. Although you could do one independently of the other, effective Scrum requires both.
Our model suggests that when teams want to support their Product Owner in focusing on stakeholder value, they should primarily invest in cross-functionality, refinement, the quality of Sprint Reviews, and making stakeholder collaboration a shared responsibility. When teams want to ship faster, our model suggests to primarily invest in cross-functionality, refinement, and release automation.
Furthermore, our model shows how important the role of management is. Stakeholders are less satisfied, less value is delivered and team morale is markedly lower when management hardly interacts with the teams, when they don’t help with the removal of impediments and when they don’t understand why teams work with Scrum in the first place.
“Stakeholders are less satisfied, less value is delivered and team morale is markedly lower when management hardly interacts with the teams”
Our model also shows that some things don’t matter. Scrum Teams can thrive in organizations of any size and age. Although the experience with Scrum does have a positive influence, the age of teams themselves does not matter.
None of these findings are earth-shattering in their novelty. For many in the industry, it will be no surprise that cross-functionality is critical to the success of Scrum Teams. What is novel, however, is that we now have more objective data to support (or reject) our claims and opinions about what matters.
“We now have more objective data to support (or reject) our claims and opinions about what matters.”
Improvements to the Zombie Scrum Survey
Now that we have data and a model, we can refine the Zombie Scrum Survey by keeping what is valuable and creatively destroying what isn’t. This allows us to improve the reliability of the measurements, the relevance of the feedback, and the value for Scrum Teams. We’ve made the following changes:
- We removed 50 items (60%) that didn’t statistically contribute to the model behind the Zombie Scrum Survey. Some of these items overlapped so strongly with other items that they basically measured the same thing, while others didn’t correlate with anything at all. Other items were too open to different interpretations.
- By removing many items, we effectively dropped the following topics: “Scrum Effectiveness”, “Celebration”, “Team Composition”, “Motivation for Scrum” and “Team Learning”.
- For some topics, like “Sprint Goals”, “Quality” and “Autonomy” we found that the original questions didn’t work as well as we’d like. We replaced the questions with new ones, often deriving from existing scientific instruments.
- The topics of “Psychological Safety” and “Cross-Functionality” were indistinguishable in the data. In nearly all cases, we could predict the scores on one topic based on the scores on the other. We don’t know how to explain the overlap unclear at the moment. We may need better ways to measure both. Or perhaps people stick to their own skill-sets in teams with low safety whereas people in teams with high safety simply find ways to help each regardless of their skills.
- In an effort to expand the explanatory power of the model, we added more topics that make sense from a theoretical perspective: “Cross-Team Collaboration”, “Continuous Learning”, “Sprint Retrospective Quality”, “Metric Usage” and “Inquiry and Dialogue”. When more teams have participated in the updated survey, we can see how they influence the model and if they indeed expand the explanatory power.
- In an effort to boost the reliability of the various measurements, we added new questions to topics with only a few items. This included “Refinement”, “Stakeholder Collaboration” and “Release Frequency”.
- More detailed improvements, refactors and adjustments are available in the changelog.
Impact on Existing Profiles
We’ve updated the profiles that are created for your team. New topics are now included and the feedback is based on the findings from our research.
We also retroactively updated profiles that were created earlier. Although we considered versioning, we wanted all current and former participants to benefit from our improvements. Since former participants haven’t entered scores for the new topics, some will be missing from the profiles. But teams can easily re-do the Zombie Scrum Survey to receive an updated profile.
The Road From Here
In the coming months, we will continue to gather data and analyze the model. Because this process itself is inherently empirical, the survey will change as new topics are added, existing ones are revised and more feedback is added. Here are some ideas we have:
- We would like to support teams that want to analyze trends over time.
- We want to offer more directed recommendations on specific interventions and exercises to try based on the results. Afterward, we’d like to measure how well they worked so we can empirically update the recommendations for teams.
- Right now, team scores are compared against a sample of all other teams. We would like to give teams the flexibility to select a specific sector, region, team size or another characteristic to base the comparison on.
- Right now, what we measure is based on self-reporting by the teams. Even though we tried hard to “objectify” the questions, we want to include more objective measures too. This is particularly helpful for topics like stakeholder satisfaction, team value, and management support.
Why are we doing this?
Too often, conversations about the “Do’s” and “Don’ts” for Scrum Teams are based more on opinions, personal preferences and biases than on objective data. Rather than using empirical evidence about what works, recommendations are too often based on well-intended advice from thought-leaders, experts and “gurus”. Although their advice may be right, it can also be wrong. We feel that Scrum Teams are better off when they have access to data-driven recommendations that are based on how Scrum Teams all over the world are working with Scrum and what is working for them.
Help Us Make This Possible
Both hosting and developing the free Zombie Scrum Survey is an expensive affair. On a monthly basis, we spend 150 dollars on hosting and required software licenses. For the continued development of the tool and the analysis of the data, we spent over 300 hours of work to date (or 25.000 dollars if you use a market-comparable rate).
So… if you use our survey and get value from it, or if you support our cause, please consider supporting us in covering these costs. You can do so with a one-time donation on PayPal or by a recurring donation as a Patreon. If enough people are willing to support us, we can keep the advanced features (e.g. trend analyses) free and continue to support Scrum Teams all over the world. We are really hoping for your support.
The Zombie Scrum Survey is available for free at https://survey.zombiescrum.org.
