What 3 Years Of Our Own Scientific Research Brought
And the potential of future research
If you’ve been following our content for a while, you know we love to incorporate scientific research where we can. We believe it is important to ground our beliefs about Agile, organizational change, and teamwork on robust evidence instead of just our preference or personal opinion.
Because the use of scientific evidence in our field is relatively new, we’ve begun to contribute to scientific research directly through collaborations with academics such as Professor Daniel Russo, Ph.D., several PhD-students, and others with academic training.
The aim of this post is to give you insight into how this has worked out so far. We also need your help to take it further.
Our Research To Date
While we wrote the Zombie Scrum Survival Guide several years ago, we noticed that very little empirical research has been done to understand what it is that makes some teams work, and others not. We also observed that the research that had been done didn’t trickle into the professional community. So a wild idea emerged: “What if we collect data about Scrum and Agile teams, and use it to answer this and related questions?”. It was a great opportunity to dust off my scientific training and love for statistics. So the Scrum Team Survey was born to simultaneously help teams and collect anonymous, team-level data to drive our research.
While the data collection was running, we wrote over 25 in-depth blog posts to summarize scientific insights on topics such as Agile transformation, burnout, motivation, pair programming, team diversity, and team fluidity/stability. We hoped that this would reconnect our field with academic insights, as others were also starting to do around that time (like Nico Thümler and Joseph Pelrine).
In January 2020 we received an email from Professor Daniel Russo, Ph.D. from the University of Aalborg. He had tried the Scrum Team Survey and wondered if we were willing to share data for research purposes and to co-write a paper on it. So I shared the results from the Structural Equation Modelling (SEM) I had already done. I also still had a ton of structured data from interviews, workshops, and coaching sessions with teams and clients that I’d collected in the hope of writing scientific case studies, but never got around to. This was the starting shot for our first paper, “A Theory of Scrum Team Effectiveness”. It was published in 2023 in the high-impact journal “Transactions of Software Engineering and Methodology” (TOSEM). We developed a model from 13 case studies and tested it with data from almost 2,000 teams. It took two years to write the paper and one more year to guide it through the peer-review process. The core findings are shown below:
In 2023, we submitted a second academic paper with Daniel Russo, Ph.D. called “The Double-Edged Sword of Diversity: How Diversity, Conflict, and Psychological Safety Impact Agile Software Teams” to TSE. The paper has been reviewed by three academic peers and we’re now making some small improvements before it can be published. Work on this paper began in 2021. We hope it will be accepted and published in (early) 2024.
At the time of writing, we just finalized a third academic paper with Daniel Russo, Ph.D. about scaling approaches (like SAFe, LeSS, etc) and how they impact team effectiveness and stakeholder satisfaction. Work on this paper started in 2021. This paper has been submitted to the journal “Empirical Software Engineering” (EMSE) where it will be peer-reviewed by three academic peers. We hope this paper will be accepted and published in 2024. A prepublication is available on arXiv.
At the time of writing, we’ve begun data collection for a fourth study. The aim of this study is to investigate teamwork quality in Agile teams and how it is affected by organizational support structure and team fluidity/stability. Due to the scale of this study, which includes surveys and interviews at a number of organizations, we hope to publish our preliminary results in 2024 and a peer-reviewed article somewhere in 2025.
The scientific process sometimes feels frustratingly slow to me, especially compared to the speed at which new blog posts and podcasts can be published. However, we’ve learned that this slowness is deliberate and valuable. It allows reviewers to read our papers multiple times before they provide structured feedback. It also gives us time to write a paper, digest it again, and look at it again a few weeks later to improve it. Science isn’t in the business of jumping on the bandwagon of what is hip. It is about developing robust knowledge. That takes time, even though it is frustrating at times.
So, Who’s Paying For All This?
So it takes a lot of time to collect and analyze data, write papers, and publish them. You might be wondering who is paying for that time. The truth is that most of it is unpaid. Since my daily work is focused on the development of the Scrum Team Survey and The Liberators, I use my weekends and vacations for scientific research. This works out. But it does put a constraint on how much time I can spend on it, and it would be lovely to speed up at some point by having a few days a week available for it. Also, Daniel Russo, Ph.D. graciously offers his time for weekly calls, co-writing activities, reviews and to connect us with the academic community.
Fortunately, we can already fund some of our research through the financial support of the 500+ patrons of The Liberators. We are also generating a modest revenue from the Scrum Team Survey.
Research Roadmap
So where do we go from here? Last year I set up a research roadmap with Daniel Russo, Ph.D. that has already substantially changed since then, as any roadmap should. But an overarching theme remains: How can we make Agile teams and their organizations more effective? We hope to answer the following research questions over the coming years:
- A Theory of Scrum Team Effectiveness: Peer-reviewed & Published in Transactions of Software Engineering and Methodology (TOSEM).
- The Double-Edged Sword of Diversity: Submitted to the academic journal Transactions of Software Engineering (TSE). Three academic peers have reviewed the article and we are now making small changes to ready it for publication.
- Do Agile Scaling Approaches Make A Difference? An Empirical Comparison of Team Effectiveness Across Popular Scaling Approaches: Prepublication submitted for peer review to Empirical Software Engineering (EMSE).
- How Teaming Strategies Impact Team Effectiveness: Data collection is running. This includes interviews with about 10 informants and quantitative data collection through surveys. We expect this to be ready for prepublication in Q3-Q4 of 2024.
- The Impact of Working from Home on Team Effectiveness: Data collection is running. We expect this in Q4 of 2024.
We also have a number of other research questions we’d like to answer:
- What kind of support from management seems most important to create more effective Agile teams
- Which types of interventions are most likely to improve the effectiveness of teams over time?
- To what extent do frameworks actually help teams and organizations work effectively? How large or small is the correlation between the framework in use and the actual behavior on the ground?
- What are more effective ways to drive change instead of through frameworks?
Our Ambition: A Non-Profit Foundation For Agility Science
Our ambition is to move the research activities into a non-profit and independent foundation. Its board would consist of scientists in (Agile) software engineering and organizational psychology, including of course Daniel Russo, Ph.D., and professionals with a strong academic interest (like myself).
Its mission would be to translate scientific insights into practical recommendations for Agile teams and organizations, to engage with the professional community in research, and to fund and support research initiatives. We hope to fund this foundation through sponsors and contributors, which will include The Liberators. But we first have to pull ourselves up by our bootstraps, so this will take some time.
On a more personal note, I would love to turn this research into a proper Ph.D. program, but I have yet to find a solution that allows me to also continue my work for The Liberators and the Scrum Team Survey.
Closing Words: How You Can Help
In this post, I summarized the initiatives we’ve deployed at The Liberators to bring more scientific insights into our profession. Although it takes a lot of effort to collect evidence and make sense of it, we believe it results in a lot of valuable insights for practitioners.
We have many more research questions we’d like to answer. For this, we can certainly use your help. First, you can consider becoming a patron at patreon.com/liberators. We love that our research is partially funded by the community itself. Second, you can contribute to our growing dataset (now over 10,000 teams) by using the Scrum Team Survey with your team. Despite its name — which will change — we also offer versions for Agile teams and even non-Agile teams. You get a ton of valuable insights for your team, including evidence-based feedback, and we can use your team-level and anonymized results to drive more research. Third, we’re always on the lookout for partners who are willing to help with our foundation. If you have ideas, please reach out! Finally, if you have a high-quality dataset but no means to analyze it, we may be able to help.
You can stay up to date with our scientific writings here on Medium.
