How to Apply Metrics for Inclusion to your Open Source Project
I’ve recently given a couple of presentations for Open Source Program Offices, interested in better understanding and applying metrics for inclusion. During one of these presentations, which included a high level summary of CHAOSS project, someone asked: ‘but how do you actually apply those metrics in practice?’.
A great question, which I think is essentially asking:
“How do we take these long, overwhelming lists of attributes for evaluating something like governance, leadership, events or project ‘places’ and apply them in a way that feels empowering (not overwhelming), achievable(time-bound), meaningful (not random) and impactful for both people and project?”
The first part of the answer, is to remember that open and inclusion cannot be decoupled. Historically, we believed that by being open we were by default inclusive — but we now know that’s not true.
This means is that the most achievable goals for inclusion, can be easily embedded in your overall project goals. You’ve had the power all along.
Let me share how I’ve done this, using metrics for Inclusive Leadership as an example.
Connect to an Existing Need or Goal
(Inclusive Leadership Metrics)
Ensuring our projects are designed for diverse leadership (‘actually open’) is a critical part of ensuring goals can be met, and even surpassed.
When evaluating the how, of applying inclusive metrics — I start by working to understand existing project goals. Sometimes those goals are written down, and other times they’re expressed as a challenge. For example: “I don’t need more contributions, I need more help triaging contributions”.
Translated to a project goal, that might be:
An increase of 60% in contribution to maintainer-tasks (testing PRs, validating issue/big reports, merging code) frees up 30% of engineering time (tracked by rate of P1 completion on project plan).
I would then write a complimentary goal for open/inclusion + leadership (because these are advanced contribution-type tasks), for example:
As a result of actions we take to act on insights provided by the Inclusive Leadership Metrics, we see that at least 50% of those contributions come from demographics currently under (or not) represented with all contributors rating their experience contributing as 5/5.
Understand
The second step is to review and understand these 6 guiding principles for leadership. It only takes a few minutes.
Evaluate
The third step is to apply those principles to the project goal. I created this checklist for our example.
NOTE: I highly recommend having someone not familiar with the project, go through this on your behalf. This is like ‘user testing’ for your open source project. I’ve done this with great results (make sure you provide stipend of some sort).
Take Action
Finally, for each ‘unchecked checkbox’ —you now have a call to action.
Of course it depends what resource and time you have available, but the important thing is, that you have newly visible way to achieve success for project.
Some ways I’ve done that:
- Created a ‘Team Page’ to list roles and responsibilities of those in leadership roles. I also used this as a way to create a new ‘Emeritus’ role to honour past leadership contribution.
- Added ‘good first PR’ .
- Ran an ‘Open Source’ Maintainer training course, for diverse participants. This is a higher effort, but a great value exchange to help those in underrepresented groups step into leadership roles in your project and open source overall.
- Submitted a Pull Request on repositories, requesting simple changes (inclusion bug).
- Community & team feedback (with option for anonymous feedback/participation) on how to address gaps I’ve seen.
- Added a ‘greet bot’ with invitation to get involved with leadership tasks; with a link to the team page, AND a coaching channel for learning maintainer tasks.
- Ran an ‘AMA’ with diverse open source maintainers .
Repeat
From this particular set of steps I have seen an (significant) increase in non-male, racial, neuro and age diverse contribution to projects overall. It’s important to ensure that these practices are part of software development workflows, and that you continue to track your progress over time.
There is no, one size fits all ‘inclusion’ metric for an open source project, but there are categories of metrics, like this one, that quite easily expose areas you can improve your project health and success overall. I’ll keep writing checklists like these, and sharing them when I can.
