The Truth About GitHub’s Gender bias
By now, you may have heard of the study of GitHub and bias showing that women get their code accepted more than men, except when people know that it’s a woman. If you haven’t, feel free to look at these blogs: Quartz, BBC News, Business Insider, Tech Times or one of the many other sources that have covered this. The story, though, when you actually read the research itself, is not quite as black and white as everyone is making it out to be. As someone who’s done a lot of regression analysis in my day, let me take a moment to clear up some confusion surrounding the actual results of this study.
Firstly, the most important result of this study, in my opinion, is that female users are more likely, on average, to get their changes on GitHub accepted than male users. This conclusion is based on the researcher’s use of a program to link 1.4 million GitHub user profiles with email addresses with Google plus profiles, where this group of users has self-identified their gender. Female users’ acceptance rate (or merge rate) is 78.6% and men’s is 74.6%, a difference that is statistically significant.
This result is really impressive when you consider the fact that women make up approximately 11.2% of open-source programmers while men make up around 88.8% (according to a 2013 study).
In addition to these general results, the paper further divides up the user data into “insiders: people who are owners or collaborators of a project” and “outsiders: everyone else.” Additionally, the researchers developed a novel, and seemingly solid, method of grouping users into those with “gender-neutral” profiles and those with “gendered” profiles.
What’s downplayed in the study is the fact that for insiders, when their gender is known, women still have a higher rate of acceptance than men. With gender-neutral profiles, women have a slightly lower rate of acceptance than men. It’s unclear whether or not these results are statistically significant or not.
For outsiders, both women and men’s rate of acceptance drops when their genders are known. To my dismay, the authors didn’t publish the results for men outsiders, which is highly problematic. However, I found the numbers in a Nature article: “The acceptance rate for women coders was 71.8% when they used gender-neutral profiles, but dropped to 62.5% when their gender was identifiable. Men with gender-neutral profiles had an acceptance rate of around 69%, which declined to 63.3% for those whose gender was clear.”
The authors gloss over this similarity by saying “There is a similar drop for men, but the effect is not as strong” due to the fact that “women have a higher acceptance rate of pull requests overall, but when they’re outsiders and their gender is identifiable, they have a lower acceptance rate than men.”
Ok, so this paper hasn’t been peer-reviewed, which is the scientific method of maintaining quality, and it has some issues, which we’ve covered, but the authors have submitted it online for comments, which is great. There’s one last problem to resolve here. How do we know that the people reviewing the requests actually considered the person’s gender? Actually, we don’t. We can therefore not say for sure that these results were definitely bias-related. However, there’s still a troubling correlation here. For outsiders, women coders who use gender-neutral profiles get their changes accepted 2.8% more of the time than men with gender-neutral profiles, but when their gender is obvious, they get their changes accepted 0.8% less of the time.
In the interview with Nature, one of the researchers noted: “The difference is statistically significant, but whether the difference is substantial is another question that’s open for interpretation.”
The other thing to consider here is that there is a lot of other research (and I mean hundreds, if not thousands of academic articles), that has been peer-reviewed and accepted by the scientific community, showing that unconscious gender bias is extremely prevalent in our society, and it affects all of us profoundly. It affects both women and men. It affects us on an unconscious level — which means that you might consciously disagree with it. (“What? But I would never associate women with humanities and men with science?!”) We get it as a sort of unwanted present from all the millions of images and stories that we’ve been exposed to throughout our lives — on TV, in advertising, and in movies. No one’s immune to society-wide bias — you might have less of it in one area, but you’ll have more of it in another area.
For a bit of fun, check out Harvard’s Implicit Association Test for free to test your own biases — it takes about 10 minutes to complete. Our best tools regarding bias are to increase our awareness and try to create more structure that eliminates potential bias to the extent possible.
This study, despite its flaws, shows us some good things and some bad things. Female users are more likely to get their code approved on GitHub than male users. However, there is a correlation, when you’re an “outsider” and a female GitHub user with a profile that shows your gender, to having a tougher time getting your changes approved than a male user with a profile that shows he’s a man.
