Wiseman and Burch’s Flawed Study

In the latest Game Developers Conference, political scientist and author of Queen Bees & Wannabes Rosalind Wiseman and voice actress for Mortal Kombat X and Borderlands 2 Ashly Burch presented the results of a study which, they contend, shows that, as The Guardian phrased it, “even teenage boys are sick of sexist video games.” Presumably, once they collected the data, Wiseman and Burch proceeded to use common modalities of statistical analysis to decipher the feelings of survey participants towards a number of questions by quantifying the answers and displaying the results of their analysis in easy to understand charts. While I have no point of contention with what I discern is the process of data analysis nor with the results presented — given previous ESA studies it is no surprise that girls play puzzle and adventure games [1] more than MOBAs, and given the perception of gamer as someone whose primary hobby is to play video games on console or on the PC that more boys would identify as such — there is one major problem with the study that must be addressed: the data collection methodology.

In order for the study to have been credible in regards to data collection, Wiseman and Burch should have either personally gone to a number of schools and asked students to participate in the surveys. If they did choose this approach [2] they would have had to contact each school district or school, depending on policy, and ask for permission to conduct the survey, as well as asked each student for assent and each of the students’ parents for consent. Alternatively, after having obtained permission from the districts or schools, they could have contacted the teachers and let the teachers deal with the students’ assent and the parental consent. However, it does not seem that this was the case.

The Problem

Wiseman and Burch’s survey was posted to the Girls Leadership facebook page with a request that only teens take the survey. It was also posted on more than one Tumblr page, with some making it explicit that the intent was to provide data to fight against “problematic attitudes” online and make the internet into a “safe space.” The survey, hosted on SurveyMonkey, was then promoted via Twitter by both Wiseman and Burch. This mode of distribution leads to two major problems that render the data and the results presented useless.

The first major problem with Wiseman and Burch’s collection methods is the complete lack of control when obtaining data from the sample. Because the survey was widely spread online, and because SurveyMonkey provides no tools to identify the age or gender of the participants beyond voluntary disclosure, there is the very likely possibility that the survey was answered by individuals outside of the target population of teens or that the survey might have been answered more than once by a single individual. With no way of accounting for these variables, Wiseman and Burch cannot say that the results were collected by their target sample. In other words, the survey was neither valid nor reliable.

Additional readings on the validity and reliability of surveys can be found HERE, HERE, HERE and HERE as well as on a number of survey, sampling, and statistics textbooks.

The second major problem is, of course, the fact that surveys are studies that require human participation. Although they are not usually harmful to the participant, research ethics dictates that participants must explicitly consent to participating in a study, and in the case of minors, the target demographic for Wiseman and Burch’s study, that a parent must consent to the child’s participation. Although I have already outlined one of many procedures that can be followed when carrying out face-to-face surveys, when the surveys are performed online there is no way to ensure parental consent. This further complicates the potential use of the data collected.

In a recent follow-up commentary Wiseman says that her research was not intended to be considered as rigorous academic research and that it only intended to spark interest so that people would “look at larger issues this survey represents.” These comments, I think, are outright dishonest. It is inaccurate, to say the least, that the survey represents issues when the validity and reliability of the essay intended to demonstrate said issues is in question. If anything, the only issue represented is poor survey planning and implementation. But more importantly, Wiseman suggests that the survey was not meant to be taken as academic research. However, the investigation was presented as such. Wiseman presented her findings in a formal panel at the GDC, a conference now known as a place where the gaming industry and gaming intelligensia converge to share ideas. Furthermore, her results were presented as fact by Polygon, Raw Story, Time, The Mary Sue, The Guardian, Toronto Star, MTV, and other outlets.

What saddens me is that in her GDC talk Wiseman says that “I really went around the country to all different kinds of schools and said: “Please, have your students fill this out.” If this is true, then there was no reason for Wiseman or Burch to spread the survey online and taint the sample.

The Role of the Media

There has been some discussion that many media outlets acted unethically when reporting on Wiseman and Burch’s “findings.” I am partly in agreement. While it is obvious to me that some outlets might have acted unethically in their reporting, it is more a question of resources and the framing of the study rather than the fact that they reported on the study itself. The Mary Sue, for example, is an ideologically feminist blog with the intent of pushing feminist ideology and preaching to their order. When their writers picked up on an “academic study” that reinforces their bias, they report on it. The Mary Sue is clear in their intent and makes no pretense of being unbiased. Although they did report the findings as a legitimate survey, I find it hard to believe that they have a statistician on call to verify the validity of a given investigation. As such, they simply repeated what they heard. The same is true of Polygon. [3] However, outlets like The Guardian and Time are large and incredibly influential publications tasked with the purpose of bringing the news to their readers. These outlets have the resources to fact-check Wiseman and Burch’s claims, and as such they are more to blame for the spreading of flawed research than the staff at Polygon or The Mary Sue who just don’t know any better.

Closing Remarks

I appreciate what Wiseman and Burch are trying to do. With an industry where bland, unmarked protagonists outnumber female protagonists, it is understandable why someone would want scientific data to back up the claim that there should be more female representation. As someone who cried when Aeris died, whose favorite game is Portal (featuring a woman), and who is looking forward to Horizon: Zero Dawn the most out of all the E3 games (perhaps with the exception of Final Fantasy 7 Remake), I am someone who wants to see more female characters in video games. However, making up numbers and presenting flawed research as fact is not the way to go about doing this.

I strongly encourage Wiseman and Burch to engage in a follow-up survey where they can control for sample rather than launching a survey into the wilds of the internet and present those as findings. I’m sure that they won’t be disappointed by their findings.