And you may ask yourself, how did I get here? An investigation of reproducibility and open research
Supporting open research is central to my role as a member of the Research Services team within the Research and Digital Horizons Directorate. Our team champions Open Access and aims to help researchers disseminate their findings as widely as possible. Many researchers are on board with the principles of Open Access, particularly as it has become part of the criteria for submission to the Research Excellence Framework (REF). This is heartening to see, as there are many benefits to making research outputs available to readers at no cost. These benefits include a higher likelihood of citation, improved efficiency and practitioners, educators and policy makers can use the information. However, I would like to see the adoption of openness go even further by researchers sharing not only their research outputs but also all underlying data sources, methodologies, documentation and metadata. Azberger (2006) explained the need for open sharing of data, stating that the:
“…focus on publications often overshadows the issues of access to the input of research — the research data, the raw material at the heart of the scientific process and the object of significant annual public investments. In terms of access, availability of research data generally poses more serious problems than access to publications.”
Sharing in this manner would help to tackle one of the most prominent challenges in research: reproducibility. As defined by The Turing Way:
“Reproducible research is work that can be independently verified. In practice, it means sharing the data and code that were used to generate published results”
In this blog, I will take a closer look at a prominent case that exemplifies the ‘reproducibility crisis’ and how open sharing of datasets and methodologies may be able to combat this threat to research integrity.
Reproducibility and the Curious Case of Diederik Stapel
In 2011, the social sciences community was rocked by the discovery that eminent Dutch psychologist Diederik Stapel had been falsifying data over a number of years. The case, reported in the New York Times , reads like the plot of a Netflix thriller. Alerted to allegations of research of fraud by an old friend, Stapel initially dismisses the claims as political manoeuvrings by academic rivals. After another friend in the academic community presses him on the specifics, he continues to assert that he is innocent. Professor Stapel then travels to the University of Groningen — where he had previously carried out research — to gather evidence to dispute the allegations. Things look promising as he recognises structures that he can use as details to back up his data from that time. On his way back, Stapel stops at the train station in Utrecht. This station was the scene for the study in which he had found that participants were less likely to sit close to an individual of another race in a row of seats if the surrounding area was full of litter. At this point, Diederik Stapel is confronted with the truth: the platform does not match anything he has described in the study. He has no choice but to admit the falsification of research data going back to at least 2004. Diederik Stapel now has 58 retractions to his name, placing him in 5th position on the Retraction Watch leaderboard. This case did not just mark the end of Stapel’s career but it also had a detrimental impact on the many PhD candidates that had been unknowingly using falsified data.
But how could Stapel get away with faking his data for so long? In their summary of the interim report published by Tilberg University in 2011, Mieke Verfaellie and Jenna McGwin outline the two factors that allowed him to continue unchecked. Firstly, the report paints a picture of Stapel manipulating his students with a mixture of charisma and intimidation. But charm and good reputation can only get you so far; I think that the more significant factor in this case was the lack of scientific integrity or critical oversight. As Verfaellie and McGwin state:
“A number of “red flags” should have been raised, including:
Colleagues were not aware of the fact that doctoral students did not collect their own data;
When students asked to see completed questionnaires, they were told that neither the schools nor Stapel had room to store them;
Insufficient clarity in the manuscripts as to how data were collected;
Data too good to be true (e.g., large effect sizes, small standard deviations);
Strange or improbable data patterns (e.g., cutting and pasting of identical scores in difference columns);
The culture in social psychology where data is not made publicly available”
The last four of these red flags all relate to a lack of openness with regard to data. If Stapel’s full datasets had been publicly available, he may not have been able to get away with falsifying data over such an extended period.
The Reproducibility Crisis
As part of my role as a Research Services Officer, I have helped to develop and deliver training on research data management during which we highlight ‘improved reproducibility’ as a benefit of open sharing of research. The case of Diederik Stapel provides a stark cautionary tale for any researchers unconvinced about the importance of openness and transparency. We also use the following infographic from a Nature survey of over 1,500 researchers to highlight the concerns about reproducibility within the academic community.
This study also asked researchers what led to problems in reproducibility and more than 60% said that selective reporting was a contributing factor. This would be negated if datasets and not just outputs were made openly available. However, many respondents also pointed to poor analysis and unavailable methods as potential causes of irreproducibility. This links into another key theme in our research data management training: the importance of documentation and metadata.
Documentation is detailed information about what is contained in a dataset and the methods used to create the dataset. Metadata is structured information about the data that is machine-readable. We encourage researchers to begin documenting as early as possible in the research process as this task becomes more difficult the more time has passed between carrying out the research and recalling all of the steps taken in the process. Documentation does not have to be complex or laborious — a simple README text file stored alongside the data can be used to record any action taken.
A move to greater transparency
In her excellent blog post Show Me the Data: research reproducibility in qualitative research, Louise Corti of the UK Data Service, expands upon the need not just for open sharing of data but also for production transparency and analytic transparency. Production transparency means that the methods used to collect data are fully detailed and shared. As Corti states, in the context of qualitative research:
“Evidence to demonstrate production transparency will vary from a description of research design, sampling, fieldwork, fieldwork materials such as topic guide and thumbnail sketches of interviews can provide a good deal of context. These could be sufficient for a future researcher to undertake a restudy or to revisit the raw data.”
By openly sharing all aspects of their methodology, researchers are providing greater context for their data, which makes it more usable. This also means that future researchers may wish to apply the methods to a different sample group or even use the research design for an entirely different research area. In this sense, openly sharing methods could increase the likelihood of citation for the original researcher and demonstrate the impact of the original research. As stated in this blog, UK research councils argue that publicly funded research must be able to demonstrate the good that comes of that money. The re-use of methodology across different research projects and perhaps even across disciplines would demonstrate efficiency savings in not ‘reinventing the wheel’.
Pre-registration: The gold standard for reproducibility?
In her analysis of the Nature survey on the reproducibility crisis, Monya Baker states:
“One of the best-publicized approaches to boosting reproducibility is pre-registration, where scientists submit hypotheses and plans for data analysis to a third party before performing experiments, to prevent cherry-picking statistically significant results later.”
It is easy to see how this practice would stop a researcher from shaping their data (either knowingly or unknowingly) to fit a desired outcome by selecting the data that agreed with their predictions and minimising or disregarding ‘outliers’. It would also help to minimise HARKing — hypothesising after results are known. In How to Crack Pre-registration: Toward Transparent and Open Science, Yuki Yamada articulates this as the practice of adjusting hypotheses to fit the data that has been collected thus creating false positives. This ‘puttting the cart before the horse’ would not be possible with pre-registration, as the hypotheses could not be altered after data collection.
Pre-registration seems to be a neat solution to a raft of questionable research practices by which the integrity of data can be damaged but it is not without drawbacks. Yamada outlines a number of ways in which researchers could circumvent pre-registration by leaving out data that does not fit the hypothesis or running the experiment until they get the result they want. There is also PARKing — the sidekick to HARKing — which is pre-registering after the results are known. Many of these practices seem quite labour-intensive and give a very dim view of researchers intentions — I would like to think that the number of researchers who would go to such lengths to ‘cheat the system’ or pre-registration is very small. A more fundamental problem posed by pre-registration is that it may not allow for exploratory research. Yamada outlines an instance in which their research team came across an interesting finding during the experiment phase of a project. If the project had been pre-registered, they would not have been able to explore this new finding, as it was not what they had set out to look at. In other words, pre-registration could be detrimental to the pursuit of knowledge as a narrow focus on a set hypothesis does not leave room for exploring unexpected findings. However, I think that pre-registration could be very positive for research integrity but it would require nuanced implementation to allow researchers to pursue interesting and significant findings even if these deviated from the original hypothesis.
What are our researchers doing about reproducibility?
Last year, Times Higher Education reported that 10 UK universities had signed up to the UK Reproducibility Network (UKRN) to promote reproducibility and research integrity. The network
“… is a peer-led consortium that aims to ensure the UK retains its place as a centre for world-leading research.
This will be done by investigating the factors that contribute to robust research, providing training and disseminating best practice, and working with stakeholders to ensure coordination of efforts across the sector.”
This group aims to change the way that research is conducted by appointing academic leads for research integrity at each stakeholder institutions who are tasked with championing the implementation of policies and procedures that will lead to a more consistent adoption of the openness and reproducibility across the HE sector.
The University of Manchester has not signed up as an institutional stakeholder of the UKRN. However, important work in promoting reproducibility is being carried out in the form of the Open Science Working Group (OSWG) founded by Andrew Stewart and Caroline Jay. The group has run a number of events to promote open research practice within the sciences including a regular journal club called ReproducbiliTea, which encourages early career researchers to meet regularly to discuss best practice, raise any issues and share resources to support open research. The OSWG has also built strong links with other institutions in the area. I contacted Andrew Stewart to get his perspective on what the University can do to overcome obstacles to openness and reproducibility and agree with his sentiment that the it is important to appoint, promote and reward staff for adopting reproducible workflows. When researchers are working with limited time and resources, it is imperative that openness and reproducibility are incentivised and supported by giving researchers the opportunity to undertake training in these practices. I also asked Andrew how the Library can better support researchers and he suggested Carpentry-style lessons in open-source programming languages and more general workshops on creating reproducible workflows. I will follow up on these suggestions with the appropriate teams within the library to see how we can implement them.
Another way that some of our researchers — along with colleagues across many different institutions — are working to promote good practice is through involvement in The Turing Way, an online guide to reproducible data science. This resource not only outlines what reproducibility is and why it is important but it provides practical, step-by-step guidance for using a variety tools to support research data integrity. In their own words:
“The Turing Way is a handbook to support students, their supervisors, funders, and journal editors in ensuring that reproducible data science is “too easy not to do”.”
Resources like this are invaluable for researchers who want to adopt best practices but do not know where to start. It is interesting that the research student is at the forefront of their intended demographic as this echoes the advice that we give with regard to data management that the earlier you get into good habits, the easier it becomes to manage your data in the long run.
The creation of the Open Science Working Group and The Turing Way demonstrate that as an organisation, we recognise the importance of reproducibility in research data. This is a promising start but much more needs to be done to bring research integrity to the forefront of our practices as a university. There needs to be more formal investment in reproducibility initiatives such as signing up as an institutional stakeholder of the UK Reproducibility Network and appointing an academic lead for Research Integrity.
So, what I can do to support reproducibility?
Investigating the issue of reproducibility has led me to reflect on what I and my colleagues within the research data management team can do to combat this. Openly sharing datasets that underpin research outputs and providing comprehensive metadata to accompany these datasets will help researchers to ensure the integrity of their data. It is our team’s responsibility to highlight the importance of open sharing of datasets and to continue to find new ways to make this as easy as possible for researchers.
Two key strands of our current work support these objectives. The first of these is the training we provide in research data management. This is what I see as the ‘winning of hearts and minds’, an opportunity to outline the many benefits of open sharing and the importance of good metadata. In addition to our workshop that covers all aspects of data management, we have also developed a file management session that focuses on adopting an organised, proactive approach to all data created during a project. This encourages researchers to build documentation into their practices and to create file architecture that will lead to the production transparency outlined by Corti without placing too much burden on researchers. The other aspect of our work that we hope will lead to greater sharing of datasets is the procurement of a data repository that allows researchers to upload and manage datasets easily. We are currently going through a tender process with data repository suppliers but the functionalities we have requested have researcher’s interests in mind. If we are to encourage busy academics to upload their data, we have to ensure that this process is intuitive, efficient and has demonstrable benefits.
