Eight take-aways from the first Australasian Open Science conference
This week saw Australia’s first open science conference hosted by the University of Queensland in Brisbane.
The two days saw a mixture of traditional presentations, 5 minute “lightning talks”, hands-on workshops, hackathons, and an Open Science Jam in which small groups competed to have the idea with the greatest traction on twitter.
The meeting was the brainchild of psychologist Eric Vanman who attended the Society for the Improvement of Psychological Science in the United States and wanted to recreate something similar in Australia.
Like SIPS, there was a diverse crowd, skewing very much towards the younger generation — PhD students and early career researchers.
It meant that there were a few cultural references that older attendees (myself included) found ourselves scratching our heads at.
Memes aside, there were a number of key points that I took from the meeting.
#1 Open science is flourishing
The idea of open science has certainly taken hold in recent years and is growing in a number of exciting and complementary directions. Presenters introduced a range of new initiatives — some being pushed by small groups of young researchers, others by the scientific establishment.
In her lightning talk, Fiona Fidler introduced SCORE, an initiative from DARPA (an agency of the United States Department of Defense) aimed at developing measures to “enable someone to understand the degree to which a particular claim or result is likely to be reproducible and/or replicable.”
Nik Steffens, Harry Manley, and Mike Phillip talked about their experience with the Psychological Science Accelerator, described as psychology’s answer to CERN. It’s an initiative to enable rapid large scale replication studies through researchers working together and pooling the data they collect.
Cooper Smout introduced another collective effort — Project Free Our Knowledge. The idea is for researchers to commit to boycotting closed access journals as a collective with names held anonymously until critical mass has been reached.
Nick Tierney introduced ROpenSci, a community of R users working towards more open scientific and statistical practices.
Meanwhile, Rebecca Willen talked about IGDORE, the scientific institute for independent researchers that aims to make research more global and accessible.
#2 There are lots of tools to help open scientists
In parallel to the presentations and hackathons were a number of workshops that provided hands on introductions to the various tools available to the open scientist and strategies for using them transparently.
Topics included machine learning…
…R Markdown for creating reproducible (and comprehensible) workflows…
…and integration of R with GitHub.
For those (currently) less savvy with programming, Joel Larwood gave a demonstration of Jamovi, a free stats package built on R that doesn’t require coding.
The meeting was also an opportunity for me to talk about developments in science and blockchain (having first explained what blockchain is) and let researchers know about what we’re doing at Frankl.
#3 Preregistration is here to stay
One of the main reasons identified for the low reproducibility of a lot of research findings is that scientists are encouraged to keep working on their data until they find something interesting. It’s a recipe for false positives.
Preregistration combats this by committing researchers to a particular course of action before they start. It makes it clear what was planned and what was exploratory.
Kristina Horne introduced Registered Reports, which take this a step further. The study is reviewed and accepted in principle before the data are collected, regardless of the results.
Sean Murphy spoke about adversarial preregistration in which two teams of researchers agree upon the design of a study that will test the competing predictions of their favoured theories. One lesson he learned when doing this is that teams also need to commit to the implications for their theory if the results did not turn out as they had predicted.
Matt Williams noted that preregistration can be undermined when the data collected necessitate a change in the analysis plan. Decisions about removing outliers or transforming the data can allow researchers to p-hack — choosing the analysis that gives the most favourable result.
Still, it’s clear that preregistration is becoming an increasingly important part of research — at least in psychology — with a number of discussions addressing how best to introduce the concept to students.
#4 Resources are needed to help teach open science
Beyond preregistration, there was a recognition that open science requires a broader shift in university teaching practices.
Blake McKimmie led a discussion of how psychologists should teach classic experiments when so many findings have recently been questioned. It’s a particular problem for first and second year teaching where lecturers are often presenting research that lies outside their expertise.
One suggestion was an online resource that would rate the studies covered in undergraduate textbooks according to how robust the methods were — and whether subsequent studies had replicated their findings.
Other hackathons resulted in a list of open science resources for students.
Jennifer Beaudry argued that research is needed to understand how best to encourage and help students and their supervisors to adopt open science practices.
Along similar lines, the winning proposal in the Open Science Jam came from Emily Kothe and her team. They suggested (and began recruiting collaborators for) a study investigating what students already know about open science.
#5 Moving to open science means acknowledging imperfections of past research
Teaching new scientists is one thing but it’s important to recognise that for established scientists, moving towards open science involves admitting problems with past scientific practices.
Rebecca Willen described how she has added disclosure statements to her CV, acknowledging which publications were not as open as they could have been or involved what she now recognises to be questionable research practices.
However, Kai Sassenberg reminded us that the natural reaction of most researchers is to be defensive in the face of criticism and to worry about the reputational damage done by admitting problems in old research.
#6 Open science isn’t just about psychology
Psychology has been at the forefront of both the replication crisis and attempts to improve scientific practices. But many of the issues affecting psychology research apply to other areas of research. And while most people at the conference had a background in psychology or neuroscience, there were representatives of other areas of science.
Ecologist Elise Gould noted that her field typically involves a more exploratory approach than psychology — there’s less testing of null hypotheses. But many of the questionable research practices identified in psychology can be mapped onto similar concerns in ecology.
Her colleague Hannah Fraser noted that HARKing (hypothesizing after the results are known) appears to be a particular problem.
#7 We need a global perspective on open science
Just as we should take lessons from different fields of science, it’s also important to consider how open science plays out in different countries and cultures.
Taciano Milfont, a Brazilian working in New Zealand, talked about the importance of conducting replication studies in different cultures.
For me, the most powerful perspective came from Samiul Hossain, a clinical psychologist and Masters student at Macquarie University. Samiul articulated the importance of open science for the developing world, focusing on his home country of Bangladesh.
#8 Open science is the way science should be done
On my way home from Brisbane I listened to a podcast discussion between PeerJ co-founder Jason Hoyt and Chris Chambers, an Australian now at Cardiff University and a driving force behind multiple open science initiatives including Registered Reports.
Chris talked about being a junior researcher struggling to understand why his research was being rejected from journals, not because it wasn’t rigorous, but because the results didn’t tell an exciting enough story. It taught him to play the academic game. And it was only later in his career that he realised his initial “naïve” reaction was the correct one.
For young researchers who haven’t yet been indoctrinated in the perverse norms of academia, open science is so obviously the right way to be doing things. For those like me who have been around a bit longer, meetings such as Australasian Open Science are a reminder of why we got into science to begin with.
Next year, the conference moves to Melbourne. I for one can’t wait.
Resources
Materials from the conference are available via the Open Science Framework
I’ve also uploaded my slides to SlideShare
At Frankl, our mission is to make open science easy and rewarding for scientists. If you’d like to know more, you can read our whitepaper, check out our website, follow us on Facebook and Twitter, or join our Telegram channel.
