User Testing Synthesis | KERNEL Week 1 Update
By Bianca Trovò | biancasama.eth
Ants-Review aspires to be a peer review platform focused on reviewers’ recognition and a transparent peer review process that nevertheless reduces bias in academia.
The protocol, based on the Ethereum blockchain, allows authors to issue public peer-reviews, the scientific community to openly access them and evaluate them and reviewers to be rewarded for their efforts.
Key words for our system are: recognition, (pseudo)anonymity, openness.
We wanted to know if the current features in Ants-Review’s design were meeting the needs of our potential users and how they felt about the peer-review system in general. For that purpose we run a qualitative survey targeting as users researchers (graduate students, postdoctoral researchers, principal investigators…) with practical experience in peer-reviewing academic papers and possibly with familiarity with web 3 tools.
We searched for interviewees through social medias (Twitter, LinkedIn, Facebook, Instagram) and Gitcoin Town Square. So far 11 users responded to the survey, with 4 of them fitting the ideal criteria of “academic researcher with knowledge in blockchain” (normally, we would look for ~5 users, following the Adjusted-Wald Binomial Confidence Interval, but we decided to broaden the target audience due to an initial lack of suitable samples with expertise in both domains).
The resulting reports were interesting as they depicted, in line with our view, the peer-review as an “independent” process ensuring “quality”, “validation/evaluation/validity” of scientific outcome with the drawback of representing a significant amount of work in terms of time and efforts that does not receive proper appreciation and compensation within academia.
Specifically, for what it concerns the 3 key features of Ants-Review (recognition, anonymity, openness), users gave positive feedback regarding monetary reward as a form of recognition for the time invested in performing a qualified peer-review. Surprisingly, having a reputation index incorporated in a novel peer-review metric system was not the highest voted feature.
Anonymity for reviewers was judged important to protect the identity of younger researchers if coupled with accountability of reviewers: this represented a salient trend in the reports denoting how preferences are still split between the advantages of a double-blind and an open-peer-review process which Ants-Review aims at conciliating with the integration of IPFS hash that guarantees traceability while reducing the biases associated with exposing researcher’s identities.
Finally, among the users with knowledge of web3 self-rated as >7/10 in a scale from 0 to 10, there was appreciation for the intuitiveness of Ants-Review’s UX and the elimination of the ‘middle’ roles represented by editors and publishers which add layers to the review process without increasing its value (“less bureaucracy” and more “freedom” for authors and reviewers).
However, at the same time, among those users there was some confusion regarding the flow of the hashing (judged “too complicated”) and doubts concerning the reappraisal of a peer-review systems without editors and publishers.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
Useful (re)sources for running qualitative user tests:
