Reproducible Research in Computational Science: An Overview
The Definition of Reproducibility
The reproducibility of research is an outsider’s ability to reproduce the presented results of that research with reasonably small effort. According to a highly cited paper by Roger D. Peng, “the standard of reproducibility calls for the data and the computer code used to analyze the data be made available to others.” Peng suggests this standard is just the bare minimum needed and claims that reproducibility is a spectrum.
Rozier et al., at her paper, further builds on top of this idea and examines the reproducibility in terms of its association with the four elements of computer science research: theory, algorithms, code, and data. She has specific suggestions around what reproducibility means for each component, and how a researcher can achieve it.
Rozier says the theory is reproducible if all theorems the proposed theory relies upon are stated in the paper with proper citation. The theory itself should also be clearly described in the main text.
For algorithms to be considered reproducible, she claims it should be listed in its entirety, preferably in the main text if the algorithm is short enough or as an attached appendix. Rozier further suggests each line in the algorithm should be commented on, and an English-prose description of the algorithm should be provided alongside the formal presentation.
The third element, code, can be considered fully reproducible if the full code with comments is in an online repository. It should also provide documentation specifying why/what the code does, the requirements needed to build the code (like the operating system), and step-by-step instructions for building and running at least one illustrative example.
According to Rozier, “a key element of reproducibility for any empirical study is the availability of data.” Though she acknowledges that certain data sets could be dealing with sensitive data with customers or private records, and therefore cannot be published. But even with such datasets, Rozier claims one can anonymize the data and make it public.
On Reproducibility of “Bilateral Maps for Partial Matching”
According to the criteria described above, let’s evaluate a paper in terms of its reproducibility. Kaick et al., at Bilateral Maps for Partial Matching, represents a new local shape descriptor whose region of interest is defined by two feature points. Then they further demonstrate how this new feature descriptor is particularly useful for partial shape matching.
The paper, in my opinion, does a great job in terms of its reproducibility for the theory. They state all of the previous research on the topic and have a clear definition of their contribution. Kaick et al. are also very detailed when it comes to providing the underlying algorithm that fuels their research. They offer both formal and plain-English explanations of each step and further simplify understanding the algorithm by giving visualization around how it runs.
What Kaick et al. fell short in terms of reproducibility comes at the code aspect. Unfortunately, as most research paper on the field of Computer Graphics does these days, they do not provide any source code or a link to an online repository on their writing. I further tried tracking down whether they made any efforts to make the source code public. In that end, I visited each author’s websites and looked for a link to a code for the said paper. It was not surprising to see, none of the authors provided any such links. I was only able to find the paper’s main text.
Although Kaick et al. did a sub-par job providing reproducibility for the code, they did a decent job when it comes to the data. They cleverly used publicly available SCAPE datasets for their research. And they also provided a lot of examples of their results, though not the actual data of the results, just picture representations.
Overall, I think Kaick et al. ’s paper is more reproducible than many other Computer Graphics paper I read on my relatively short research years. At least they made every effort to make their theory and algorithm perfectly understandable. Although they did not provide the actual implementation, their clear explanation of the research made it easy for me to replicate what they code on my own. That is not to say there is a lot of room for improvement.
References
Peng, Roger D. (2011). Reproducible Research in Computational Science. American Association for the Advancement of Science. SCIENCE 2011. 10.1126/science.1213847.
Rozier, Kristin & Rozier, Eric. (2014). Reproducibility, Correctness, and Buildability: the Three Principles for Ethical Public Dissemination of Computer Science and Engineering Research. 2014 IEEE International Symposium on Ethics in Science, Technology, and Engineering, ETHICS 2014. 10.1109/ETHICS.2014.6893384.
Oliver van Kaick, Hao Zhang, and Ghassan Hamarneh. 2013. Bilateral Maps for Partial Matching. Comput. Graph. Forum 32, 6 (September 2013), 189–200. DOI: https://doi.org/10.1111/cgf.12084
Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: shape completion and animation of people. ACM Trans. Graph. 24, 3 (July 2005), 408–416. DOI: https://doi.org/10.1145/1073204.1073207
