Reproducible Robotics Research: Back to the Basics of the Scientific Method
In general, as any Robotics and/or AI PhD student knows very well, replicating the research results of other labs is quite difficult. The information that you can customary find in a reputed journal paper is usually not enough to reproduce the experimental results claimed by the authors, let alone to make comparisons of the strengths and weaknesses of the different methods proposed in the literature in term of performances.
This is a quite serious issue as the possibility to reproduce results it the cornerstone of the scientific method. ‘Science’ is defined by the possibility to experimentally verify, Karl Popper would say ‘falsify’, ‘theories’. This is even more striking when at least part of robotics research is more and more regarded as ’Science’, as for example it is witnessed by the recent launch by AAAS of the new journal ’Science Robotics’. By the way, what’s a ‘theory’ in Robotics and AI???
This uncomfortable situation doesn’t only make very difficult the cumulative process of research, it also severely impairs technology transfer and industrial exploitation. It is worth notice that while everybody includes a ‘State of the Art’ section in grant applications or business plans, the sad truth is that the state of the art is more or less insightfully guessed as published results cannot in many if not most cases be checked objectively. And what about the TRLs (Technology Readiness Levels) at the core of SPARC Strategic Research Agenda??? The community has been aware of this problem for a long time. In 2008 the European Robotics Network (EURON) started a Special Interest Group on Good Experimental Methodology and Benchmarking (coordinated by me and co-chaired by John Hallam and Angel P. Del Pobil) , the following year within IEEE RAS the TC Pebras was started, that it is still very active today, in 2012 the Euron GEM SIG led to the establishment by euRobotics aisbl of the Topic Group on Replicable Robotics Research, Benchmarking and Competitions (actually coordinated by me).
A very long series of workshops (more than 20) at various IROS, ICRA and RSS has debated the related issues and proposed examples of reproducible experiments and measurable results.
We are now at the point that we can provide concrete directions and guidelines for reproducible research in robotics and AI. In September 2015 the first ever Special Issue on Reproducible Robotics Research was published on the IEEE RAS Robotics and Automation Magazine. Reproducible Research is now becoming an IEEE priority. There are no more excuses to indulge in ‘proof by video’ and ‘it worked once in my lab’ attitudes :-)
It is time to act! It’s time for Robotics and AI to come of age!
Do you want to know more? check the links here below!
- Bonsignorio F., Del Pobil A., (Eds.), Replicable and Measurable Robotics Research, IEEE Robotics & Automation Magazine, 22(3), 2015 http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7254310
- Bonsignorio F., del Pobil A.P.,Toward Replicable and Measurable Robotics Research [From the Guest Editors], Robotics & Automation Magazine, 22 (3), 32–35, 2015 http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=7254280&punumber=100
- Stoelen M. F. , de Tejada V. F., Huete A. J., Balaguer C., Bonsignorio F., Distributed and Adaptive Shared Control Systems: Methodology for the Replication of Experiments, IEEE Robotics & Automation Magazine, 22(4),137-146, 2015, http://ieeexplore.ieee.org/document/7307132/
- The IEEE RAS TC Pebras website: http://www.ieee-ras.org/performance-evaluation
- The Euron GEM Sig website page still listing related events http://www.heronrobots.com/EuronGEMSig/gem-sig-events