When to Update Systematic Literature Reviews in Software Engineering
by Emilia Mendes, Claes Wohlin, Katia Felizardo and Marcos Kalinowski
The following blog post briefly describes the main findings and implications of our Journal of Systems and Software (JSS) article, which provides recommendations on when to update systematic literature reviews to the software engineering research community.
The Need for Well-Justified Updates
During our investigations, we found a total of 436+ SLRs published in Software Engineering (SE), within the period from January 2004 to May 2016. Furthermore, we identified 20 updated SLRs. These searches provide evidence that many SLRs in SE are potentially outdated; thus affecting our current aggregated understanding of the state-of-the-art in those SLRs’ research topics.
When new evidence is added as part of updating SLRs, different findings and conclusions from those reported initially may be identified. Therefore, updating SLRs may contribute to different purposes. For example, (a) providing a continuous update of the state-of-the-art on a research topic; and (b) identifying how that particular research topic is evolving.
How to keep SLRs current, more specifically concerning how and when to update SLRs, has also been a topic of discussion in other fields, such as medicine and environmental sciences. Within the context of SE, SLRs also need updating; however, the discussion on this important topic was rather limited, which led us to conduct this investigation to provide recommendations for our research community.
Recommendations on When to Update SLRs
To propose guidance on when to update SLRs in SE, we have investigated existing decision frameworks and detailed guidelines in other more mature disciplines with respect to SLRs. We put forward that the decision framework proposed in (Garner et al., 2016) — 3PDF henceforth, can be used as a basis to guide the decision of when to update SLRs in SE.
To reach this conclusion, we conducted a literature review of SLR updates in SE and contacted the authors to obtain their feedback relating to the usefulness of the 3PDF within the context of SLR updates in SE. Second, we used these authors’ feedback to see whether the framework needed any adaptation; none was suggested. Third, we applied the 3PDF to the SLR updates identified in our literature review. The 3PDF showed that 14 of the 20 SLRs did not need updating.
These findings reinforce the importance of using a decision support mechanism (such as the 3PDF) to help the SE community decide when to update SLRs, allowing them to properly justify the need for updates and avoiding wasting scientific effort with SLRs that do not need to be updated. The 3PDF is shown in below, and further details on how it can be applied within the SE context can be found in our paper.
Implications for the Software Engineering Research Community
In our paper, we provided recommendations on when to update SLRs in SE. We put forward that these recommendations should be followed by researchers active in the community when updating SLRs to assess and justify the need for updating.
We believe that one of the several takeaway messages for the SE research community is that the ‘maturation’ time between the publication of the original/previous SLR and its update should be longer than two years, although exceptions exist. Such a suggestion does not however bypass the need to apply the 3PDF to assess whether an SLR update is genuinely needed. Our results show that many resources are wasted in carrying out updates that did not need to be conducted.
Another important aspect that has been brought to light is that, whenever SLR updates do not include new relevant methods, they should at least provide results that reflect a change in findings, conclusions, or credibility, when compared to the original SLR. The point here is that the effort to carry out an SLR update should be justified by some novelty, in either the methods used, new findings or conclusions, or increased credibility. If nothing has changed, those SLRs should not be accepted for publication, as one may as well rely on the findings already presented in the original SLR.
The implications also reach out to newly starting Ph.D. students, i.e., they should not waste any effort to carry out an SLR update if such an update is not justified by a novelty in some manner. And the justification for the decision to not update (or to update) an existing SLR can be based upon applying the 3PDF to the original SLR, and additionally supported by the evidence and recommendations provided in our paper.
Furthermore, based on the evidence, we believe that an additional take away message on “How ” to update would be to employ the same mechanisms used in the original SLR being updated. To complement our suggestions regarding “How” to update SLRs in SE, we have made recommendations elsewhere as to what search strategies to employ when updating SE SLRs.
We expect these findings to help the SE research community to optimize their efforts when employing SLR methodology to provide a current aggregated understanding of the state-of-the-art on relevant SE research topics.
