Jul 6, 2020
4 Ways to find review papers, systematic reviews, meta-analysis, and other rich sources of references — 2D Search templates, Connected Papers & more.
Starting your research in a totally new area and unsure where to start? Wouldn’t it be nice if someone had done the work for you already, surveying the research landscape of papers of importance and interest in the area you are interested in and even commenting and evaluating on the results?
Sounds like a pipe dream? Not really. Find one of the following types of research and things becomes tons easier as they bring together citation rich resources on a specific area as assessed by an expert in the area. Say the concept you are studying is “Servant leadership”. What type of items would help you quickly gain an understanding of the literature written on that concept?
- Review articles (sometimes known as survey articles, they are more narrative reviews and have less formal inclusion critera) e.g. Servant leadership: A review and synthesis
- Systematic reviews (similar to review articles but tend to be more methodical in setting out how studies are found, screened and selected) e.g. A systematic literature review of servant leadership theory in organizational contexts
- Meta-analysis (Systematic reviews + pooling together results/data from multiple studies for additional analysis) e.g. Do ethical, authentic, and servant leadership explain variance above and beyond transformational leadership? A meta-analysis
- Phd and Masters Theses (Theses tend to have very comprehensive reviews) — Some examples
- Bibliographies (References lists often annotated with comments on a certain topic) — e.g. The Hine Bibliography of Resources on Servant Leadership
But these citation rich items don’t exactly grow on trees. So how can you find them?
Here are the automated methods I know of (in order of most known to lesser known methods). There are also various more manual ways like hand searching journal titles (some fields will have particular outlets that publish such content e.g. Annual Reviews of … ), conference proceedings that aren't covered here of course.
- Keyword Search in Google Scholar
- Search within a database with a suitable Subject or Publication type
- Structured search template in 2D search (NEW!)
- Citation mapping tools like Connected Papers, CoCites. (NEW!)
In my examples below I am going to use “Creativity” as a sample topic.
The accuracy (or rather precision and recall) of these different techniques to identify reviews etc differ depending on the technique used. Simple minded boolean string matching is likely to have higher recall (less false negatives) but lower precision (more false positives), while with machine learning techniques which are increasingly used , depending on how the training is conducted it may lead to higher precision (less false positives) at the cost of lower recall (more false negatives) if trained that way. Lastly of course there is simple human curation or hybrid human+ML may get the best of both worlds, but human input is expensive and is unclear how much this is used today.
1. Keyword search in Google Scholar
Edit : Jan 2022 — Google Scholar now has a “review articles” filter.
You can still use the keyword based method below, I personally find the keyword search based methods tends to give more precise results than Google Scholar’s new filter.
There isn’t much to say about this. To find a systematic review, just go to Google Scholar and enter your keyword <topic keyword> systematic review.
This generally works because most reviews and systematic reviews have those words in the title, and Google Scholar generally prioritizes matches with titles. e.g. “Determinants of organizational creativity: a literature review”
Still you can try a more refined search like intitle:review <topic keyword> to see if it gives you better results. e.g. intitle:review creativity.
Or even combine terms with boolean like this.
creativity intitle:review OR intitle:meta-analysis OR intitle:”a survey”
Sidenote: Interestingly, China’s Baidu Xueshu (百度学术) / Baidu Academic or their answer to Google Scholar, does have a automated function to try to find review papers via their analysis function
It also tries to find “classic papers” (seminal papers), recent works and theses. I’m still trying it out, while Baidu Academic seems to have a smaller index than Google Scholar, it does seem to be in the same ballpark as Microsoft Academic, Lens.org and acts as a big web scale cross disciplinary database that should be sufficient for most needs. And yes, it covers the usual international journals.
2. Search within a database with a suitable document type or subject type filter
Of course the limits of using keyword searching in Google Scholar is that you are going to get a lot of irrelevant results past the first few results even with the refined search syntax above.
A database which has a filter for reviews, meta-analysis etc would take the guess work out of it wouldn’t it? Indeed some databases do have such a filter, typically as a publication type filter.
Some of these databases include
- Web of Science — use the “Review Article” filter
- Scopus — use the “Document type — Review“ filter
- Semantic Scholar — use the “Publication Type — Review or Meta Analysis”
- Pubmed — use the “Article type — Review, Systematic Review or Meta-analysis”
- Psycinfo — use the “Methodology — Literature review, systematic review, meta analysis, metasynthesis”
- ProQuest Research Library — use “Document Type: Literature review”
TIP : In fact, almost all the methods below do not give 100% accuracy because the ‘reviews’ are often identified using complicated search filter patterns (‘Hedges’ in Pubmed speak) or Machine learning (Microsoft Academic, Semantic Scholar) using methods akin to #3 and to to some extent #4 below.
Take PubMed, where there is a filter for “Reviews”, “Systematic reviews” and “Meta-analysis”.
Unfortunately not many databases have such a filter, most are on the life sciences side (e.g. Psycinfo via methodology facet).
Again while this is useful, a cross-disciplinary database that has such controlled terms would be useful.
Of the Cross-disciplinary databases Scopus and Web of Science do indeed have filters for this. For example, you can use the Document type “Review” for Scopus and Web of Science.
Note : When looking at filters with the label “review”, one must be careful to test if this means review papers or things like “book reviews”
However both databases are not only pay to access but also have selective coverage.
Another big cross disciplinary index that has a review filter is Semantic Scholar, though it shares a lot of the same underlying data with the next entry below via a data partnership.
Sometimes, a database might not have a outright filter for such items but a somewhat hidden way to do such searches is available if they support controlled vocabulary and the controlled terms includes a subject for reviews or bibliographies.
This means someone has helpfully classified these items and all you need to do is to search with these subjects.
TIP : In some vocabularies, a controlled term for say “Systematic reviews” might be use for articles/content written on the theory and practice of doing systematic reviews, rather than for labelling the item as a systematic review (i.e. as a publication type), while others may include both. You can check by running the search and looking at what appears.
One example is ERIC the education database, and in their thesaurus you can see Literature Reviews, combine that with your topic and you can do a search for literature reviews!
Is there one with as broad a coverage as possible (competitive with Google Scholar) AND some way to filter this way?
And indeed one exists — Microsoft Academic (which sadly will be sunset 31 Dec 2021).
As noted in this article, Microsoft Academic is one of the largest sources of academic content out there and they use NLP and Machine learning to auto-classify over 200 million pieces of content into subjects including “Systematic review”, “Review article”, “Meta-analysis” and “Bibliography”
So do searches like
- Creativity Systematic Review
- Creativity Review Article
- Creativity Meta-analysis
- Creativity Bibliography
A slightly more advanced technique would be to combine them all together with Boolean but as Microsoft academic doesn’t support Boolean, you can do this in Lens.org which has Microsoft Academic data and does support Boolean. However, I recommend you look at the next method first before trying it out.
3. Structured search template in 2Dsearch
As noted above, the PubMed filter for reviews is actually a complicated search pattern to try to identify reviews.
Can we do the equalvant for a Cross disciplinary database like Microsoft Academic or Lens.org?
Indeed we can. Using both methods employed in #1 and #2 we can create a complicated Boolean search in Lens.org that maximizes the recall of the search (so we can get as many correct items) while also keeping the precision of the search high (so we can avoid wrong results).
Why Lens.org? It is one of the largest open sources of academic records out there (>200 million as of 2020), including data from Crossref, Pubmed, PMC, JISC CORE, Microsoft Academic and more AND unlike other large search indexes has a very powerful Boolean and search function that allows you to craft a powerful search strategy to maximise recall and precision. For example the search template is powerful enough to include a section to search within specific journal like Annual reviews or Cochrane Database of Systematic Reviews
Due to the complexity of the search, I employ the use of 2Dsearch to create a search template for your ease of use.
I detail the method in — Finding reviews on any topic using Lens.org and 2d search — a new efficient method. You should really read it, but the cliff notes is as follows.
- Go to the saved 2D search template here (no sign in required)
- Scroll to the bottom and locate the block in the search strategy that is on “creativity” and change the terms to your topic (e.g. “Servant Leadership”)
3. Generate the search results in Lens.org
4. Enjoy!
That’s it.
4. Citation mapping tools like Connected Papers, CoCites.
I’ve been tracking the rise of tools that are designed to help literature review by mapping the literature. However while tools like Citation Gecko and CoCites are useful and have the potential to identify review or seminal papers, they generally do not outright try to identify them.
The only exception to this is the new Connected Papers tools. I’ve done a long review here but in this piece I will focus on the relevant portions.
Connected Papers, allows you to put in one paper and it will generate a map of similar papers using a similarity function based on a combination of co-citations & bibliometric couplings.
For example take this 2008 paper of mine — Improving Wikipedia Accuracy: Is edit age the answer? and throw it into Connected Papers to see what it generates.
The twist here is that you can use the papers found in this generated network to try to identify not just seminal works by clicking on “Prior works” button but also try to “find surveys of the field or recent relevant works which were inspired by many papers in the graph” by clicking on the Derivative works.
While this method doesn’t always work, it works a lot more than I expected and you get the added bonus of finding similar pages as well as seminal papers on top of review papers!
Methods compared roughly
At this point you may be bewildered by the methods available and wondering which to use. There is no one sized fit all answer, it depends on how comprehensive you want the search to be and how long you want to do the search.
In general, the fastest and quickest way would be to use keyword searching in Google Scholar, it won’t be the most complete, but it tends to give you good results on the first page. If you are in a discipline where there is a dominant database like PsycInfo, then you should definitely use that as well.
If for some reason you only want to find such items in a restricted set of “high impact” journals, you can use the filter functionality in Web of Science and Scopus, though there is no reason I see why you might do that.
Semantic Scholar’s review publication type seems to give me more mixed results, and while Baidu Academic analysis function can help surface some relevant China paper missed by other methods, I find like Semantic Scholar it can be a bit hit or miss in classifying review papers.
The saved 2D search template here is something I recommend and use when I want to be comprehensive, it tends to give me the best balance between precision and recall and works well in my experience for most disciplines.
Connected Papers isn’t particularly geared for this purpose , though it’s other functionality like mapping out related papers and seminal papers is helpful.
If you want to do a comprehensive overview for a thesis say, I would try all of these methods! Some might find additional relevant items missed by others.
Conclusion
I hope the techniques shown here are of use to you. Once you have found all these review papers, systematic reviews, meta-analysis, bibliographies what do you do next? Stay tuned for my next post!
