4 Ways to find review papers, systematic reviews, meta-analysis, and other rich sources of references — 2D Search templates, Connected Papers & more.

Starting your research in a totally new area and unsure where to start? Wouldn’t it be nice if someone had done the work for you already, surveying the research landscape of papers of importance and interest in the area you are interested in and even commenting and evaluating on the results?

Sounds like a pipe dream? Not really. Find one of the following types of research and things becomes tons easier as they bring together citation rich resources on a specific area as assessed by an expert in the area. Say the concept you are studying is “Servant leadership”. What type of items would help you quickly gain an understanding of the literature written on that concept?

But these citation rich items don’t exactly grow on trees. So how can you find them?

Here are the automated methods I know of (in order of most known to lesser known methods). There are also various more manual ways like hand searching journal titles (some fields will have particular outlets that publish such content e.g. Annual Reviews of … ), conference proceedings that aren't covered here of course.

  1. Keyword Search in Google Scholar
  2. Search within a database with a suitable Subject or Publication type
  3. Structured search template in 2D search (NEW!)
  4. Citation mapping tools like Connected Papers, CoCites. (NEW!)

In my examples below I am going to use “Creativity” as a sample topic.

The accuracy (or rather precision and recall) of these different techniques to identify reviews etc differ depending on the technique used. Simple minded boolean string matching is likely to have higher recall (less false negatives) but lower precision (more false positives), while with machine learning techniques which are increasingly used , depending on how the training is conducted it may lead to higher precision (less false positives) at the cost of lower recall (more false negatives) if trained that way. Lastly of course there is simple human curation or hybrid human+ML may get the best of both worlds, but human input is expensive and is unclear how much this is used today.

1. Keyword search in Google Scholar

Edit : Jan 2022 — Google Scholar now has a “review articles” filter.

New Google Scholar Review articles

You can still use the keyword based method below, I personally find the keyword search based methods tends to give more precise results than Google Scholar’s new filter.

There isn’t much to say about this. To find a systematic review, just go to Google Scholar and enter your keyword <topic keyword> systematic review.

This generally works because most reviews and systematic reviews have those words in the title, and Google Scholar generally prioritizes matches with titles. e.g. “Determinants of organizational creativity: a literature review”

creativity intitle:review OR intitle:meta-analysis OR intitle:”a survey” in Google Scholar

Still you can try a more refined search like intitle:review <topic keyword> to see if it gives you better results. e.g. intitle:review creativity.

Or even combine terms with boolean like this.

creativity intitle:review OR intitle:meta-analysis OR intitle:”a survey”

Sidenote: Interestingly, China’s Baidu Xueshu (百度学术) / Baidu Academic or their answer to Google Scholar, does have a automated function to try to find review papers via their analysis function

Analysis function in Baidu Academic that surfaces review papers

It also tries to find “classic papers” (seminal papers), recent works and theses. I’m still trying it out, while Baidu Academic seems to have a smaller index than Google Scholar, it does seem to be in the same ballpark as Microsoft Academic, Lens.org and acts as a big web scale cross disciplinary database that should be sufficient for most needs. And yes, it covers the usual international journals.

2. Search within a database with a suitable document type or subject type filter

Of course the limits of using keyword searching in Google Scholar is that you are going to get a lot of irrelevant results past the first few results even with the refined search syntax above.

A database which has a filter for reviews, meta-analysis etc would take the guess work out of it wouldn’t it? Indeed some databases do have such a filter, typically as a publication type filter.

Some of these databases include

  • Web of Science — use the “Review Article” filter
  • Scopus — use the “Document type — Review“ filter
  • Semantic Scholar — use the “Publication Type — Review or Meta Analysis”
  • Pubmed — use the “Article type — Review, Systematic Review or Meta-analysis”
  • Psycinfo — use the “Methodology — Literature review, systematic review, meta analysis, metasynthesis”
  • ProQuest Research Library — use “Document Type: Literature review”

TIP : In fact, almost all the methods below do not give 100% accuracy because the ‘reviews’ are often identified using complicated search filter patterns (‘Hedges’ in Pubmed speak) or Machine learning (Microsoft Academic, Semantic Scholar) using methods akin to #3 and to to some extent #4 below.

Take PubMed, where there is a filter for “Reviews”, “Systematic reviews” and “Meta-analysis”.

Creativity in Pubmed filtered to Review, Systematic Review and Meta-analysis

Unfortunately not many databases have such a filter, most are on the life sciences side (e.g. Psycinfo via methodology facet).

Again while this is useful, a cross-disciplinary database that has such controlled terms would be useful.

Of the Cross-disciplinary databases Scopus and Web of Science do indeed have filters for this. For example, you can use the Document type “Review” for Scopus and Web of Science.

Document type — Review in Scopus

Note : When looking at filters with the label “review”, one must be careful to test if this means review papers or things like “book reviews”

Review Articles in Web of Science

However both databases are not only pay to access but also have selective coverage.

Another big cross disciplinary index that has a review filter is Semantic Scholar, though it shares a lot of the same underlying data with the next entry below via a data partnership.

https://www.semanticscholar.org/

Sometimes, a database might not have a outright filter for such items but a somewhat hidden way to do such searches is available if they support controlled vocabulary and the controlled terms includes a subject for reviews or bibliographies.

This means someone has helpfully classified these items and all you need to do is to search with these subjects.

TIP : In some vocabularies, a controlled term for say “Systematic reviews” might be use for articles/content written on the theory and practice of doing systematic reviews, rather than for labelling the item as a systematic review (i.e. as a publication type), while others may include both. You can check by running the search and looking at what appears.

One example is ERIC the education database, and in their thesaurus you can see Literature Reviews, combine that with your topic and you can do a search for literature reviews!

Creativity + Literature reviews in ERIC

Is there one with as broad a coverage as possible (competitive with Google Scholar) AND some way to filter this way?

And indeed one exists — Microsoft Academic (which sadly will be sunset 31 Dec 2021).

As noted in this article, Microsoft Academic is one of the largest sources of academic content out there and they use NLP and Machine learning to auto-classify over 200 million pieces of content into subjects including Systematic review”, “Review article”, “Meta-analysis” and “Bibliography

So do searches like

A slightly more advanced technique would be to combine them all together with Boolean but as Microsoft academic doesn’t support Boolean, you can do this in Lens.org which has Microsoft Academic data and does support Boolean. However, I recommend you look at the next method first before trying it out.

3. Structured search template in 2Dsearch

As noted above, the PubMed filter for reviews is actually a complicated search pattern to try to identify reviews.

Search Strategy Used to Create the PubMed Systematic Reviews Filter

Can we do the equalvant for a Cross disciplinary database like Microsoft Academic or Lens.org?

Indeed we can. Using both methods employed in #1 and #2 we can create a complicated Boolean search in Lens.org that maximizes the recall of the search (so we can get as many correct items) while also keeping the precision of the search high (so we can avoid wrong results).

Why Lens.org? It is one of the largest open sources of academic records out there (>200 million as of 2020), including data from Crossref, Pubmed, PMC, JISC CORE, Microsoft Academic and more AND unlike other large search indexes has a very powerful Boolean and search function that allows you to craft a powerful search strategy to maximise recall and precision. For example the search template is powerful enough to include a section to search within specific journal like Annual reviews or Cochrane Database of Systematic Reviews

Due to the complexity of the search, I employ the use of 2Dsearch to create a search template for your ease of use.

Saved search template

I detail the method in — Finding reviews on any topic using Lens.org and 2d search — a new efficient method. You should really read it, but the cliff notes is as follows.

  1. Go to the saved 2D search template here (no sign in required)
  2. Scroll to the bottom and locate the block in the search strategy that is on “creativity” and change the terms to your topic (e.g. “Servant Leadership”)
Edit the part of the box that says Creativity with your topic keyword

3. Generate the search results in Lens.org

Some results generated by Lens.org

4. Enjoy!

That’s it.

4. Citation mapping tools like Connected Papers, CoCites.

I’ve been tracking the rise of tools that are designed to help literature review by mapping the literature. However while tools like Citation Gecko and CoCites are useful and have the potential to identify review or seminal papers, they generally do not outright try to identify them.

The only exception to this is the new Connected Papers tools. I’ve done a long review here but in this piece I will focus on the relevant portions.

Connected Papers, allows you to put in one paper and it will generate a map of similar papers using a similarity function based on a combination of co-citations & bibliometric couplings.

For example take this 2008 paper of mine — Improving Wikipedia Accuracy: Is edit age the answer? and throw it into Connected Papers to see what it generates.

Connected Papers generated network for the paper

The twist here is that you can use the papers found in this generated network to try to identify not just seminal works by clicking on “Prior works” button but also try to “find surveys of the field or recent relevant works which were inspired by many papers in the graph” by clicking on the Derivative works.

Derivative Works generated

While this method doesn’t always work, it works a lot more than I expected and you get the added bonus of finding similar pages as well as seminal papers on top of review papers!

Methods compared roughly

At this point you may be bewildered by the methods available and wondering which to use. There is no one sized fit all answer, it depends on how comprehensive you want the search to be and how long you want to do the search.

In general, the fastest and quickest way would be to use keyword searching in Google Scholar, it won’t be the most complete, but it tends to give you good results on the first page. If you are in a discipline where there is a dominant database like PsycInfo, then you should definitely use that as well.

If for some reason you only want to find such items in a restricted set of “high impact” journals, you can use the filter functionality in Web of Science and Scopus, though there is no reason I see why you might do that.

Semantic Scholar’s review publication type seems to give me more mixed results, and while Baidu Academic analysis function can help surface some relevant China paper missed by other methods, I find like Semantic Scholar it can be a bit hit or miss in classifying review papers.

The saved 2D search template here is something I recommend and use when I want to be comprehensive, it tends to give me the best balance between precision and recall and works well in my experience for most disciplines.

Connected Papers isn’t particularly geared for this purpose , though it’s other functionality like mapping out related papers and seminal papers is helpful.

If you want to do a comprehensive overview for a thesis say, I would try all of these methods! Some might find additional relevant items missed by others.

Conclusion

I hope the techniques shown here are of use to you. Once you have found all these review papers, systematic reviews, meta-analysis, bibliographies what do you do next? Stay tuned for my next post!

--

--

--

Thoughts on open access, focusing on discovery, delivery from the academic librarian’s point of view.

Recommended from Medium

What Are the Most Important Elements of Data Storytelling?

What are Small-world Network Models?

Viziflu: Initial Release

Feature Selection For King County Housing Data

The Column formula in Excel is used to find out the number of a column where a reference cell is…

How to deliver your model result to non-technical stakeholders

Ultra Clean Resume

Limit5 — Experimenting with Forecasting Visualizations in Looker

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aaron Tay

Aaron Tay

A Librarian from Singapore Management University. Into social media, bibliometrics, library technology and above all libraries.

More from Medium

From reactive to proactive: meet the entrepreneurs using AI to transform how we fight wildfires

Helping Computers Find Food in Text

Screenshot of an article, where the words that humans would consider to be food have been boxed in red.

Data: The Fuel of the Digital Economy

Language of logistics: How user voices help us build better assistants