How to Quickly and Confidently Review Research

Tools and systems you need to set yourself apart without drowning in citations

If a wall of studies leaves you in a cold sweat, this method will help you. Photo by author.

Much of my work as a coach, educator, and writer has come from reviewing articles, ‘body-hacks’, and celebrity exercise and diet plans, all of which point to citations to support their ‘scientifically proven’ claims.

In my reviews, I frequently find that these article exaggerate study findings(at best) and often point to ‘supporting’ studies which are in fact completely irrelevant or even outright contradict the claim.

Readers are hungry for sources who can wade into the technical details, provide context, and better enable them to make good decisions. In my experience, most writers and coaches shy away from engaging with the primary sources for several key reasons:

• They can’t access the full text of the original work.
• They lack an efficient workflow to get through the information.
• They lack the confidence to question research by experts in a field.

Although these are legitimate challenges, they can be overcome with a few useful tools, and the benefits of a reputation as an honest, thoughtful, critical reviewer of the science more than outweigh its costs.

The paywall problem

If you’ve ever tried to check a claim posted in a major news source or across your newsfeed, you likely ran into something like this:

Screenshot by author.

The journal that published this study charges 40$ to access the full text, and without paying that price, you’re left with nothing but the abstract- the researcher’s summary of the study. Since most writers aren’t employed by institutions willing to pay extortionate subscription prices to countless journals, most writers will simply take the abstract at face value, and getting deeper is what will distinguish you from the hack.

Finding a study is relatively simple, even when the source article fails to link the study it’s reporting on. The press release, article, or op-ed will usually include comments from one of the study’s authors or the university. A search of Google Scholar using key words from the study and the author or university’s name, sorted by most recent, will usually do the trick for breaking research.

When you scan Google Scholar, if the work is freely available, you will see it listed to the right of the study abstract.

Screenshot by author. Studies to the right are available in full text.

If it’s not listed to the right and you can’t get access, you have a few options:

• Google Search: Often, ‘paywalled’ studies are publicly posted. Typing the study’s name followed by filetype:pdf in Google may uncover it.
• Unpaywall: This free Google Chrome extension searches its database for studies which are published “Green” open access. Publishers are still charging for the work from their site, but it’s available free, legally, via the author’s academic institution.
• “Black” Open Access: There are public projects available which provide open access to research. If you use these, know their continued availability is uncertain. But when researchers have to cheat to do the hard peer review needed to further scientific thought, it’s an open question whether they’re a net harm or a public good.

Reviewing efficiently

Shoddy salesmen and lazy writers know they can get away with poor evidence because the barrier to check them is so high. Even if you can access the 20 studies they cite to support their claim, the combined weight of hundreds of pages of technical papers will deter most critics. Fortunately, you don’t have to become an expert in the research to get through it efficiently and provide your audience meaningful advice.

To get through the mass of papers, I recommend Zotero, a research organization app students and academics use to collaborate on and cite research. With it, you can directly download and organize the research by project.

You will certainly find a workflow that works for you, but I find this approach works as a start.

I conduct my reviews in three phases:

Collect

• Look through the study titles and select only those that are relevant to your audience.
• Download all the studies into Zotero within a ‘collection’ for each article you intend to write.
• Skim the abstract of the first study and ask: what does my audience need from this?
• Make a note of what you need and attach it to that study.
• Don’t read the full text of the studies yet. Once you’ve got your note attached to the first study, move onto the next until you know what you need from each.

Interrogate

• Don’t go back to the source article.
• Scan each study for the specific information you recorded in your notes. Some studies will be worth an in-depth review, but most can be skimmed. If you’re looking to check a specific claim or learn more about an idea, focus on the ‘discussion’ and ‘results’ sections. If you have a question about how a certain fact was collected, look to the ‘methods.’
• You should finish your review of each study with the key takeaways that are relevant to your question, limitations of the study that affect your confidence in it, and further knowledge you may need to usefully inform your audience.
• Oftentimes, these studies will point to other studies- usually systematic reviews or meta-analyses- which can help you understand the big picture, especially if you’re not deeply knowledgeable about that field. If the topic is important enough, it may be worth doing another round of collection and interrogation.
• Regardless, stop after two rounds. If you’re not expert enough in the topic to be confident in your conclusions after two rounds, I recommend either letting the project go or bringing on a collaborator who has the knowledge you lack, as I did here.

Evaluate

• Collect your finalized notes on each finding like index cards, gathering them into evidence for the claim, against the claim, and about the claim,
• Start writing.

The authority problem

If you’re reading this far, I’m willing to bet you already see the value of this skill, but even the most expert journalists, writers, and coaches sometimes struggle with the idea of “challenging” published research. In our current media environment, threatened by climate change, pandemics, and ‘fake news,’ no one wants to be branded with the title of ‘science denier.’

That’s fair, but this isn’t a battle between “Science” and “Anti-Science.” Research is complex, and even when the findings of a study are correct, those findings are useless if they don’t reach the right audience or are wrapped in so many technical details that the audience doesn’t understand or trust those findings.

As writers interpreting scientific findings for the lay public, we’re advocates, not adversaries. We give a voice to researchers doing the hard work of drawing useful conclusions from noisy data. We arm our readers with the information they need to protect themselves predatory scientific publishers, deceptive advertising and misinformation. And when we do so with integrity, clarity, and attention to detail, rather than assuming a footnote is Proof-of-Truth, we help build public trust in scientific findings that can save lives.

To see an example of this process, evaluating the evidence for fish oil from a reputable source, click here: