How we’re adapting a technique for keeping the law reliable to science
When I first heard the term “Shepardizing,” I ignored it because I had no idea what it was and wrongly assumed it was irrelevant to what we are working on at scite. Indeed, it sounds like something that has to do with corralling sheep or some kind of meat pie, not scientific publishing, but it kept coming up over and over in conversations. It’s safe to say that many scientists, if not most, don’t know the term. So what is Shepardizing?
Shepardizing
Shepardizing is a term named after Frank Shepard who in 1873 introduced a process by which, “An individual checking a citation..will be able to find out various information, such as how often the opinion has been followed in later cases and whether a particular case has been overruled or modified.” In short, it allows lawyers to search and cite reliable precedents without needing to read every court case and opinion ever referencing it. Today, various websites, like Casetext, make it easy to check to see if a case has been overturned or not with a simple search.
And while Shepardizing has become a standard practice in law, there’s been no parallel in scientific research despite the idea being discussed as early as the 1950s.
The fact that no such system in science exists has had serious negative effects on research as well as society. In one case, a letter from 1980 claiming to show opioids are not addictive went on to become cited as fact even though it was only five sentences long.
A critical look at the report in 2017 (nearly forty years later!) revealed that it was uncritically cited and the letter now features a prominent warning to readers:
Such uncritical citation highlights the key issue: unlike in law, in science, there is no easy way to tell if a scientific report has been supported or contradicted, there is simply a number of citations and no easy way to figure out how something has been cited without reading each referencing paper in full. At the extreme researchers even continue to cite work after it is retracted!
At scite, we’re working to solve this problem by introducing a way for researchers to automatically identify how a scientific report has been cited by displaying citation contexts and classifying them by where they appear in a paper and if they are supporting or contradicting. To date, we’ve analyzed over 12 million full-text publications extracting over 430 million citation contexts targeting over 30 million distinct articles. scite reports can be accessed by visiting scite.ai or by downloading a browser plugin for Chrome or Firefox, which brings scite analytics to virtually any scientific publication on the web.
