Using OwlFactor to evaluate news quality
With the rampant spread of misinformation on social networks people are more aware they must evaluate the news with a critical eye.
Helpful guidelines, like the one below from The International Federation of Library Associations, are useful in spotting fake news and generally evaluating news quality:
But doing this with every article you see on Facebook or Twitter is difficult (see this example of evaluating by hand). So tools like OwlFactor make this easier by automating the guidelines above.
Let’s vet the news!
If you’ve installed OwlFactor’s chrome extension on your desktop browser, you can do this practical example together now.
Mosey on over to NPR and pick a random article to analyze, say this one: Google Employees Walk Out To Protest Company’s Treatment Of Women. The article gets an OwlFactor of 81, which denotes a high-quality article. Tapping on the OwlFactor icon gives you more detail:
You’ll see the boxes match up to IFLA’s infographic quite nicely:
- “Check the source” — Source box for NPR is green — indicating a high-quality site based on the scores of articles historically. This matches the About Us page of NPR which shows 50 yrs of high quality reporting.
- “Check the author” — Author box for Emily Sullivan is green — indicating that the historical scores of her articles is high and she has expertise in this topic based on her previous writings. This matches with her bio page which shows she “holds bachelors degrees in psychology and women’s, gender, and sexuality studies from Fordham University” — very relevant to the topic at hand.
- “Supporting sources” — Sources box is green — indicating the article has several diverse and extensive sources.
- “Read beyond” — The Tone box is green — indicating the tone of the article is generally neutral and not overly opinionated. This is calculated based on a machine learning model and is careful to exclude quotes.
Since all four boxes are green you know this article is high quality.
(Note: if you want to do this on your phone you’ll have to import the article into the CivikOwl app. Go to the Bookmarks tab and follow the instructions).
Details
If your spider-sense is going off and you’re thinking “how exactly are these scores being generated” that’s good! Your critical thinking skills are already being put to use.
Mouse-over the “i” bubble on the boxes and you’ll see an additional explanation of what the grade factors mean. And if you tap on “Show Details” you’ll get even more information on how these are calculated:
As you can see there is a heavy focus on links and quotes because The American Press Institute describes “journalism as a discipline of verification.”
The details section shows how OwlFactor counts the number of unique links, whether these are repeated too much, and whether they point back to the site too often. You can see the list of the individual links extracted from the article below.
OwlFactor also tallies the number of meaningful quotes (quotes with more than 4 words) and in the case of political articles looks to see if links are from across the political spectrum as an indication of if multiple viewpoints considered.
If you’re still curious about OwlFactor’s scoring model you may want to see our FAQ. If you have concerns or questions please email us: arjunm@owlfactor.com.
Another liberal-biased software tool?
There’s a perception that software companies have a liberal bias due to the background of their employees and that bias is baked into software they build. So a fair question to ask is will OwlFactor skew towards certain viewpoints?
While no model is free of bias, OwlFactor’s model is unique in that it wasn’t trained on articles designated as “good” or “bad”. Doing so would encode our bias into the model. Rather, it evaluates articles from a structural view, looking for measurable attributes like the number of sources and how diverse they are etc. This means you’ll find low- and high- scoring articles on nearly every news source, across the political spectrum.
As an example, here’s a medium scoring article on NPR and a high-scoring article on Reason, a popular right-leaning news outlet.
All that said, there’s no accepted methodology to assess political bias. So the best habit is to read multiple diverse well-written articles on a topic. OwlFactor makes it easy to find additional viewpoints on a story with its suggested articles feature just below the score details screen.
Conclusion
News has always had its share of misinformation, sometimes intentional and others times accidental. Getting into the habit of evaluating news quality will ensure you form opinions based on facts instead of fiction.
Get the most credible news on the hottest news topics via The Factual Owl newsletter — free at Owlfactor.com.
