You asked for journalist rankings Elon — OwlFactor delivers
Update 11/10/18: The original post below was written around a concept with ratings for publishers, not journalists. We received positive encouragement as well as some thoughtful criticism. So we’ve now built the real thing while acting on the feedback. A site with ratings for 5000+ journalists across 50+ sites. A more detailed post is forthcoming.
I’ve long been an unabashed Elon Musk fan, particularly after reading the brilliant four-part blog post on Wait But Why. So I was quite interested in Elon’s recent tweet that there should be a site to rate the credibility of journalists and publishers — an idea we’ve been working on for a while.
The broad concept is a good one, in light of the all-time low levels of trust the public has in the media, and at least 600,000 people seem to agree with him.
But his suggested implementation — that people rate individual journalists and outlets — seems questionable as far as yielding a useful signal. Quite simply, too few people know how to assess and vet content.
In addition, everyone has biases so they may rate a publication or journalist poorly simply because they disagree with the content rather than the content being inaccurate. Indeed, Elon himself was accused of this with the recent investigation into safety practices at Tesla. People may also rate a piece of content without knowing enough about the subject so their ignorance factors into their rating.
I suspect Elon thinks the “wisdom of crowds” statistical approach addresses both problems but it’s not clear that people’s ratings would be independent, or that the population will be diverse enough, both of which are important for the wisdom of crowds approach.
So, we modified Elon’s suggestion to account for the limitations above and replaced his implementation with an algorithm we’ve been working on for about a year now.
Without further delay, we’re thrilled to announce CivikOwl’s first publisher rankings! (EDIT: now goes to our new journalist ratings site: owlfactor.com/ratings)
I hope you like it folks… and if you’re reading this Elon, a CivikOwl-branded Tesla Model 3 is thanks enough.
We’ve ranked 250 publishers across nearly 200,000 articles in May 2018. The results reveal some wonderful surprises on great sources of journalism that you may not have heard of. Check it out and see where your favorite publication ranks.
Wait, who the heck are you, CivikOwl, to rate publishers?
Good question. We are a small startup in San Mateo who is passionate about quality news and has been working on news-related tech products for some time now. We’re largely self-funded, with no ties to the media or any political party, and do our best to be transparent on everything about our product and company.
About that algorithm, how does it work?
CivikOwl defines the quality of an article based primarily on how well it has been researched. This is calculated by looking at the quantity, quality, and diversity of citations (links) in the article — a methodology commonly taught in high schools and college essay writing.
In addition, CivikOwl looks at the prevalence of the article with “Owls” — CivikOwl users who are the most well read on over 1000 topics that we classify every news article in — so that rankings are not driven by pure popularity but rather by those who understand the topic. We gather this anonymous data from users who use our Chrome extension and our beta app.
You can see details of our ratings for every “top article” in the rankings table — just scroll to the bottom and you’ll see all the grade factors and individual citations. Our hope is that our score matches your intuition for how good the article is.
It’s worth reiterating that the ratings are on an article-specific basis rather than a publication-specific basis. This means you’ll see a range of quality for every publication. Within many publications some journalists work very hard to source their work meticulously, while others do not. Our goal is to highlight the work of the best journalists.
For now our rankings aggregate all articles for each publisher. We’ll soon have journalist-specific rankings, as well as topic-specific rankings, e.g. who is the top journalist for foreign affairs or for healthcare policy etc. And we’ll update these rankings regularly, at least once a month.
Important Caveats
Because our scores assess how well-researched articles are, and the prevalence of each article with our best-read readers, they aren’t a guarantee of a publisher’s neutrality or even their absolute accuracy.
Hence, a low average score doesn’t necessarily mean a publisher is poor quality. And while our “Owl” community is small right now we lean on the link analysis, which can mean good articles that are lacking in links, for example breaking news, may score poorly.
But a high average score does mean that the publisher tends to post well-researched articles, which CivikOwl’s “Owl” experts may be reading. This is a good place to start as you look for quality news sources. In fact, journalists would do well to make it easy for their readers to vet their content (which would lead to higher CivikOwl Rankings).
Do you find the rankings useful? Want to see something more? Let us know: arjunm@civikowl.com or comment below.
And please clap for this post so others discover it. Thank you.
Alright Elon — we’ve done our part to save the human race. Good luck with that Mars thing.
