Aggregating Signals of Quality in News

  • Work together on a common data model to facilitate sharing of signals.
  • Continue convening the coalition of initiatives working on the problem of trust and credibility in news.
  • Encourage platforms and others to share data with researchers to inform this work.
  • Support networks to share alerts about targeted campaigns of disinformation.
  • Investigate opportunities to use blockchain to enable this work.
  • Observed and algorithmic: As a backbone of this project, we will have access to relevant Trust Metrics signals the company already gathers with human raters aided by algorithmic prediction and tracking. These will be combined with signals from similar work by other entities. See examples of these criteria in the latter portion of this paper.
  • Self-reported: The Trust Project worked with scores of news organizations — including The Washington Post,the Globe and Mail, La Repubblica, SkyNews — to determine eight indicators of journalistic quality and set standards these organizations will follow. Reporters Without Borders is also endeavoring to set standards for news organizations with its Journalism Trust Initiative. These self-reported signals will require some auditing of compliance against the standards. We hope organizations such as NewsGuard may be able to help provide some of that verification in the course of their efforts to rank sites and there are other proposals afoot to providing auditing.
  • Endorsement: If a news organization has passed muster to join an industry or professional body with entrance standards, that can provide a signal of quality. Similarly, winning certain prizes is a positive signal.
  • Public opinion: Public opinion — like TV ratings and newsstand circulation — has limited value and indeed can lead to pandering to a market. But in a larger set of signals such as we are gathering, the public’s opinion must have a place. In this regard, we admire the work done by the Reuters Institute for the Study of Journalism in its annual report, which surveys the public across political viewpoints regarding their trust of specific news organizations.
  • Diversity: The greatest danger with particularly endorsements and public opinion is that diverse, specialized, and new sources of news will be disadvantaged in favor of large, well-known, legacy, and mainstream sources. We must seek out ways to prevent this. We plan to gather data about the particular and diverse communities various sites serve (e.g., in New York, that the Amsterdam News serves African-American communities and The Forward serves Jewish communities). Trust Metrics has a methodology of viewing sites against various “lenses,” which can include communities of various definitions. We will also seek out means for new sources to be brought to our attention for review. Diversity flags will include capturing information about sites that serve specific ethnicities, religions, localities, political viewpoints (across a wide spectrum, describing both a site and its audience), sexual orientations, social groups, demographics, interests, and professions.
  • Behavioral: If a site consistently fails fact-checks — in spite of other measures — that is certainly a valuable signal that should be accounted for. We want to work with the many good fact-check efforts underway to gather data about their findings at a site level. The ClaimReview process and schema adopted by fact checkers around the world is a model for collaboration.
  • Alerts of disinformation campaigns: Similarly, if a site regularly spreads disinformation as well as conspiracy theories and incitement that has been identified not only by fact-checkers but by initiatives like Data & Society’s manipulation campaign, that should be accounted for. As we all know, some sites bury such disinformation amid a larger volume of apparently legitimate content.
  • Does the profanity occur in the editorial content or user-generated content? If it is the latter, profanity would be more of an indication of poor UGC moderation rather than an indicator of site profanity.
  • Do certain words need to carry more weight than other words?
  • A site geared towards an audience of younger adults might have a higher tolerance for what others might regard as profanity given its target demographic (e.g. Vice), but the same density of certain language might not be acceptable on a site geared towards a different audience.
  • Profanity occurring in a quote or in the context of objective news is not an issue.
  • Recent updating of a site is generally a good indicator of quality. However, it’s important to note that time stamps can be faked, so that signal can’t hold too much weight on its own.
  • On their own, the presence of lots of content recommendation ads and sponsored links is not a negative feature. However, when these links appear all over a low-quality site, this signal is more meaningful.
  • Having a “Contact Us” or “About Us” page is often a sign of professional journalism. However, we see lots of examples with no content on these pages or just standard copy and pasted lorem ipsum from other domains.




Working to support quality news, innovative journalists, and news media leaders at the Craig Newmark Graduate School of Journalism

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Seattle independent journalists oppose SPD subpoenas

10 tips to help kickstart your journalism career on Twitter

Hashkey Funds DAOrayaki to Build a More Decentralized Media Platform

>+HBO Max “The Fresh Prince of Bel-Air Reunion 2020 — Movie Full Onlie (TV)

Zeno Spotlight: Bombo Radyo Philippines

Joy Reid Slams Tucker Carlson, Fox News, and the Trump Cult

Journalism: A Brief History of America

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tow-Knight Center

Tow-Knight Center

Working to support quality news, innovative journalists, and news media leaders at the Craig Newmark Graduate School of Journalism

More from Medium

A Good Feedback

Understanding new tech: What is behind the latest technological disruptions

The Life of a Gymnast

Irredentism: How Far Are Nationalists From Their Goals?