What do we mean by “quality news?”

Aviv Ovadya
Nov 16, 2018 · 4 min read

Imagine this: You work at Facebook and you have been instructed by Zuck himself to “fix news on the platform” — to do “the right thing” for users, even if it cuts into Facebook’s profits. So what do you do? And how do you measure if you’ve succeeded?

This scenario gets at much of the purpose of journalism and news consumption. To help answer it and elucidate some the implicit complexity, we propose a conceptual framework for news and news ecosystem “quality,” which can then form a basis for news ecosystem health measurement. Without such a framework and measurement capability, it is difficult to even judge, either as an insider or outsider, if Facebook and its brethren are doing the “right thing” — or to meaningfully suggest substantive improvements.

Underpinning this framework are several elements for which it might be valuable to measure quality: the content, the source, and the effect on the receiver. These are defined as follows:

  • Content: The actual content and message of the news

News Source Quality

In evaluating a news source, there are several related but distinct values that one might consider and evaluate.

  • Integrity — How well does this source adhere to a set of ideals and processes? These ideals might include not plagiarizing or slandering, while processes would include fact checking and correcting errors. This value speaks most directly to the element of source, exploring the methodology used as the source creates content.

These are all values which an ideal news source would attempt to maximize, and which are deeply correlated and build upon one another; e.g. a news source cannot be credible if it is not accurate, and is unlikely to be accurate without integrity. As a shorthand, we can say that the quality of a news source hinges on the extent to which it fulfills these properties. Each of these can also be the basis for a metric or classification scheme.

News Consumption Quality Metrics

How do we measure these different properties of news sources and the news ecosystem as a whole? One measure of news ecosystem health might simply be “relative news quality”the degree to which a population’s news diet comes from sources that value the news properties that are indicators of overall quality.

However, there is some crucial nuance here when we place all the news sources consumed by an individual within their role in civic society. Quality is not necessarily enough — various civic categories of news need to provided and consumed in order to ensure a robust society. A 2012 report for the FCC, “Regarding Critical Information Needs of the American Public”, proposed one version of these, suggesting 8 key categories of news: economic opportunities, transportation, health, emergencies, environment, civic information, and political information.

Keeping that in mind, there are several types of “news health” that an individual or population might aspire to:

  1. Significant quality news consumption. Consuming a significant amount of high-quality news. This does not imply anything about simultaneous consumption of low-quality news, which may also be significant.

Evaluating if a news ecosystem fulfills these criteria requires a way to classify or score the quality of news and/or the completeness of a civic news diet. Being able to measure (D) is ideal, as it represents the highest possible level of “news health” for each person or community — but is also the mostly costly to classify.


If you believe there are important aspects not captured by these frameworks, please get in touch via email at tow@aviv.me or on twitter at @metaviv. For updates on my work, follow me and subscribe at list.aviv.me.

Tow Center

The Tow Center for Digital Journalism at the Columbia Graduate School for Journalism

Aviv Ovadya

Written by

Fascinated by the effects of tech. Former Chief Technologist at the Center for Social Media Responsibility (UMSI) & Knight Tow Fellow. av@aviv.me

Tow Center

The Tow Center for Digital Journalism at the Columbia Graduate School for Journalism