What do we mean by “quality news?”

Imagine this: You work at Facebook and you have been instructed by Zuck himself to “fix news on the platform” — to do “the right thing” for users, even if it cuts into Facebook’s profits. So what do you do? And how do you measure if you’ve succeeded?

This scenario gets at much of the purpose of journalism and news consumption. To help answer it and elucidate some the implicit complexity, we propose a conceptual framework for news and news ecosystem “quality,” which can then form a basis for news ecosystem health measurement. Without such a framework and measurement capability, it is difficult to even judge, either as an insider or outsider, if Facebook and its brethren are doing the “right thing” — or to meaningfully suggest substantive improvements.

Underpinning this framework are several elements for which it might be valuable to measure quality: the content, the source, and the effect on the receiver. These are defined as follows:

• Content: The actual content and message of the news
• Source: The entity that creates or packages content for distribution, generally associating their identity with the content, e.g. a publisher, author, account, blog, etc.
• Reception: The effect the content has on individuals, their communities, etc.

News Source Quality

In evaluating a news source, there are several related but distinct values that one might consider and evaluate.

• Integrity — How well does this source adhere to a set of ideals and processes? These ideals might include not plagiarizing or slandering, while processes would include fact checking and correcting errors. This value speaks most directly to the element of source, exploring the methodology used as the source creates content.
• Accuracy — How closely do the articles published by the source match the true state of the world? This value is more content-focused, examining the output of the source.
• Reliability — What is the likelihood that any particular story from this source will be accurate? This value ties content and source together over time.
• Credibility — What is the likelihood that the source will inform recipients — or at least avoid misleading them? This value includes the direct impact of the source on recipients — which may be different for different contexts or recipients.
• Informativeness — How much does this source inform readers? This value captures the degree to which a source increases the recipient’s understanding (which is also usually context and recipient dependent).
• (Net-positive) Impact — How much does this source improve its readers and the world? (This could include informing readers, fostering community, improving accountability, etc.). This value is focused entirely on reception to the content.

These are all values which an ideal news source would attempt to maximize, and which are deeply correlated and build upon one another; e.g. a news source cannot be credible if it is not accurate, and is unlikely to be accurate without integrity. As a shorthand, we can say that the quality of a news source hinges on the extent to which it fulfills these properties. Each of these can also be the basis for a metric or classification scheme.

News Consumption Quality Metrics

How do we measure these different properties of news sources and the news ecosystem as a whole? One measure of news ecosystem health might simply be “relative news quality” — the degree to which a population’s news diet comes from sources that value the news properties that are indicators of overall quality.

However, there is some crucial nuance here when we place all the news sources consumed by an individual within their role in civic society. Quality is not necessarily enough — various civic categories of news need to provided and consumed in order to ensure a robust society. A 2012 report for the FCC, “Regarding Critical Information Needs of the American Public”, proposed one version of these, suggesting 8 key categories of news: economic opportunities, transportation, health, emergencies, environment, civic information, and political information.

Keeping that in mind, there are several types of “news health” that an individual or population might aspire to:

1. Significant quality news consumption. Consuming a significant amount of high-quality news. This does not imply anything about simultaneous consumption of low-quality news, which may also be significant.
2. High relative news quality. Consuming a significant proportion of quality news, without implying anything about the amount and comprehensiveness of the news consumed, which may be limited in number of articles, frequency of consumption, and variety of sources.
3. A complete civic news diet. Consuming a sufficient proportion of each civic category of quality news, without implying anything about the quality of other news being consumed. As in (1), consumption of low-quality news may also be significant.
4. A complete and fulfilling civic news diet. Consuming a sufficient proportion of each civic category of quality news and avoiding overconsumption of low-quality news — a combination of (1) and (2).

Evaluating if a news ecosystem fulfills these criteria requires a way to classify or score the quality of news and/or the completeness of a civic news diet. Being able to measure (4) is ideal, as it represents the highest possible level of “news health” for each person or community — but is also the mostly costly to classify.

If you believe there are important aspects not captured by these frameworks, please get in touch via email at av@aviv.me or on twitter at @metaviv. For updates on my work, follow me and subscribe at list.aviv.me.