Hey Alexa, Can you tell me how well you answer news queries?

Alexa, Siri, Google Home, Cortana — smart speakers and agents are now used in nearly a quarter of U.S. homes. According to NPR’s Smart Audio Report, 35% of U.S. adult smart speaker owners are listening to more news and information since the COVID-19 outbreak, and 50% of those aged 18–34 say the same. But do these devices deliver high-quality news and information or could they be misinforming and sharing “junk” news? We sought to find out in our recent study that investigates how smart speakers respond to requests for timely information about trending topics that people may be searching for news about.

To evaluate the information quality of news-related queries on the Alexa voice assistant, we first surveyed people on Amazon Mechanical Turk (MTurk) to provide different ways they would ask Alexa for information about a news-related topic. From their responses we identified four most-commonly used generic question phrasings:

• What happened {during / to / in / on / with / at the } ________ ?
• What is going on {during / with / at / in / on} ________ ?
• Can you tell me about {the} ________ ?
• What is new {with / on} ________ ?

Using the above query phrases, we then developed an automated process to query the Alexa voice assistant for news-related information over a two week period. (See the paper for the full methodological details).

After collecting and evaluating a set of 1,112 Alexa responses to the automated voice queries, here is what we found.

The same question, phrased differently, yielded responses of varying information quality. For example, the question phrases “Can you tell me about” and “What happened” had higher proportions of relevant responses compared to “What is going on” and “What is new”. This observation means that subtle differences in how people interact with the voice assistant (e.g., how different people ask questions differently) could lead to certain Alexa users potentially and unknowingly suffering worse user experiences and information access compared to others.

The Alexa voice assistant could provide reliable and up-to-the-minute updates about questions related to sports, but often failed to provide relevant and timely information about other events, especially those that are politically relevant. For example, when asked “What is going on with Panthers vs Packers” the Alexa voice assistant responded, “Currently, the Packers are beating the Panthers, 24–16 with 19 seconds left in the fourth quarter”. However, when asked “What is going on with the impeachment” — an important political question — the voice assistant provided a bland definitional answer about what an impeachment is based on a Wikipedia entry.

When further comparing how relevant and timely the smart speaker responses were to the questions asked, we observed that hard news had a 15.9% lower response relevance and 16.9% lower response timeliness rate compared to soft news. Alexa users may therefore face additional constraints with respect to finding information about societally and politically relevant events compared to sports, entertainment and lifestyle events.

The same question, phrased differently also resulted in responses from different information sources. Interestingly, the question phrasing “Can you tell me about ‘’, provided the least number of unknown sources and most number of Wikipedia sources. Similarly, the majority of the known sources for all question phrases also originated from Wikipedia. However, we also found that the majority (60.4%) of the query responses didn’t indicate where the information came from. The lack of source information makes it more difficult to evaluate the reliability and credibility of the information provided by smart speakers.

Wikipedia is the most prevalent individual information source, providing 18.6% of the responses. While Wikipedia can often provide reliable information, it is important to highlight that, at any given time, Wikipedia information may be vandalized, incomplete, or inaccurate. In fact, prominent news subjects as well as politically and culturally contentious topics in Wikipedia are often vulnerable to vandalism during ‘edit wars’. Although edit errors and conflicts may be resolved, some errors can go unnoticed for long periods of time, especially considering that Wikipedia is volunteer run. Additionally, the prevalence of Wikipedia sources in Alexa responses further raises concerns over what commercial entities such as Amazon may “owe” to the creators of freely available and volunteer-created information that their technologies depend on.

An important lesson and insight from our study is that smart speaker users may suffer differential information access based on subtle variations in how they interact with smart speaker devices. Considering the challenges that media platforms have had in curating quality information and combating disinformation, the growing use of smart speakers for news and information consumption presents new questions related to the quality and source diversity of the information they provide.

Given the current absence of gate-keeping and regulations on content production on the Alexa platform, frequent audits of news information quality are going to be necessary. In the future, periodic auditing of information quality from voice assistants should become fully integrated into the engineering development process of these devices to safeguard from potential biases in information access for broad demographics.

See the full paper: Henry Dambanemuya and Nicholas Diakopoulos. Auditing the Information Quality of News-Related Queries on the Alexa Voice Assistant. Proceedings of the ACM (PACM): Human-Computer Interaction (CSCW). 2021. https://dl.acm.org/doi/abs/10.1145/3449157