COVID-19 Multi Forecast Visualizations: Trust or Distrust?

Examining the factors affecting trust and reliability

In the context of a global pandemic, characterised by the urgent need for information propagation, numerical data emerges as a particularly potent tool to encourage safety and preventative measures amongst the public. The efficacy of conventional verbal communication is surpassed by the compelling impact of visualizations — especially when leveraging visualizations and numerical data proves pivotal in motivating behavioural changes that promote public health. However, the accuracy and reliability of such visual representations demand meticulous scrutiny in terms of trust and comprehension among diverse audiences. Moreover, the lessons gathered from this analysis could extend beyond immediate crisis management, offering invaluable insights for the refinement of communicative strategies during prime future events.

What are Multiple Forecast Visualizations?

Multiple Forecast Visualizations (MFVs) represent a specific type of chart where uncertainty is subtly conveyed through the agreement or disagreement among various forecasts presented in the same graph. This method provides insights into the range, shape, and concentration of predictions. Unlike ensemble plots or hypothetical outcome plots, which sample forecasts from a distribution, an MFV showcases distinct predictions from diverse forecasting sources. At present, there is no established framework for judiciously choosing forecasts in MFVs. While one option is to incorporate all available forecasts, the potential drawback is over-plotting, especially when dealing with numerous forecasts, possibly numbering in the dozens.

The caveats of the MFVs design in terms of trade offs among trust, intelligibility, and performance in a trend prediction task are examined during the pandemic in this 2022 study. To understand more about these factors, three experiments were conducted examining three visualization design choices in the context of MFVs — number of forecasts, colour and best case or worst case forecasts.

Covid-19 mortality forecasts used in Experiments 1 and 2. Image taken from Multiple Forecast Visualizations (MFVs): Trade-offs in Trust and Performance in Multiple COVID-19 Forecast Visualizations (Padilla et al., 2022)

Understanding the Research Experiments

• The first experiment aimed to examine how varying the number of forecasts and using colour-coded visuals impacted participants’ trust.
• In the second experiment, the focus was on observing how the inclusion of worst-case and best-case forecasts influenced the trust of viewers.
• The third experiment served as a follow-up to the first two, assessing whether participants sensibly adjusted their trust levels in response to the reported range of confidence intervals in the visualizations.

Outcomes of the Experiments

1. The study uncovered that trust generally rises as charts incorporate more models but levels off when displaying approximately 6–9 forecasts.
2. Invariably, individuals expressed higher confidence in visualizations presenting less visual information, such as those featuring a 95% Confidence Interval, a median forecast, and forecasts encoded in grayscale. The coloured visualizations proved to be ineffective in terms of adding more clarity but rather causing more confusion.
3. Incorporating the best or worst-case forecasts with confidence intervals seems to erode trust and leaves individuals uncertain about how to interpret the forecasts.
4. Participants noted an uptick in trust with the increasing number of models shown but also reported a perceived lack of trust when confronted with an excessive number of models simultaneously.
5. Viewers tend not to appropriately calibrate their trust based on the size of the indicated confidence interval on the visualization.

It is important to keep in mind the limitations of the research when evaluating the results. Though the research inevitably finds the number of forecasts that could be shown to evoke trust in participants, the threshold tested is 15 forecasts at once. Apart from the factors(number, colour and best/worst case scenarios) investigated, there are a multitude of factors that could sway the trustworthiness of visualizations.

In conclusion, it is inevitable for visualizations to stand as pivotal tools in elevating public comprehension and guiding decision-making during global events, holding the potential either to facilitate understanding or to bring about confusion and breed distrust. Within this context, the research unravels methods geared toward bolstering trust while concurrently upholding proficiency in trend interpretation, achieved through the portrayal of guidelines for Multiple Forecast Visualizations (MFVs). The findings of this study can be reliably extended for application in situations where there is a public imperative to comply with official regulations, effectively conveying the intended message.

Reference

• Padilla, L.M., Fygenson, R., Castro, S.C., & Bertini, E. (2022). Multiple Forecast Visualizations (MFVs): Trade-offs in Trust and Performance in Multiple COVID-19 Forecast Visualizations. IEEE Transactions on Visualization and Computer Graphics, 29, 12–22.