Viziflu 2 release
Explore flu forecasts and forecast accuracy in a simple user interface
IQT Labs is pleased to release Viziflu 2, an interactive web app that visualizes seasonal influenza forecasting data. It builds on a prior collaboration with B.Next and the Centers for Disease Control and Prevention (CDC) Influenza Division, and it simplifies the original Viziflu interface we released in December 2018.
In addition to presenting multiple forecasts of the timing of peak week (the week with the highest percentage of outpatient visits for influenza-like illness in the U.S.), Viziflu 2 also shows each forecast’s skill score, an accuracy-related measure described below.
The Viziflu web app is designed to help public health analysts answer the question “When is seasonal flu likely to peak in the U.S.?” It uses publicly available flu forecasts submitted to the CDC FluSight Challenge. Forecasting models provide important insights about disease activity/incidence, such as the predicted total and location of new cases in seasonal outbreaks and emerging pandemics. Visualizing the uncertainty inherent in this data enables more transparent risk communications and a more nuanced understanding of forecast results.
Viziflu highlights two types of uncertainty that are critical for analyst to understand: (1) uncertainty within each forecast, about the likelihood that flu will peak in a particular week, and (2) uncertainty about the reliability of forecasting models, created when multiple infectious disease forecasting models generate conflicting predictions.
Once we released the original version of Viziflu in December 2018, we ran a user study to evaluate how variations in the design of the Viziflu interface impacted viewers’ interpretation of the data — and uncertainty — displayed. The goal of Viziflu 2 was to improve the effectiveness of our existing forecast uncertainty visualizations based on results from our user study.
In developing Viziflu 2, the team sought to simplify the interface (and therefore, the cognitive load for viewers). However, we did not want to distort or oversimplify our representation of the data, and we worked closely with our partners at the CDC Influenza Division to validate various ideas.
Our approach, shown above, involved using information about the historical accuracy of each forecasting model to determine the order in which forecasts were displayed (with the top-performing peak week forecasts shown at the top and additional forecasts shown in a separate tab). To accomplish this, we divided forecast models into two groups: “Highest-scoring forecasts” and “Additional forecasts.” Moreover, if a single forecasting team submits multiple forecasts, we only display the most accurate forecast for that team. Finally, we made data about forecast skill scores accessible to viewers through a modal / pop-up window.
As a default setting, we display only the “Highest-scoring forecasts” and users interested in accessing more peak week predictions can click on the “Additional Forecasts” tab, which opens an accordion UI component immediately below.
All accuracy data is based on forecasts’ “skill score,” a performance measure that used by the CDC. This measure is based on both accuracy (log scores) and confidence (distributional sharpness) over the entire flu season, as modeling teams update their forecasts each week. It reflects which probabilistic forecasts assigned the highest confidence to the eventually observed outcome. The measure also reveals which forecasts correctly predicted an outcome with high confidence vs. those that underestimated that outcome. Below is another Viziflu 2 visualization (available here) showing the range and variation of skill scores.
Skill scores are “a function of both accuracy (a measure of point estimates) and confidence (a measure of distributional sharpness)” (Osthus and Moran, 2019). They vary as modeling teams update their predictions over time, and they range from 0 (low) to 1 (high). Skill scores are calculated at the end of the flu season using a logarithmic scoring rule that assigns high skill scores to forecasts that correctly predict the eventually observed outcome with high confidence. Conversely, forecasts that underestimate the observed outcome and/or equivocate by submitting multiple low-confidence predictions tend to receive low skill scores.
By displaying forecast results alongside forecast accuracy-related data, Viziflu 2 seeks to inform cross-model comparisons by analysts. It also encourages forecast consumers to think critically about which models ought to inform judgments about outbreak response. Of course, past performance does not guarantee future results, and new forecasters often lack accuracy-related track records, but historical accuracy provides important context for interpreting — and building confidence in — the output of forecasting models.
For more information, please see the companion blog piece on forecast skill scores here on high-stakes design.
Please note the forecasting models presented in Viziflu are not official CDC forecasts and are not endorsed by either CDC or IQT Labs. While Viziflu 2 focuses only on forecasts of population-level influenza incidence during U.S. peak week, the tool can be adapted to other forecast targets, tasks, and use-cases if developed appropriately. If you are interested in using or adapting Viziflu, please check out this code repo and readme, which are available to the public for reuse/modification under the Apache 2.0 License.
To learn more about IQT Labs and B.Next’s uncertainty visualization research, please explore the following related blog posts:
- Bioviz Under the Microscope (Part 1)(high-stakes design, Feb. 2020)
- Bioviz Under the Microscope (Part 2)(high-stakes design, Feb. 2020)
- Visualizing flu forecast data with Viziflu (high-stakes design, Dec. 2019)
- Viziflu — Discussion on the collaborative effort between IQT Labs and the CDC Influenza Division (BioQuest, Mar. 2019)
- The Walk of Life (BioQuest, Nov. 2018)
