Viziflu — Discussion on the collaborative effort between IQT Labs and the CDC Influenza Division
Viziflu is a visualization tool developed by IQT Labs that displays multiple predictions about the timing of national “Peak Week,” the week with the highest predicted number of flu cases, in the United States. The visualization aims to make interpreting the uncertainty associated with the output of multiple influenza models fast and easy.
Seasonal influenza (flu) can be serious and deadly. The U.S. Centers for Disease Control and Prevention (CDC) estimates that seasonal flu caused 960,000 thousand hospitalizations and 79,000 deaths during the 2017–2018 flu season.[i] While seasonal influenza is clearly a large public health concern, public health officials are also worried about the emergence and global spread of a new influenza virus that people have little immunity against (called a pandemic). For example, the 1918 flu pandemic that killed an estimated 50M people around the world was caused by a new influenza A(H1N1) virus.[ii] In short, both seasonal and pandemic flu can be deadly, and each poses a real risk to health security.
CDC advances and translates science to keep the public safe from public health threats including flu. One area of focus in recent years has been advancing the science of forecasting seasonal influenza activity. CDC does this through a collaboration with academic and private sector researchers who have been developing flu forecasts to help anticipate when flu activity will occur and how bad each flu season will be. Simply stated, these researchers apply various forecasting methods to infectious disease data to understand seasonal flu dynamics in order to inform public of the risks and guide prevention efforts.
Other groups within the U.S. Government and academia have used similar forecasting techniques to help guide response and medical countermeasure development efforts in recent infectious disease outbreaks, such as Ebola in West Africa and Zika in the Americas. [iii],[iv]
One challenge of this work is translating forecasts so that people can easily interpret and act on them. IQT Labs took on a project to design and evaluate different visual translations of seasonal flu forecasts to see which ones made the information easier to interpret.
This effort resulted in the development and release of Viziflu, a visualization of “Peak Week” predictions of seasonal flu from multiple forecasts. You can read more about Viziflu here.
Chelsea Lutz, Matt Biggerstaff, and Carrie Reed, who are epidemiologists with CDC’s Influenza Division, have offered insight and feedback on flu forecasting in general and Viziflu in particular.
What is flu forecasting and how has it been useful?
Our traditional ways of monitoring flu include collecting information on people that have become ill, which measures influenza activity only after it has occurred. Influenza forecasting combines this kind of data with mathematical and statistical models as the flu season is unfolding to predict the characteristics of influenza epidemics before they occur, and describe the chance that the predictions are correct. For example, when will influenza season begin or peak, and how confident are we in our prediction? Currently, we share influenza forecasts with public health leadership at CDC and at state and local public health departments, as well as a sharing public summaries of forecasts on the CDC website.[v] During a flu season, information like this could help our medical system plan better for the increased staff and capacity needs during the peak of activity or determine whether vaccination efforts should be ongoing. In the event of a public health emergency or pandemic, influenza forecasts could be useful to help inform key preparation and prevention efforts, like school closures.
Why has CDC supported flu forecasting and what is FluSight?
We know that influenza causes a substantial health and economic burden in the United States, but how bad a flu season will be and when the worst of it will happen varies from year to year, which makes flu difficult to predict.[vi],[vii],[viii],[ix],[x] We launched the “Predict the Influenza Season Challenge,” now known as FluSight, during the 2013–2014 influenza season in order to better understand the current accuracy of real-time influenza forecasts and advance the science and utility of flu forecasting.[xi],[xii] FluSight is a collaboration between CDC and members of the scientific community who work very hard to predict the timing, intensity, and short-term trajectory of influenza activity in the United States each season.[xiii] The public and other users can view real-time results of these challenges at the FluSight webpage, which has an interactive display of the weekly individual forecasts for current and past influenza seasons (back to the 2016–17 influenza season).[xiv]
FluSight contains ensemble models and historical averages. What are the ensemble models? How do you use them and historical averages to understand the current flu season?
Ensemble modeling takes two or more forecasts and synthesizes them into a single forecast. This could be as simple as taking an average of multiple forecasts, or could involve a more complicated process of using recent or historical performance to weight (give the individual forecast more or less importance relative to other forecasts) the individual forecasts.[xv] This is desirable not only because it is often easier to understand and to communicate a single forecast, but because ensembles may help improve the accuracy of forecasts.[xvi] Starting with the 2015–16 influenza season, CDC created a simple average ensemble of forecasts to use as the basis of CDC’s forecast communication. While this ensemble has been one of the top performing forecasts among those submitted, the FluSight Network was formed in 2017 to test whether an ensemble weighted by past forecast performance would perform even better.[xvii] This ensemble was submitted to CDC on a weekly basis during the 2017–2018 season and was more accurate than the simple average ensemble. The FluSight Network ensemble is now the basis of CDC’s forecast communication beginning with the 2018–2019 season.
CDC also produces a historical average forecast based solely on the distribution of the onset and peak weeks, peak intensity, and weekly values of prior influenza seasons. The historic average does not consider any new information from the current season, but provides context for when these events have occurred in past seasons and when current season forecasts begin to diverge from historic trends.
What data sources have been most useful in forecasting flu?
The forecasts that are currently submitted to CDC through FluSight are based on CDC’s U.S. Outpatient Influenza-like Illness Surveillance Network (ILINet). ILINet comprises more than 3,500 enrolled outpatient health care providers, and each week providers report the number of patients with ILI and the total number of patients seen in their practice.[xviii] In addition to these data, teams are free to use any other data sources to inform their forecasts. Examples of data used by teams includes Internet search queries, Wikipedia page visits, Twitter, and weather data. The most accurate forecasts have used a variety of data sources in their approaches.[xix],[xx],[xxi],[xxii],[xxiii]
What visualizations have you developed for forecasting in the past? And what were their intended use cases?
We’ve made a number of visualizations in the past, mainly to help explain the different aspects of the forecasts (e.g., season start, peak week, and peak intensity) to CDC leadership and public health colleagues at the state and local level. We’ve provided a couple of examples below from forecasts received February 19, 2019. The first example shows the chance that flu activity will peak in February from our ensemble (the FluSight Network on the left) and the Historical Average (on the right). The second shows forecasts for short-term influenza activity and the chance the flu activity will be higher over the next four weeks.
What challenges and successes have you had with visualizing forecasting results?
Influenza forecasts are complicated to interpret visually because they not only tell us when something will happen but also the chance that it will happen. This adds a level of complexity that we have struggled to present in a way that is easy to understand but still accurately represents the forecast information. Trying out different visualizations has helped us identify some approaches that work better and hone in on the key messages that we want the visualizations to convey.
Why was advising IQT Labs on Viziflu interesting to you?
We think the most appealing part of the collaboration was having people who are not forecasters or epidemiologists look at our forecasts and what we were trying to say about them. We are so close to the forecasts and surveillance data that it can be difficult for us to take a step back and be sure that a visualization is conveying the intended message clearly to people who are not so involved in the process. By collaborating with IQT Labs, we got access to experts who identified the best practices for visually representing forecasts, created new forecast visualizations, and developed a study framework for approaching future communications and visualizations efforts.
What will improving forecasting visualizations allow CDC to do differently?
Better forecast visualization is key to improve the use of forecasts by CDC, state and local public health officials, and the public. Most of our forecast users do not have 10 or 15 minutes to try and understand what a forecast is saying, so effective visualizations will make it more likely that a forecast will be used and interpreted correctly. We are intrigued about using the visualizations in Viziflu and testing how people respond to it.
[i]Disease Burden of Influenza. Centers for Disease Control and Prevention. https://www.cdc.gov/flu/about/burden/index.html
[ii]https://wwwnc.cdc.gov/eid/article/12/1/05-0979_article
[iii]Meltzer, M. I. et al. Modeling in Real Time During the Ebola Response. Centers Dis. Control Prev. Mortal. Morb. Wkly. Rep. 65, 85–89 (2016).
[iv]Chretien, J. P., Riley, S. & George, D. B. Mathematical modeling of the West Africa ebola epidemic. Elife(2015). doi:10.7554/eLife.09186
[v]Centers for Disease Control and Prevention. FluSight: Flu Forecasting. https://www.cdc.gov/flu/weekly/flusight/index.html. Accessed January 28, 2019.
[vi]Molinari NA, Ortega-Sanchez IR, Messonnier ML, Thompson WW, Wortley PM, Weintraub E, et al. The annual impact of seasonal influenza in the US: measuring disease burden and costs. Vaccine. 2007;25:5086–96.
[vii]Reed C, Chaves SS, Daily Kirley P, Emerson R, Aragon D, Hancock EB, et al. Estimating influenza disease burden from population-based surveillance data in the United States. PloS one. 2015;10:e0118369.
[viii]Davlin SL, Blanton L, Kniss K, Mustaquim D, Smith S, Kramer N, et al. Influenza Activity — United States, 2015–16 Season and Composition of the 2016–17 Influenza Vaccine. MMWR Morb Mortal Wkly Rep. 2016;65:567–75.
[ix]Blanton L, Alabi N, Mustaquim D, Taylor C, Kniss K, Kramer N, et al. Update: Influenza Activity in the United States During the 2016–17 Season and Composition of the 2017–18 Influenza Vaccine. MMWR Morb Mortal Wkly Rep. 2017;66:668–76.
[x]Garten R, Blanton L, Elal AIA, Alabi N, Barnes J, Biggerstaff M, et al. Update: Influenza Activity in the United States During the 2017–18 Season and Composition of the 2018–19 Influenza Vaccine. MMWR Morb Mortal Wkly Rep. 2018;67:634–42.
[xi]Epidemic Prediction Initiative. https://predict.cdc.gov/. Accessed November 11, 2018.
[xii]Centers for Disease Control and Prevention. FluSight: Flu Forecasting. https://www.cdc.gov/flu/weekly/flusight/index.html. Accessed January 28, 2019.
[xiii]Epidemic Prediction Initiative. https://predict.cdc.gov/. Accessed November 11, 2018.
[xiv]Epidemic Prediction Initiative. https://predict.cdc.gov/. Accessed November 11, 2018.
[xv]Lutz CS, Huynh M, Schroeder M, et al. Applying infectious disease forecasting to public health: A path forward. 2019. Unpublished manuscript.
[xvi]Ray EL, Reich NG. Prediction of infectious disease epidemics via weighted density ensembles. PLoS Comput Biol. 2018;14(2):e1005910. Published 2018 Feb 20. doi:10.1371/journal.pcbi.1005910
[xvii]Reich NG, Brooks LC, Fox SJ, et al. A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States. Proc Natl Acad Sci USA. Epub ahead of print. January 15, 2019. doi: https://doi.org/10.1073/pnas.1812594116.
[xviii]Centers for Disease Control and Prevention. Overview of Influenza Surveillance in the United States. https://www.cdc.gov/flu/weekly/overview.htm. Accessed November 1, 2018. Updated October 19, 2018.
[xix]Shaman J, Kandula S. Improved Discrimination of Influenza Forecast Accuracy Using Consecutive Predictions. PLoS Curr. 2015;7:ecurrents.outbreaks.8a6a3df285af7ca973fab4b22e10911e. Published 2015 Oct 5. doi:10.1371/currents.outbreaks.8a6a3df285af7ca973fab4b22e10911e
[xx]Shaman J, Karspeck A, Yang W, Tamerius J, Lipsitch M. Real-time influenza forecasts during the 2012–2013 season. Nat Commun. 2013;4:2837.
[xxi]Lu FS, Hattab MW, Clemente CL, Biggerstaff M, Santillana M. Improved state-level influenza nowcasting in the United States leveraging Internet-based data and network approaches. Nat Commun. 2019;10(1):147. Published 2019 Jan 11. doi:10.1038/s41467–018–08082–0
[xxii]Farrow DC, Brooks LC, Hyun S, Tibshirani RJ, Burke DS, Rosenfeld R. A human judgment approach to epidemiological forecasting. PLoS Comput Biol. 2017;13(3):e1005248. Published 2017 Mar 10. doi:10.1371/journal.pcbi.1005248
[xxiii]McGowan CJ, Biggerstaff M, Johansson M, et al. Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016. Sci Rep. 2019;9(1):683. Published 2019 Jan 24. doi:10.1038/s41598–018–36361–9
