Incorporating Domain Knowledge into Human-centered Behavior Models for Time-series Forecasting

Co-authors : Anindya Das Antar, Anna Kratz, Nikola Banovic

This post summarizes a research paper where we presented a computational behavior modeling approach that formalizes clinical knowledge about people with Multiple Sclerosis (MS) to forecast their end-of-day physical functioning and support timely interventions. This paper has been accepted for publication in the March 2023 edition of the Proceedings of the ACM on Interactive, Mobile, Wearable, and Ubiquitous Technologies (IMWUT), and this will be presented at the UbiComp / ISWC 2023 conference. The authors of this paper are Anindya Das Antar, Anna Kratz, and Nikola Banovic.

Forecasting the end-of-day (EOD) physical functioning of people with Multiple Sclerosis (MS) (a chronic disease) can inform “just-in-time” interventions [1]. These interventions include “day planning” [2] which allows people with MS to plan their activities for the day in advance and “activity pacing” [3] which involves taking rest breaks in between physical activities to manage symptoms and physical functioning. People with MS who can manage their symptoms and functioning experience few barriers to social interactions and professional careers.

However, tracking and forecasting the functioning of people with MS remains challenging because the symptoms (e.g., pain and fatigue) that influence their functioning fluctuate over time [4] and depend on different factors (Fig. 1). Recent advancements in personal, mobile, and wearable devices enable tracking of people’s symptoms and functioning (e.g., lower extremity functioning (LEF) and upper extremity functioning (UEF)) at their homes. Existing Machine Learning (ML) models [5, 6] can use such data to predict people susceptible to or already experiencing low functioning. However, they do not forecast when MS symptoms will worsen or when people will experience low EOD functioning ahead of time.

Fig. 1. Our expert-informed conceptual behavior model of people with MS is structured on the World Health Organization International Classification of Functioning (WHO-ICF) model [7].

Here, we present a human-centered computational behavior modeling method that captures the behaviors of people with MS to forecast their EOD functioning. We structured our behavior model based on the WHO-ICF model of functioning and disability [7] (Fig. 1). Also, we based our model on clinical knowledge from an MS disease expert (a clinical psychologist and researcher) on our team that informed our feature engineering, model building, and subsequent evaluations (Fig. 2).

Fig. 2. Stages of our computational behavior modeling method to forecast EOD functioning (LEF and UEF).

Method for Forecasting EOD Functioning

Our computational modeling method extends an existing behavior modeling approach [8] based on Inverse Reinforcement Learning (IRL) and applies it to MS. We first selected and co-engineered features with our MS domain expert; a domain-specific step that is not immediately obvious from existing work [8, 9].

We further extended the existing approach [8, 9] and replaced “off-the-shelf” ML algorithms with a Bayesian Network (BN) that we co-designed with a domain expert to estimate the probability of initial state P(s₀), changes (e.g., worsening or improvement) in MS symptoms (e.g., pain, fatigue) probability 𝑃 (𝑠 ′| 𝑠, 𝑎) at different intervals, and predict EOD functioning.

Then, we used the engineered features and BN to train our probabilistic generative IRL-based model (Fig. 3) to compute the conditional probability 𝑃 (𝑎 | 𝑠) of different actions preceding those symptoms, estimate different functioning levels, and forecast the one with the maximum likelihood.

Fig. 3. A behavior instance of people with MS for the Inverse Reinforcement Learning (IRL) model.

Quantitative Evaluation of Model Performance

We validated our method in a series of quantitative performance evaluation experiments. We trained and tested our model using an existing clinically validated dataset [10] containing data (demographics, actigraphy, self-reported symptoms, and functioning) from 107 ambulatory adults with MS collected over 7 consecutive days in their homes.

We showed that our model is more accurate at forecasting EOD LEF and UEF compared to existing ML baselines in two experiments: 1) starting at each daytime interval and using both self-reported and passive-sensing data (Fig. 4), and 2) starting when participants woke up and using their self-reported data then and only passively sensed data afterward (Fig. 5).

FIg. 4. Forecasting performance comparison of EOD LEF starting at each daytime interval.

Fig. 5. Forecasting performance comparison of EOD LEF with actigraphy data up to a daytime interval.

We found that when forecasting EOD functioning, prediction error compounds at each forecast step irrespective of the model choice. Also, time granularity (e.g., start and end intervals of forecasts) and information available to the model (e.g., self-reports, actigraphy) during different intervals of forecasting, could influence the forecasting performance.

Qualitative Visual Model Exploration

Our qualitative visual model exploration allowed two model designers on our team to investigate the model inputs and outputs in domain expert-suggested what-if scenarios. This allowed us to explore how well the model captures the actual behaviors of people with MS and perform error analysis of instances where our model made the wrong forecasts. Here, we describe a scenario from our visual model exploration.

Influence of Personal Factors (Gender and Age) on EOD Functioning

Despite MS not being directly inherited or contagious, personal factors (e.g., gender, age) could impact the physical functioning of people with MS [11]. The Sankey visualizations (Fig. 6) demonstrate that our model accurately captured the gender at birth and age proportions in the data.

We observed that older individuals, both in the data and model-generated samples, were more likely to report low functioning, suggesting a potentially lower susceptibility to MS among older adults. However, the model tended to overestimate instances of low functioning compared to the data, likely due to predicting functioning values that were not recorded.

Also, our model could sample from parts of the state space unrepresented in the data. For example, in the data, males younger than 40 never reported low EOD LEF; our model can sample what happens when they do.

FIg. 6. Visualizing behavior instances based on the influence of gender and age on EOD LEF.

Discussion and Takeaways

Our work contributes a computational modeling method that improves on the existing ML approaches. Our evaluation showed evidence that our method outperformed existing ML baselines due to carefully engineered and expert-informed features and algorithms. We showed that it is feasible to forecast functioning before it worsens (i.e., when people with MS need it the most), with few false alarms.

Our feature ablation experiment showed that collecting self-reported symptoms helps in forecasting EOD functioning in addition to demographics and contextual factors. However, despite its value, collecting self-reported symptoms five times a day could be burdensome for people with MS. Results from our two forecasting experiments showed that our method can forecast EOD functioning starting from any daytime interval without the need for repetitive self-reported symptom collection.

Our findings also showed the value of using passively collected activity intensity and pacing data to forecast physical functioning. However, our evaluation showed that activity features contributed only a small increase in the LEF and UEF forecasting performance despite clinical research suggesting a stronger relationship. Instead, in our forecasts, EOD UEF depended more on the previous day’s UEF, and EOD LEF depended more on self-reported symptoms closer to bedtime.

Conclusion and Future Work

Our human-centered method for modeling the behaviors of people with MS enables forecasting their EOD physical functioning as early as when they wake up.

Our comprehensive model evaluation, which included quantitative and qualitative visual model exploration has implications for human-centered applied ML approaches to support future ubiquitous computing applications (e.g., activity-based ubiquitous computing, human-centered machine learning (HCML), eXplainable Activity Recognition (XAR)).

Our results are the first necessary step towards more accurate forecasting of the functioning of people with MS that can be used to deploy behavior-aware user interfaces that deliver just-in-time interventions.

References:

1. Elena D Koch, Talar R Moukhtarian, Caroline Skirrow, Natali Bozhilova, Philip Asherson, and Ulrich W Ebner-Priemer. 2021. [Using e-diaries to investigate ADHD–State-of-the-art and the promising feature of just-in-time-adaptive interventions](https://pubmed.ncbi.nlm.nih.gov/34090919/). *Neuroscience & Biobehavioral Reviews* (2021).

2. Kathleen Matuska, Virgil Mathiowetz, and Marcia Finlayson. 2007. [Use and perceived effectiveness of energy conservation strategies for managing multiple sclerosis fatigue](https://www.sciencedirect.com/science/article/pii/S0003999306010025). *American Journal of Occupational Therapy* 61, 1 (2007), 62–69.

3. Ulric S Abonie, Femke Hoekstra, Bregje L Seves, Lucas HV van der Woude, Rienk Dekker, and Florentina J Hettinga. 2020. [Associations between Activity Pacing, Fatigue, and Physical Activity in Adults with Multiple Sclerosis: A Cross Sectional Study](https://www.mdpi.com/2411-5142/5/2/43). *Journal of Functional Morphology and Kinesiology* 5, 2 (2020), 43.

4. National MS Society. 2021. [MS Signs and Symptoms](https://www.nationalmssociety.org/Symptoms-Diagnosis/MS-Symptoms).

5. Diana Ohanian, Abigail Brown, Madison Sunnquist, Jacob Furst, Laura Nicholson, Lauren Klebek, and Leonard A Jason. 2016. [Identifying key symptoms differentiating myalgic encephalomyelitis and chronic fatigue syndrome from multiple sclerosis.](https://pubmed.ncbi.nlm.nih.gov/28066845/) *Neurology (E-Cronicon)*.

6. Liliana Barrios, Pietro Oldrati, Marc Hilty, David Lindlbauer, Christian Holz, and Andreas Lutterotti. 2021. [Smartphone-Based Tapping Frequency as a Surrogate for Perceived Fatigue: An in-the-Wild Feasibility Study in Multiple Sclerosis Patients](https://dl.acm.org/doi/10.1145/3478098). *Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.* 5, 3, Article 89 (Sept. 2021), 30 pages.

7. World Health Organization (WHO). 2021. [International Classification of Functioning, Disability and Health (ICF)](https://www.who.int/standards/classifications/international-classification-of-functioning-disability-and-health).

8. Nikola Banovic, Tofi Buzali, Fanny Chevalier, Jennifer Mankoff, and Anind K. Dey. 2016. [Modeling and Understanding Human Routine Behavior](https://dl.acm.org/doi/10.1145/2858036.2858557). In *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems* (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 248–260.

9. Nikola Banovic, Anqi Wang, Yanfeng Jin, Christie Chang, Julian Ramos, Anind Dey, and Jennifer Mankoff. 2017.[Leveraging Human Routine Models to Detect and Generate Human Behaviors](https://doi.org/10.1145/3025453.3025571) In *Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems* (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 6683–6694. 

10. Anna L Kratz, Tiffany J Braley, Emily Foxen-Craft, Eric Scott, John F Murphy III, and Susan L Murphy. 2017. [How do pain, fatigue, depressive, and cognitive symptoms relate to well-being and social and physical functioning in the daily lives of individuals with multiple sclerosis?](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5660943/) Archives of physical medicine and rehabilitation 98, 11 (2017), 2160–2166.

11. National MS Society. 2021. [Who gets MS?](https://www.nationalmssociety.org/What-is-MS/Who-Gets-MS)

Check out the full paper here!
https://dl.acm.org/doi/10.1145/3580887