What to expect on accuracy & reliability for self-monitoring Multiple Sclerosis using the MS sherpa App — a literature review
Summary
Multiple sclerosis (MS) is a highly unpredictable disease of the central nervous system which can affect patients’ social functioning as well as neurological functioning. To help people with MS gaining control over their life again Orikami’s Healthcare Data Scientists developed MS sherpa. The MS sherpa app examines the functional mobility and cognitive impairment of a person with MS, and the ultimate goal for the app is to be used in clinical practice. The accuracy and reliability of these smartphone tests is currently being examined by the MS Zelf study. We describe the SDMT and 2MWT results from literature and what to expect from the MS Zelf study based on literature.
Introduction on Multiple Sclerosis
Multiple sclerosis (MS) is an unpredictable, often disabling disease of the central nervous system in which the immune system attacks and damages the insulating covers (myelin) of the nerve cells in the brain and spinal cord. This disrupts communication within the nervous system, resulting in a range of signs and symptoms [1], including physical, mental, and sometimes psychiatric problems. Such impairment has important implications in the clinical management of patients with MS [2]. Signs and symptoms of MS vary widely and depend on the amount of nerve damage and which nerves are affected.
About half of the MS population is cognitively impaired [3–5], affecting their employment status and social functioning. Walking limitations are also among the most visible manifestations of MS [6]. Some people with severe MS may lose the ability to walk independently or at all, while others may experience long periods of remission without any new symptoms. The Symbol Digit Modalities Test (SDMT) is a particularly sensitive measure of cognitive dysfunction in patients with multiple sclerosis (MS). The 2MWT (2-Minute Walking Test) has been proposed as a variant of the frequently used 6-Minute Walking Test to assess walking endurance [7]. It takes less time to perform and is a smaller burden to persons with MS, especially to those with decreased mobility.
On the MS Zelf study
MS sherpa (www.mssherpa.nl) is an app developed at Orikami (www.orikami.nl) for self-monitoring of MS. It examines the functional mobility and cognitive impairment of a person with MS, and monitors its evolution. MS sherpa performs an online version of SDMT by counting the correctly matched symbols in 90 seconds, as seen in Fig. 1. The 2MWT is performed by measuring the distance walked during 2 minutes. The MS Zelf study was carried out with the purpose of seeing whether these new smartphone tests and possibly also Fitbit data can be used for self-monitoring for people with MS. The intention is that these instruments can then be used in other scientific studies and in clinical practice. The analysis of MS Zelf study is ongoing, and the results will be presented in a future blog post. For now, we will present what can be expected from MS Zelf study, by examining some results in the literature.
Expectations from Literature
There is a large body of literature dedicated to the reliability of the SDMT and 2MWT. The MS patients tested in these studies have very different disability levels, measured by the Expanded Disability Status Scale (EDSS), and that is also reflected in the wide range of walking speeds found. The SDMT has been administered in several versions, oral, paper, and computerised.
As seen in Fig. 2, the mean results vary between 43–52 for the MS patient group (MS) and 47–59 for healthy control group (HC)[8, 9, 10]. This difference is significant, given a SEM between 4.9–5.9. The time between tests in the literature varies between a few hours [10] to several months [8], and that has influence in investigating possible learning effects. The test-retest reliability as measured by ICC varies between 0.71–0.83 for HC [8, 10] and 0.82–0.97 for MS [8, 9, 10], in a maximum of 1.0. The effect size varies between 0.7–1.1 [8, 9, 10], which is considered large [11]. There are also reports of a SDMT score increase with educational level [12].
Along with the 2MWT, there are also many other walking tests, such as the 25FWT (25-foot Walking Test). The average speed found in the literature for 2MWT and other walking tests vary between 0.77–1.25 for MS patients. The test-retest reliability, as measured by ICC varies between 0.87–0.98 [5, 10, 13, 13, 15, 16].
Age and disability level (EDSS score) seem to be a strong factor influencing SDMT scores and 2MWT distances. The average SDMT scores can vary between 55.0 in the age group 18–34 and 40.0 in the age group >65 [11]. The average distance walked during the 2MWT for the age group 18–35 is ~200 meters and for ages >65, an average of ~150 meters was measured [15]. This is possibly reflected in the differences in average walking speeds seen in Fig. 3.
Conclusions
1 — The SDMT scores and 2MWT average speeds are in general lower for MS patients than for the healthy population. This difference is significant, with a large effect size.
2 — The factors with the largest influence on the SDMT scores and the 2MWT results are the disability levels, measured by EDSS and the age of the participants. It is therefore important for the age distribution of the healthy control sample to be close to the MS sample. An increase of the SDMT score with educational level was also reported.
3 — To detect a real difference between MS and HC groups in SDMT scores and 2MWT distances it is useful to use the T-test, which produces the p-value and the effect size, as measured by Cohen’s d.
References
1 . Compston A, Coles A (2008): “Multiple sclerosis”. Lancet. 372 (9648): 1502–17
2. Olazarán J, Cruz I, Benito-León J, Morales JM, Duque P, Rivera Navarro J (2009): Cognitive dysfunction in multiple sclerosis: methods and prevalence from the GEDMA Study. Eur Neurol 61:87–93
3. Benedict RH, Cookfair D, Gavett R, Gunther M, Munschauer F, Garg N, Weinstock-Guttman B (2006): Validity of the minimal assessment of cognitive function in multiple sclerosis (MACFIMS). J Int Neuropsychol Soc 12:549–558
4. Nocentini U, Pasqualetti P, Bonavita S, et al. (2006): Cognitive dysfunction in patients with relapsing-remitting multiple sclerosis. Mult Scler 12:77–87.
5. Gijbels D, Alders G, Van Hoof E, et al. (2010): Predicting habitual walking performance in multiple sclerosis: relevance of capacity and self-report measures. Mult Scler 16: 618–626.
6. Brooks D, Davis AM, Naglie G (2007): The feasibility of six-minute and two minute walk tests in in-patient geriatric rehabilitation. Can J Aging 2007, 26:159–162.
7. Gijbels D, Eijnde BO, Feys P (2011): Comparison of the 2- and 6-minute walk test in multiple sclerosis. Mult Scler 17: 1269–1272.
8. Benedict RHB, Duquin JA, Jurgensen S et al. (2008): Repeated assessment of neuropsychological deficits in multiple sclerosis using the Symbol Digit Modalities Test and the MS Neuropsychological Screening Questionnaire. Mult Scler 14: 940–946.
9. Drake AS, Weinstock-Guttman B., Morrow SA, Hojnacki D., Munschauer FE and Benedict RHB (2010): Psychometrics and normative data for the Multiple Sclerosis Functional Composite: replacing the PASAT with the Symbol Digit Modalities Test. Mult Scler 16: 228.
10. Rudick RA, Miller D, Bethoux F et al. (2014): The Multiple Sclerosis Performance Test (MSPT): An iPad-Based Disability Assessment Tool. J. Vis. Exp. (88), e51318.
11. Cohen J (1988): Statistical Power Analysis for the Behavioral Sciences. Routledge. ISBN 1–134–74270–3.
12. Ruet A, Deloire MSA, Charré-Morin J, Hamel D, and Brochet B (2013): A new computerised cognitive test for the detection of information processing speed impairment in multiple sclerosis. Mult Scler 19 1665–1672.
13. Toomey, E, Coote, S (2013): Between-Rater Reliability of the 6-Minute Walk Test, Berg Balance Scale, and Handheld Dynamometry in People with Multiple Sclerosis. Int J MS Care 15: 1–6.
14. Bennett SE, Bromley LE, Fisher NM, Tomita, MR, Niewczyk P (2017): Validity and Reliability of Four Clinical Gait Measures in Patients with Multiple Sclerosis. Int J MS Care 19:247–252.
15. Feys P, Bibby M, Romberg, A et al. (2014): Within-day variability on short and long walking tests in persons with multiple sclerosis. J Neurol Sci .
16. Kieseier BC and Pozzilli C (2012): Assessing walking disability in multiple sclerosis. Mult Scler 18: 914–924.
