
Your monitoring system should address two questions: what’s broken, and why? The “what’s broken” indicates the symptom; the “why” indicates a (possibly intermediate) cause.“What” versus “why” is one of the most important distinctions in writing good monitoring with maximum signal and minimum noise.
Remote tests can use both qualitative and quantitative methods. Quantitative methods can review interactions, animations, navigation, task times, completion rates, preference tests, and A/B tests. Qualitative methods can include desirability, visual look and feel, and content and messaging. I’ve previously written about readability of notifications based on the amount of time users prefer to see the banner before it disappears and how this test can be done using remote methods.