Nov 9, 2021
Into the BLUE (framework): Designing user centered in-product feedback
Lessons learned from Microsoft Azure Data UXR
Coauthors: Samira Jain, Vichita Jienjitlert, Dr. Paula Bach
Based on lessons learned from developing feedback mechanisms for Azure Data, we created a framework, dubbed BLUE, for designing and implementing in-product feedback that supports the end-to-end design and implementation process.
In-product feedback is a valuable tool for gaining insight into how users think and feel about a product or service. The key benefit is unlike simulated test environments (e.g. usability), users provide feedback in their live environment while using the product.
Further, common issues that arise from not following a framework include ad hoc mechanisms designed by individual teams, metrics that do not align to business objectives, and inconsistent ways of capturing metrics. In some cases, these inconsistencies can result in disruption to the user experience.
To provide some context to the framework these are three common feedback mechanisms:
1. System-initiated feedback mechanism (pop-up for overall product use)
This mechanism relies on sampling and is generally focused on capturing metrics to better understand overall sentiment using a toast or pop-up message.
2. Behavior-initiated feedback mechanism (intercept based on user action)
The user provides feedback about the actions they just completed using this mechanism. An example of this in use would be the question prompt, How helpful or unhelpful was this error message? The corresponding response item used with this question type is a scale from “Not helpful” to “Extremely helpful.”
3. User-initiated feedback mechanism (always available)
This mechanism is an ‘always available’ channel for users to provide feedback and is useful for bug reporting and for capturing feature ideas that can be explored through additional user research efforts. It’s important to note, the objective of this mechanism is not to measure satisfaction of a product but rather provide users with a channel to be heard.
The Framework
The BLUE Framework guides UX practitioners through the UX health metrics process. It builds on best-in-class frameworks, for example HaTS (Muller & Sedley, 2013) and UX Outcomes (Spool, 2021), and The Customer-Driven Culture (Lowdermik & Hammontree, 2020) to provide direction on:
- What UX metrics to capture and how to capture them
- How to create customized UX health metrics for content
- Which guiding principles to apply to the relationship between Business and UX metrics
- How to embed in-product feedback efforts in the context of real-world collaboration, socialization and the UX research process
Build
Build UX metrics. Building metrics is more than thinking about outputs — it’s about getting to UX outcomes, translating the outcomes into objectives, and mapping those to metrics that matter within your product and organizational context. For example, if an organizational goal was to build quality features that will increase satisfaction by X%, a UX outcome could be that users feel empowered by completing jobs easily within the system, and a metric could be perceived ease of use (Overall, how easy is it to use <product>, 5-point rating). The aim of the Build pillar is to capture metrics that are tied to organizational and UX outcomes that can be translated into actionable insights later for stakeholders to build products and features that make users’ lives better.
Build principles:
✓ Align with business goals
✓ Investigate product UX outcomes, goals, and metrics
✓ Focus on consistency of scales and ratings
Leverage
Leverage proven mechanisms. We recommend starting with the HaTS framework (learn more) as a guide for the mechanism behaviors, specifically around utilizing random sampling based on individual users instead of page product views. Design the mechanisms such that users can provide both feedback on high-level satisfaction questions and feedback on more granular, task-based questions. In addition to the system-initiated, it’s also important to give users a voice by providing a user-initiated option.
Leverage principles:
✓ Utilize HaTS and adapt as needed
✓ Customize needs around sampling/cadence
✓ Include a user-initiated feedback mechanism
Unlock
Unlock the results. This pillar speaks to supporting stakeholders for how they currently or ideally would like to access the UX health data. After all, they are the users of the metrics, and being user-centered, this helps drive success. The goals for unlocking should make the UX health metrics accessible so stakeholders can empathize with the users’ stories. Planning for the best way to serve up the results ahead of time is an important part of ongoing success and engagement with the data, contributing to a data-driven culture.
Unlock principles:
✓ Make the data accessible
✓ Keep the user story central
✓ Contribute to a data-driven culture
Embed
Embed in practice. The final pillar requires embedding the UX health metrics into your UX research practice and socialization. Showing how we can prioritize the user with these metrics and tracking their impact across the entire UXR process is a powerful story to share. Finally, with unique stakeholders, user goals and flows, there is no one-size-fits all approach. Embedding business data with UX health data helps everyone make better decisions with the user in mind.
Embed principles:
✓ Socialize for understanding so that everyone advocates for users
✓ Combine with UXR process and track impact
✓ Get feedback on the feedback
Conclusion
We believe this framework provides the guidance necessary for research teams to lead the implementation of UX in-product feedback giving their users a stronger voice. View the full paper on the BLUE framework.
