Features & Feedback: Co-Designing the Future with CR

Reflecting on Feedback from Collaborative Session and Concept Validation

Welcome back to MHCI Consumer Reports Capstone Team! After taking a short break after our Spring Presentation, we kicked off the summer term with a successful collaborative session with our clients in New York. As well as collecting feedback from our clients, we have begun the concept validation phase to evaluate consumers’ sentiment towards our design concepts and whether their needs are met by these prototypes.

Collaborative Session, Co-creating Solution

With the purpose of understanding the priorities of our stakeholders, their thoughts on the feasibility of the design concepts and the potential tradeoffs that they may need to make, we conducted three co-design activities during our collaborative session. The first activity was a Rose, Bud, Thorn (RBT) session, wherein we recapped our three design concepts using videos and elicited feedback and critique using the RBT framework (Rose = positives, Bud = areas of opportunity/improvement, Thorn = challenges).

Wait… What Are These Design Concepts?

To give all stakeholders a recap on the prototypes we have built to address consumer pain points uncovered in the research process, we showed them the demo videos. We developed the following design concepts at the end of the Spring term:

#1: Negotiation Helper
Leveraging the capabilities of AI, LLMs, and cloud technologies, Negotiation Helper enables real-time policy alerts upon potential violation by company customer service, post-call tips to strengthen argument and advocacy, and centralizing businesses’ contact channels and call history for easy access.

Negotiation Helper.mp4

Rose, Bud, Thorn Feedback:

• Rose: Stakeholders appreciated the feature that compiles contact information, eliminating the need to search for company numbers. They also liked the coaching aspect, which helps users navigate customer service calls.
• Bud: They believe there is potential to generate high-integrity datasets from customer service call transcripts. This could enable CR to provide valuable insights to companies about their customer service performance, potentially creating a new revenue stream for CR.
• Thorn: Stakeholders mentioned legal concerns regarding the recording of conversations.

#2: Policy Assistant
Policy Assistant flags dark, deceptive patterns and predatory policies to the user, explains policy implications in a clear, comprehensible manner, and provides actionable advice and recommendations. It can be initiated by the user through their phone’s Control Center of their phone, granting access to the content that the user is currently viewing.

Policy Assistant Presentation Video Clip.mp4

Rose, Bud, Thorn Feedback:

• Rose: Stakeholders like the concept’s clear explanation of policy implications and actionable advice.
• Bud: They believe that combining this with the Negotiation Helper could enhance its utility.
• Thorn: They were concerned about the complexity of sifting through extensive information.

#3: CR Wallet
The CR Wallet stores users’ products along with their respective policies and terms of service by allowing the user to connect to their Consumer Reports account, or other online retail accounts via a user opt-in. Engage the smart agent on command, enable it to inform and guide you through troubleshooting, and even submit service requests on your behalf. The wallet leverages an LLM in order to communicate effectively with the user.

CR Wallet.mp4

Rose, Bud, Thorn Feedback

• Rose: Stakeholders like how the concept supported users throughout the post-purchase journey, making it easier for them to manage their products and service requests.
• Bud: They believe there is an opportunity to integrate this design within the current CR app.
• Thorn: Stakeholders raised concerns about whether consumers would feel comfortable using the augmented reality (AR) feature to identify product issues and overall adoption rate of this technology. They believe marketing this feature will be difficult.

Buy A Feature and Build Your Own Service!

The RBT critique session was followed by the “Buy A Feature” activity, wherein our clients were provided with a set of 17 “features” and a limited budget of $10. We then requested our clients to buy features that they believed to be the most important. Once they had “bought” their desired features, we asked them to explain their choices in order to understand the needs and priorities of each stakeholder. Following this, we facilitated a final “Build Your Own Service” activity, wherein the various stakeholders were requested to envision their own service across the pre-purchase, purchase, and post-purchase phases using the features they bought (plus, a wildcard feature!). Stakeholders were asked to pitch their service designs with the group, and all participants were asked to pick a winning service idea in a quick Venture Capital competition! :-)

Key Takeaways:

• Stakeholders noticed significant overlap between the three design concepts, suggesting a potential for integration. For instance, the negotiation helper’s real-time coaching and the policy assistant’s policy explanations could be merged into a comprehensive tool.
• The idea of focusing on specific transactions like warranty claims, repairs, or replacements was highlighted as a way to scale the service effectively. Stakeholders recommended using these transactions as focal points to address major consumer pain points.
• The features that were prioritized by a majority of stakeholders were:

1. Centralizing Company Contact Information — This feature emotionally resonated with stakeholders, as it simplifies a common consumer frustration. However, stakeholders expressed concerns about the technical challenges involved in compiling contact details from various companies due to the frequent changes in contact numbers.
2. Providing Customer Service Reviews and Ratings — Can be present at both pre- and post-purchase phases. Stakeholders believe this can be added to current CR subscription as an additional benefit for CR subscribers. Additionally, can empower users to make informed choices about how to interact with companies, plus can have upstream impact on companies.
3. Building Arguments using LLM — Can equip users with the knowledge and confidence to advocate for themselves effectively.
4. Acting on Users’ Behalf (with reservations) — There were mixed opinions on using AI to directly interact with companies on behalf of users. While stakeholders saw the potential for efficiency, they were concerned about low user adoption rates and costs.

• The client was particularly enthusiastic about Negotiation Helper’s real-time coaching and transcript storage features. These capabilities enable CR to generate high-integrity datasets from customer service calls, allowing CR to have upstream impact on companies and create a new revenue stream through data-driven insights.
• We discovered that CR stakeholders are open to collecting user data, provided it offers a fair exchange of value to consumers. They also stressed the importance of being transparent about data collection and ensuring users have the option to opt-out.
• Stakeholders highlighted the importance of thinking about “entry points” to “lock-in” the consumer. They suggested some new interesting ideas for the same:

1. Community/Social Engagement — Leveraging the family unit and recommendations/tips from the hobbyist communities to “lock-in” consumers to the service.
2. Leveraging the Unboxing High — Stakeholders emphasized the need to capture consumers at high points in the customer journey to build trust. For example, catching consumers at the unboxing stage when they are excited and happy can build trust with the agent, which can encourage them to seek out the agent when they need help in the post-purchase stage. One stakeholder came up with a feature in the unboxing phase called “Help Me Play”. This feature provides users with an opportunity to share tips and tricks they discover as they are exploring their new product. This allows users to share their moment of excitement and also helps other users learn new and interesting things about the product.
3. Personalization — One stakeholder envisioned a service called “Before You Do,” which analyzes a user’s purchase and customer service history to preemptively alert them of potential issues that they may face, before they make a purchase. This feature mirrors users’ behavioral patterns and blind spots back to them, providing insights that help them understand and improve their purchasing decisions. Users appreciate learning new things about themselves, and this personalized feedback can serve as a powerful way to engage and “lock-in” users by continuously adding value to their consumer experience. Stakeholders also liked the idea of providing personalized tips advising users on the best communication strategies to resolve customer service issues effectively.

Concept Validation, Need Validation

The other big accomplishment for Sprint 6 and 7 is that we completed our first phase of concept validation play-test sessions. The goal of this activity is validating that our designs are meeting what consumers:

• To gather insights on users’ preferences and needs when contacting customer service.
• To identify any challenges or difficulties users face in processing the design of the Negotiator Helper prototype.
• To investigate potential differences in user experience based on age groups, particularly regarding multi-modal/sensory interactions.

Wizard-of-Oz’ing A Customer Service Call

Because from our collaborative session, Negotiation Helper has garnered significantly more interest from our clients than the other concepts, we decided to move forward with this idea and conduct concept validation for this solution. In order to simulate the experience of being on a call with a company’s customer service representative, we thought, “what’s a better way to test this than Experience Prototyping to communicate this concept to the users?”

We thus design an experience where users are given a scenario (being overcharged by an ISP without authorization) and instructed to explore the app interface of Negotiation Helper to call the customer service representative. Two team members are in another room, one of which pretends to be the representative taking the call, while the other is acting as the CR Agent (Negotiation Helper) behind the scenes.

What was the feedback, you ask?

Stay tuned for our updates next sprint where we bear news about our testing results!

(note: The work and knowledge gained from this project are only intended to be applicable to the company and context involved and there is no suggestion or indication that it may be useful or applicable to others. This project was conducted for educational purposes and is not intended to contribute to generalizable knowledge.)