Understanding how OEM’s use customer feedback to improve vehicle quality
Going from a being reporting platform to an investigation tool.
The Enprecis CQi reporting platform is a customer feedback analysis tool that connects automotive manufacturers, and dealers with vehicle owners (via new vehicle owner surveys) in the hopes of gathering insights into vehicle quality and overall customer satisfaction. The goal of the automotive manufacturer is to improve vehicle quality and design, while preventing recalls and increasing owner loyalty. The goal of Enprecis is to provide the tools and data they needed in a meaningful and compelling way.
One major issue Enprecis faced as a company was that they lacked a clear product vision. In addition, (other than the account reps) very few people had insight into how Enprecis automotive partners used their tools and data. Because of this, they became reactive to every client request, which ultimately led to disparate platforms for each of their clients. As a small company, this wasn’t sustainable.
So for this startup, they needed to invest in being more proactive vs. reactive in creating a product vision and understanding client needs. This was a huge step for Enprecis as a company, and one they needed to take.
In addition, Enprecis had a somewhat inexperienced product team who had never been through any type of visioning process. So with the help of a company called InContext Design, we worked together to educate and facilitate the learning of a UCD process. InContext are the creators of a UCD methodology called Contextual Design (CD), which is a design process that incorporates ethnographic methods for gathering data relevant to products via field studies, rationalizing workflows and designing of HCI.
I was introduced to InContext and CD back in 2000 while Design Lead for MSDN and knew if I wanted to educate my team about how to conduct proper Contextual Inquiries, building Affinity Diagrams, and defining a product vision, that I wanted them by my side to do so.
Project Team Specifics
Project: Define next generation reporting platform
Team: Davin (Product Owner), Jessica (Researcher), Michael (Solutions Architect) and Myself (Project Owner and UX Lead)
Methodology: Agile, Contextual Design
Our Goal — Create a vision for Enprecis Reporting
- Create a vision which will inspire, rally, guide focus and unify our team as we move into planning and design
- Articulate and define user experience and business goals for stakeholders, planning and engineering
- Defined design principles, user insights and development of deep product knowledge that will help us find collective agreement on product direction and identify strategic opportunities.
- Cohesive understood plan of both our product and business goals as we head into Phase 2: Reporting UX Initiative — Planning & Design process.
- Affinity Diagram
- Pain Points
- A Vision for Enprecis reporting
RESEARCH and DISCOVERY
Our plan was to approach this process in phases — with phase 1 being Research. Phase 1 contained three milestones, over four 2-week sprints.
NOTE: Enprecis was purchased by a private equity firm during this process, and the office was for the most part shut down not long after Milestone 2 was complete. So for the purposes of this case study, I will take you through the process we went through up until Milestone 2 (M2).
We started out with two OEM site visits. I’m not at liberty to discuss specific names of the auto manufactures we interviewed, so for the sake of an NDA, I will refer to them as OEM1 and OEM2.
OEM1 is an Asia based automotive manufactures with their North American headquarters based in the US. While at OEM1, we performed contextual inquiries with five product quality team members.
The quality team roles interviewed consisted of:
Model Line Engineers focused on monitoring quality for a particular vehicle line, across all years. Model Line engineers tend to have more experience and are able to pull their own data from various internal and external systems for their respective vehicle lines. Once they determine there is a potential issue, they open up a case file and start the investigation process. Once they believe they have a solid case, they create a PPT presentation of their case and present it to the HQ Coordinator and their Senior Group Manager. If the case is solid, the HQ Coordinator presents it to HQ where the fix and countermeasure is determined and implemented.
System Engineers essentially focus on vehicle systems and/or parts. This could be anything from the infotainment/navigation system to brakes. These systems typically span multiple vehicle lines. Their goal is to monitor and report on system issues across vehicle lines. System Engineers are less familiar with finding or pulling data from internal systems, e.g. CQi and often rely on more senior members of the team — especially the Fleet Manager for help. Process wise, once they’ve determined an issue, they open, investigate, and build their case for HQ. Their process is similar to Model Line Engineers, but often times are more junior and have less research experience.
The Fleet Manager manages all departmental test/fleet cars. This person often works at assisting System Engineers in pulling the data they need for investigation and case building. In addition the Fleet Manager works with the legal team, providing warranty data and customer comments. This person is the keeper of all data. They manage partner relationships with Enprecis, and know every system inside and out. Often times the same part codes are different from system-to-system, and this person manages the matrix of every system.
There is a HQ Coordinator for every department. This persons role is to work with engineers to understand their cases, and to determine if there is enough in their case to present to OEM1 headquarters in Asia. They are the liaison from the USA to Asia and back.
The Senior Group Manager, monitors all issues being managed by their team and HQ. This person works closely with the HQ Coordinator to determine which cases get presented to HQ. This person also works with the Warranty Group, and will often times give the engineers a heads up of potential issues popping up in new warranty claims.
OEM1 Quality Case Management Process
- Warranty data is king. Enprecis data is seen as supplemental and is used to give context to warranty data when investigating issues and building cases
- Users proactively look for issues by checking each data source first thing, every morning. There is no alert mechanisms in place.
- Enprecis CQi reporting system is very slow
- Enprecis data is always exported, edited in excel to either clean/scrub the data, or reformat it so that it can be read properly and/or exported into another system
- Users used many different tools and data sources to investigate — because of this users memorize codes and create cheat sheets in order to interact between tools. e.g the same part may have different codes in different systems.
- Collaborating on an issue is difficult due to so many systems
- OEM1 HQ determines all fixes/countermeasures and implementation time frame. The case engineer has no direct input on countermeasures.
- All investigation research is pulled from various systems, saved to excel or word documents and stored in a case file on an internally built quality management system.
- Survey changes sometimes affect reporting data, and engineers often times don’t have insight into those changes. e.g. a survey question wording changes, or gets added and there is now a flood of complaints, and the engineer is wondering, “why now, what changed?”
OEM2 is also a long term client of Enprecis and based in Detroit. At OEM2, we interview six quality team members. Overall roles and processes were somewhat similar to that of OEM1, with exceptions in a few key areas.
- Role wise, they had engineers just dedicated to researching potential issues that would then be handed off to the appropriate owners for further investigation.
- Engineers managed determining fixes and manage implementation and tracking of countermeasures, as they worked at HQ. They worked directly with manufacturing plants to rollout countermeasures.
- OEM2 tracks all cases on their Quality Wall. Each case presentation and status report is printed out and hung on the wall. As a group, they meet up each week to review progress and overall status.
- Engineers used and relied on Enprecis data much more so than at OEM1.
- CQI Reporting is impossible to use on screen, as corporate standard pc monitors only supported 1024x768, and the reporting tool was designed to work on a higher resolution screen.
- All data was exported into excel in order to data mine comments.
- Users manually data mine through comments to create keyword matrices of possible issues for tracking-daily.
- They manually look for data relationships as the current CQI reporting system doesn’t provide that functionality
- Users track all changes (e.g. survey question changes and/or counter measures manually, as there is no way to visually track changes in the reporting tool.
- Building a case involves many people, some of which don’t have access to the tools so they have to export and format the data.
After the OEM visits, we worked together as a team to interpret each interview and document the data points from our interviews. In the end we interviewed 11 users, across two OEMs and captured over 800 data points. A bit overwhelming, so I will summarize our findings below.
Affinity Diagram End-to-End Process, Findings and Summaries
Triggers come from a variety of sources, like unusual data presented in a reporting system, an alert from an internal reporting system, an emergency request from the legal department, or an engineer receives research from others about issues they’re responsible for.
Investigating an Issue
In the case of an issue, engineers use multiple sources for investigation. They investigate by validating issues on company/employee vehicles. They data mine through customer feedback/comments. They manually create tools and matrices. They query multiple reporting platforms. They also often times just rely on their experience to determine whether or not an issue is valid.
Compiling Data/Building a Case
Users currently aggregate data manually as their is no centralized data source. Because of this they have to memorize lots of codes to efficiently look up info. Different codes are used to identify the same thing in different tools, and they need to reference multiple sources to translate codes.
Aggregating data manually leaves room for human error, and the process is painfully repetitive. When aggregating comments to look for patterns in data, often times a comment could reference multiple areas of the vehicle. Therefor if a vehicle owner comments about their brakes and seats in one comment, a system engineer could potentially miss a relevant comment.
To get an issue resolved, cases go through multiple levels of escalation. Engineers are ultimately responsible for figuring out an action plan for a counter measure resolution. Escalation happens when issues require new parts or money.
Once a countermeasure is in place, engineers monitor the case via monthly data reports. And, because the current system doesn’t allow users to track when countermeasures were implemented, engineers manually track when they’ll expect to see a change in vehicle owner feedback. Engineers typically look for countermeasure effectiveness around the 60-day mark of when the countermeasure took place.
Communicating Current State Through Corporate Reports
Engineers create monthly reports to ensure a shared understanding within their team, and up through multiple levels of management. Monthly reports tells them how they’re doing across all vehicle lines.
Monthly reports are typically based on a shared template, which includes data from multiple sources and people. The type of data included in these reports relate to case status, with a comparison of previous vs. current month, and forecasting data.
Ultimately we believed the tool should support the automotive engineers workflow, and that data should be presented in a meaningful way.
We believed that the tool needed to be able to…
Alert me to the things I want/need to know
- Find connections for me — data mining & push relevant potential trends & issues to users (push not pull)
- Keep track of Watch Phrases and Top Issues/Results
Make investigation easy
- I want details on any data from my dashboard — Data Views
- Easy, fast way to get all related information from a given starting point, e.g. VIN, Part Codes, allowing users to pivot through the system to see data relationships
- Easy learn about me system (saved queries)
- Allow me to search on anything I know and see all the related naming conventions
- Within my search results, allow me to see data relationships
Allow me to track changes in the data
- Flag data milestones (e.g. Survey Change (system), Counter Measure Implementation, etc… giving me a way to track changes.
Make sharing data and collaboration easy
- Pin data views to my sandbox or to reports and/or cases
- Allow me to create dynamic reports where the data is always current
- Allow me to see reports based on time periods
- I want to share my reports and/or cases — support my workflow
- Allow me to follow data — subscribe to alerts based on thresholds
- I want to be able to export, save and share
- Research assembly plant workers to capture all case management users
- Develop personas
- Paper prototype and test scenarios that support vNext Vision
- Create vNext Vision Roadmap