Buckle Up for Virginia Beach!
With sprint 8 in the books, our team is ready to hit the road at full speed! Jumping off of our testing of the test drive experience in sprint 7, we started this sprint by testing the pre-test drive experience of “kicking the tires”. After that, our team traveled to Virginia Beach to test our conversational agent with customers at a real CarMax store. Beyond all of the learnings we were able to take away from our in-store research, we had a great time with the process of hands-on testing with customers and look to use that momentum as we start refining our design for the end of the semester!
“Kicking the Tires”
Testing the Pre-Test Drive Experience
Having completed testing and synthesis around the test drive, we shifted focus to the interactions that shoppers have with a car before starting their test drive. We saw this as an area of opportunity for rich conversation between customers and our agent.
We started exploring this space by planning the design of our conversational agent and thinking of how we might use this time of “kicking the tires” to help customers better prepare themselves for their test drive. In doing so, we landed on the idea of a guided tour around the car — one wherein our conversational agent would walk shoppers around the car to show off the features of the car and demonstrate their functionality.
We hypothesized that this experience of “kicking the tires” with our conversational agent would enable shoppers to learn more about the car and get information that would help make their test drive even more enjoyable. An area of interest we identified from prior testing was shoppers’ questions around the operational use of features in the car; questions like “How do I turn on adaptive cruise control?” or “How do I connect my phone to the car’s Bluetooth?”. Working in tandem with this, we also focused on using the experience to contextualize features of the car to shoppers’ personal interests to better help themselves visualize their future in the car.
With a specific interest on providing customers operational details of the car and contextualized information, our primary goals with this research endeavor were to….
1. Understand how we can best highlight car features in a guided tour before the test drive
2. Understand how we can best contextualize the car information for each customer
3. Understand the limitations of a conversational agent while facilitating the “kicking the tires” process
For this round of testing, we continued to use a wizard of oz approach to running our conversational agent. This allowed us to use our script for the more prescribed parts of the experience and gave us the flexibility to field people's’ questions as they were learning about the features of the car. Our tests were conducted over two days with six members of our cohort.
From a high-level, the experience was found to be helpful, and participants felt better prepared for a test drive after “kicking the tires” with our agent. Specifically, we found that people liked the contextualized information that the agent provided and enjoyed being guided around the car before taking more control during the test drive. We also found that people wanted more time and space to explore the features of the car after being introduced to them.
Having completed this round of testing, we quickly began to brainstorm and iterate on the experience of “kicking the tires”. Beyond thinking of how the agent could be used most effectively, we started to design screens to help visually facilitate the experience.
These findings, combined with data from our prior testing of the test drive, were used to shape our prototypes for in-person testing with CarMax shoppers in Virginia Beach!
Virginia Beach, here we come!
In the second week of this sprint, our team was very fortunate to have the opportunity to travel to Virginia Beach to do in-store testing with real CarMax shopper. This was our first time, as a team, at a CarMax store and our first (in-person) meeting with members of the Conversational Experiences team!
Meeting the CarMax team and Learning the Store
We kicked the first day off by introducing our team and meeting members of the sales team at their morning “huddle” meeting. This provided us the opportunity to meet members of the sales team and introduce our work to them. After wrapping up, we started to finalize our plans for the prototype testing with our clients (ConvEx team!) and got to interview sales consultants to better understand how to engage with shoppers.
Then, on day two, we got the opportunity to take an in-depth tour of the store. Learning about store operations, like the production bay and the business center, was informative to our teams’ understanding of how our project could fit into the current CarMax service offerings.
Setting-Up & Running the Prototype
For our testing with shoppers, we were going to be testing the prototype in the car they were actually test driving. We thought about having a designated “test” car but ultimately decided to use shoppers’ real test drive cars so that we could most accurately evaluate how people would interact with the agent while driving a car they are interested in purchasing.
However, this decision to use shoppers’ real test drive cars required our team to make a prototype “rig” that could quickly be set-up and taken down for testing. For our prototype rig, we had 3 phones set-up (see above): One used for the interface and voice input, one used for capturing video feed of the shopper, and a third (not pictured) for tracking GPS location.
The decision to test with shoppers’ real test drive cars required quick thinking and planning on our team. To actually run the test, we had to have all 5 team members working to pull it off. We started each test once we found out what car the shopper would be test driving — from there, we had to quickly update our research tool with data relevant to the car that was being driven. We’d then set-up the rig in the car and get the test started. While running the test (see above), we had 3 team members running the experience. One member was controlling the tool that acted as the conversational agent, another was controlling the visual interface, and lastly another teammate researching facts about the car as they were needed throughout the test.
Testing with Customers
For the first run of our prototype, we had one of the sales consultants do the prototype test during one of the morning “huddle” meetings. We had the live video feed of the sales consultant driving the car playing for the whole sales team and we were able to gather the teams’ feedback before we started testing with customers. From this experience, we were able to make some quick adjustments to the prototype. Specifically, we added an audio cue for when the agent would be interacting with the shopper.
With all of the prep work and initial tests completed, we were ready to put our work in front of shoppers! We ended up testing with 6 shoppers (several of which had family/friends joining them on their test drive). We started each test with a pre-survey to learn more about their interest in the car they were test driving and get at some of the context for their visit to the store. Then, after a brief onboarding, shoppers started with the “kicking the tires” experience and then, finally, the test drive.
During the tests, our team was all hands on-deck running the conversational agent, pulling information about the car being driven, and recording notes. We completed these test in collaboration with the shoppers’ sales consultants and members of the ConvEx team (see below)!
Overall, we received positive feedback from the testing - we even “sold” 5 cars! We learned a lot about how shoppers will interact with our agent and are taking away actionable insights that will help as we iterate on our design.
Next Steps!
With our trip to Virginia Beach in the books, our team is feeling a lot of momentum as we head into the last two months of the project! We’re planning to continue refinement of our prototype and work towards a functional prototype that can stand alone. We will continue to conduct validation research and synthesize our findings as we continue to polish the design and implementation of our conversational agent in sprint 9.
