Secrets To Successful Mystery Shopping
Research shows that interpersonal communication between library staff and patrons is just as important to the patron as whether a question was answered correctly. 1 But how can you accurately capture and evaluate such interactions? One possible method is mystery shopping.
In a mystery shopping exercise, a shopper poses as a patron and asks a library staff member a question. Immediately following the encounter, the shopper reports on the service provided. The corporate sector has used mystery shopping for years to measure whether employees provide quality customer service and adhere to guidelines and policies.
· Lay the groundwork
When we first discussed mystery shopping, we realized that we needed to have a clear idea of how we would use our findings and how it would most benefit the libraries and their staff. Did we want to use it as a snapshot to measure current service only to start a cyclical training program, or perhaps a combination of the two? Establishing goals would allow us to measure the success of our exercises. Ultimately, we found it best to work with the immediate desk supervisors to identify the goals; interestingly, each one settled on a different desired outcome.
Knowing that the mystery shopping exercises could reveal potentially sensitive information, we each designated people other than the desk supervisors to receive and compile the shopping data. We also came to agreement as to who could have access to the information. At both Radford and Longwood, the desk supervisors wanted reports that measured service overall, without identifying individual workers. The resulting reports would be shared library-wide. At Longwood University there was an additional component; individual staff members were shopped, and the supervisor shared the confidential results with each staff member.
Before we even mentioned the program with the desk supervisors, the authors discussed with each other how the gathered data could be used to improve service. Mystery shopping takes considerable effort to plan and carry out, not just by the researchers but also by the staff involved. We didn’t want to demoralize everyone by launching a large evaluative process that goes nowhere. Everyone involved in the exercises agreed that each area shopped should have concrete plans to use the data. At our libraries, each service desk decided to create training programs to address areas identified by the mystery shopping that could be improved.
· Involve all stakeholders
When we floated the idea of mystery shopping by our library administrators, the response at both institutions was enthusiastic. Our library administrators were eager for us to provide both quantitative and qualitative data about how well we do what we do at our service points.
Once we got the okay from the higher-ups, it was on to the desk supervisors. At Radford University, the reference and circulation desk supervisors were intrigued. Neither had the time to personally observe all of his or her workers on the desk and loved the idea of getting a better sense of what goes on at the desk. At Longwood University, the desk supervisors were interested in generating staff accountability at service desks and proving what we believed to be true: that our staff did provide quality customer service.
Employees being shopped
This was our biggest challenge. All of the groundwork we had done earlier prepared us to answer their questions and assuage their concerns. At the same time, we had to be flexible with our planning. We knew a successful exercise would involve the employees as much as possible. Therefore, staff were involved from the ground up. They developed the customer service guidelines by which they would be evaluated. This generated buy-in from the start. Although we put together a framework ahead of time, we solicited feedback about each step of the exercise. We made many modifications to the exercise based on feedback from the employees, and, because transparency is imperative, we made sure these modifications were communicated. In our first round of mystery shopping, we also told the employees the beginning and end dates of the exercise. They wouldn’t know when exactly they were being shopped, but they knew the window. This increased their comfort level with our “beta” mystery shop. Over time, once staff were comfortable with the process, we were able to stop giving advance notice, and let them know mystery shopping would happen regularly.
Institutional review board and human resources
We ran the exercises as research studies and went through formal IRB review. If you are mystery shopping unionized employees, be sure to discuss your plan with the union.
· Establish model behaviors
Quick quiz: how many of you have written guidelines or standards for appropriate customer service at the reference and circulation desks? We didn’t either. But we all know what proper desk behavior is, right? Individually, sure. But coming to consensus was surprisingly challenging. Consistent service is imperative. We were going to evaluate everyone using the same shopping instrument, so we needed everyone to understand what behaviors are expected.
The conversations among the desk employees as to what constitutes model behaviors were fascinating. We set ground rules for open but courteous discussion. We used existing model behavior lists, such as RUSA’s “Guidelines for behavioral performance of reference and information service providers”3 to generate discussion about generally accepted standards, to guide us in the process, and to help us from re-creating the wheel.
· Design an effective mystery shopping instrument
In both libraries, we had lists of model behaviors that were far longer than could be measured by one shopper. Our first mystery shopping exercises revealed that if the shopping evaluation tool were too long, some of the shoppers would take the form with them to the desk and fill it out right in front of the employee. In the end, we worked with desk supervisors to prioritize the behaviors to be measured and shorten the length of the shopper questionnaire.
For each behavior to be measured, we created a scale identifying what is unacceptable, acceptable, and optimal for each measured behavior. Example: Behavior to be measured: Were you greeted promptly? Possible answers: Unacceptable: I was not greeted. Acceptable: I was greeted after I said hi or hello. Optimal: Employee immediately greeted me.
We then formulated the questions the mystery shoppers would ask the staff. We wanted the question to be involved enough to elicit the behaviors being evaluated, without being so complicated the shopper would not be able to sustain the transaction.
Examples that have worked for us: Do you have any books by [author name]? Can you help me find a scholarly article on images of women in advertising? Examples of the types of questions to avoid: Directional questions such as, Where is the bathroom? These answers do not require much interaction. Complicated questions that lead to a lengthy reference interview should be avoided as well. Often the shopper will not have the time for a personal consult, or will not have the background on the topic to fake it for long. At Radford University, we worked with business majors. We had the most success feeding them questions like “Can you help me find an article about outsourcing factory work to Asia?” At Longwood University, we had a variety of multi-disciplinary questions designed to elicit the behavior being evaluated and determine if the staff person could answer the question correctly.
· Set your shoppers up for success
We decided that we needed to use shoppers from outside the library to maintain objectivity. At both universities, we worked with business faculty to use their students as shoppers. The faculty members graciously gave extra-credit or participation points to students who shopped us. Although professional mystery shoppers could be hired, we were satisfied with our arrangement. The advantage with using on-campus shoppers is that they are familiar with the academic environment and their expectations may be more in line with our average user than an outside shopper might be.
We presented the project to the student shoppers in their classes and described mystery shopping in general and the specifics of our plans. Even after sharing these details, we had to emphasize to the students that the point was to evaluate the service, not to test the students’ library knowledge. Some students expressed discomfort at approaching the reference desk with a simple “Do you have this book” question, explaining that they knew how to use the catalog. Role playing a mystery shopping occurrence helped them better understand their function, as well as teaching them appropriate shopper behavior.
As we continued with our mystery shopping exercises, we were able to also share some changes in the libraries that had come about because of past results. These examples often made the project ‘real’ to the students and increased how seriously they took their roles.
We reviewed the mystery shopping instrument the students would use, reminded them not to bring it to the desk when conducting their shop, and clarified the procedure for returning the completed instrument. At Radford University, we eventually moved to an online form, which simplified the process greatly. To make sure the same librarian wasn’t shopped repeatedly, we scheduled our shoppers. At Longwood, we provided staff pictures to our shoppers that were to help them identify whom they were to shop. This greatly helped ensure accuracy of who was shopped.
· Share the findings
At both libraries, librarians who did not staff the reference or front desks during the mystery shopping study were given the raw data. Since Radford University was using an online survey tool that aggregated the shoppers’ data, the Radford University author focused on eliminating identifying details from the free comments section. Final reports, assessing overall the service provided, were given to the desk supervisors. The supervisors then met with the staff to review the results. Successes were celebrated and areas for improvement were identified.
For more information, visit our website — tophawks.com.