When it comes to star ratings, the devil is in the detail
The Verge has just published a fascinating article, “The Rating Game”, about the evaluation systems used by matching platforms such as Airbnb, Uber, Task Rabbit, Postmates, and a growing number of other companies. The article details how these systems convert the customer into companies’ quality control department, allowing them to concentrate on growth without having to employ a legion of supervisors. At the same time, the article highlights the problems using these mechanisms create in terms of managing the people who provide the service in question, leading to relationships that are not far off from being servile.
I have spent the last 26 years working within just such a system and am convinced that this evaluation-oriented approach is precisely what sets my employer, IE Business School, apart from the competition. The students at IE Business School play a key role in assessing professors’ performance: a rating of below four on a scale of five means that your area chair might well call you in for a meeting to discuss the problem; it can also be a factor in deciding how much you get for an in-company session, or may lead to having to seat in another professor’s course, or even another professor taking over your course in case the problems persist. IE Business School works on a matrix basis, with program directors asking area chairs to provide professors based above all on the evaluation they have received from their students. Professors who have consistently been given low ratings quite simply have no place here. As we like to say: “IE, not your average university”.
Obviously, we have been doing this for a lot longer than the companies mentioned above, most of which are less than five years old. The problems mentioned in the article do exist, but I think they can be fixed relatively easily. However, it should be said that an underperforming business school professor is not quite so easily replaced as a driver, which is of course not meant as any disrespect to people who make their living in this way.
The way to address these issues is through analysis of the system in question. For example, it’s not the same to score four out of five from a group that is really happy overall, compared to one in which the members are all extremely critical. Class dynamics depend on many factors, among them the type of students, the combination of professors, or simply the way the course develops over the academic years: students that might have started out reasonably tolerant can become more critical as the months roll by. This is why normalization is so important when it comes to making decisions: if we are all informed of our score as soon as it is calculated, without it undergoing any “creative” analysis, we should also have measures that allow us to appreciate just what the score means in the bigger context. Being given four in a group in which your fellow professors have an average higher score is not the same as a four in a group in which your average score is significantly higher than the others, and this affects, among other things, your morale.
Feedback at IE Business School is detailed to the limit. Every comment written by an student goes straight to the professor, unedited. Sometimes this can be difficult for the area chair, and my own experience in the job has occasionally put me in some delicate situations when having to discuss a particular professor’s problems, but overall, the system works, and provides a good way to help professors make the necessary adjustments and changes required.
Managing such systems is essential to avoid one of the main problems the article in The Verge pointed out: that of creating a servile relationship between say, a driver and Uber. As mentioned, comparisons between a professor at a business school and said Uber driver are not that helpful, but there are also cases of academics who compensate for their deficiencies in the classroom by being very popular with students. Then there is the question of how detailed an assessment is being provided of the service. Again, a student at a business school is probably quite prepared to provide a detailed evaluation of their professor: somebody in the back of a taxi will not provide anything like that level of analysis. Any attempt to get anything more detailed than a tap in a ranking from one to five stars will probably fail.
The key to success is how the system is managed: evaluation is not the goal, but simply a means to an end, and processing them means taking into account a number of factors. Success comes as a result of years of practice and experience, as well as consistency and in having a clear idea of the goals in mind. My business school’s approach may have its failings, but I do know that it works hard to identify and isolate the main problems, and also that professors and students alike are aware of its importance.
The point here is that it is essential to remember that what is taking place is the use of a relatively simple mechanism to measure a complex process. But this doesn’t have to mean that the way the information gathered is analyzed is also simple. In the final analysis, it’s not the same to be given five stars from a client who always gives five stars as it is from a client who usually gives two or three. Both can be subject to normalization, and the results should also taken with a pinch of salt. Without losing objectivity or rigor as a way of providing feedback, the system has to be able to correct any deviations beyond service providers’ control, and which can be highly discouraging when included in an evaluation. But the fundamental question remains: as an evaluation system, it is infinitely better than any of its alternatives, and when properly managed provides better quality and satisfaction.
We live in a world in which these types of evaluation systems are increasingly being used: after passing through security or when leaving a shop, we now find large panels with easy-to-understand instructions inviting us to evaluate our experience; and we do, but without putting too much thought into it, continuing on our way. And we are going to be asked to do this more and more often. If your company isn’t doing so already, then you need to start thinking if you need to and what you would gain from doing so. But as with so many other similar questions, the system is neither the problem or the answer: the devil is in the details.
(En español, aquí)
