Using Clicks In Ranking
A green curry dish is only as good as the paste that one uses. The secret to a great paste lies in the core ingredients such as green chilies, coriander, lemon grass and kaffir lime leaves. Equally as important is having the right quantity of each ingredient. I occasionally make the paste from scratch at home when I have the time. In an era where convenience is rated highly, naturally I was asked why I insist on making the paste myself. I was told that even certain restaurants with paying customers rely on pre-made paste. My answer to the question is control.
Making the paste myself allows me to adjust the quantity of the different ingredients and tweak the taste to match the individuals’ preferences. The benefit of this is reflected in consumer satisfaction, paying or not.
How is green curry related to clicks and ranking?
The ranking of results in search is a lot like green curry paste. Both have different ingredients that go into them to ensure that the things we present to users are top notch and most likely to their liking. In results ranking, the core ingredients include topicality, novelty and authority. These different factors, if combined properly, will ensure that the most relevant jobs are presented to the users first, for a lot of the cases. This control is important to a search master, the same way making the green curry paste from scratch is to a chef who is serious about satisfied consumers.
If we liken the search results that we present to users to a green curry dish, then the clicks that the results attract are like the implicit feedback that you get once your dish has been tasted.
For instance, you can look at the amount of leftovers or their desire to come back for more as signs of approval of your green curry. However, the thing with implicit feedback is that it is always tainted with biases. As an example, even if your dish was really spot-on for a person, the fact that they have eaten prior to tasting your dish may have influenced whether they actually finished your dish or not. Similarly, the mere fact that you have cooked the green curry dish with a cut of meat that the person does not like may influence whether they finish it or not, which has got nothing to do with how good your paste was.
Biases in clicks
In the context of clicks in search, there are two main forms of biases, namely, position bias and presentation bias. What this means is the implicit feedback that is clickthrough rate, while contains elements of relevance as perceived by the users, is heavily influenced by a range of biases.
“Our findings show that clicks are measurably biased by attractive titles inflating perceived relevance.” [1]
Regarding position bias, studies show that users are almost always more likely to click on results positioned at the top. The trust in a search engine plays a big role in this behaviour. What this means is that, there is very little to no change in the way users click even after the most relevant results at the top are swapped out and replaced with less relevant results from lower positions.
“just putting a search result high on the page tends to get it more clicks even if that search result is less relevant than ones below it.” [2]
Presentation bias, on the other hand, affects the likelihood of user clicks depending on how the individual documents are summarised and presented to the users. For example, it has been observed that the quality of snippets that search engines generate to represent the individual documents on the result pages has a big influence on the user’s decision to click or not. The bolding of keyword matches also disproportionately make an otherwise less relevant result more attractive.
Should clicks be used in ranking?
Search engine companies have been using clicks for evaluating the effectiveness of search for some time now. As an example, there has been work done on using large volume of click data combined with some explicit relevance judgments to build prediction models. These models are used to make educated guesses about what the relevance might be for documents that have not been judged. As acquiring explicit judgments can be costly, this approach is useful for comparing the effectiveness of ranking as we incorporate new or tweak existing factors.
The use of clicks as another ingredient into ranking has also been reported. This use of clicks, however, is not a straightforward thing and for good reasons. The control that a search master or a chef has erodes as the ranking relies more and more on past clicks. If the clicks are based off a poor ranking function to begin with, than the inclusion of the clicks into further ranking perpetuates the sub-optimal condition.
The situation above is like adding a green curry paste that was initially too spicy and salty into subsequent paste making. It just makes the chef’s task of tweaking the taste unnecessarily difficult to impossible. In this scenario, the chef may now have to compensate with more sugar and coconut milk just to counter the saltiness and the heat. Well, this of course assumes that the chef knows what went into the previous paste and the implicit feedback tells the chef that the dish was not overly great. What if you had no clue want went into the original green curry paste? It would not be too hard to extend this reasoning to the use of clicks in ranking.
It is important to call out that not all is lost. In scenarios where the basic factors such as topicality do not quite work, clicks can be very useful. For example, if a query is vague, past clicks can be used to provider further context to the search. Past clicks can also be a great factor to arbitrate between two or more documents that come up equal based on the core factors such as topicality, novelty and authority. If you subscribe to my reasoning so far about the need for care when using clicks as a factor for ranking, then the sole reliance on past clicks for ranking is an amateurish move.
Conclusion
While relying on pre-made green curry paste seems OK for your neighbourhood takeaway, chefs that require control in their cooking need to make it from the base ingredients. Search is no different.
Search engines that are continuously striving for better experience are either using or have started experimenting with click data. Past clicks are a valuable resource for evaluating effectiveness of search and as another signal in ranking. The use of clicks as a ranking factor requires thinking through the circumstances in which it might be of help. The blind use of clicks, especially as the sole ranking factor, is not recommended.
