At Wattpad search is used millions of times a day by people looking to discover stories they want to read. We use Elasticsearch to power these searches over tens of millions of documents. One of the techniques we use allows the search system to re-rank documents over time to better suit the needs of people using the system. Let’s start at the beginning and see why this technique works and a simple implementation of how it can be used in your search system.
In an alternative reality, if you stole (don’t actually do this) the algorithm used by the very best search engine in the world and decided to start your own search engine with your newly found secrets surely your search engine would perform just as well right? Of course, the answer to this question is no, but if the algorithms used are exactly the same between the two systems then what has changed?
All a search engine is doing is comparing documents against a handful of words and trying to find the documents that best match these words. Our ill-gotten algorithm is capable of indexing documents and contains the logic to compare our query to our index of documents. Sure, we have to spend the time indexing the pages but that is just part of the cost of running a search engine and the results we show to users should be the same after that is done.
The clue is in what a search engine does…users type in queries and then they click on the results you show them. How many users does the very best search engine in the world have? How many users does your brand new search system have? Probably not nearly as many. The very best search engine in the world isn’t ignoring how users interact with the system and your new system doesn’t have those users so the results aren’t as good. (And because the results aren’t very good getting those users is going to be very difficult.)
Using the power of the crowd is crucial in developing a good search system. One simple way to use this power is to do A/B testing: show some users results from your current algorithm and some users your new algorithm. This allows you to slowly build and improve your algorithm in response to the whims of the crowd (assuming we actually haven’t taken part in corporate espionage).
You can also use the power of the crowd to assess the quality of your search results. In particular you can see which queries are performing well and which queries are not. If a user types a search term in but skips the first few items in the results and clicks on the tenth item there may be nothing wrong with the order of your search results. If, however, most people are doing the same thing then maybe the results are bad. This allows you to narrow down on the queries with bad results, tune your search algorithm and run another round of A/B testing. Again this is using the power of the crowd to focus on which improvements are most needed.
These ideas are good, unfortunately both figuring out which results are poor and testing a possible solution takes times and some intuition about what changes might work. (Also, I’m lazy.) It would be much better if we could simply ask the search system to watch how users interact with it and have it reorder results when needed. Below is a very simple way of doing this using Elasticsearch but the ideas can be implemented in any search system.
People often use Wattpad to search for stories so it makes sense that we should have a stories index to allow this:
{
"stories": {
"mappings": {
"story": {
"properties": {
"description": {
"type": "string"
},
"title": {
"type": "string"
}
}
}
}
}
}And when, for example somebody searches for “The Necromancer” we could query this index with something like:
{
"query": {
"bool": {
"must": {
"match": {
"title": {
"query": "The Necromancer"
}
}
},
"should": [
{
"match": {
"description": {
"query": "The Necromancer"
}
}
}
]
}
}
}This will match all documents that contain our desired term. In order to use the power of the crowd to make our search system better we need to do a couple things. First of all we need to keep track of when somebody searches, what they search for, and what they click on. For example we could have a series of events for a particular person:
action,datetime,query,document,position
query,2016-06-09 11:04:01,the necromancer,,
click,2016-06-09 11:04:23,the necromancer,56534248,10So we know that after searching a particular document was clicked on. We notice that this document is the 10th item in the results. If one person clicks on it the search results might not be bad however if most people are skipping the first 9 results to click on it we want to the search system to pick up on this and boost this document in the ranking.
In order to do this we need to make three changes. First of all we need to add a queries field to our index mapping:
{
"stories": {
"mappings": {
"story": {
"properties": {
"description": {
"type": "string"
},
"title": {
"type": "string"
},
"queries": {
"type": "string",
"norms": {
"enabled": false
}
}
}
}
}
}
}This new field will keep track of which queries were used by people to end up at this document, in order to take advantage of this and modify our results we should modify our query:
{
"query": {
"bool": {
"must": {
"match": {
"title": {
"query": "mystery"
}
}
},
"should": [
{
"match": {
"description": {
"query": "mystery"
}
}
},
{
"match": {
"queries": {
"query": "mystery"
}
}
}
]
}
}
}Note that we might want to adjust the weights of these fields and not give this new field the same weight as every other field.
Finally we need to populate this new field. The way to do this is once every so often (hourly, daily, weekly) go through your query logs and, for each document, get a list of queries that people used to arrive at this document and then update your index with this list.
This technique will get the results users want quicker and will help pinpoint the correct results to show for vague or ambiguous queries. It is also helpful to not only look at the queries that people used to arrive at the document but also previous queries people tried to use that didn’t give good results within the same session, this helps people finds documents that don’t necessarily contain terms used in the document.
Using this technique we have found a significant improvement in the quality of our search results. Over the course of a week when we compared these two algorithms we saw a increase of 1.7% in the amount of time people spent reading stories on Wattpad and a 1.4% increase in the DCG score across all of our search results using this technique. In another test we saw an increase in precision at rank 10 of 5%.
Originally published at engineering.wattpad.com.
