Evaluating Site Search Relevance: What, Why, and (One Way) How

Photo by Scott Graham on Unsplash

Evaluating the relevance of results for your top internal site search terms may sound intimidating. (It’s a mouthful to say. I’ll grant you that.) But the practice doesn’t have to be a headache.

If you’re willing to take a granular, term-by-term, hands-on approach, this evaluation method offers a low-budget, generally low-tech way to identify high-impact problems. The process has also given me more insight into users’ experiences and increased my familiarity with my clients’ products and web offerings.

(Note: This method is oriented towards the obstacles and goals of e-commerce websites. That’s the primary context in which I’ve used it. But it is applicable to other types of websites as well.)

What do I mean by site search relevance evaluation?

A site search relevance evaluation is the process of assessing whether your site’s search engine is showing users the items that they expect, want, and need to see.

Think of the search functionality on your website or websites you use. Picture that little search box, maybe in the upper corner, often with a magnifying glass button by it. A site search relevance evaluation focuses on what happens when users — perhaps optimistically — interact with that little box.

• What are users searching for?
• What comes up for them when they search for various terms?
• Most importantly, how relevant are those search results? In other words, how well do the search results relate to the user’s query?

Let’s take the example of an online home goods store. If someone searches for “pink shower curtain,” does the results page show all the pink shower curtains that the store offers? If so, the results are very relevant. Now let’s imagine that the search results page displays a variety of curtains — pink window curtains along with pink shower curtains. In other words, the results are much less relevant on average, possibly causing annoyance among users.

Why do this kind of site search relevance evaluation?

You might be thinking, “Couldn’t I just look at the metrics? Or buy a smarter search add-on to show more relevant results automatically?” Well, of course you could! Those options aren’t mutually exclusive with evaluating site search relevance.

Here are a few reasons to do a site search evaluation term by term:

• Human understanding can often arrive at a more nuanced assessment of relevance. When a person goes through and looks at search results, they can often see subtle distinctions that an algorithm might’ve overlooked. Relevance is both an art and a science. Especially when you’re dealing with a highly complex and specialized group of products.
• You can learn more about your client’s products — or you can teach your consultant to think like you do. I often work closely with clients to understand their product offerings so that I can judge the relevance of search results. I’ll ask them to go through some tricky searches with me and see if they agree with my logic to separate the relevant from the irrelevance. The process has enabled me to learn a lot and become more valuable to my clients. The better you understand your client’s business, the more you can help them on any project, not just site search issues.
• Seeing is believing; you get to wince on behalf of the customer. Look, you know how the navigation is set up on your (or your client’s) website. You know how to find what you need on the website. Consider yourself fluent in a language that site visitors are still struggling with. It can be an eye-opening experience to use the site search functionality in the way users do, with the terms they search, and see what they see. Running searches on a website builds empathy for the user; the exercise gives you a more concrete, urgent grasp of a common task and how it makes users feel.
• You can quickly review a wide swath of top terms and results and start thinking about specific refinements as well as broader next steps. By contrast, you’d probably have to conduct an absurd amount of UX testing sessions to cover most of the top search results.

A quick caveat: this kind of evaluation is not a substitute for moderated UX testing. If you can run sessions and watch prospective customers search for items on your website — or hire a consultant to do that — I definitely recommend it. You will learn things that you couldn’t anticipate by doing this relevance analysis alone. It’s a “better together” situation. A site search relevance evaluation and UX testing can complement each other beautifully.

How to do this kind of evaluation?

It depends on the website, but I generally break the process down into 4 main steps.

1. Get a list of the website’s most frequently searched terms. This part will depend on the website platform that you’re using. If it’s a major CMS or analytics platform, there should be a simple way to export a list of the most commonly searched terms within a given time frame. Put them all in the first column of a spreadsheet. Leave the second column blank.
2. Go through each of the top keywords. Ask yourself, “What is the user intent here? What does the user probably want or expect to see among the company’s offerings?” This is both the most challenging and the most fun part. You get to put yourself in the user’s shoes and simulate their thought process. In the third column of your spreadsheet, take a few notes on what you think would be relevant results for this search.
3. Rate the results based on how they align with the user expectations. Take notes on what you think users should see. Search each term and compare the top results with what you think would be relevant. I use a very simple scale to rate the results — Poor, Fair, and Good — and put the rating in the second column of my spreadsheet. If all or almost all of the search results are relevant, I rate the relevance Good. If the majority of the search results are not relevant, I rate them Poor. If it’s somewhere in between, I rate it Fair. This basic rating plan can be helpful for making charts and slicing and dicing the data. What patterns emerge if I apply a filter in the spreadsheet and look only at the Poor relevance results? What percentage of the top 50, 100, or 200 terms are Good, Fair, or Poor? The 3-level system also enables me to prioritize updates to the site search results, which I may be passing on to another team.
4. Adjust your site search system to return more relevant results. Yes, that’s maddeningly vague. But how you need to update your site search will vary based on the relevance issues you observe and how the in-site engine is controlled. How can the search engine be tweaked and “taught” to do better? Can you customize the results for a given term? Do you need to set up any redirects? Are there platform add-ons that you should evaluate and recommend to the client? Any or all of the above might be the right solution(s) for the situation.

What if the problem isn’t exactly relevance?

As you evaluate the relevance of search results, some overlapping issues may arise. (Trust me, this isn’t a totally bad thing. After all, if you’ve got a cavity in your tooth, you want a dentist to catch it before you need a root canal.) Here are a few patterns and problems related to relevance that point to a more complex situation.

Are you seeing a lot of oddly broad search terms? If you have an online clothing store and tens of thousands of users a month search for something as broad as “women’s clothes” or even “dress,” that’s worth investigating. Your navigation may not be as intuitive as you need it to be. Even if the search results for “dress” are fairly relevant, those results won’t solve a potential problem.

Next step: Look into possible navigation issues. Watch session replays through a service like HotJar or Fullstory. Consider conducting moderated UX testing with a focus on navigation and search.

Are users searching for content that isn’t included in the site search? Many e-commerce websites’ internal search engines only include products. But are your users searching for something other than products? Jared Spool gives the example of a woman who couldn’t find a company’s return policy to illustrate the point that a narrow view of “content” can get you in trouble when it comes to in-site search.

Next step: Tag or index those items, pages, or resources that users are searching for so that they will show up in the site search.

Are people looking for something that you don’t currently offer? You might get hundreds of searches a week for a product you don’t have or a resource you haven’t yet developed. In other words, doing site relevance evaluation could help you identify opportunities for growth.

Next step: Consider filling the gap in what you offer. Or, if that’s not applicable, recommend similar products or resources. “We don’t currently offer {product X}. But we think {product Y} might help.”

As these issues suggest, evaluating site search relevance is only one piece of the puzzle when it comes to the overall picture of site search usability. I recommend reading Baymard Institute’s search-related reports and the Nielsen Norman Group’s report on site search (or this highlights article), which have helped me in the past.