Introducing New Search Modalities on Mobile
At ClassPass, we’re always looking for ways to make search feel more intuitive to our users. For the last several months, the Search & Discovery team worked on introducing search modalities in our mobile apps based on user research. We introduced phrase-based search (i.e., search by any arbitrary text), multi-location search, and many other improvements through a series of experiments and feature flags via Optimizely to ensure the new features didn’t harm core metrics. We chose to develop these features using milestones to save upfront engineering investment in case the results from the first milestone were subpar.
Milestone One: Implicit Full Text Search
The first milestone was to broaden the results we get back for existing queries. We started at the Opensearch level by adding text fields containing venue (e.g., fitness studios) and class names and descriptions to our schedule and class indexes. In tandem, we created analyzers that direct Opensearch’s tokenization and search strategy on these new search fields. Now we were able to perform queries that match phrases on our text fields, whereas we were previously limited to filtering on our internal tag IDs.
We wanted to start testing our new capabilities in a way that wouldn’t disrupt the existing user experience. The solution was to execute a new full-text search alongside our existing tag filter search — an implicit full-text search. The user would search as normal, but when a tag was selected as their search term, we would also execute a full-text search on the backend and the results would be combined with the tag filter results.
After 7 days of running an A/B experiment, we found that the variant (i.e., additional text fields indexed) had too many inaccurate results. We decided to remove class descriptions from the index because they were often the longest body of text and sometimes contained words that were irrelevant to the tag searched. We re-ran the experiment with the modified variant and found that our metrics–conversion from search to reservation (CVR) and click through from search to a venue or schedule page (CTR)–were now trending positive, giving us the green light to expand our capabilities to milestone two.
Milestone Two: Phrase-based Search
Once we’ve expanded the backend’s capability to match more terms, we felt confident building a brand new mode of searching on mobile: phrase-based search. Previously, when a user starts typing in the search field, we would return a list of suggestions based on our internal tag hierarchy (e.g., yoga, pilates).
This worked for obvious genres like yoga but it wasn’t very helpful when it didn’t map to something we already had a tag for. For example, searching for balayage, a type of hair highlighting treatment, used to yield no autocomplete suggestions and merely a few places that had “Balayage” in their name.
It’s not that we don’t have balayage appointments on our platform, it’s that our own tagging system was too restrictive. This limited inventory discoverability and wasn’t an intuitive search experience. We therefore introduced a new row on the autocomplete screen that lets the user search for any arbitrary word or phrase even if we return no autocomplete results.
When a user taps on that row, it will execute a search and return results that contain that word or phrase in various places (e.g., class name).
What if the inputted term returns no results? That’s likely to happen and is one of the reasons we had our previous implementation in place. However, we addressed this by adding a fallback feature, whereby we map ahead of time common words and phrases to underlying tags in our system in case we get no results back for the user’s original search term. This often happens with misspellings.
This allows us to have the flexibility of searching by any arbitrary text (and returning results in situations where we may not have prior) while mitigating the increase in zero-result states.
Milestone Three: Multi-location Search
In addition to phrase-based search, another frequently requested search function was the ability to view all the locations of a particular brand or studio chain at once. Let’s use Barry’s in New York City for example. Previously, our UI would only display a list of Barry locations and users would have to tap into each one to view class times. This proved tedious and cumbersome and we knew we could do better.
We made search smarter by detecting when a user searches by a brand like Barry’s and displaying a new “See all locations” row. Tapping this will take you to search results filtered by just that brand. Now you can browse availability in a consolidated list and view on our map!
Experimentation Setup and Results
These features were gated behind Optimizely feature flags and experiments to control rollout and to ensure we didn’t harm our existing metrics. We used Optimizely’s Full Stack framework and mobile SDKs to integrate their platform with our code. The first experiment for milestone one was simply an A/B test to ensure our change to leverage additional metadata fared better than the control. In this stage, the key metrics we looked at were CVR, CTR, and recall (ability to retrieve more relevant results from all relevant results). Recall was measured by manually checking for relevancy and results volume using a sample set of frequent queries and calculating the mean average precision @ position K (Map@K).
After validating our new metadata hypothesis through improvements across CVR, CTR and recall, we ran an experiment for user-facing changes in milestones two and three that split users into four groups based on what metadata was used–control, explicit_class, explicit_venue, and explicit_class_venue. At this stage since the user experience was noticeably changing, we added usability scores and user research for qualitative feedback metrics. We found that explicit_class, the variant that only included class name metadata, performed the best in terms of CVR, CTR, and recall. However, the usability scores and user research sessions highlighted improvements needed in our multi-location and fallback search experience. For example, when users in the explicit_class variant were searching for ‘Barrys’ or ‘Crunch’ and pressing enter, we were taking them to a fallback search for the nearest tag-based genre ‘Barre’ and ‘Core’ instead of understanding their intent to look for those business locations.
After making tweaks to ensure user intent was being better handled (‘Barrys’ showed ‘Barry’s’ locations and ‘Crunch’ showed ‘Crunch Gym’ locations) we heard from users through user research sessions that the multi-location experience better met their expectations. Once we felt confident that we had addressed the major concerns, we fully rolled out the feature set so all of our mobile users could experience the new and improved search experience.
Acknowledgements
This work wouldn’t have been possible without close collaboration between backend engineers (Bree and Rachel), Android engineers (Rich and Ridwan), Madhu our product manager, Jenn our engineering manager, Josh for QA, and George our designer. We’re thrilled with the results and testimonials so far and I’m excited to see what further improvements to search we can make in the new year!
