Improving job search accuracy through test and learn
White-collar workers will have worked five jobs (including two internships) before they reach the age of 35, as they stay on average 2.8 years in one position according to a 2015 study by Deloitte and Nomination. Therefore finding a job needs to be a simple and easy process!
At Welcome to the Jungle (WTTJ) we are committed to making work an exciting experience for everyone. This involves offering emotional support for work life through our media, making sure we provide high-quality job postings on our platform, and continually improving the accuracy of job search results. With 43% of users’ behaviors revolving around the job search feature on our website, improving the accuracy of results quickly became a priority for our tech team.
Here we will explain how we improved our job search results for both job seekers and hiring companies by exploring functionalities on the Algolia search engine and releasing a new feature to boost the visibility of job postings.
Spotting the issues
We run regular NPS campaigns at WTTJ to measure the global satisfaction of our users. And the results are good: Our NPS score is currently about 50. However, we felt that there was something wrong with the job search experience.
A specific campaign that focused solely on the job search process quickly confirmed this, giving a lower score. A series of user interviews was then conducted and three types of issues emerged regarding the job search feature on our website:
- The results were not always relevant.
- Several filters were missing.
- Job seekers, especially students, were struggling to find the right job titles.
Additionally, we quickly implemented a new KPI to measure the search satisfaction by displaying like and dislike buttons on search results pages. By using them, people can tell us directly what they think of their search experience, allowing us to measure the impact of changes in the experience more frequently.
A new pop-up appears if they click the dislike button:
If they click the like button, they still have the option to share their thoughts:
Now let’s look at how we addressed the issues.
Deep diving into the Algolia search engine
During the last quarter of 2022, the Algolia search engine indexed 5 million searches on the WTTJ website, with 40% of them using filters.
Search engine optimization
To begin with, we conducted 14 AB tests on the Algolia search configuration in order to improve search results, which included removing stop words, optimizing searchable attributes, and adding synonyms. The experience made us question our decisions, such as regarding whether the search should be based on job titles only or on the entire content, and whether linking words should be removed. We also enriched our synonym suggestions for job titles in response to feedback from users as well as data science.
Optimizing the Algolia search engine for our needs required going through a lot of trial and error. We tested a wide variety of configurations before releasing them and discovered that those that work well individually could be inefficient when combined. Having a large number of searches helped us iterate quickly and it usually took us two days of testing to decide whether an optimization was worth it.
Significant results were achieved! By working on synonyms, we were able to increase the click-through rate by 1.2 points and the conversion rate by 1 point.
After optimizing the search engine, we worked on enhancing the user experience, taking advantage of Algolia’s features.
User experience rework
Historically, the search feature on our website had offered users a real-time search experience: Typing keywords into the search bar would update results immediately. But we quickly realized it wasn’t ideal, especially for students who don’t always know what to type.
To inspire and guide users, we switched to Algolia’s auto-complete feature to show similar job titles as well as the number of open roles.
By making just this one change, the click-through rate increased by 2 points.
This wasn’t enough, though. While improving the search experience, we also wanted to address recruiters’ stakes. So last year the tech team decided to work on a new feature to make job postings more visible.
Promoting job postings
In 2022, we decided to introduce a new feature that allows hiring companies to boost their job posts. Nothing new there: Similar features had already been proposed by some of our competitors. After confirming that our customers would be willing to pay for that feature, we conducted several interviews with users and companies to understand what they expected from such a feature.
Identifying expectations
Based on our conversations with recruiters, we concluded that there were two situations where boosting was needed:
- Urgent hirings, such as when someone in a key position leaves the company, or when the person in charge of a crucial event is due to leave the company in the near future.
- Difficult hirings, such as for profiles where there is a shortage of talent.
To make this feature successful, it also had to meet the expectations of job seekers. Job boards are like marketplaces: Demand and supply must be balanced. To target the right audience, we decided to use the Google model rather than some sociodemographic data to reach people based on their intentions.
Taking into consideration both client and job seeker expectations, we decided to market this new feature as an easy-to-use and efficient solution for our clients that gives job seekers visibility of the promoted job postings in search results.
Implementing the solution
A new index containing only boosted job postings was created in Algolia. Whenever a search is performed on the WTTJ website, a multi-query is used to request this index and organic search results. Job seekers will only see boosted jobs that correspond to their searches.
A UX analysis then helped us optimize the positioning of boosted job postings. We used Contentsquare to have a deeper understanding of candidates’ behaviors, not only on our website as a whole, but also on individual pages. Thanks to heat zones, clicking-recurrence areas, and more, we were able to identify potential UX issues and understand how our visitors interact with the content. It turned out that the top of the page had the best visibility. As a result, we display up to 3 boosted job postings at the top of each page, when available. The job postings that follow these are organic.
Last but not least, we wanted to ensure that job seekers understand why boosted jobs appear before more recent ones in our system. Transparency is important to us beyond legal requirements. As a result, we created a “promoted” tag to be fully transparent in our engagement with users.
Using this tag also allows us to get some feedback as users are encouraged to share their thoughts after clicking it.
A 3-step launch to confirm the impact
Our team tested this feature in 3 steps in real life before releasing it. The users’ reactions were hard to predict and we wanted to be careful not to misread them or create a negative experience.
We began by promoting some of our own job postings as phase 1, as well as adding specific tracking actions through our tracking solution Snowplow to have a deeper insight into the impact. The results were impressive: Page views increased by 8 and applications increased by 4. By this point, we knew promoting jobs worked, but we weren’t sure how people would react to other companies’ openings. Because we were using our own job board for our posts, we didn’t want the results to be biased.
So we moved on to phase 2. Some of our customers were invited to take part in the experimentation and try this new feature. These results were conclusive as well.
However, we were still not confident enough to proceed with the release. So, for phase 3, additional tests were carried out on a job vacancy with a lot of competition to see how well the boost works under those circumstances. Once more, a strong impact was confirmed: Page views increased by 4 and applications by 1.6.
Improving the experience for hiring companies
From the hiring companies’ perspective, the process was quite manual at the beginning. To promote a job posting, they would need to manually set a Boolean. This wasn’t ideal, since we wanted companies to be able to manage their boosts themselves. So we worked on creating a product out of the experience.
How does it work? Think of a boost as a parking spot. Once you activate it, you can only place one job in it at a time, and you can change jobs whenever you want (and even leave the spot empty for a while if you want to). We wanted to give recruiters as much latitude as possible to experiment.
We also provide hiring companies with some data to assist them. For example, they can see how many page views were generated during boosted and non-boosted periods. As a result, they have control and can make informed decisions about which jobs they want to boost.
A never-ending job
Because the search for improvements is a never-ending job, every week we track the reasons some job seekers click dislike about their search on WTTJ.
We are currently testing a new feature that allows people to search only in the titles of job postings to restrict the search results and only show very precise matches. In addition to improving the results, this feature has taught us the importance of clarity in wording and design, as we realized this new feature was not being seen or understood correctly when we first released it.
A rework of the mobile and desktop UX was also prioritized to address filtering slowness and mobile UX issues. These were among our top pieces of feedback, as well as a design and tech prerequisite that would enable us to address other important issues, such as missing filters (salary, start date, contract duration, etc.).
After all this work, our dislike/like ratio improved with +4 likes on desktop and +7 likes on mobile. With the most recent improvements, we reduced the dislike reasons “filtering slowness” and “mobile UX issues” by 2.5.
On top of that, we made sure to not lose sight of our strengths along the way! Having a great design sets us apart from the competition and makes our users satisfied. This was something we were not going to compromise on.
Written by Jean-Nicolas Fine, Team Lead Data Analysis and Aurèle Duterme, Product Lead @ WTTJ
Edited by Anne-Laure Civeyrac
Illustration by Myriam Wares
