Personalizing Fiverr:
From Machine Learning to User Experience
Personalizing the Fiverr experience is a long journey and we invest a lot in order to know our buyers, understand their problems and tailor their experiences to meet their needs quicker.
In this blog post, I will share our process and the insights gained from personalizing Fiverr’s banner content, showing that personalized content selection increases buyer engagement and better exposes our buyers to the variety of services offered in the marketplace.
To achieve this, we conducted a number of AB tests, each with a dedicated goal of increasing engagement and getting us one step closer to our end goal.
Have we reached that goal? Continue reading and you’ll find out…
Moving Away From One-Size-Fits-All
If you have a static page that you want to personalize, you must be familiar with the problem of content that’s pretty much one-size-fits-all. You know that it’s wrong, but don’t know where to start or how to make it right.
A year ago, we had a very limited number of banners that had been created by the Marketing and Editorial teams and presented to all audiences, equally.
Our hypothesis was that by increasing the amount of content and personalizing the banners’ selection, we could increase the overall engagement with buyers. This would expose them to a greater variety of services and assets within our marketplace, and guide them towards their “next step” of fulfilling their broader needs.
Our Plan Was Simple:
- Content creation — in order to test more content, we needed to have more content.
The Marketing team started working on creating content, including a mix of guides, articles, promotions, inspirational pieces, catalog announcements and, of course, service (Gig) recommendations - Data collection — to personalize the experience, we needed to make sure we collected the right data on both the buyer and the content. Allowing us to perform better analysis and apply data-driven decisions moving forward in our journey
- Personalized algorithm — personalizing the experience, meant we needed a personalized algorithm.
Wait, how can we jump from a one-size-fits-all solution to a fully personalized experience? The answer: in small steps
It is crucial to start simple, and not move into a full-blown, complex algorithm.
So we decided to proceed step by step, and the first step was to deliver more content, using the easiest thing we could use, randomness.
Step 1: Hard-Coded vs. Random
Testing a random selection model is important, because it’s easy to implement and can also serve as a benchmark for any future optimization.
The need for randomness is mainly for collecting more data on all banners, so it will help us to verify the hypothesis that fresh content drives engagement.
We ran an AB test for a period of 14 days, enough to get statistical significance and test our new experience.
The test performed great! Content CTR (click-through-rate) improved by 20%, meaning that it was more interesting, attracting engagement. We also improved our ability to track engagement with the component and identify key features.
For what purpose? Let’s see…
Step 2: Random vs. Reinforcement Learning
We now had more content than before, and it made sense that some content was more engaging than others. The next step was to show more engaging content and less content that underperformed.
To do so, we applied a Reinforcement Learning model — Thompson Sampling — to optimize content CTR, while balancing new content exploration.
In simple words, content that is more engaging will appear more often than less engaging content.
We conducted an AB test, with the Random group performing as our control group and the Variation group getting the Thompson Sampling model for banner selection.
The test results were very good! We saw an additional 6% increase in content CTR, and this model helped us to identify many insights.
Firstly, we were able to analyze buyer features and identify top-performing topics and domains.
Secondly, and most importantly, we learned that there is a huge need for content personalization, as we saw that the chances of generating engagement dramatically decreased as a function of the content visibility frequency.
It’s worth mentioning that another key insight was that not all content banners are the same.
For example, an announcement banner that celebrates a new category’s availability, may improve buyer awareness even without clear engagement (click). While a recommendation banner, which leads to a dedicated recommendation page, must be engaged to deliver value.
This learning led us to define additional engagement KPIs, which I won’t cover in this blog post.
Step 3: Reinforcement Learning vs. Personalized Model
Reinforcement Learning, by definition, exploits some content more than others. This is why it’s still not personal; to deliver a personalized model, we needed to define a list of relevant features that should affect the chances of engagement.
To do so, we trained a DeepFM (Deep Factorization Machine) model that took into account a list of 15 hand-picked features, including:
- Buyer-related features, such as purchasing history and browsing activity
- Content-related features, such as content type and KPI
- Behavior-related features, such as chances for a click in function of visibility frequency
Then we ran an AB test, comparing the Thompson Sampling model as our control group versus our DeepFM model in the variation group:
Happily, we saw an additional 12% improvement in engagement (CTR).
The new model was able to optimize the content selection, personally, simultaneously balancing between exploiting high-performing content and exploring new content.
Achievements So Far
With that, our “zero to one” process reached an important milestone. We were able to see improvement in a few areas:
- increase in multiple business KPIs
- orchestrating buyer traffic to more categories and marketplace assets
- improving the overall experience
- improving content engagement
Having said that, our personalization efforts haven’t ended and we still have a long journey ahead of us.
Our journey so far has been very insightful, allowing us to gather more data and come up with tons of new ideas and hypotheses.
We have decided to boost our content scaling plans, to incorporate better content tagging techniques, and to expand our capabilities to target premium content for a specific audience and segment.
I’ll have more to share on that in my next blog post.
__
This article was created with the help of Fiverr Sellers:
lirazbeno (Illustration) & dudleybh (Proofreading).
Fiverr is hiring, learn more about us here
