Context is the King — The Art and Science of Content Discovery
— Exploring the power of Transformers, GPT and Hybrid Search engines to revolutionise content discovery
With the digital revolution in full swing, navigating the vast amounts of content across the numerous apps we use on a daily basis has become a daunting task for individuals. While there are a variety of advanced technologies and tools available for product developers to design solutions that will have an impact on users, the constantly changing demands and behaviours of users are driving product owners to continually evolve their systems and adapt as necessary. Advent of AI enabled products like ChatGPT has made it more crucial for products to adopt and ride the surf!
The service industry operates on the principle of “customer is the king,” while the media and entertainment industries emphasize that “content is the king.” However, it is more accurate to say that every app user’s actions are driven by their context. That’s right — “context is the king.” As every app captures every click and action, there is already a substantial trail of data that can be used to infer context and provide value to the user.
In today’s world of diminishing attention spans, understanding the context of users has become more critical than ever before, particularly for content products and platforms. The concept of “content discovery” can take many forms in a software product, including search, recommendations, and external events that trigger notifications. Rather than treating each of these as separate use cases and applying different strategies, I draw inspiration from insightful book, “AI Powered Search.” The authors argue that search and recommendation are not mutually exclusive but fall on different ends of the personalisation spectrum. I find myself in full agreement with the author’s perspective.
While the objective of this blog is not to repeat myself, rather recommend every product owner to check the book and contextualise as applicable.
In this post I am planning to share couple of aspects of content discovery that we have achieved by harnessing the advanced NLP frameworks like GPT, Transformers and Semantic Search augmented with classic search engines. Structured data analytics engines are employed to enrich the context for both user as well as content. As a product developer myself, I’d attempt to establish the possible business context in this post before I deep dive into technical details for each in separate posts.
Types of product discovery
In real life context, we could organise the product discovery into following categories:
a. Active /Solution centric :
Active content discovery involves users searching for solutions to their specific needs, driven by their explicit demands.
- Searching for products on e-commerce websites
- Finding a restaurant
- Looking for a file on a computer or cloud drive
- Seeking help with specific issue fixes or sample code
- Learning resources on a specific topic on online learning platforms
In each case, the user’s objectives are typically specific and short-term, and once the purpose is achieved, the need disappears. In this case, the user is at the center stage and is assisted by the app in efficiently navigating through hundreds of products/content in the system to bring relevant information to their attention.
Approaches for Implementation:
- Provide a streamlined search experience that helps users quickly find what they’re looking for
- Create search experience that are tailored to a user’s specific needs and interests
- Offer targeted assistance to users who are experiencing specific issues or problems
b. Passive / Serendipitous discovery:
This refers to a more passive approach where the user’s objective may not be well defined or may have some level of obscurity. The app’s goal in this case is to serve the user’s short or long-term interests and surprise them with unexpected discoveries.
- Personalised movie recommendations on streaming apps
- Relevant news feed on online news reader apps
- Personalised feeds on social media apps
- Music stream discovery on music apps
- Fitness regime recommendations and online diet planner/tracker
- Course recommendations on skill development apps like Coursera or Duolingo
In these scenarios, the app strives to anticipate the user’s needs and provide compelling content to ensure long-term engagement.
Approaches for Implementation:
- Foster a sense of discovery and exploration that keeps users engaged with the platform
- Make intelligent content recommendations that are unexpected but relevant to the user
- Encourage users to try new things and expand their interests by offering diverse and varied content options
c. Discovery lead by External events:
This type of discovery is based on external events, such as promotional campaigns or online advertisements, which recommend items or content to users.
- Upcoming shows on BookMyShow or other entertainment apps
- Travel/holiday offers on travel booking apps
- Product/content suggestions based on promotions and discounts advertised online
- Discounted pricing and offers for items in the wish list
The primary objective of this type of discovery is to increase sales or services by strategically pushing products or content to the targeted audience.
Approaches for Implementation:
- Leverage promotional campaigns and discounts to drive sales and increase revenue
- Use targeted advertising and email marketing to reach specific segments of the audience
- Partner with other businesses or influencers to promote products and services to their audiences.
Although these three paradigms serve a broad range of objectives and employ distinct implementation strategies, their ultimate goal is to present the most relevant content to the user at the right time, whether the need is explicitly stated or implicitly inferred. Moreover, this must be done efficiently while navigating through numerous options. Context lies at the core of the intersection of these three use cases.
From my personal experience, I have observed products at various maturity levels. Focusing on Search, Recommendations, and Campaign/Catalog Lead Discovery as three separate use cases has limited benefits compared to considering content discovery as a holistic approach. Each of these use cases can complement one another since they share a common “context” to operate upon.
To achieve the common goal of “content discovery”, it is best for product teams to collaborate and equip each other with the necessary context that mutually can leverage. The engineering teams, NLP teams, and analytics teams should work together to tackle the shared objective - content discovery. In my upcoming posts (here), I explore the ideas, design approaches along with technical aspects and demonstrate how context can be utilised in each of these scenarios.
I am eager to hear your ideas and approaches based on your own experiences. Please feel free to share them in the comments below. I am looking to reflect on them and continue to evolve my understanding.
Citations:
AI Powered Search : https://www.manning.com/books/ai-powered-search