Insights from Netflix: An Interview with Ding Tong, Senior Research Scientist
Ding Tong presented a session at our third annual RecSys event and we asked Ding to share some additional insights. To check out Ding’s session in addition to the interview below, visit the event replay.
Question: As a Senior Research Scientist at Netflix, what do you and your team focus on?
Ding Tong: Our team is dedicated to advancing recommender systems at Netflix through machine learning. We research, develop, and optimize recommendation algorithms, which span personalizing rankings, modules, and pages for users, as well as personalizing how we display them. We handle the end-to-end process, from algorithm design to implementation, evaluation, and deployment. I led the area of cold-starting new modules with the page algorithms as well as the nomination stage of the page.
Question: What is a recent success for you and your team at Netflix?
Ding Tong: While I can’t share confidential details, I can highlight some recent achievements that we are proud to share publicly. Our team had two pieces of work accepted at RecSys 2023’s industrial track, titled Reward Innovation for Long-term Member Satisfaction and Navigating the Feedback Loop in Recommender Systems: Insights and Strategies from Industry Practice.
Additionally, our machine learning platform team had a research paper accepted by RecSys 2023 titled InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models. Our colleagues in the search area recently published a blog post titled Lessons Learnt From Consolidating ML Models in a Large Scale Recommendation System.
Question: Your experience with predictive systems, recommenders, and personalization is extensive, why the focus on recommender systems for the past 6+ years?
Ding Tong: I’ve been captivated by the intricate interplay of user preferences and their interactions with the contents we present, which has been a consistent theme in my academic background and industry career journey. I found the field of recommendation systems to be rich in both research and business challenges, continually fueling my passion for innovation. Also, the recommendation area drives immense business impact and plays a pivotal role for product success, making it an exciting domain for impact-driven professionals like myself.
Further, I believe that working on recommendations aligns with the future paradigm of human-AI interaction. It’s about creating personalized experiences that seamlessly integrate AI into our daily lives and empower efficient decision-making, which in turn makes better AI. This intersection of technology and human experience is both compelling and forward-looking, and it motivates me to contribute to shaping this future.
Question: What led to you to work on end-to-end recommendations for the Netflix homepage and catalog pages?
Ding Tong: My deep passion for recommender systems, Netflix unique and fascinating challenges of recommender systems, coupled with the opportunity to join the Netflix “Dream Team” — extraordinary colleagues that highly effective working together.
Question: How does your work relate to the overall business?
Ding Tong: The research and development of recommender systems have driven tremendous business impact through my career journey. This website highlights a few significant improvements we made at Netflix.
Question: Why are understanding and measuring impact of industrial large scale recommender systems interesting problems to solve?
Ding Tong: Understanding and measuring the impact of industrial large-scale recommender systems are pivotal to their success. As the quote says, ‘If you can’t measure it, you can’t improve it’.
However, these problems come with fascinating challenges. The online and offline evaluation gaps, for instance, challenge our ability to measure real-world user experiences accurately. Additionally, the feedback loop problem in industrial recommender systems, which we explore further in the NVIDIA’s RecSys at Work: Best Practices and Insights event, presents another vital challenge.
Question: How can practitioners address the challenge of measuring impact of large scale recommender systems?
Ding Tong: Addressing this challenge is a dynamic and evolving research domain, and practitioners can draw valuable insights from both academic and industry literature. For industrial practitioners, adopting the ‘build -> measure >- learn -> adapt’ cycle can be immensely helpful. It allows us to tailor our measurement approaches to the unique challenges posed by each product. This iterative process enables us to innovate and refine our metrics and evaluations continually.
Question: How about performance? Any tips for optimizing performance of recommender systems?
Ding Tong: This is one of my favorite topics, and I can go on and on for days because it’s a domain filled with both challenges and opportunities :) One of the key strategies I think is alignment: aligning with business objectives is paramount, particularly for those recommender systems aiming for long-term user satisfaction. Recommendations that resonate with users not only drive immediate engagement but also foster loyalty over time. This involves the delicate task of navigating through (often) delayed feedback from users, achieving a balance between accuracy and diversity, as well as short-term and long-term engagements.
Moreover, as we discussed earlier, the use of reliable offline evaluations can be a game-changer to accelerate the optimization speed. These evaluations provide us with invaluable insights, acting as a guiding compass in our development cycles to optimize for performance.
Question: Given your prior experience as a tech team lead, what advice would you provide for a new team lead trying to build a personalization product roadmap from 0 to 1?
Ding Tong: 1. Build a personalization MVP (Minimum Viable Product) as quick as possible, which helps you gather feedback and iterate faster. It’s perfectly fine to make mistakes and correct them. These mistakes are not failures but rather valuable opportunities for learning, growth, and improvement. 2. Invest early on algorithm and architecture explorations. Take the time to thoroughly evaluate available options for algorithms and system architectures. Careful consideration and investment in the ML system design at the early stage can save a lot of time and effort down the road.
Question: What did you cover at NVIDIA’s RecSys at Work: Best Practices and Insights event?
Ding Tong: I talked about our paper recently accepted by RecSys 2023 industry track Navigating the Feedback Loop in Recommender Systems: Insights and Strategies from Industry Practice. Understanding and measuring the impact of feedback loops in industrial recommender systems is challenging, leading to the underestimation of their deterioration. In this study, we define open and closed feedback loops and investigate the unique reasons behind the emergence of feedback loops in the industry, drawing from real-world examples that have received limited attention in prior research. We highlight the measurement challenges associated with capturing the full impact of feedback loops using traditional online A/B tests. To address this, we propose the use of offline evaluation frameworks as surrogates for long-term feedback loop bias, supported by a practical simulation system using real data. Our findings provide valuable insights for optimizing the performance of recommender systems operating under feedback loop conditions.
