An Intern’s Machine Learning and Generative AI Journey at foodpanda

November 2022 marked both the start of my internship with foodpanda’s regional data team, and interestingly, the mainstream adoption of LLMs (large language models). Here’s my experience contributing to two distinct AI projects over the past 8 months!

Automate the Boring Stuff with ML

Machine learning is already commonplace in foodpanda and its parent company DeliveryHero to automate processes like deciding pricing strategies, cuisine classification, and content personalization. These ideas allow us to make our business less capital-intensive and free up resources for other areas of interest.

I worked with a team to develop an XGBoost model to identify abnormal food delivery claims. Our hypothesis includes the assumption that the type of abnormal claims will change over time; however, it’s unclear how that would influence model degradation/ drift, and what type of re-training schedule is most appropriate. My team had to determine our model’s retraining frequency before bringing it to production. Since we considered the latest trained model the most accurate, our training frequency should be slightly more frequent than the typical time it takes for a model to degrade, such that its predictions (for a common dataset) begin to deviate from the latest model.

Sample degradation of 3 identical models with increasing training data age; Model 1 > Model 2 > Model 3

To that end, I wrote a script to do the following:

1. Train a set of models on customer and order features of varying ages (e.g. X days, X weeks, X months)
2. Execute an SQL query to get the order features we’ll test the various models on from BigQuery
3. Run payload tests on a set of models and save each model’s prediction in a separate JSON file
4. Plot the predictions as histograms
5. Create a GIF based on the histograms to visualize the model degradation across time

We recommended a retraining frequency to our team’s ML engineer based on the period where we observed the largest deviation of predictions between the benchmark and outdated model. If the model’s retraining frequency needs updating after major package releases, our engineer could re-run the script before re-deploying the model.

Our XGBoost model automates part of the regional operations’ process of detecting abnormal orders by combining machine learning recommendations with customer service agents’ manual assessments. As a result, we have reduced our reliance on human labor and the overall number of fraud cases that were losing money for our company.

We can make further improvements including:

1. Zooming in on ambiguous risk score thresholds (after optimizing the model) and investigating the model drift for those thresholds
2. Scheduling a task that runs the script regularly and automatically makes a recommendation on an appropriate retraining schedule
3. Increasing the number of time intervals plotted to deduce training frequency even more accurately

Innovation with Generative AI

Evaluating the value of an idea can be tricky, but I think the best ideas can spin off new innovations. The spike in interest in Generative AI coincided with my move to a new team a few months into my internship, letting me build new data POCs for foodpanda. I began diving into the basics of LLMs with the aim of creating an internal chatbot for business stakeholders to create SQL queries from natural language to retrieve data answers on BigQuery.

An overview of the approach I took to prompt engineering:

Modifying user prompts to input into our GPT model

Sample SQL query created with the GPT model:

Initial POC: The question was fed into the GPT model and the query was returned (in terminal)

Seeing how our GPT model can respond with largely accurate queries in this initial POC motivated me to test our microservice on GCP. Our next goal is to explore serving the chatbot on a larger scale by integrating it into our Slack channels with the help of the infrastructure and engineering squads.

Our GPT-powered SQL generation tool could bridge the gap between technical and business stakeholders: business users can get answers to their data-related questions more quickly through a chatbot, while analysts can focus on reducing time to insights. This moonshot project has helped us exemplify the impact of leveraging innovations in generative AI in our business, prompting further exploration in areas like content improvement, customer service automation/ standardization, and many others.

Embracing Both Automation and Innovation

In Peter Thiel’s book Zero to One, he argues that startups should favor bold ideas over incremental developments on existing solutions. I think larger organizations can take a page from Thiel too. Especially with the availability of AI tools today, companies that remain on the lookout for “revolutionary” ideas will set themselves apart–as is the case for Microsoft with Generative AI.

Experiencing the flexibility of a startup to build new things and having the resources of a medium-large company while at foodpanda empowered me to get my hands dirty building data solutions–both at an operational level (i.e. determining ML retraining cycles in reaction to changing fraud patterns) and at an exploratory stage (i.e. prototyping a chatbot for business stakeholders to write SQL queries using only prompts). With the continued rapid development of ideas in AI and many other technologies that potentially impact billions of people, I’m more excited than ever to grow my career in the tech industry.