Sometimes Curiosity Makes the Cat Smarter
As a Senior Data Scientist, curiosity is in my DNA. I love using data to answer difficult questions or confirm something we intuitively suspect. Luckily, the insights we gain from experimenting and analyzing data at Noom sometimes lead to more questions than answers, so there is always something new for me to investigate.
I’m Gregory Matchado Picco, and I work on Noom’s Incredibles team, which is primarily responsible for everything related to our users’ food logging habits. We know that keeping track of what you’re eating is a big step in making healthier choices and ultimately losing weight, so making this task easier is a crucial part of what we do. It’s a team I am very proud to be a part of, and the impact we can make on the health and happiness of our users is huge.
Sometimes, however, in the course of analyzing data, more questions than answers arise. Because Noom employees are encouraged to follow their curiosity and pursue the answers to these deeper questions, even if they don’t directly relate to the team’s mission at the time, I was able to sate my curiosity with further analysis and experimentation.
The Original Question: Does App Engagement Affect User Success?
There are many variables that play into a Noom user’s success (in this case, “success” is defined specifically as “weight loss”). Engagement with our mobile app, education about behavior patterns and nutrition, food logging, increased consumption of foods with low-calorie density, levels of exercise, and water intake are just a few examples of the data points we might take interest in aggregating and measuring. In this particular case, we were focused on mobile app engagement and its effect (if any) on a user’s weight loss.
Our initial analysis looked at Noom users who continued to engage with the mobile app a few months past the initial 2-week trial period. Among this group, we found a better weight loss outcome in those that engaged with the app regularly. We found that this group of users lost the most weight in body percentage, but because correlation does not always equal causation, we wanted to dig a little deeper and analyze the relationship between these two things. While app engagement seemed to be a pre-requisite for weight loss, it wasn’t a guarantee.
There was a group of people who engaged with the app for the same amount of time and didn’t experience the same level of weight loss, and I wanted to find out why. What were the two groups doing differently? What was preventing this second group from achieving significant results? Why are these users doing everything “right,” according to what should be successful strategies, but they aren’t experiencing the same level of success?
These were the questions I really wanted to answer.
The Next Step: Breaking Down the Disparity with Proxy Metrics
Every company has its own definition of “mobile app engagement” and in this case, the engagement I was looking for consisted of one of three criteria:
- A user needed to read new daily content
- A user needed to log at least one food item, or
- A user needed to log their weight.
We also look at consistency; a user needed to do one of those three things consistently over the past 120 days. We hone in on this set of criteria because past data indicates that these are the highest forms of engagement that are meaningful and conducive to the outcome we hope for our users (losing weight). Traditionally, our definition of “weight loss success” meant a loss of at least 5% of a user’s body weight.
But this quest for deeper knowledge required me to redefine this high-level criterion and our definition of “success.” I decided to focus on comparing our power users (7% or more weight loss results instead of 5%) with users who experienced more moderate levels of success (2%-4% weight loss). And I decided to look closer at these engagement variables:
- What types of content were people reading?
- What was the role of exercise in successful versus less successful weight loss?
- Were users logging only one food item or full meals?
- Were users accurately and honestly logging meals?
- What is the ratio between the different foods logged and the number of meals?
- What proportion of “low-calorie density” foods are people consuming?
When I looked at the content the two groups of users were consuming, and whether or not they consistently logged their weight, I found no discernible difference between the habits of the power users and the habits of those who experienced moderate weight loss.
I then turned my focus to food logging. Were these power users who logged at least one food item 70 out of the past 120 days also logging all their meals? Were they being honest, accurate, and consistent with what they were logging, or were they skipping some meals? As data scientists, we know that the quality of self-provided data will never be 100% accurate. There is no mechanism by which we can explicitly understand the answer to these questions, so we have to rely on proxy metrics that try and mimic a user’s behavior. In this way, we can start to draw some conclusions about the level of accuracy, honesty, and consistency in their logging practices.
First, I looked at meal logging over the course of the day and focused on users who logged all three meals with a total calorie count of over 1000. This indicates that a user wasn’t just logging random foods here and there, but that they were logging them honestly, accurately, and consistently. We also looked at if the users were eating green foods (low-calorie density foods) in the right proportion to other less healthy foods for a certain amount of days. This indicates that the user is paying attention to what they are eating, and not only logging. I also looked at whether or not users were staying inside their calorie budgets for the day. One final thing I reviewed was the total number of meals logged compared to the total number of food items logged, to see if people were including a variety of foods or sticking with the same familiar ones.
The A-Ha Moment!
After comparing the data from the two engaged groups (power users with at least 7% weight loss and those with moderate weight loss of 2–4%), both who have met one of our three criteria for 70 out of 120 days, I found that the huge differentiator between the two groups was the number of days where users met their green food calorie proportion (low-calorie density foods).
If you aren’t familiar with Noom, our system separates foods into three categories: red, yellow, and green. Foods are divided into these three categories based on their nutritional value, calorie count, and how filling they are. Green foods are the healthiest foods that also satisfy your appetite, so we refer to them as “low-calorie density foods.” Conversely, red foods are higher in calories and not as filling or healthy, so we recommend limiting the intake of these foods. While no food is off limits, the ratio of red, yellow, and green foods our users eat is one of the cornerstones of the Noom system.
What was interesting about my deep dive into our data was that although the other variables generally improve a user’s chances of experiencing weight loss, none of them seemed to make a significant difference in the two groups. Sticking to the recommended percentages and ratios of calorie-dense foods was a much bigger driver of success.
This outcome wasn’t surprising to us. In May 2021, members of our Research team published an observational study around self-reported nutritional factors.
“We found that users who logged more green foods had better weight loss success, made healthier food choices (e.g., when asked to choose what they wanted to eat right now, they said they’d choose an apple over a cake), had better nutrition knowledge, and had healthier quality diets (according to a survey)”. — Annabell Ho, Noom Research Team
It was definitely great to validate that this is a key factor for a user’s successful outcome.
So if we want to visualize a path to success for a user, at the very basic level we want them to show up and engage as often as they can, but also to engage meaningfully. We want them to log their foods, but also log all their foods honestly, accurately, and consistently. We want them to exercise but log their exercise honestly. We want them to stay within their calorie budgets, but the most important thing they can do is to follow the recommended ratio of how much of each type of food they are eating.
The Implications for Other Teams
Not only did this data analysis confirm what we’d already suspected, but it will also help other teams across Noom. From Product to Engineering, from the website team to the mobile app team, we can help improve the path of the user and nudge them toward changing their behavior in the ways that are going to bring them the most success.
Of course, correlation does not mean causation, and so of course there is a chance that a user could do everything right and still not have as successful of an outcome as someone else. But the data is pretty strong in suggesting that the common behaviors of our most successful users are something to be encouraged as much as possible.
