How Noom Uses Technology to Herd the Experimental Cats
You never know how a thing’s going to go until it happens. You can make an educated guess and predict how you think something will work, but until you actually implement and test, you can’t know for sure. So you run an experiment, you collect and analyze the data, and then your educated guess just got a little more educated. You iterate and repeat the process, and, with each iteration, you become stronger, smarter, and better. It’s why experimentation and reliance on data to reduce risk and guide the next iteration is at the heart of everything we do at Noom.
I’m Vic Moreno, and I am the Tech Lead for a group named Funnel Cake, on Noom’s Growth team. We focus on the funnel of new users from the United States visiting the noom.com website for the very first time. Above all things, every team at Noom wants to gently guide our users toward making better choices for their health, and that journey starts the moment they come to the noom.com landing page. After all, it can be overwhelming to start a weight loss journey. It’s going to involve a lot of hard work and our team’s goal is to serve users with the best experience we can, shepherding them along a path of discovery that we ultimately hope will turn them into Noom customers who will live happier healthier lives.
The Challenge: Managing All Those Experiments
Even though people generally come to Noom to lose weight, their motivations, goals, backgrounds, experiences, health issues, location, and other demographic characteristics vary greatly. They also differ in how much they know about Noom, how much they know about themselves, how much they know about living a healthy lifestyle, and how ready they are to commit. Thus, our biggest challenge is gleaning that information in an unobtrusive way. Because we often have limited information about a new user, we rely heavily on testing and experimenting with different scenarios to continuously improve a user’s experience.
Because experimentation is baked into everything Noom does, it’s not just our team that relies heavily on it. This presents an entirely new set of challenges. How do we keep experiments from clashing with one another? How can we make sure one experiment isn’t tainting the results of another? How can we increase the velocity of our experimentation without affecting any of the other moving parts? And how can we enable other teams to do the same? How can we enable multiple teams to experiment independently even though much of the platform is shared across teams?
The other goal of Noom’s Growth Team (and thus all of the subteams inside it) is to make experimentation the fastest, easiest, and highest quality it can be, for us and our Product team. As a result, we have effectively created a meta-layer of experimentation, where we analyze the performance of and iterate on our own experimentation platform itself. This allows our team and others to be as confident as possible when using experimental data to guide future iterations of the site.
The Solution: Collaboration through The Experiment Assignment System
Some data points can be gleaned through technology (location, for example) but deeper insights require collaboration with other teams, such as the core mobile app team, the product, and the research teams. We work with these groups to help run their own experiments to keep people engaged, interested, and continuing on their Noom journey.
One of the ways in which we enable that collaboration is the Experiment Assignment System (internally called Meristem from the stem cells that can be found in plants), which contains a list of experiments populated and configured by Product Managers. This allows them to create definitions for various experiments using a specialized web interface. Some examples of these experiments might be:
- Changing the predictive algorithms that determine the user experience in realtime
- Improving the billing process
- Using previous customer data to improve the multiple-choice questions in the Buy Flow questionnaire
- Adding more inclusive language
Each experiment can have multiple variations, weights, and percentages of traffic for which those “experiment arms” should be served. Each experiment arm, in turn, corresponds to its own frontend code bundle and is stored by the Experiment Assignment System in a SQL database.
As a result of this system, our website is completely dynamic in nature: Not only do we serve a unique experience to each user, we serve unique code that is generated on the fly by Meristem. When a user first visits noom.com, the index page only makes a very small HTTP request to our backend systems. Here, a serverless function that is part of the Experiment Assignment System decides what happens next. If the requests’ cookies indicate that person is already part of an experiment variation, the lambda sends the corresponding code bundle to the frontend, ensuring that the user will receive the same experience as before for consistency and data integrity. If the user has not been assigned an experimental variation yet, the lambda decides which experiment the user will get based on the parameters set by the Product Managers in the system.
EAS Sequence Diagram:
EAS Architecture Diagram:
How the System Works
Every experiment has its own index.html file stored in an S3 bucket. Once the lambda knows which experiment will be assigned to a user, it selects the appropriate index.html file and returns it to the client, which then loads the appropriate code bundle from AWS CloudFront.
In this way, we have created a sort of code vending system: The only code that is served to each client is the code that applies specifically to its user, and code for all the other experiments is never included. There are no bloated and buggy pages that are trying to solve everything — only clean, fast, consistent, and tailored experiences for each visitor.
Using a lambda also ensures that communicating with the Experiment Assignment System (EAS), a complex application that is not designed for the kind of heavy traffic that our website experiences, is not in the critical path of a response to a request to noom.com. The lambda collects experimental data from the EAS at a time interval, so the only condition that would cause a delay for the customer is a cold start — which is something that we avoid by keeping the container in which the lambda runs “warm.” When the lambda does connect to the EAS’s API to refresh its data, it downloads only the currently running experiments and caches them in memory to maximize performance.
By creating this system, we’ve enabled our Product Managers to run an incredibly sophisticated and complex set of experiments, and we have given them complete control over how and how often those experiments are run. This allows them to confidently iterate over their ideas and continuously improve our site’s user experience for new Noomers.
Because Noom’s Engineering Team built the system, the resulting code is clean, fast, and integrates flawlessly with the rest of our website, rather than operating as an add-on that rewrites parts of our web pages after they have been served to our users. As a result, even experiments that involve huge changes do not require cleanup later and will not pollute the codebase with dead code if that does not work. We can ship even large refactors under an experimental arm, analyze their performance, iterate over them until we’re satisfied with the result, and then seamlessly baseline them without writing any additional code. The resulting partnership between Product and Engineering is something I really enjoy, especially when it leads to such successful outcomes!
Upcoming Challenges
What’s next for Funnel Cake? The work never ends, and there are a few challenges on the horizon that we are just starting to experiment with. First, we are always striving for a more cohesive experience for our users; Noom has a separate team that experiments with different images and language in our advertising and social media campaigns, and this sometimes results in a disconnect between what users see elsewhere on the web and what they experience when they visit our website. In the months to come, we plan to collaborate with that team to develop a mechanism providing consistent messaging that will carry from external properties onto noom.com.
The other challenge before us has to do with striking the right balance between education and action. The Buy Flow questionnaire is an invaluable tool that helps us give our audience an experience that is closely aligned with their Noom and weight loss journeys. This works well for “low intent” users, who tend to come to noom.com primarily to learn more about our product and understand how it can help them achieve their health goals. Users who are more highly motivated and ready to use our product, however, may find the questionnaire tedious, leading them to lose their enthusiasm or drop off altogether before they even begin.
The question that we want to answer then, is “how do we identify a user’s intent as early as possible in their Noom journey so we can tailor their experience differently?” This is a surprisingly hard problem to solve because we need to ask questions in order to determine the user’s level of interest, and then customize the rest of the Buy Flow questionnaire in real-time based on the answers — all the while allowing for the same kind of flexible experimentation that we have today. This adds a whole new dimension of complexity to the EAS, and it’s a really tricky problem to solve!
Thankfully, our Product and Engineering teams enjoy a great collaborative relationship, and so a lot of smart people here at Noom are working together to try and figure out how to solve these big challenges.
