Select the right MMM candidate based on your specific criteria and business knowledge

Choosing the right Marketing Mix Model (MMM) is a daunting task, especially when faced with hundreds of potential models to consider. The complexity of the model selection phase is not just about crunching numbers or running simulations; it’s also about incorporating your business knowledge, understanding market dynamics, and aligning with your specific criteria and needs.

When using Robyn, after running thousands of iterations, you’ll find several methodologies are automatically applied to reduce the number of models the user must select from. We use Pareto-front lines to find those models with the lowest errors and clustering to find the most different models to pick. But in practice, sometimes that’s not enough for you to find the right model based on your specific needs and customized criteria.

Navigating the Complexity of Model Selection

Setting aside the massive challenge of gathering clean, relevant data and selecting the right variables to add to your MMM, the model selection step is one of the most critical and non-automatable phases.

In addition to the complex math and methodology behind it, we must always incorporate business knowledge and conscious biases. This isn’t about skewing results to fit a preconceived narrative but integrating real-world insights and expectations into the model.

The Role of Bias and Business Knowledge

Incorporating bias might sound counterintuitive, but in the context of MMM, it’s about leveraging your expertise to select the model that best describes your business and matches your expectations.

To be able to find a data-driven and business-relevant model, I’ve been testing with real-world data and non-technical brand experts a different new approach that lets the user define their own criteria importance to rank the models and start digging into the top candidates. This is a cherry on top of the other methodologies applied in Robyn to reduce the number of useful different models to pick from.

Leveraging the new Model Selector Function in lares

I’ve included a new function within the lares R package (already installed if you have Robyn installed given it is a dependency) called robyn_modelselector() to streamline this process. This function uses a new simple approach to select the best models by using weighted criteria scores and suggesting a single best model followed by the second, and so on.

The following visual is provided by the function. It helps overlook the common trends and relative differences in each criterion for each model. The ranked and sorted best models are identified by the number of stars (*) beside the models' IDs. Notice that the models considered are all the Pareto-front models and picked out of any cluster.

The function calculates the ranked score based on several key metrics and criteria, such as: R squared, performance (high ROAS or low CPA), potential improvement in budget allocation, the number of non-zero beta coefficients, distance to expected baseline, number of models per cluster, multi-objective optimization errors (NRMSE, DECOMP.RSSD, MAPE). Depending on the team and the knowledge (and interests) of the people involved, the weights will vary accordingly.

Here’s part of a screenshot of a Shiny app* that uses the criteria to sort a dropdown menu with the available models to check from. The idea is to closely study the models one-pagers and evaluate if they make sense with your business by reviewing all the details:

Screenshot of a Shiny App* using the function to select models

Here's a brief overview of the key parameters you can customize in the function:

• metrics: You can choose which metrics to consider (mentioned above).
• wt: Assign custom weights to each metric to prioritize them according to your specific needs. Any numeric positive value will work, they will be normalized afterward.
• baseline_ref: Set a baseline expectation for non-media channels, helping to align the model with realistic business scenarios.
• Check out the rest of the documentation and parameters here, or running: ?robyn_modelselector

With this novel approach, you can now efficiently narrow down the best models to study further, ensuring that your final selection is well-informed and tailored to your unique needs.

Some Extra Comments and Disclaimers

• Be sure also to check the functions robyn_hypsbuilder() here and robyn_performance() here. You can install the latest CRAN (stable) version or GitHub’s (dev) version of the package; both already contains all functions.
• I developed and applied this function to a Shiny app called GeMMMa* and with my colleague Iván del Valle introduced it earlier this year during ShinyConf2024.

Screenshot of GeMMMa’s session during ShinyConf2024

• I don’t have any strings attached to Meta since exactly a year ago when I left the company. Since then, all my collaborations have been to share improvements, fixes, and new features with the MMM community. Also, I’d like to add that I’m fully writing this post as an MMM community contributor.
• This is an enabler to simplify the complex and time-consuming process of modeling selection; with it, I don’t intend for users to pick the first-ranked model and stop further analyses and model comparisons.
• This approach has been successfully tested in several markets but not several industries. I don’t see why the logic behind it wouldn’t apply to other business types.
• Maybe adding specific weights per channel (similar to the baseline approach) could be useful but not sure if that’s too much bias. Let me know your thoughts.
• If you can think of or would like to add new criteria to the function, ping me! I’m eager to hear from other MMM experts about this.

Conclusion

Selecting the right MMM is both an art and a science. By using tools like Robyn and the robyn_modelselector() function, and integrating your business knowledge, criteria, and expectations, you can navigate the complexities of model selection with more confidence. This approach ensures that your final model is not just a theoretical construct but a practical tool that helps you find the right models for your business.