Machine Learning Platform at Endeavor

By Saad Zaheer

Endeavor is a global sports and entertainment company at the intersection of all forms of culture. Whether you stream a UFC fight, attend New York Fashion Week, experience premium hospitality at the Super Bowl, or enjoy the latest Hollywood flick, there is an Endeavor business behind your experience, powering lifelong memories for you and your loved ones. Our goal is to reach, engage, and delight our customers wherever they are and however they choose to engage with us.

We have developed a machine learning platform (MLP) at Endeavor that combines the best features of the most modern data technologies to enable data scientists to develop and deploy machine learning applications at scale. This article explains the underlying architecture and components of Endeavor’s MLP. So far, we have deployed MLP-based solutions for UFC, On Location, Endeavor Streaming, and IMG Events. Examples of deployed models include ranking and scoring of leads for On Location, the likelihood of UFC users to stream additional content, the likelihood to re-engage lapsed UFC customers, and personalized event recommendations for IMG events such as Hyde Park Winter Wonderland and Hall Des Lumieres.

Diagram 1: Machine Learning Platform Architecture

The diagram above describes the technologies and data flow processes that comprise MLP. These should be familiar to machine learning practitioners. Endeavor’s MLP is an attempt to create a closed-loop around the disparate tasks of ML engineering, such as data ingestion, feature engineering, model research and development, deployment, inference, and export. Let us deep dive into each area.

Overarching Principles

Historically, typical machine learning workflows start out with data scientists building models in Python, and then machine learning engineers rewriting them in more performant libraries for production, along with data engineers rewriting entire feature engineering pipelines. A typical Machine Learning model can take months from research to deployment while changing multiple hands. Our goal here is to simplify and speed up this process by choosing technologies that can help close the loop from data ingestion to export of predictions within simply connected processes. Endeavor’s MLP consists of six distinct processes that have been designed with the following principles in mind:

1. Data scientists should be able to research, develop, and deploy models in production with minimum support from engineering.
2. Processing of data into features, training sets, and inference sets is best handled by a large-scale data processing engine such as Snowflake or BigQuery and expressed as DBT models which data scientists can develop in standard SQL syntax.
3. Orchestration of data pipelines handled by standard orchestration frameworks such as Prefect or Airflow deployed on a cloud computing service, such as Kubernetes on AWS or GCP.
4. Model inference and training are best handled by standard machine learning libraries in Python.
5. Snowflake or BigQuery erves as a landing zone for raw data as well as materialized model inferences, while cloud storage such as AWS S3 or GCS serves as a storage zone for model objects.
6. Artifacts follow strict naming conventions, e.g. {artifact}_{client_name}_{model_name}.extension across all systems. This includes DBT models describing features, training, validation, inference sets, and model objects. This allows complete traceability of model artifacts from ingestion to inference and exposes operational analytics on system health.

Our data engineers have written four generalizable and configurable data pipelines in Prefect for data ingestion, model deployment, model inference and data export. The data science team can add new models and data sources by modifying config files in Prefect pipelines:

The above snippet corresponds to the likelihood_to_stream model config in the Model Inference process. Pre-configured Prefect pipelines can connect to Snowflake and S3 using pre-defined roles. Secrets are declared as Kubernetes environment variables which read from AWS secrets manager. Model deployment takes as long as a simple code review by an engineer instead of weeks of rewriting Python code.

1. Data Ingestion

Data ingestion pipelines are perhaps the most customized components of the MLP. Largely driven by bespoke data sources such as application event streams coming through Segment, point of sale data from ticketing companies, customer relationship data from CRM systems such as Salesforce, demographic attributes from data enrichment providers, or custom data sources such as the Endeavor Streaming platform or On Location ticketing and reservations systems, our ingestion pipelines can break down into two types:

1. Near real-time streams coming through a data platform such as Segment
2. Batch streams coming via SFTP or S3 that are read into Snowflake via a Prefect pipeline.

Segment data follows well-defined schema patterns and our data scientists utilize pre-defined DBT models to organize this data for feature engineering.

Prefect ingestion pipelines usually require the data scientist to declare unique_keys and schema_configuration as well as one of the three ingestion modes merge , append , or full_refresh .

2. Feature Engineering

Here, we introduce the concept of a feature store. Each data source, e.g. point-of-sale data or segment data, follows consistent schema patterns and semantic definitions. Therefore, a recency feature for a specific data source applies to different business units such as On Location or Hyde Park Winter Wonderland. We organize features as DBT models inside a DBT folder called feature_store under the data source segment with naming convention as feature_feature1_defn.sql and so on. These features are documented in feature_feature1_defn.yml files and can be discovered in an internally hosted dbt-docs website (see here). This lineage diagram from the Endeavor Streaming feature store illustrates how different features are used for the likelihood_to_stream model. Some of these features are clearly applicable to likelihood_to_churn or likelihood_to_repurchase_ppv models.

Diagram 2: Feature Store Architecture

3. Model Research and Development

Rapid prototyping and iteration are the most critical process of the MLP. Our data scientists experiment in a Kubernetes hosted Jupyter Hub environment where they use standard Python ML libraries such as sklearn, xgboost etc. Hyperparameter tuning requires significant computing resources so the data scientists have access to a Dask cluster deployed on Kubernetes. Ultimately, the outcome of this process results in a model training object that is configured to read a DBT model that lives in Training store and a joblib object that is stored in AWS S3.

4. Model Deployment

The joblib object output from the model R&D process is stored in AWS S3 under the following directory structure:{client_name}/{model_name}/active/joblib_{client_name}_{model_name}_{model_version}.pkl . The model version deployed in production lives in the /active directory. The deployment pipeline ensures that, a) only one model object lives in the /active directory and, b) the correct model has been deployed by comparing the filename.pkl string with the S3 directory path. A second object that performs feature calculations and filtering before inference step also gets exported to the S3 bucket under the same directory: {client_name}/{model_name}/active with a similar naming convention to the model object.

Finally, model deployment includes writing ToScore and ToExport DBT models in a Marts DB in Snowflake which respectively perform the following functions:

1. ToScore models: this object organizes new data and features for ongoing scoring.
2. ToExport models: this object assumes that Model Inference materializes scored data into a predefined schema and organizes that data into a format that is ready for export to a destination. It includes, for example, logic for selecting the most recent predictions for a given user.

5. Model Inference

This process is executed by a Prefect data pipeline that takes the location of the model objects in AWS S3 and ToScore data models as input parameters and, after performing pre-defined data transformations, applies a model.predict transformation to the inference set, and publishes new scores to a Snowflake landing database. The Prefect pipeline is completely generalized and can be adapted to new models and inference sets by declaring configuration variables as described in the Overarching Principles section above.

Package Dependency Management

Different ML models have different Python package dependencies, and since inference takes place on dockerized Kubernetes VMs, these dependencies need to be installed at runtime. However, requiring bespoke pip install package commands to run on demand can impact cluster network uncontrollably. Therefore, we pre-install the specific packages for each model into Python wheel objects when the docker image is built.

Each model’s dependencies are declared in a file titled requirements.{model_name}.txt while the general pipeline dependencies are declared in requirements.txt. Both files live in the \root directory of the Prefect inference pipeline and are read in by the above shell commands upon docker build.

6. Model Export

This process is executed by a Prefect pipeline that gets triggered by the Inference process whenever export_to_destination = True . Destinations can vary based on the business need, e.g. an AWS S3 bucket that connects with Snowflake via storage integration, a CRM system such as SalesForce that gets written through AWS AppFlow, an email serving platform such as DotDigital that gets written into via API orchestrated by Prefect, or a pre-configured destination in Segment such as MailChimp that can accept data via SQL computed traits. We use dbt post_hooks to grant appropriate export roles Select access to the ToExport views as well as apply any decrypt masks on PII data so personalized marketing actions can be taken in the appropriate platform as required by business needs (and as legally permissible).

Conclusions

Endeavor has developed a closed-loop Machine Learning Platform that processes customer interactions and organizes them into features used to make predictions against well-defined business objectives such as reducing customer churn or recommending event tickets to purchase. These predictions are actioned via personalized marketing messages or personalized user experiences, which generate more interactions that feed back into the MLP. Through the MLP, our data scientists have generalized data pipelines and compute and orchestration engines at their disposal which can be easily configured for multiple models, clients, and use case combinations. The novelty of our design is in utilizing two different systems: Snowflake for data storage and large-scale data processing, and Prefect on Kubernetes for pipeline orchestration, deployment, and configuration management. The versatility of this system has allowed us to unify the role of data scientist, data engineer, and machine learning engineer into one person who can concentrate on model development and feature engineering while simplifying deployment and inference. This has twofold benefits: machine learning teams can move faster without the cognitive load of translating technical concepts across different skill sets, and don’t need to hire specialized skill sets other than data scientists who are skilled in SQL and Python.

To learn about what the Endeavor Digital team is up to, reach out to us via digital_comms@endeavorco.com

Interested in continuing the conversation further with the author? Contact Saad Zaheer directly on LinkedIn!