Economics of Scaling Machine Learning Workloads — Part I
Architectural Lessons from Data Science Engagements
The rise of the digital era has been fueling a data explosion and the adoption of AI across all industries. With access to growing volumes of data, organizations are looking to build scalable machine learning (ML) solutions that can process massive datasets. Processing at scale imposes a unique set of challenges, and enterprises of all sizes are compelled to engineer solutions than not only scale but optimize the associated costs of running the workloads.
Based on learnings from Data Science engagements, the two-part blog series attempts to summarize solution architectures that help address the critical challenge of optimizing costs associated with scaling machine learning workloads.
Scaling Machine Learning
An end-to-end machine learning workload is a multi-stage workflow with stages across data preparation, feature engineering, model training, and model scoring. Scaling machine learning requires capabilities to scale every phase of the workflow.
Multiple factors impact the scalability of an end-to-end machine learning solution.
Scalable ML Framework
Machine learning has existed for years, but the renewed adoption of AI at scale has led to a massive increase in the evolution of multiple ML Frameworks. In recent years, the choice of innovative ML algorithms has only accelerated further with significant contributions from open source communities.
Typically, factors like use case, support for a specific language (Python, R, Java, or C/C++), performance, and community support influence the choice of ML framework. While ML frameworks like Spark MLlib, TensorFlow, Caffe, and PyTorch scale inherently across systems, the popular Python framework Scikit-learn is limited to a single machine. This limitation imposes constraints for training or scoring datasets that don’t fit into the memory of a single node.
For some models and datasets, the learning curve levels off after a specific data size and training on more data do not yield improved performance. In such cases, using a sampling approach, the ML pipeline can train on sampled datasets that are representative subsets of the entire data. Other alternatives to scale Scikit-learn require using either Scikit-learn’s out-of-core API with mini-batches of data or ‘Dask.’ Dask implements a flexible library to scale Python across a cluster of nodes by parallelizing computations. Python packages like Numpy, Pandas, Scikit-learn, XGBoost can be scaled with minimum rewrites.
The inherent capability of a ML framework to scale beyond a single system is critical for building scalable machine learning solutions.
Scalable Data Processing
The data preparation and transformation stages of an end-to-end machine learning workflow drives data processing requirements. As the data scale increases, machine learning solutions must build on scalable data pipelines for data processing.
Distributed computing frameworks like Apache Spark, Apache Flink, and Presto parallelize operations across a cluster to drive data processing at scale. Apache Spark offers a unified analytics engine for SQL, ML, Graph, and Stream processing of data. Apache Flink is best known for its continuous flow operators for realtime processing of streaming data. Presto is primarily a distributed SQL query engine that highly parallelizes queries for best performance on big data.
Apache Spark leads the pack as a popular choice for most implementations. While enterprise database vendors offer optimized Spark connectors to push down SQL processing for boosting performance, popular ML packages like XGBoost and LightGBM have distributed implementations for Spark, making it a natural choice. With support for an extensive range of data sources, data formats, and language environments, Apache Spark has become the standard for scaling data processing in ML pipelines.
Dynamic, High Compute Capacity
High compute resources are essential to driving performance at scale. The economics associated with owning high compute resources compels organizations to either share on-premises HPC clusters across workloads or leverage cloud computing. Cloud offers flexible, on-demand compute capacity at economic price-points.
Scaling machine learning requires engineering pipelines to either leverage shared, on-premises HPC clusters or burst to cloud and use dynamically provisioned, on-demand compute resources.
Beyond the technical essentials, the enterprise IT landscape imposes additional factors that impact the viability of machine learning workloads.
Flexible Deployment for On-Premises or Hybrid Cloud or Multi-Cloud
For most enterprise organizations, all their data does not exist in one place. And for many industries, regulatory compliance and security requirements restrict the moving of data. The enterprise constraints, coupled with the gravity of data at scale, demands that ML processing must move closer to data rather than moving data around. Some workloads are best fit for public or private cloud while some are best fit for on-premises or hybrid cloud.
For enterprises, business needs may mandate flexibility in the deployment of ML workloads. In addition to architecting for scale, machine learning solutions need to execute in on-premises or hybrid Cloud or multi-cloud environments without significant rework.
Skilled Staff
Open-source contributions are fueling rapid innovation. Organizations aspiring to adopt the latest innovations must keep up with the need for advanced technical skills. On one end of the spectrum, born in cloud organizations like Netflix, Uber, or Facebook ave been building innovative open software stacks to address their business needs. This approach requires staff with cutting-edge technical skills to build, run, and manage their ML workloads. On the other end of the spectrum are the traditional businesses that rely on off-the-shelf, vendor-built enterprise platforms to build ML solutions as the tools may help lower the skills barrier.
For organizations of all sizes, access to highly skilled staff is a critical barrier that limits technology choices for building scalable machine learning applications.
Summary
Enterprises are faced with building scalable machine learning solutions to process growing volumes of data. Multiple factors like the choice of ML and data processing frameworks, access to scalable compute resources, flexibility in deployment options, and skilled staff impact the scalability of machine learning workloads.
In the 2nd part of the two-part blog series, we’ll look at architectures that optimize computing costs of driving ML workloads at scale.