AWS Certified Machine Learning Cheat Sheet — SageMaker Features 2/3
Glad you are back! We got more to come in this second installment of the SageMaker Features series. The AWS really put engineers first with their features, and in this installment will cover Autopilot, Model Monitor, Deployment Safeguards and Canvas.
Machine Learning certifications are all the rage now and AWS is one of the top cloud platforms.
Getting AWS certified can show employers your Machine Learning and cloud computing knowledge. AWS certifications can also give you life-time bragging rights!
So, whether you want a resume builder or just to consolidate your knowledge, the AWS Certified Machine Learning Exam is a great start!
This series has you covered on the features in SageMaker, from Model Monitoring to Autopilot, this series has it all!
Full list of all installments:
- 1/3 for Automatic Model Tuning, Apache Spark, SageMaker Studio and SageMaker Debugger here
- 2/3 for Autopilot, Model Monitor, Deployment Safeguards and Canvas here
- 3/3 for Training Complier, Feature Store, Lineage Tracking and Data Wrangler here
TL;DR
- Autopilot is a feature set that can automate various steps in the machine learning workflow. It supports datasets up to hundreds GBs. A report is created on what factors influence predictions, model performance, how the data was processed and how the candidate model was chosen. It also integrates with SageMaker Clarify, which tells you how different features affect the prediction and what bias exists. Works for Regression, binary, multi-class classification, time-series forecasting and text generation.
- Model Monitor tracks the quality of your SageMaker model in production via CloudWatch. Can be used for continuous monitoring or a batch transform job. Monitoring jobs are scheduled via a Monitoring Schedule. To visualize the results you can use Tensorboard, QuickSight, Tableau or SageMaker Studio. It computes metrics and statistics for tabular data only, it supports endpoints with a single model, monitor the inference of an entire pipeline only, AWS suggests to keep disk utilization below 75%. If SageMaker studio is launched in a VPC you need to create VPC endpoints to enable Model Monitor to communicate with S3 and CloudWatch.
- Deployment Safeguards allows you to catch things in a deployed model quickly. It controls shifting traffic to a new model, known as Blue/Green deployments. You can also do auto-rollbacks and has Shadow Tests, which compares performance of shadow variant to production.
- Canvas can build common machine learning models without having to write any code or have access to a ready-to-use models. You can register it in SageMaker Model Registry. You can share model predictions with QuickSight to build dashboards and create new predictions within the dashboard.
Autopilot
What problem does it solve?
Some machine learning tasks can be tedious and time consuming. Each step of building a model can be a challenge and can take many iterations to get right, with lots of trial and error. You may be new to machine learning and not know all the steps or the right action to take when building a model.
Why is it the solution?
Autopilot is a feature set that can automate various steps in the machine learning workflow. Autopilot supports datasets up to hundreds GBs. Autopilot also helps to remove the black-box aspect of the model by providing a report on the importance of each feature (uses SHAP baselines), what factors influence predictions, model performance, how the data was processed and how the candidate model was chosen.
A typical workflow would be to load your data in S3, select the target column for prediction, automate model creation, view candidate models on a leader board and then deploy and monitor your model. The model notebook is available for visibility and can refine the model through the notebook.
Autopilot allows for human guidance if you don’t want to automate every step in the process, and can do Model Monitoring with or without writing code. It also integrates with SageMaker Clarify, which tells you how different features affect the prediction and what bias exists.
Some key tasks Autopilot can do are:
- Autopilot can handle missing data, normalize data, select features and can get the data ready for the model
- It can explore different algorithms and use cross-validation resampling technique to assess metrics of model performance based on pre-defined metrics
- Automate hyperparameter tuning
- Can split data into train and test sets, train and test the model and evaluate model performance. It can also rank model candidates and identify the best optimized model.
- Lastly, Autopilot gives the option to deploy the model automatically
It supports the following algorithms:
- Regression, binary and multiclass classification with tabular data in a CSV or Parquet format
- Text classification in CSV or Parquet format
- Image classification in png or jpeg (or combination of both)
- Time-series forecasting in CSV or Parquet format
- Text generation fine-tuning LLMs in CSV or Parquet format
Model Monitor
What problem does it solve?
Monitoring the model after its been deployed is just as important as building your model. Monitoring the model for data drift, wrong predictions, quality, bias, etc. is all necessary. As a machine learning engineer you may not have enough time or resources to build one model, while monitoring another model and it can be overwhelming.
Why is it the solution?
Monitors the quality of your SageMaker model in production via CloudWatch. You can set up continuous monitoring for an endpoint, a batch transform job that runs regularly, or on-schedule monitoring for asynchronous batch transform jobs. Monitoring jobs are scheduled via a Monitoring Schedule.
Baselines can be created when creating the Monitor Model job. You can set alerts when the model quality starts to decline, early detection allows you to take early interventions. Metrics are emitted to CloudWatch. To visualize the results you can use Tensorboard, QuickSight, Tableau or SageMaker Studio.
Model Monitor also integrates with SageMaker Clarify to help explain model behavior and understand which features contribute to the predictions. You can also integrate it with Ground Truth to see differences between model predictions of labels vs human predictions of labels.
Model Monitor provides monitoring of the following features:
- data quality/statistical properties of features, for data drift
- model quality, when metrics such as accuracy, RMSE, etc. starts to decline
- bias drift, when bias starts to impact model predictions
- feature attribution drift, to see a drift in what features impact the predictions, uses NDCG score to compute feature attribute drift and compare feature ranking of training vs live data
Some guidelines:
- Model Monitor computes metrics and statistics for tabular data only, if you have an image classification model it can assess the metrics for the output, but not the input
- Supports endpoints with a single model
- It can monitor the inference of an entire pipeline, but not individual contains within a pipeline
- AWS suggests to keep disk utilization below 75% to ensure data capture continues capturing requests
- If SageMaker studio is launched in a VPC you need to create VPC endpoints to enable Model Monitor to communicate with S3 and CloudWatch
Deployment Safeguards
What problem does it solve?
When putting a new model in production some things can be unpredictable. You may realize the model isn’t functioning in production as expected and want to rollback to a previous version of the model. You also may want to use the new model on a subset of production data.
Why is it the solution?
SafeGuards allows you to catch things in a deployed model quickly. It can be used for asynchronous or real-time inference endpoints. It controls shifting traffic to a new model, known as Blue/Green deployments, where previous model is running on Blue fleet and new model is running on Green fleet.
Blue/Green deployments are:
- All at once: can shift traffic all at once,
- Canary: shift a small portion and monitor
- Linear: shift in linear spaced steps.
You can also do auto-rollbacks if you want to switch the model to a previous version. Safeguards also has Shadow Tests, which compares performance of shadow variant to production. Can monitor it in SageMaker console and decide when to promote.
Canvas
What problem does it solve?
If you are a business analyst or a multi-disciplinary team you may want to create your own machine learning model, but you don’t know how to code. You may have an understanding of machine learning, but not have a strong background in it.
Why is it the solution?
You can build common machine learning models without having to write any code or have access to a ready-to-use models.
Some features:
- Access to foundation models for generative AI through a no code interface, ready to use NLP or computer vision models or create custom models
- Browse, import and combine datasets. You can understand data distributions and correlations within the data
- Create EDA visuals
- You can create a model by selecting which feature you want to predict and share model predictions with QuickSight to build dashboards and create new predictions within the dashboard
- Can generate machine learning predictions in singular or in a batch, manually or on a schedule
- Share your model in SageMaker Studio
- Bring your own model if it is registered in SageMaker Model Registry or from SageMaker JumpStart
- Finally, if you create a model in Canvas you can register it in SageMaker Model Registry
Canvas can also evaluate bias metrics such as:
- class imbalance, such as one facet/demographic group has fewer training values
- difference in proportions of labels, imbalance of positive outcomes between facet values
- KL and JS diverge, how much outcome distributions of facets diverge
- LP norm and TVD, which measures difference between distributions of outcomes from facets
- KS maximum divergence between outcomes in distributions from facets
- CCD measures disparity of outcomes between facets as whole
Want more AWS Machine Learning Cheat Sheets? Well, I got you covered! Check out this series for SageMaker Built In Algorithms:
- 1/5 for Linear Learner, XGBoost, Seq-to-Seq and DeepAR here
- 2/5 for BlazingText, Object2Vec, Object Detection and Image Classification and DeepAR here
- 3/5 for Semantic Segmentation, Random Cut Forest, Neural Topic Model and LDA here
- 4/5 for KNN, K-Means, PCA and Factorization for here
- 5/5 for IP insights and reinforcement learning here
and high level machine learning services:
- 1/2 for Comprehend, Translate, Transcribe and Polly here
- 2/2 for Rekognition, Forecast, Lex, Personalize here
and this article on lesser known high level features for industrial or educational purposes
and for ML-OPs in AWS:
- 1/3 for SageMaker and Docker, Production Variants and SageMaker Neo here
- 2/3 for Instance Types, SageMaker and Kubernetes, SageMaker Projects, Inference Pipelines and Spot Training here
- 3/3 for Availability Zones, Serverless Inference, SageMaker Inference Recommender and Auto Scaling here
and this article on Security in AWS
Thanks for reading and happy studying!