Planning a Machine Learning project
As a Head of the Data Science team, I am continually challenged with planning for a Machine Learning project and estimating the amount of time and effort necessary to complete it.
In order to make an informed decision about each machine learning project, I prepared a template that can assist me with thinking about important elements before planning.
To simplify the explanation of the most important points, each machine learning project is divided into three main parts, prototyping, deployment, and monitoring. Each part describes the items that you should consider in planning.
Prototyping:
The goal of prototyping is to decide if the application is workable and worth deploying. During this phase, it is acceptable to consider manual preprocessing of data. Take extensive notes/comments. The prototyping process should include the following steps:
1. Obtain dataset:
Define the list of the datasets which are needed for the project
Define the strategy for labeling data, in-house/outsourced/ crowdsourced
Describe other datasets you believe are important to this project, especially meta-data (the data about data).
2. Define a baseline: a baseline is a simple model that provides reasonable results on a task and does not require much expertise and time to build. A baseline gives an estimate of the irreducible error and indicates what might be possible, it also helps to determine the amount of time and effort required to develop the final model. It will also be useful to make clearer the required data and the missing data and even hardware needs.
Way to establish a baseline:
Human-level performance (HLP)
Literature search for state-of-the-art/open source
quick-and-dirty implementation
Performance of the older system
Common baseline models include:
Linear regression when predicting continuous values
Logistic regression when classifying structured data
Pretrained convolutional neural networks for vision-related tasks
Recurrent neural networks and gradient boosted trees for sequence modeling
3. Clarify Auditing framework: Check for accuracy, fairness, and bias.
Brainstorm the ways the system might go wrong.
Performance on subsets of data (e.g., ethnicity, gender).
Prevalence of specific errors/outputs (e.g., FP, FN).
Performance in rare classes.
Establish metrics to assess performance against these issues on appropriate slices of data.
Get business/product owner buy-in.
Deployment:
The deployment is an iterative process. Consider the following points to complete the deployment process:
1. Clarify the software engineering requirements and issues involved in designing a prediction service
Realtime or Batch
Could vs. Edge/Browser
Compute resources (CPU/GPU/memory)
Latency, throughput (QPS)
Logging
Security and privacy
2. Clarify the data pipeline to make sure the data is replicable
Tools like TensorFlow transform, Apache beam, Airflow,…
Keep track of data provenance ( where it comes from) and lineage (sequence of steps)
3. Clarify the type of deployment:
New product/capacity
Automate/assist with the manual tasks (shadow deployment)
Replace the previous ML system
4. Clarify the deployment pattern:
Canary deployment
Monitor the system and ramp up traffic gradually.
Blue-green deployment: The old version can be called the blue environment while the new version can be known as the green environment. As you test and deploy to your green environment, you keep your blue environment running seamlessly for production users, until successful deployment and testing on a green environment.
Monitoring:
1. Examine the concept drift and data drift according to the investigation of the Productization phase.
2. Provide a monitoring dashboard
Software metrics( Memory, Compute, Latency, Throughput, Server load)
Input metrics( Avg image brightness, Num missing values, Avg input volume)
Output metrics(Numebr of times return null, user redoes, etc.)
Success criteria
A final point concerns the success criteria of each project. Both the technical and business teams should agree on success criteria that they are comfortable with. In order to achieve this, the machine learning team might stretch a little bit further to business metrics, and the business teams might stretch a little bit further to the machine learning metrics. Generally, the closer one gets to business metrics, the harder it becomes for a machine learning team to make a guarantee.
Key metrics:
ML metrics (accuracy, precision/recall, etc.)
Software metrics (latency, throughput, etc. given compute resources)
Business metrics (revenue, etc.)
Reference:
MLOps Specialization course which recently developed by DeepLearning.AI and Coursera
