Member-only story
Monitor Your Dependencies! Stop Being A Blind Data-Scientist.
Reasons for monitoring your model dependencies.
In my previous article “Monitor! Stop Being A Blind Data Scientist”, I mentioned the many use cases and the monumental importance of monitoring & alerts on our field, specifically from a data-science-researcher point-of-view. I looked at use-cases and reviewed several companies that provide varying solutions for this huge, but strangely undiscussed problem, i.e., the lack of model monitoring, a problem that atm overruns our industry.
Update: this article has now been published as article #49 in O’reilly’s book “97 things every cloud engineer should know”
Use Cases
In the original article, I listed several use cases for data monitoring & alerts, I am sure there are others, but these ones that have the most impact on a data scientist. The list was written in chronological order. In this article, we’ll look into package dependency-versions and explore several incidents that can affect your clients directly.
- Annotators Performance
- Annotation Distribution
- Data Integrity, i.e., schemas and test-driven data analysis.
- Data Distributions, i.e., concept drifts