Member-only story

The Most Expensive Data Science Mistake I’ve Witnessed in My Career

Why true success in machine learning goes beyond optimizing a single metric

Claudia Ng
Towards Data Science
6 min readNov 28, 2024

--

Image by Clker-Free-Vector-Images from Pixabay

“The new credit model didn’t perform as expected,” he said.

My colleagues from another team came out from the “war room” looking dejected.

This was my first time witnessing a data science “mistake” in a business context, so I didn’t quite understand the scale of it.

As I would learn over the next six months, this was a very costly error from both a financial and operational perspective, because of the downstream impact and reverberating effects it had across the company.

This issue impacted every team and we all worked tirelessly for months after:

  • Analysts worked on metrics to track portfolio health,
  • Data scientists worked on diagnosing the issue and patching it,
  • Engineers worked with data scientists to deploy the new model,
  • Product marketers worked on campaigns and messaging,
  • Operational folks worked on customer outreach and education on repayment.

The first sign of trouble was much higher default rates relative to expectations, but this was just the…

--

--

Towards Data Science
Towards Data Science

Published in Towards Data Science

Your home for data science and AI. The world’s leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.

Claudia Ng
Claudia Ng

Written by Claudia Ng

Data Scientist | FinTech | Language Enthusiast

Responses (28)