Ceiling Analysis for Prioritizing Work on ML Systems

Sean Gahagan
3 min readOct 21, 2022

--

My last note shared how a system of machine learning models can be used to identify and classify text within an image in optical character recognition (OCR). This note shares how to prioritize which parts of a machine learning system are most important to work on using ceiling analysis.

What is ceiling analysis?

Ceiling analysis is a method for assessing the relative impact that improving a single part of a machine learning system will have on the overall performance of that system. The ML system in my last note for OCR had three stages:

  1. text detection,
  2. character segmentation, and
  3. character recognition.

How does ceiling analysis work?

Step 1: Check the overall system’s performance metric.

Let’s go with accuracy for our OCR example, and say that our overall system has an accuracy of 72%.

Step 2: Give first component the “right answers” to have 100% accuracy, then see how that improves the overall accuracy.

If we have the “text detection” stage only output exactly right answers, let’s say that boosts the overall system’s accuracy up to 89% (a 17 percentage point gain).

Step 3: Repeat for each subsequent stage, by giving it 100% accuracy also.

Next, let’s also have the “character segmentation” stage only output right answers, and let’s say that boosts overall accuracy up to 90% (a 1 percentage point gain).

Lastly, we’ll also have the “character recognition” stage only output right answers, which would boost the overall system’s accuracy up to 100% accuracy, because all of its stages are now only outputting the right answers (a 10 percentage point gain).

Step 4: Assess max lift for each stage.

Based on what we observed in steps 2 and 3, we’ve learned that working on the “text detection” model could give us the most performance lift in our system (up to 17 more percentage points in system accuracy).

Working on the “character recognition” model could also give us a lot of performance lift (up to 10 more percentage points in system accuracy).

Working on the “character segmentation” model would not yield a lot of performance lift (only 1 more percentage point in system accuracy at best).

Leveraging ceiling analysis in prioritization

This insight helps us prioritize efforts and resources as we work to improve our machine learning system. This analysis suggests that we should prioritize work on our “text detection” model first, then prioritize work on our “character recognition” model second, and working on our “character segmentation” model should be our last priority.

Up Next

This is the last note in this series that will focus on a specific topic from Andrew Ng’s Machine Learning course. The next note will give a very high-level summary of some of the more technical topics from the course that weren’t covered in the previous posts.

Past Notes in This Series

  1. Towards a High-Level Understanding of Machine Learning
  2. Building Intuition around Supervised Machine Learning with Gradient Descent
  3. Helping Supervised Learning Models Learn Better & Faster
  4. The Sigmoid function as a conceptual introduction to activation and hypothesis functions
  5. An Introduction to Classification Models
  6. Overfitting, and avoiding it with regularization
  7. An Introduction to Neural Networks
  8. Classification Models using Neural Networks
  9. An Introduction to K-Means Clustering
  10. Anomaly detection with supervised learning
  11. An Introduction to Machine Learning for Content-Based Recommendations
  12. Machine Learning with Real-Time Data via Online Learning
  13. An Introduction to Optical Character Recognition (OCR)

--

--