Systems optimize for inspection points

I recently had a discussion with my peers about engineering productivity and metrics around our fiscal allocations. some concerns about the rate at which certain software defects were being resolved. There was one concern: defects labeled P3/P4 (lower priority) were being closed at a slower and slower rate over time.

The realization I had was the system was performing exactly as intended. We had inspection points in the system: the inspections were about delivering value to the customers (product roadmap) and ensuring quality attributes (availability/performance/security/product quality) were where we expected it to be. Product quality was assessed by the number of P1 (highest priority) and P2 (2nd highest priority) defects. We were meeting all our SLA’s with P1/P2 defects. Availability was where we needed to be.. just P3/p4 (lower priority defects) were not being fixed at the rate we wanted. The SLA’s were not being met.

The system was indeed performing as expected: because a system optimizes for inspection points. There was no inspection of P3/P4 defects (though we have an SLA for those as well) — decision making on these was local, and inspection was local. On the other hand, all the execs were all over the numbers related to P1/P2, Availability… When a trade off needs to be made, what will a team optimize for? The metrics that are going to be inspected, of course.

Design your input metrics carefully. Systems will optimize for those. Be ok with letting other things slide. Focus on your critical few metrics, and ensure everyone does a great job on those.

If you try to optimize for everything, nothing will get optimized.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store