Pen Testing Black Box ML Models: A Visual Guide

A simple-to-follow example about QA-ing your ML Models

I’ve been thinking a lot recently about how to better systematically understand and explore complex, “black box” machine learning models

“Black Box” ML models referring to statistical models that it is unclear what exactly the learned activation function is.

There are, of course, a host of incredibly important reasons as to why it’s a good idea to better understand your ML models:

• Allows you to better explain what your model is learning from the data to endusers and stakeholders
• Allows you to address potential edge cases before they appear in production and blow up your model
• Allows you to iterate faster on the next model, knowing what signal your model was picking up.

But how do you crack open Black Box models to see how they work?

One method to better understand your ML model is to use a form of Penetration Testing, or PenTesting.

Hopefully this article is a quick intro to this concept, and helps you think through how you could implement this QA technique for your future modelling efforts 🚀

What is Pen Testing and why should I care?