Non-reproducibility: The unspoken danger of AI

AI or machine learning systems are beginning to infiltrate every segment of the economy. With their ability to quickly process large datasets, they can be used to produce new insights and automate tasks previously unachievable. There is a dark side to these modern marvels — there is no way to reproduce many of the decisions made by a machine learning system.

Machine learning systems are so adaptable because they evolve when presented with new data. As they take in new datasets, the algorithm actually changes in response to the newly presented data. Because the algorithm is constantly changing, it is very difficult to backtest to reproduce a given result once the algorithm has processed more data. To further complicate reproduction of decisions made by a machine learning system, most machine learning systems cannot provide an explanation of a decision even as the decision is being made.

The combination of these factors makes a system that resists explanation and characterization and actually changes in response to experimental data fed into it. The physical analogue is the Schrodinger’s Cat quantum mechanics thought experiment that posits a system that resists an experimenter’s attempts to determine whether a cat is alive or not within a sealed box.

So why does it matter if we can’t reproduce a decision made by a machine learning system?

It matters if the results of a decision need to be preserved or examined later, as is always the case in a legal setting. How does a fact finder determine why an AI-driven car ran over a toddler instead of a squirrel? How does a civil rights agency prove that mortgage decisions were made with a racial bias?

Our legal system requires accountability that can be examined after an incident. If we let a technology weave itself deep within our economy with little availability to examine the factors in those decisions, we will cripple our legal system’s ability to adjudicate disputes arising from those decisions. Some serious effort needs to be put into requiring accountability for decisions made by such systems, and regulation that effectively requires a receipt whenever a machine learning system is used to make decisions that can directly affect a person’s life or wellbeing.