Oops. I’ve been trying to warn about this precise problem in AI for three years, and I have to say without much luck. It’s obvious, but only apparently to those with stars in their eyes (that’s a joke, duh) rather than hands on experience coding stuff.
And believe me, the military is FULLY informed about it too, as DARPA’s stance tells you very well. AI cannot explain how it acts, and NEVER WILL BE EITHER. You have to trust it — and in a military race, which we have, in the greatest last march in history, running pell-mell to AI before the others do, we cannot wait until AI explains itself, and we will reap the consequences. Get it through your heads.