The cardinal sequence to follow when testing your AI
If you want your system to perform better, you ought to teach is better; and, how do you verify if you have trained it well? By testing it better!
Any testing, whether it is traditional software testing or AI testing, should start with the premise — there is a problem (or a bug) until proven otherwise by way of tests.
A typical software testing sequence is quite logical and almost immutable due to its limited nature.
However, in the case of AI testing, change in sequence is possible, and this change has a full potential to make or break the system performance. If you do not test AI in proper sequence, it can increase the risk exposure of the business significantly.
1. First decide what to test, i.e. define your population of interest
You may not be required to define test cases or individual test scenarios; however, you will need to specify how the test dataset would look. You will need to define overall characteristics of the test dataset, a broad collection of instances, and potentially, where to get it? These have to be representative ones, such that if your system demonstrates a good performance on their testing, you will approve it for production.
2. Define how would you test, i.e. testing criteria
Testing criteria should be reasonably straight-forward to define — how would you test your AI system. What would you do to test it?
3. Decide what will you accept, i.e. quantified acceptance criteria
Define the minimum acceptable performance that you will be ready to sign off. Describe, quantitatively, what level of performance is good enough for you. Define, what does good enough means to you; and once you do it — stick to it!
4. Complete AI testing
Once the above three steps are complete, it is time to test. Do all the testing and accept or reject the system based on the criteria set earlier. Avoid bargaining with the acceptance criteria once you fix it. As humans, we tend to fall in love with what we have poured time and effort into, even if it is all junk. In economic jargon, we call it as the endowment effect. You should stay away from it!
Decide what to test — how to test — what to accept or reject — then test. Stick to this sequence, and you would be fine.
If you find yourself deciding or negotiating acceptance criteria after the testing, then you’re doing it wrong. Avoid exposing your business to a high financial and reputational risks; these are irreversible.
No matter how much you test, do not test excessively, with high passing bar lest you could miss out on a good solution.
About the Author: I am many things packed inside one person: a serial entrepreneur, an award-winning published author, a prolific keynote speaker, a savvy business advisor, and an intense spiritual seeker. I write boldly, talk deeply, and mentor startups, empathetically.