The cardinal sequence to follow when testing your AI

If you want your system to perform better, you ought to teach is better; and, how do you verify if you have trained it well? By testing it better!

Anand Tamboli®
Aug 24, 2019 · 3 min read
Image courtesy:

Any testing, whether it is traditional software testing or AI testing, should start with the premise — there is a problem (or a bug) until proven otherwise by way of tests.

A typical software testing sequence is quite logical and almost immutable due to its limited nature.

However, in the case of AI testing, change in sequence is possible, and this change has a full potential to make or break the system performance. If you do not test AI in proper sequence, it can increase the risk exposure of the business significantly.

You may not be required to define test cases or individual test scenarios; however, you will need to specify how the test dataset would look. You will need to define overall characteristics of the test dataset, a broad collection of instances, and potentially, where to get it? These have to be representative ones, such that if your system demonstrates a good performance on their testing, you will approve it for production.

Testing criteria should be reasonably straight-forward to define — how would you test your AI system. What would you do to test it?

Define the minimum acceptable performance that you will be ready to sign off. Describe, quantitatively, what level of performance is good enough for you. Define, what does good enough means to you; and once you do it — stick to it!

Once the above three steps are complete, it is time to test. Do all the testing and accept or reject the system based on the criteria set earlier. Avoid bargaining with the acceptance criteria once you fix it. As humans, we tend to fall in love with what we have poured time and effort into, even if it is all junk. In economic jargon, we call it as the endowment effect. You should stay away from it!

In short

Decide what to test — how to test — what to accept or reject — then test. Stick to this sequence, and you would be fine.

If you find yourself deciding or negotiating acceptance criteria after the testing, then you’re doing it wrong. Avoid exposing your business to a high financial and reputational risks; these are irreversible.

No matter how much you test, do not test excessively, with high passing bar lest you could miss out on a good solution.

About the Author: I am many things packed inside one person: a serial entrepreneur, an award-winning published author, a prolific keynote speaker, a savvy business advisor, and an intense spiritual seeker. I write boldly, talk deeply, and mentor startups, empathetically.

If you liked this article, subscribe to my newsletter for more such articles and connect with me on LinkedIn.


It is time we started thinking beyond tomorrow…

Anand Tamboli®

Written by

Award-Winning Author • Keynote Speaker • Transformation Specialist • Tech Futurist ⋆


It is time we started thinking beyond tomorrow…

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade