How to Evaluate your AI

Metrics to Test Your AI Algorithm

So you’ve finally finished training the last epoch of your AI. It’s always a great feeling after hours of cleaning data and waiting for those epochs to finish up.

Now comes one of the most important parts of finishing up a new model … evaluating its performance.

Most tutorials online use % accuracy to measure a model’s performance. Yet despite its popularity, that’s not always the best option.

The Downfalls of Accuracy

To show accuracy can be a misleading metric, I’m going to use a model that detects COVID-19 as an example.

Let’s take the world’s COVID data as an example

The current population of the Earth is 7.8 billion people.

Now imagine a model that diagnosed every single person as COVID free. It’s pretty obvious that this is the worst possible model to deploy … right?

19,700,000 / 7,800,000,000 = 0.253% error

This model would have a 0.252% error. That means a 99.747% accuracy.

Take that in … a model can achieve 99.747% accuracy by saying nobody in the world has coronavirus!