There are different metrics for assessing a classifier ‘s performance, they are called evaluation matrices. They can be categorized as matrices that satisfy and optimize. It is necessary to remember that these evaluation matrices must be evaluated on a training set, a production set or on the test set.
Example: Cat vs Non-cat
Let’s say you’ve decided that you care about the classification accuracy of your cat’s classifier, this could have been an F1 score or some other measure of accuracy, but let’s say you also care about running time in addition to accuracy.
You may want to select a classifier that maximizes accuracy, but subject to running time, that is the time it takes to classify an image that must be less than 100 milliseconds or equal to it. So in this case, we would say that accuracy is a metric that optimizes, because you want to maximize accuracy. In terms of accuracy you want to do as well as possible so that run time is what we call a satisfying metric.
That is, it just needs to be nice enough, it just needs to be less than 100 milliseconds and beyond that you just don’t care about, or at least you don’t care that much. That will therefore be a fairly reasonable way to trade off or put off Both accuracy and running time together.
And by defining matrix optimization and matrix satisfying, This allows you to choose the best classifier, Which will be classifier B in this situation, because of all those with It has better running time than 100 milliseconds and the best accuracy.
In this case, accuracy and running time are the evaluation matrices. Accuracy is the optimizing metric, because you want the classifier to correctly detect a cat image as accurately as possible. The running time which is set to be under 100 ms in this example,is the satisficing metricwhich mean that the metric has to meet expectation set. The general rule is: