Comparing Classification Models. Part1

Deep learning is a trend and applying it to your product backend/frontend is kind of cool. But when it comes to limited set of available data, general Machine learnings algorithms(Logistic Classifier, RandomForest, Support Vector Machines, etc..) show far better results. Usually when we talk about accuracy difference in 2–8%, increasing/tuning the training set is better than messing out with Machine Learning algorithm.

In this post, we would like to share results on ML model comparisons for Question classification task from Question-Answering problem.


Dataset with 1500 binary features, 1400 elements and 315 possible classes are given. So each class has 2–6 examples. We took 676 instances as a test set and rest as training set.


  1. RandomForest with bagSizePercent(Size of each bag, as a percentage of the training set size) - 500, gives 634 Correctly Classified Instances and 42 Incorrectly Classified Instances
  2. MultiClass Classifier with Logistic function gives error rate of 52/676
  3. Logit Boost with Decision Stump function gives error rate of 42/676
  4. MultiClass Classifier Updatable with Stochastic Gradient Descent function is a leader with 36/676 error rate.
(Test set error visualization. X-axis — Class of an instance, Y-axis — Predicted Class)

Despite high accuracy of Classifier, there’s another big issue called confidence. How can we know that user question should not be classified among available clasess (meaning it’s out of set)? To better handle this issue, transfer learning in combination with deep learning seems to be a good solution. So planning to go deeper into this field.