AutoML comparison
Automatic Machine Learning (autoML) is a process of building Machine Learning models by algorithm with no human intervention. There are several autoML packages available for building predictive models:
Datasets
In this post we compare three autoML packages (auto-sklearn, h2o and mljar). The comparison is performed on binary classification task on 28 datasets from openml. Datasets are described below.
Methodology
- Each dataset was divided into train and test sets (70% of samples for training and 30% of samples for testing). Packages were tested on the same data splits.
- The autoML model was trained on train set, with 1 hour limit for training time.
- Final autoML model was used to compute predictions on test set (on samples not used for training).
- The logloss was used to assess performance of the model (the lower logloss the better model). The logloss was selected because is more accurate than accuracy metric.
- The process was repeated 10 times (with different seeds used for splits). Final results are average over 10 repetition.
Results
The results are presented in table and chart below. The best approach for each dataset is bolded.
Discussion
The poor performance of auto-sklearn algorithm can be explained with 1 hour limit for training time. Auto-sklearn is using bayesian optimization for hyper parameters tuning which has sequential nature and requires many iterations to find good solution. The 1 hour training limit was selected from business perspective — in my opinion, user that is going to use autoML package prefers to wait 1 hour than 72 hours for result. The h2o results compared to auto-sklearn are better on almost all datasets.
The best results were obtained by mljar package — it was the best algorithm on 26 from 28 datasets. On average it was by 47.15% better than auto-sklearn and 13.31% better than h2o autoML solution.
The useful feature of mljar is user interface, so all models after the optimization are available through web browser (mljar is saving all models obtained during optimization).
The code used for comparison is available at github
The mljar package can be used in python or R or by web browser.