AutoML comparison

Automatic Machine Learning (autoML) is a process of building Machine Learning models by algorithm with no human intervention. There are several autoML packages available for building predictive models:

• auto-sklearn
• autoML from h2o
• TPOT
• auto_ml
• mljar

Datasets

In this post we compare three autoML packages (auto-sklearn, h2o and mljar). The comparison is performed on binary classification task on 28 datasets from openml. Datasets are described below.

Datasets used in comparison. All datasets are accessible from openml.org by id provided in the table.

Methodology

1. Each dataset was divided into train and test sets (70% of samples for training and 30% of samples for testing). Packages were tested on the same data splits.
2. The autoML model was trained on train set, with 1 hour limit for training time.
3. Final autoML model was used to compute predictions on test set (on samples not used for training).
4. The logloss was used to assess performance of the model (the lower logloss the better model). The logloss was selected because is more accurate than accuracy metric.
5. The process was repeated 10 times (with different seeds used for splits). Final results are average over 10 repetition.

Results

The results are presented in table and chart below. The best approach for each dataset is bolded.

The average logloss for each method on test subset of data, computed with 10 times repetition.

AutoML packages comparison (the lower logloss the better algorithm).

Discussion

The poor performance of auto-sklearn algorithm can be explained with 1 hour limit for training time. Auto-sklearn is using bayesian optimization for hyper parameters tuning which has sequential nature and requires many iterations to find good solution. The 1 hour training limit was selected from business perspective — in my opinion, user that is going to use autoML package prefers to wait 1 hour than 72 hours for result. The h2o results compared to auto-sklearn are better on almost all datasets.

The best results were obtained by mljar package — it was the best algorithm on 26 from 28 datasets. On average it was by 47.15% better than auto-sklearn and 13.31% better than h2o autoML solution.

The useful feature of mljar is user interface, so all models after the optimization are available through web browser (mljar is saving all models obtained during optimization).

The view with all models trained during optimization.

The details of selected model. The information about used hyper parameters and learning curve for train and test folds is available.

The code used for comparison is available at github

The mljar package can be used in python or R or by web browser.