Image Classification on Kaggle using AutoGluon

Authors: Yue Sun, Chongruo Wu, Zhongyue Zhang, Tong He, Jonas Mueller, and Hang Zhang

AutoGluon is a new open-source AutoML toolkit for deep learning (DL) that simplifies the workflow of scientists, engineers and students who are interested in using deep learning models. This blog demonstrates how to quickly get started with image classification challenges using AutoGluon, which makes it very easy to produce high quality models. For our examples, we choose 4 Kaggle competitions that have already ended with many strong results on their leaderboard from models produced by DL experts. Without bells and whistles, AutoGluon can achieve around top 10% rank with only a few lines of code. Below, we show how to apply AutoGluon in the [Dog Breed Identification](https://www.kaggle.com/c/dog-breed-identification) competition as an example.

Install AutoGluon

We use a Linux GPU machine in this experiment. To install AutoGluon in a different environment, please visit the AutoGluon website. Our machine has CUDA 10.0 pre-installed. To install AutoGluon with GPU support, use the following command:

pip install --upgrade mxnet-cu100
pip install autogluon

Download and Load the Dataset

Let’s first download the dataset for this competition from the Kaggle website. Some example images are shown below.

After downloading and uncompressing the file, we should have the following directory structure with the train/ and test/ folders containing the training and test images, respectively:

dog-breed-identification/
├── labels.csv
├── train/
    ├── xxxx.jpg
    ├── ...
├── test/
    ├── xxx.jpg
    ├── ...

Now, we open Python, import AutoGluon, and specify that image classification is the task of interest. The training data are easily loaded using AutoGluon, we just need to specify the file containing the labels (`label_file`) and the folder containing the images to use for training (called ‘train’ in this case).

from autogluon import ImageClassification as task
dataset = task.Dataset('train', label_file='labels.csv')

Start Training Models

If you have many machines available, AutoGluon provides an easy interface for distributed model/hyperparameter search. See details in this tutorial. Here, we just use single machine training (with 8 GPUs) for purposes of demonstration. We tell AutoGluon to try training at most 40 models (`num_trials`), each for at most 40 passes over the training data (`epochs`), and start fitting these models to the provided training dataset:

import autogluon as ag
classifier = task.fit(dataset, net=ag.Categorical(
    'standford_dog_resnext101_64x4d', 'standford_dog_resnet152_v1'),
    batch_size=32, epochs=40, final_fit_epochs=180,
    ngpus_per_trial=8, num_trials=40, ensemble=5)

To achieve better performance on this dog breed identification dataset, we utilize transfer-learning from models that were pre-trained on a different dataset from a similar domain, namely the Stanford dogs dataset [1]
AutoGluon makes it very simple to leverage powerful techniques like domain transfer, which ordinarily would require significant manual effort to get working successfully.

The AutoGluon training process inside of task.fit() performs the following steps:
1) AutoGluon automatically partitions the provided training data into disjoint training/validation sets, split at a 4:1 ratio (here the split is done randomly, but you could instead manually specify which data to reserve for validation). AutoGluon will then train many different models under many different hyperparameter configurations (e.g. different neural architectures, learning rates, etc.) on the training set, and evaluate the resulting predictive performance of each model on the held-out validation set.
2) AutoGluon automatically launches multiple training jobs using different hyperparameter configurations based on system availability. Each model is trained for up to 40 epochs (based on the epochs argument specified above) and its validation performance gets recorded. As computation resources become available, additional training jobs are then scheduled with new hyperparameter values that AutoGluon finds promising to try out.
3) After all training jobs finish, the best hyperparameter configuration is selected for a final training run. During this final run, all of the originally provided training data (i.e. both the training and validation splits) are used to train a final model using this best hyperparameter configuration. Based on the final_fit_epochs argument specified above, we train for up to 180 epochs in this final training run. When the ensemble argument is specified, the final model will be an ensemble of multiple models, which can typically further boost predictive performance.

Make Predictions and Submit to Kaggle

After the training process finishes, we get a classifier. Let’s load the test data with multi-scale evaluation on these images. Certainly, you are also allowed to specify a fixed scaling ratio by (`crop_ratio`) instead of (`scale_ratio_choice`) :

test_dataset = task.Dataset('test', train=False, scale_ratio_choice=[0.7, 0.8, 0.875])
inds, probs, probs_all = classifier.predict(test_dataset, set_prob_thresh=0.001)

AutoGluon’s predict() method outputs both the indices of the predicted class-labels, as well as the predicted class probabilities. In the calculation of the evaluation metric of log-loss in multi-classification problems, the small probability (close to 0) estimated by many categories usually affects the final scores. We can set thresholds (`set_prob_thresh`) to filter out the impossible categories, and improve the performance. With these predictions, we can generate a CSV file to submit to the Kaggle competition:

import autogluon as ag
ag.utils.generate_prob_csv(test_dataset, probs_all, custom='./submission.csv')

Finally, we can upload this submission file to Kaggle. We achieved a score of 0.00375, which ranked in the top 3.28% of all submissions (42/1280). Not bad for under 10 lines of code!

Results on Other Datasets

We also applied AutoGluon in a similar fashion to datasets from three other Kaggle competitions. In each case, AutoGluon produced strong performance detailed in the table below.

Note that not all datasets are suitable for ImageNet pretrained models. AutoGluon has models pretrained on various datasets allowing us to select from. Just specify the name of the pretrained model in task.fit(), and AutoGluon will automatically handle the rest of the transfer learning process. We hope you find AutoGluon useful for your own image classification applications!

Resources

AutoGluon Website
AutoGluon Github Repository

Reference

[1] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao and Li Fei-Fei. Novel dataset for Fine-Grained Image Categorization. First Workshop on Fine-Grained Visual Categorization (FGVC), IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.