Hands on hyperparameter tuning with Keras Tuner

This post will explain how to perform automatic hyperparameter tuning with Keras Tuner and Tensorflow 2.0 to boost accuracy on a computer vision problem.

Here you are : your model is running and producing a first set of results. However they fall far from the top results you were expecting. You’re missing one crucial step : hyperparameter tuning!

In this post, we’ll go through a whole hyperparameter tuning pipeline step by step. Full code is available on Github.

What is hyperparameter tuning and why you should care

A machine learning model has two types of parameters:

• trainable parameters, which are learned by the algorithm during training. For instance, the weights of a neural network are trainable parameters.
• hyperparameters, which need to be set before launching the learning process. The learning rate or the number of units in a dense layer are hyperparameters.

Hyperparameters can be numerous even for small models. Tuning them can be a real brain teaser but worth the challenge: a good hyperparameter combination can highly improve your model’s performance. Here we’ll see that on a simple CNN model, it can help you gain 10% accuracy on the test set!

Thankfully, open-source libraries are available to automatically perform this step for you!

Tensorflow 2.0 and Keras Tuner

Tensorflow is a vastly used, open-source, machine learning library. In September 2019, Tensorflow 2.0 was released with major improvements, notably in user-friendliness. With this new version, Keras, a higher-level Python deep learning API, became Tensorflow’s main API.

Shortly after, the Keras team released Keras Tuner, a library to easily perform hyperparameter tuning with Tensorflow 2.0. This post will show how to use it with an application to object classification. It will also include a comparison of the different hyperparameter tuning methods available in the library.

Hyperparameter tuning with Keras Tuner

Before diving into the code, a bit of theory about Keras Tuner. How does it work?

Hyperparameter tuning process with Keras Tuner

First, a tuner is defined. Its role is to determine which hyperparameter combinations should be tested. The library search function performs the iteration loop, which evaluates a certain number of hyperparameter combinations. Evaluation is performed by computing the trained model’s accuracy on a held-out validation set.

Finally, the best hyperparameter combination in terms of validation accuracy can be tested on a held-out test set.

Getting started

Let’s get started! With this tutorial, you’ll have an end-to-end pipeline to tune a simple convolutional network’s hyperparameters for object classification on the CIFAR10 dataset.

Installation step

First, install Keras Tuner from your terminal:

pip install keras-tuner

You can now open your favorite IDE/text editor and start a Python script for the rest of the tutorial!

Dataset

CIFAR10 random samples. The dataset is composed of 60000 images belonging to one out of 10 object classes.

This tutorial uses the CIFAR10 dataset. CIFAR10 is a common benchmarking dataset in computer vision. It contains 10 classes and is relatively small, with 60000 images. This size allows for a relatively short training time which we’ll take advantage of to perform multiple hyperparameter tuning iterations.

Load and pre-process data:

The tuner expects floats as inputs, and the division by 255 is a data normalization step.

Model definition

Here, we’ll experiment with a simple convolutional model to classify each image into one of the 10 available classes.

Simple CNN representation, from this great blog post about CNNs

Each input image will go through two convolutional blocks (2 convolution layers followed by a pooling layer) and a dropout layer for regularization purposes. Finally, each output is flattened and goes through a dense layer that classify the image into one of the 10 classes.

In Keras, this model can be defined as below :

Search Space definition

To perform hyperparameter tuning, we need to define the search space, that is to say which hyperparameters need to be optimized and in what range. Here, for this relatively small model, there are already 6 hyperparameters that can be tuned:

• the dropout rate for the three dropout layers
• the number of filters for the convolutional layers
• the number of units for the dense layer
• its activation function

In Keras Tuner, hyperparameters have a type (possibilities are Float, Int, Boolean, and Choice) and a unique name. Then, a set of options to help guide the search need to be set:

• a minimal, a maximal and a default value for the Float and the Int types
• a set of possible values for the Choice type
• optionally, a sampling method within linear, log or reversed log. Setting this parameter allows to add prior knowledge you might have about the tuned parameter. We’ll see in the next section how it can be used to tune the learning rate for instance
• optionally, a step value, i.e the minimal step between two hyperparameter values

For instance, to set the hyperparameter ‘number of filters’ you can use:

The dense layer has two hyperparameters, the number of units and the activation function:

Model Compilation

Then let’s move to model compilation, where other hyperparameters are also present…

…read the full article here