Neural Networks Hyperparameter tuning in tensorflow 2.0

Published in

ML Book

6 min readAug 31, 2019

When building machine learning models, you need to choose various hyperparameters, such as the dropout rate in a layer or the learning rate. These decisions impact model metrics, such as accuracy. Therefore, an important step in the machine learning workflow is to identify the best hyperparameters for your problem, which often involves experimentation. This process is known as “Hyperparameter Optimization” or “Hyperparameter Tuning”. Typically people use grid search, but grid search is computationally very expensive and less interactive, To solve such problems TensorFlow 2.0 provides HParams dashboard in TensorBoard, which can be visualized within notebook easily.

The HParams dashboard in TensorBoard provides several tools to help with this process of identifying the best experiment or most promising sets of hyperparameters.

This tutorial will focus on the following steps:

Experiment setup and HParams summary
Adapt TensorFlow runs to log hyperparameters and metrics
Start runs and log them all under one parent directory
Visualize the results in TensorBoard’s HParams dashboard

Note: The HParams summary APIs and dashboard UI are in a preview stage(Aug 2019) and will change over time.

Sections 1.0 to 4.0 are described properly here Google Calab or GitHub

Start by installing TF 2.0 and loading the TensorBoard notebook extension:

# !pip install -q tf-nightly-2.0-preview
!pip install tf-nightly-gpu-2.0-preview# Load the TensorBoard notebook extension
%load_ext tensorboard# Clear any logs from previous runs
!rm -rf ./logs/

0.0 The Dataset

I have used Titanic: Machine Learning from Disaster from kaggle, you can download and find description of dataset on kaggle. I have used google colab and hence uploaded data in google drive.

0.1 Mount google drive

I have uploaded data on google drive, Learn How to use data from google drive here

from google.colab import drive
drive.mount('/content/drive')

1.0 Import libraries

from __future__ import absolute_import, division, print_function, unicode_literals

import numpy as np
import pandas as pd


import tensorflow as tf
from tensorboard.plugins.hparams import api as hp

from tensorflow import feature_column
from tensorflow.keras import layers
from sklearn.model_selection import train_test_splitprint("GPU Available: ", tf.test.is_gpu_available())

2.0 Load and preprocess Data

2.1 Use Pandas to create a dataframe

data = pd.read_csv('drive/My Drive/collab data/titanic/train.csv')
data.head(5

2.2 Missing Data

2.2.1 Check missing values

data.isnull().sum()

2.2.2 Missing value handling

mean_value = round(data['Age'].mean())
mode_value = data['Embarked'].mode()[0]

value = {'Age': mean_value, 'Embarked': mode_value}
data.fillna(value=value,inplace=True)
data.dropna(axis=1,inplace=True)
data.shape

2.3 Explore data with pandas_profiling library

import pandas_profiling as pdpf
pdpf.ProfileReport(data)

3.0 Train, val, test Split

We will divide data into train, validation, test data with 3:1:1 ratio

train, test = train_test_split(data, test_size=0.2)
train, val = train_test_split(train, test_size=0.25)
print(len(train), 'train examples')
print(len(val), 'validation examples')
print(len(test), 'test examples')>>534 train examples
178 validation examples
179 test examples

4.0 Input pilpe line

4.1 Create an input pipeline using tf.data

4.2 Feature columns

Know more about feature columns here

4.2.1 Decide which types of features you have in data

#numarical features
num_c = ['Age','Fare','Parch','SibSp'] 
bucket_c  = ['Age'] #bucketized numerical feature#categorical features
cat_i_c = ['Embarked', 'Pclass','Sex'] #indicator columns
cat_e_c = ['Ticket'] # embedding column

4.2.2 Scaler function

4.2.3 Create Feature Columns

5.0 Experiment setup and the HParams experiment summary

Experiment with five hyperparameters in the model:

Number of units in the first dense layer
Number of units in the first dense layer
Dropout rate in the dropout layer
Optimizer
L2 Regularization parameter

List the values to try, and log an experiment configuration to TensorBoard. This step is optional: you can provide domain information to enable more precise filtering of hyperparameters in the UI, and you can specify which metrics should be displayed, here we are using ‘accuracy’ metric, you can chose any other matric and you can use as many hperparameters as you want. In hp.HParam function you have to specify the name of parameter and values.

If you choose to skip this step, you can use a string literal wherever you would otherwise use an HParam value: e.g., hparams['dropout'] instead of hparams[HP_DROPOUT]

6.0 Adapt TensorFlow runs to log hyperparameters and metrics

The model will be quite simple: a input feature layer and two hidden dense layers with a dropout layer between them and a output dense sigmoid layer. The training code will look familiar, although the hyperparameters are no longer hardcoded. Instead, the hyperparameters are provided in an hparams dictionary and used throughout the training function:

6.1 Create a feature layer

Now that we have defined our feature columns, we will use a DenseFeatures layer to input them to our Keras model.

feature_layer = tf.keras.layers.DenseFeatures(feature_columns)

6.2 Define model in a function

6.3 Log an hparams summary

For each run, log an hparams summary with the hyperparameters and final accuracy:

7.0 Start runs and log them all under one parent directory

You can now try multiple experiments, training each one with a different set of hyperparameters.

For simplicity, use a grid search: try all combinations of the discrete parameters and just the lower and upper bounds of the real-valued parameter. For more complex scenarios, it might be more effective to choose each hyperparameter value randomly (this is called a random search). There are more advanced methods that can be used.

Run a few experiments, which will take a few minutes:

8.0 Visualize the results in TensorBoard’s HParams plugin

%tensorboard --logdir logs/hparam_tuning

Following photos are only for demo purpose, run code to see actual tensorboard

Table view

Parallel coordinates view

The left pane of the dashboard provides filtering capabilities that are active across all the views in the HParams dashboard:

Filter which hyperparameters/metrics are shown in the dashboard
Filter which hyperparameter/metrics values are shown in the dashboard
Filter on run status (running, success, …)
Sort by hyperparameter/metric in the table view
Number of session groups to show (useful for performance when there are many experiments)

The HParams dashboard has three different views, with various useful information:

The Table View lists the runs, their hyperparameters, and their metrics.
The Parallel Coordinates View shows each run as a line going through an axis for each hyperparemeter and metric. Click and drag the mouse on any axis to mark a region which will highlight only the runs that pass through it. This can be useful for identifying which groups of hyperparameters are most important. The axes themselves can be re-ordered by dragging them.
The Scatter Plot View shows plots comparing each hyperparameter/metric with each metric. This can help identify correlations. Click and drag to select a region in a specific plot and highlight those sessions across the other plots.

A table row, a parallel coordinates line, and a scatter plot market can be clicked to see a plot of the metrics as a function of training steps for that session (although in this tutorial only one step is used for each run).

See full tutorial on Google colab or GitHub

Please clap if find this tutorial helpful :)

Join our Telegram channel for more updates, study resources and discussion

Join and earn ₹31