Tutorial 5: Cross-Validation on Tensorflow Flowers Dataset

Prerequisite: Tutorial 0 (setting up Google Colab, TPU runtime, and Cloud Storage)

Many deep learning tutorials provide two datasets: one for training and one for validation. In practice, however, it is much more common that we only have a single training set and no validation set. While we could randomly partition the training set into trani/validation sets, there is a more common approach in traditional machine learning: cross-validation (CV). CV shuffles the data and splits it into k partitions called folds. Let’s say k is 5. Then, each time CV takes 4 folds as the training set, and the remaining one as the validation set:

5-fold cross-validation. © Datarobot

If the dataset fits in memory, cross-validation can be done easily with Scikit-Learn. How about a large dataset that doesn’t fit in memory? In this tutorial, we show how to do cross-validation using Tensorflow’s Flower dataset.

Setup. First we set up Fenwicks, and provide options for hyperparameters:

import tensorflow as tf
import os
import numpy as np
if tf.io.gfile.exists('./fenwicks'):
  tf.io.gfile.rmtree('./fenwicks')
!git clone -q https://github.com/fenwickslab/fenwicks.git 

import fenwicks as fw
ROOT_DIR = 'gs://gs_colab'
PROJECT = 'tutorial5'
MODEL = "ResNet50" #@param ["InceptionResNetV2", "ResNet50", "ResNet50V2", "InceptionV3", "MobileNetV2", "Xception"]

BATCH_SIZE = 128 #@param ["64", "128", "256", "512"] {type:"raw"}
EPOCHS = 11 #@param {type:"slider", min:1, max:100, step:1}
LEARNING_RATE = 0.001 #@param ["0.001", "0.01"] {type:"raw"}
WARMUP = 0.05 #@param {type:"slider", min:0, max:0.5, step:0.05}

And set up Google Cloud Storage (GCS):

fw.colab_utils.setup_gcs()

Preparing the pre-trained model and the dataset. In this tutorial, we use a relatively small model, namely ResNet50, pre-trained on ImageNet. This gives us around 90% validation accuracy. If we instead use a BFN such as InceptionResNetV2, we can get a much higher accuracy such as 96%. We use ResNet50 in this tutorial since it is much faster. We do 5-fold CV, which repeats the training process 5 times.

Let’s load the pre-trained model:

fw.colab_tpu.setup_gcs()
data_dir, work_dir = fw.io.get_project_dirs(ROOT_DIR, PROJECT)
base_model = fw.keras_models.get_model(MODEL, ROOT_DIR)

Preparing the dataset. We download Tensorflow’s flower dataset to the local machine:

data_dir_local = fw.datasets.untar_data(
  fw.datasets.URLs.FLOWER_PHOTOS, './flower_photos')
data_dir_local = os.path.join(data_dir_local, 'flower_photos')

Then, we convert the full data to a single TFRecord, and upload it to GCS:

data_fn = os.path.join(data_dir, 'all.tfrecord')
paths, y, labels = fw.data.data_dir_tfrecord(data_dir_local,
  data_fn, shuffle=True)
n_classes = len(labels)
n_all = len(y)

Tensorflow Flowers contains 5 types of flowers. Let’s plot a pie chart for different class labels.

fw.plt.plot_counts_pie(y, labels)

To run the command below, we need to update a Python package called cufflinks, since Colab only includes an outdated version, as of April 2019. To do so, we use the pip command:

!pip install -qq -U cufflinks

Since Colab is already running an old version of cufflinks, the above command prompts us to restart the Colab runtime. Alternatively, we can put this command at the beginning of the notebook, in which case we don’t need to restart the runtime. Here’s our pie chart:

The 5 classes of flowers are evenly distributed — a good condition for building our classifier.

Building ConvNet with the pre-trained base model. Similar to the last tutorial, we freeze the pre-trained base model, and add a new head block. Since cross validation takes a long time, we use the simplest possible head block, with only one fully-connected layer:

def build_nn():
  base = base_model.model_func()
  base.trainable = False
  model = fw.Sequential()
  model.add(base)
  model.add(tf.keras.layers.Flatten())
  model.add(tf.keras.layers.Dense(n_classes, use_bias=False))
  return model

Next, we build the training optimizer. To do so, we need to know the size of the training set, from which we derive the total number of training steps. In our setting, we have only one dataset and it is up to us to decide on the respective sizes of the training and validation sets. One constraint on Colab’s TPU is that the batch size must be a multiple of 8, since this TPU has 8 cores. For convenience (and for speed), we put the validation set in one batch. So we force its size to a multiple of 8:

n_valid = n_all // 5 // 8 * 8
n_train = n_all - n_valid
steps_per_epoch = n_train // BATCH_SIZE
total_steps = steps_per_epoch * EPOCHS
warmup_steps = int(total_steps * WARMUP)

Now, we are ready to set our learning rate schedule and our optimizer:

cosine_decay = tf.train.cosine_decay_restarts
lr_func = fw.train.one_cycle_lr(LEARNING_RATE, total_steps,
  warmup_steps, cosine_decay)

opt_func = fw.train.adam_optimizer(lr_func)
model_func = fw.train.get_clf_model_func(build_nn, opt_func)
fw.plt.plot_lr_func(lr_func, total_steps)

Input pipeline and 5-fold CV. First, we create the input parsers. In Tutorial 4, we used the image transforms from Google’s Inception example. In this tutorial we try something different: a sequence of transforms similar to the default one in fast.ai. We will visualize these transforms in Tutorial 7, and compare them in Tutorial 8.

def get_input_parser(training):
  h = w = base_model.img_size
  tfms = fw.transform.get_fastai_transforms(h, w, training=training,
    normalizer=base_model.normalizer)
  return fw.data.get_tfexample_image_parser(tfms)

parser_train = get_input_parser(True)
parser_eval = get_input_parser(False)

And the TPUEstimator:

est = fw.train.get_tpu_estimator(steps_per_epoch, model_func,
  work_dir, base_model.weight_dir, base_model.weight_vars,
  trn_bs=BATCH_SIZE, val_bs=n_valid)

Then, we define the input pipeline for one fold:

def train_eval_fold(val_fold):
  train_input_func = lambda params: fw.data.tfrecord_ds(data_fn,
    parser_train, params['batch_size'], n_folds=5, val_fold_idx =
    val_fold, training=True)
  valid_input_func = lambda params: fw.data.tfrecord_ds(data_fn,
    parser_eval, params['batch_size'], n_folds=5, val_fold_idx =
    val_fold, training=False)
  est.train(train_input_func, steps=total_steps)
  result = est.evaluate(valid_input_func, steps=1)
  fw.io.create_clean_dir(work_dir)
  return result

In the above code, val_fold_idx is the index of the validation fold, which is used as the validation set. The remaining 4 folds are then the training set.

Now train the first fold:

result = []
result.append(train_eval_fold(0))

And the second fold:

result.append(train_eval_fold(1))

And so on, until we finish all 5 folds. Finally, let’s check the CV accuracy:

acc = [res['accuracy'] for res in result]
loss = [res['loss'] for res in result]
print('accuracy:', np.mean(acc), '+/-', np.std(acc))
print('loss:', np.mean(loss), '+/-', np.std(loss))

The accuracy we got here is not high (around 91%), since the 5-fold CV takes a long time, and we used a very simple model to be fast. To get high accuracy, you could do the following:

• Use the much bigger Inception-ResnetV2 base model instead of ResNet50.
• Train longer — more epochs.
• Don’t freeze the model: remove the line base.trainable = False from the model function.

With the above you can get 95%-96%.

Here is the complete notebook:

fenwickslab/fenwicks

All tutorials:

Deep Learning on a Free TPU