What I Learned at the Tensorflow Dev Summit 2020 — Part 1

Spotting weeds with Tensorflow Datasets

Introducing Tensorflow Datasets

One of the biggest obstacles to learning the techniques of deep learning is finding a good dataset. A lot of examples just cover handwriting recognition (the famous MNIST dataset). Some online data is published in a weird proprietary format and you spend just as much time extracting the data as you do with any kind of machine learning. Enter Tensorflow Datasets: go from a URL to vectors in one line of code. And you’re not stuck with MNIST!

Google discussed Tensorflow Datasets at the 2020 Tensorflow Dev Summit. They’ve made available a lot of well documented, labelled datasets that are ideal for deep learning projects. All kinds of data are available such as:

• Audio,such as books read aloud and 13.6 hours of tempo-aligned expressive drumming.
• Images, such as photos of 10,117 different celebrities or mappings from photos to facial landmarks
• Text, such as reading comprehension quizzes, summaries of scientific papers, and movie reviews
• Video, including a mapping from a video of a robot to how the robot is controlled.

For this blog, we’ll look at the deep_weeds dataset. It consists of images of different species of weeds native to Queensland. Neural Networks are great at image classification, and an interesting application of this is weed control. It’s easier to survey crops than ever thanks to drones, and so you could imagine using a weed classifier as part of a crop management system.

I chose this dataset because it’s local to Australia and I like the idea of using deep learning to help Agriculture. Please get in touch if you work in agriculture and think machine learning would be helpful.

This blog post will be about showing some code to explain how to use these datasets.

Downloading Datasets

Downloading datasets is straightforward.

import tensorflow_datasets as tfdsdata, info = tfds.load("deep_weeds", with_info = True)

The infoobject provides metadata describing the dataset in question. From it we learn that the dataset consists of 17509 images with shape 256x256x3, and that it has not been split up into training vs test sets.

Let’s do that now:

data_train, info = tfds.load("deep_weeds", with_info=True, split='train[:80%]')
data_valid  = tfds.load("deep_weeds",split='train[80%:90%]')
data_test = tfds.load("deep_weeds", split='train[90%:]')

We can preview the dataset with the show_examples function.

tf.show_examples(data_train)

Working with tf.data.Datasets

The objects that we’ve just created are Datasets. Unfortunately they don’t behave like arrays. This is because if we do away with the notion of ordering, we can parallelize processing more easily.

data_train[0] # Throws an error.

If you want to “poke inside” a dataset, I recommend taking a sample and casting it to a list. Here’s us confirming the shape of our data is 256x256x3.

>>> my_data = list(data_train.take(5))
>>> type(my_data[0])
dict
>>> my_data[0].keys
dict_keys(['image', 'label'])
>>> my_data[0]['image'].shape
TensorShape([256, 256, 3])

You could also convert the entire dataset to a list, but this is slow and uses a lot of memory.

To preprocess datasets, use the map method. The function below scales pixel values between 0 and 1 and converts to the (data, label) format that keras expects.

We can now train an image classifier. I use ResNet50 as the base because that’s what was used in the deep weeds paper. The paper trained an image classifier to achieve 95% accuracy on the validation set.

After 5 epochs, I got a training accuracy of 95%, but a validation accuracy of 50%. This is a classic case of overfitting and we’ll improve it in the next section.

Image Augmentation on Datasets

A standard method for dealing with overfitting is to feed in more data. Since going out and taking pictures of weeds would be time consuming and involve going outside, we’ll just generate new images synthetically with image augmentation.

I usually use the keras image augmentor, which has a bunch of different augmentations built in. Unfortunately it doesn’t work well with Tensorflow Datasets.

Because the default keras image augmentor is based on numpy arrays, we have to use the tf.numpyfunction to use it. This stackoverflow post elaborates. This approach is fairly slow however, and I wouldn’t recommend it. Instead it’s better to write your own image augmentation pipeline with the tf.image library.

Below is an example which applies random augmentations such as flipping, rotation and adjusting brightness to the images. Note that we should only apply image augmentation to the training set, not the validation set. This is because the validation set gives a measure for the performance of the classifier and this measure won’t change even if we adjust the training method.

Below is a sample image augmentation pipeline that takes an image, randomly adjusts brightness, randomly flips the image, adjusts the hue, contrast and rotates it by a random number of degrees.

After augmenting the images and training for a long time (50 epochs). I was able to achieve a validation accuracy of 90%! We’re catching up to the paper! The full training code is available on github.

438/438 [==============================] - 77s 176ms/step - loss: 0.1353 - accuracy: 0.9527 - val_loss: 0.3281 - val_accuracy: 0.9092

Speeding things up with prefetching

Preprocessing data can be expensive, especially if you’re using data augmentation and need to do it during the training loop. I spend a lot of my time training models waiting for things to finish training, so any features that speed things up are really helpful.

A benefit to using Datasets is that Tensorflow provides a number of options for making data processing faster. The prefetch method will overlap data preprocessing (on the CPU) and model training (on the GPU) in order to minimize the amount of time the GPU spends idle.

Sequential Processing without prefetch

Interleaved processing with prefetch. Notice the shorter gaps between training.

To use prefetching, just use the .prefetch method. Note that we don’t augment the validation set.

Without prefetching, training on a frozen Resnet model took 50 seconds per epoch in Colab. With prefetching, training took 30 seconds. The image augmentation takes time that could be spent training the model, so augmenting the next batch of images while we train is really useful!

In this blog we’ve:

• Looked at the Deep Weeds dataset from tensorflow datasets
• Trained an image classifier,
• Wrote an image augmentation pipeline,
• Used prefetching to speed up model training.

In part 2 we’ll look at the Profiling Feature in Tensorboard and how to download pretrained models online.