Adding mixed shaped inputs to a neural network.

Using images and numbers to make predictions with Keras.

Photo by Chris Ried on Unsplash

In life we have to process inputs of different types; images, text, numbers, etc. and neural networks can too.

For example, let’s suppose we want to build a network that predicts house prices using location data, attributes, and images. The process would look something like this:

Figure 1. Neural network data flow

We will work with a simplified example. We will input an image with either an X or an I, and numerical data with either a 0 or a 1. The outputs are shown in the following table:

Figure 2. Input-Output table

So, if we add an image with an X and the number 0 we get an output of 1.

Before starting to build our network, we import the required libraries:

import numpy as np
import random
import tensorflow as tf
from tensorflow import keras

Building our data

Let’s start by building our input data. We will create the two 3x3, black and white images.

#0 is black, 255 is white.
X_image = [[0,255,0],[255,0,255],[0,255,0]]
I_image = [[255,0,255],[255,0,255],[255,0,255]]

Figure 3. X and I, increased 100x in size.

We will now create our data.

We now have three datasets; our images and numerical inputs and the output array. Their shapes are shown in the following table:

Figure 4. Data shapes

Designing our model

We can now design our model’s data flow:

Figure 5. Model data flow

Building our model:

Here we use the TensorFlow concat layer to combine our outputs and pass them to the combined dense layer.

Also notice how we were able to process the image with convolutional layers and the number with dense layers, which improves our network performance.

Training our model

To train our model we follow the Keras functional API methodology:

1. Define our optimizer and loss function.
2. Define the train step.
3. Define the train function.
4. Train.

The models training progress is shown in the following graph:

Figure 6. Epochs and Loss

Hurray! Our model can predict the output with 100% accuracy.

We have now established the basis for adding mixed data types to a model using the Keras functional API. For further reading I recommend looking at the Keras functional API guide, the TensorFlow site, and seeing more examples on working with mixed data.

The code used for this article can be found here.