Custom models with TensorFlow (Part-1)->Multi-output model

TensorFlow is a wonderful package that helps in designing machine-learning models. This package makes it easy for developers to experiment with building machine-learning models.

We can use TensorFlow to design models for different use cases (e,g text, image, .. etc). TensorFlow provides a lot of built-in functions to create models for different use cases but sometimes it requires to have custom functionality.

In this custom model with the TensorFlow series, we will go through how we can implement custom model architectures using Tensorflow.

In this current article, we will go through how to build a multi-output model using Tensorflow.

For building a multi-output model, we will be using Energy efficient dataset. This dataset has building features (e.g. wall area, roof area) as inputs and has two outputs (heating load and cooling load).

Dataset can be downloaded by using the following link .

Let’s see how we can build a model to train on this data.

First, we will import all the required libraries

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, Input
from sklearn.model_selection import train_test_split

def format_output(data):
    y1 = data.pop('Y1')
    y1 = np.array(y1)
    y2 = data.pop('Y2')
    y2 = np.array(y2)
    return y1, y2


def norm(x):
    return (x - train_stats['mean']) / train_stats['std']

The above two utility functions help with normalizing data and processing output targets.

Now let’s load the dataset and split it into train and test.

dataset= './ENB2012_data.xlsx'

df = pd.read_excel(dataset)
df = df.sample(frac=1).reset_index(drop=True)

train, test = train_test_split(df, test_size=0.2)
train_stats = train.describe()

train_stats.pop('Y1')
train_stats.pop('Y2')
train_stats = train_stats.transpose()
train_Y = format_output(train)
test_Y = format_output(test)

norm_train_X = norm(train)
norm_test_X = norm(test)

Using the above syntax we will get Y1 and Y2 as the 2 outputs and format them as np arrays.

In the end, using the norm function we will normalize the training and test data.

Now lets we build the model using the functional syntax.

input_layer = Input(shape=(len(train .columns),))
first_dense = Dense(units='128', activation='relu')(input_layer)
second_dense = Dense(units='128', activation='relu')(first_dense)

y1_output = Dense(units='1', name='y1_output')(second_dense)
third_dense = Dense(units='64', activation='relu')(second_dense)

y2_output = Dense(units='1', name='y2_output')(third_dense)

model = Model(inputs=input_layer, outputs=[y1_output, y2_output])

print(model.summary())

In the above source code, Y1 output will be fed directly from the second dense layer and Y2 will come via the third dense layer.

Here we have the flexibility to adjust and create any configuration, adding more intermediate layers (first_dense, second_dense), merging different intermediate layers, and splitting it. This is the beauty of Keras functional API.

The model is defined with the input layer and a list of the output layers.

The model summary will look like this,

Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, 8)]          0                                            
__________________________________________________________________________________________________
dense (Dense)                   (None, 128)          1152        input_1[0][0]                    
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 128)          16512       dense[0][0]                      
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 64)           8256        dense_1[0][0]                    
__________________________________________________________________________________________________
y1_output (Dense)               (None, 1)            129         dense_1[0][0]                    
__________________________________________________________________________________________________
y2_output (Dense)               (None, 1)            65          dense_2[0][0]                    
==================================================================================================
Total params: 26,114
Trainable params: 26,114
Non-trainable params: 0
__________________________________________________________________________________________________
None

Using the below source code we will specify the optimizer as stochastic gradient descent and will mention the learning rate as 0.001.

After that, we will compile the model with loss functions for both outputs.

Note: We can mention different loss functions for different outputs. In the same way, we can also mention different metrics for different outputs.

optimizer = tf.keras.optimizers.SGD(learning_rate=0.001)
model.compile(optimizer=optimizer,
              loss={'y1_output': 'mse', 'y2_output': 'mse'},
              metrics={'y1_output': tf.keras.metrics.RootMeanSquaredError(),
                       'y2_output': tf.keras.metrics.RootMeanSquaredError()})

Using the below syntax we will train the model for 500 epochs.

history = model.fit(norm_train_X, train_Y,
                    epochs=500, batch_size=10, validation_data=(norm_test_X, test_Y))

Epoch 490/500
62/62 [==============================] - 0s 3ms/step - loss: 0.4838 - y1_output_loss: 0.1436 - y2_output_loss: 0.3402 - y1_output_root_mean_squared_error: 0.3789 - y2_output_root_mean_squared_error: 0.5833 - val_loss: 0.7331 - val_y1_output_loss: 0.1636 - val_y2_output_loss: 0.5694 - val_y1_output_root_mean_squared_error: 0.4045 - val_y2_output_root_mean_squared_error: 0.7546
Epoch 491/500
62/62 [==============================] - 0s 4ms/step - loss: 0.4815 - y1_output_loss: 0.1398 - y2_output_loss: 0.3418 - y1_output_root_mean_squared_error: 0.3739 - y2_output_root_mean_squared_error: 0.5846 - val_loss: 0.7368 - val_y1_output_loss: 0.1554 - val_y2_output_loss: 0.5813 - val_y1_output_root_mean_squared_error: 0.3942 - val_y2_output_root_mean_squared_error: 0.7625
Epoch 492/500
62/62 [==============================] - 0s 3ms/step - loss: 0.5719 - y1_output_loss: 0.1549 - y2_output_loss: 0.4171 - y1_output_root_mean_squared_error: 0.3935 - y2_output_root_mean_squared_error: 0.6458 - val_loss: 0.8713 - val_y1_output_loss: 0.2361 - val_y2_output_loss: 0.6352 - val_y1_output_root_mean_squared_error: 0.4859 - val_y2_output_root_mean_squared_error: 0.7970
Epoch 493/500
62/62 [==============================] - 0s 3ms/step - loss: 0.4094 - y1_output_loss: 0.1384 - y2_output_loss: 0.2710 - y1_output_root_mean_squared_error: 0.3720 - y2_output_root_mean_squared_error: 0.5206 - val_loss: 0.6825 - val_y1_output_loss: 0.1673 - val_y2_output_loss: 0.5152 - val_y1_output_root_mean_squared_error: 0.4091 - val_y2_output_root_mean_squared_error: 0.7178
Epoch 494/500
62/62 [==============================] - 0s 4ms/step - loss: 0.4045 - y1_output_loss: 0.1089 - y2_output_loss: 0.2957 - y1_output_root_mean_squared_error: 0.3300 - y2_output_root_mean_squared_error: 0.5437 - val_loss: 0.8336 - val_y1_output_loss: 0.1822 - val_y2_output_loss: 0.6514 - val_y1_output_root_mean_squared_error: 0.4268 - val_y2_output_root_mean_squared_error: 0.8071
Epoch 495/500
62/62 [==============================] - 0s 4ms/step - loss: 0.5090 - y1_output_loss: 0.1248 - y2_output_loss: 0.3843 - y1_output_root_mean_squared_error: 0.3532 - y2_output_root_mean_squared_error: 0.6199 - val_loss: 0.7651 - val_y1_output_loss: 0.2118 - val_y2_output_loss: 0.5533 - val_y1_output_root_mean_squared_error: 0.4602 - val_y2_output_root_mean_squared_error: 0.7439
Epoch 496/500
62/62 [==============================] - 0s 3ms/step - loss: 0.4560 - y1_output_loss: 0.1285 - y2_output_loss: 0.3275 - y1_output_root_mean_squared_error: 0.3585 - y2_output_root_mean_squared_error: 0.5723 - val_loss: 0.8264 - val_y1_output_loss: 0.2039 - val_y2_output_loss: 0.6225 - val_y1_output_root_mean_squared_error: 0.4516 - val_y2_output_root_mean_squared_error: 0.7890
Epoch 497/500
62/62 [==============================] - 0s 3ms/step - loss: 0.4489 - y1_output_loss: 0.1212 - y2_output_loss: 0.3278 - y1_output_root_mean_squared_error: 0.3481 - y2_output_root_mean_squared_error: 0.5725 - val_loss: 0.8420 - val_y1_output_loss: 0.2018 - val_y2_output_loss: 0.6402 - val_y1_output_root_mean_squared_error: 0.4492 - val_y2_output_root_mean_squared_error: 0.8001
Epoch 498/500
62/62 [==============================] - 0s 4ms/step - loss: 0.4725 - y1_output_loss: 0.1396 - y2_output_loss: 0.3330 - y1_output_root_mean_squared_error: 0.3736 - y2_output_root_mean_squared_error: 0.5770 - val_loss: 0.7472 - val_y1_output_loss: 0.1930 - val_y2_output_loss: 0.5541 - val_y1_output_root_mean_squared_error: 0.4393 - val_y2_output_root_mean_squared_error: 0.7444
Epoch 499/500
62/62 [==============================] - 0s 4ms/step - loss: 0.5368 - y1_output_loss: 0.1312 - y2_output_loss: 0.4056 - y1_output_root_mean_squared_error: 0.3622 - y2_output_root_mean_squared_error: 0.6369 - val_loss: 0.7359 - val_y1_output_loss: 0.1758 - val_y2_output_loss: 0.5601 - val_y1_output_root_mean_squared_error: 0.4193 - val_y2_output_root_mean_squared_error: 0.7484
Epoch 500/500
62/62 [==============================] - 0s 3ms/step - loss: 0.4197 - y1_output_loss: 0.1240 - y2_output_loss: 0.2957 - y1_output_root_mean_squared_error: 0.3522 - y2_output_root_mean_squared_error: 0.5438 - val_loss: 0.7257 - val_y1_output_loss: 0.2046 - val_y2_output_loss: 0.5211 - val_y1_output_root_mean_squared_error: 0.4523 - val_y2_output_root_mean_squared_error: 0.7219

Let’s test the model

# Test the model and print loss and mse for both outputs
loss, Y1_loss, Y2_loss, Y1_rmse, Y2_rmse = model.evaluate(x=norm_test_X, y=test_Y)
print("Loss = {}, Y1_loss = {}, Y1_mse = {}, Y2_loss = {}, Y2_mse = {}".format(loss, Y1_loss, Y1_rmse, Y2_loss, Y2_rmse))

5/5 [==============================] - 0s 3ms/step - loss: 0.7257 - y1_output_loss: 0.2046 - y2_output_loss: 0.5211 - y1_output_root_mean_squared_error: 0.4523 - y2_output_root_mean_squared_error: 0.7219
Loss = 0.7256720066070557, Y1_loss = 0.20459024608135223, Y1_mse = 0.4523165225982666, Y2_loss = 0.5210817456245422, Y2_mse = 0.7218599319458008

In this way using TensorFlow we build the multi-output model.

In this article, we used two output model, but we can extend this to three, four,..etc output model.

Reference:

https://www.coursera.org/learn/custom-models-layers-loss-functions-with-tensorflow/home/week/1