Little things about data massage for neural network

Published in

vclab

3 min readJun 7, 2016

This is part of an experiment on studying applicability of neural network.

Massagables

In neural network construction, there are mainly four areas which can be tuned to make learning process more effective and efficient.

Data to be input
Target values to be learnt
Network architecture
Weights and biases initialization inside network

Data massage

In this experiment, the first two areas, namely input data and target values, are considered. In order to ease back-propagation of neural network, we have to standardize input data, and rescale target values.

x' = ( x - x.mean() ) / x.var()
y = ( y - y.min() ) / ( y.max() - y.min() )

Standardization of input data helps stablize weight updates along time, and balance the rate of weight updates in the network. Target values should accommodate to the chosen activation function.

Experimental results

Suppose we have a dataset for our neural network to fit.

We choose a three-layer neural network with 16 nodes in the hidden layer, and train it with mean-square error as objective and adam as optimizer.

m = Sequential()
m.add( Dense( 16, input_dim=1, init='glorot_normal', activation='relu' ) )
m.add( Dense( 1, init='glorot_normal', activation='relu' ) )
m.compile( loss='mse', optimizer='adam' )

We train our neural network for 200 epochs, and record the change of weight updates, bias updates and total loss of our neural network.

After 200 epochs, our neural network appears to fit the dataset.

88output of trained neural nework (green); original dataset (blue)

On the contrary, if input data is not standardized, weight and bias are updated less evenly, and it may lead to saturation of output nodes in network.