This is part of an experiment on studying applicability of neural network.
Massagables
In neural network construction, there are mainly four areas which can be tuned to make learning process more effective and efficient.
- Data to be input
- Target values to be learnt
- Network architecture
- Weights and biases initialization inside network
Data massage
In this experiment, the first two areas, namely input data and target values, are considered. In order to ease back-propagation of neural network, we have to standardize input data, and rescale target values.
x' = ( x - x.mean() ) / x.var()
y = ( y - y.min() ) / ( y.max() - y.min() )
Standardization of input data helps stablize weight updates along time, and balance the rate of weight updates in the network. Target values should accommodate to the chosen activation function.
Experimental results
Suppose we have a dataset for our neural network to fit.
We choose a three-layer neural network with 16 nodes in the hidden layer, and train it with mean-square error as objective and adam as optimizer.
m = Sequential()
m.add( Dense( 16, input_dim=1, init='glorot_normal', activation='relu' ) )
m.add( Dense( 1, init='glorot_normal', activation='relu' ) )
m.compile( loss='mse', optimizer='adam' )
We train our neural network for 200 epochs, and record the change of weight updates, bias updates and total loss of our neural network.
After 200 epochs, our neural network appears to fit the dataset.
On the contrary, if input data is not standardized, weight and bias are updated less evenly, and it may lead to saturation of output nodes in network.
After 200 epochs, the network fails to fit the dataset.
Reference
LeCun, Yann A., et al. “Efficient backprop.” Neural networks: Tricks of the trade. Springer Berlin Heidelberg, 2012. 9–48.