How do determine the number of layers and neurons in the hidden layer?

Deep Learning provides Artificial Intelligence the ability to mimic a human brain’s neural network. It is a subset of Machine Learning. The major difference between deep learning and machine learning is the way data is presented to the machine. Machine learning accesses vast amounts of data (both structured and unstructured) and learns from it to predict the future, whereas deep learning networks work on multiple layers of artificial neural networks to predict the future.

General Structure of Neural Network

A neural network has input layer(s), hidden layer(s), and output layer(s). It can make sense of patterns, noise, and sources of confusion in the data. When the features are linearly correlated usually the neural network is not preferred, it can be done by using machine learning. Even in linear correlation if neural networks are used, there is no need for any hidden layer.

Neural networks calculate the weighted sums. The calculated sum of weights is passed as input to the activation function in the hidden layers. The activation function is a function to map the inputs to the desired output. It takes the “weighted sum of input” as the input to the function, adds a bias, and decides whether the neuron should be fired or not. The output layer gives the predicted output, and the model output is compared with the actual output. After training the neural network, the model uses the backpropagation method to improve the performance of the network. The cost function helps to reduce the error rate.

Number of Neurons In Input and Output Layers

The number of neurons in the input layer is equal to the number of features in the data and in very rare cases, there will be one input layer for bias. Whereas the number of neurons in the output depends on whether is the model is used as a regressor or classifier. If the model is a regressor then the output layer will have only a single neuron but in case if the model is a classifier it will have a single neuron or multiple neurons depending on the class label of the model.

Number of Neurons and Number of Layers in Hidden Layer

When it comes to the hidden layers, the main concerns are how many hidden layers and how many neurons are required?

An Introduction to Neural Networks for Java, Second Edition by jeffheaton it is mentioned that number of hidden layers is determined as below.

There are many rule-of-thumb methods for determining the correct number of neurons to use in the hidden layers, such as the following:

• The number of hidden neurons should be between the size of the input layer and the size of the output layer.
• The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.
• The number of hidden neurons should be less than twice the size of the input layer.

Moreover, the number of neurons and number layers required for the hidden layer also depends upon training cases, amount of outliers, the complexity of, data that is to be learned, and the type of activation functions used.

Most of the problems can be solved by using a single hidden layer with the number of neurons equal to the mean of the input and output layer. If less number of neurons is chosen it will lead to underfitting and high statistical bias. Whereas if we choose too many neurons it may lead to overfitting, high variance, and increases the time it takes to train the network.

Pruning

Pruning can be used to optimize the number of neurons in the hidden and increases computational and resolution performances. It trims the neurons during training, by identifying those which have no impact on the performance of the network. It can also be identified by checking the weights of the neurons, weights that are close to zero have relatively less importance. In pruning such nodes are removed.