Member-only story
Neural Networks: parameters, hyperparameters and optimization strategies
Neural Networks (NNs) are the typical algorithms used in Deep Learning analysis. NNs can take different shapes and structures, nevertheless, the core skeleton is the following:
So we have our inputs (x), we take the weighted sum of them (with weights equal to w), pass it through an activation function f(.) and, voilà, we obtain our output. Then, depending on how accurate our predictions are, the algorithm updates itself through the so-called ‘backpropagation’ phase, according to a given optimization strategy.
Needless to say, this is a tremendously poor definition, but if you keep it in mind while reading this article, you will better understand its core topic.
Indeed, what I want to focus on is how to approach some characteristic elements of NNs, whose initialization, optimization and tuning can make your algorithm much more powerful. Before starting, let’s see which elements I’m talking about:
· Parameters: these are the coefficients of the model, and they are chosen by the model itself. It means that the algorithm, while learning, optimizes these coefficients (according to a given optimization strategy) and returns an array of parameters which minimize the error. To give an example, in a linear regression task, you have your model that will…