Understanding Hyperparameter Tuning.
Thanks to Machine Learning for solving out the world’s complicated problems.
One of the core part of Machine Learning is finidng the right Hyperparameters.
In this article, I’ll walk you through the basics of Hyperparameter tuning and Types of Hyperparameter tuning algorithms.
Let’s begin by understanding what is a Parameter and Hyperparameter.
Parameter: Model Parameters or commonly parameters is a configuration variable that is internal to the model and its value can be estimated from the data.
The coefficients in a Linear regression or Logistic regression are one example of a model Parameter.
An example for a model parameter is: Think of a Normal Distribution. Main parameters of a normal distribution are Mean And Standard Deviation.
In Machine Learning, these parameters are estimated from the data and used for model building.
Hyperparameter: Hyperparameters are those configuration variables that are external to the model and their value cannot be estimated from the data.
These variables are usually set manually by the practitioner.
We cannot know the best value for a model hyperparameter. We may use rules of thumb, copy values used on other problems, or search for the best value by different Hyperparameter tuning techniques.
Learning rate for various models, Number of leaves in a Decision Tree, Number of hidden layers in neural network are few examples of a Hyperparameter.
Hyperparameter Tuning :
To solve a complicated problem, we need complex machine learning models.
Our main aim when trying to train a model is that we should be able to get a model that has fewer errors and high accuracy. But how do we do these when we have a lot of parameters to choose from? That’s when Hyperparameter tuning comes into the picture.
Choosing an appropriate set of hyperparameters is challenging but it is crucial for model accuracy as they control the over-fitting and under-fitting of the model.
The main difference of a hyperparameter from that of a model parameter is that they are not learned automatically by the model while training instead, they are set manually before the training process.
Up next, I’ll be explaining the two important methods for selecting Hyperparameters.
Manually, We check random sets of hyperparameters by training the algorithm and selects the best sets that fit our objective. The main drawback is that it more time consuming as it is a trial and error approach.
So we have mainly two techniques for searching purposes. They are :
1. Grid search
2. Random Search
Grid Search :
Grid search basically automates the trial and error approach. It lets us select the best parameters for your optimization problem from a list of parameter options that we provide.
Grid Search automates the process of manually searching for the best hyperparameters, as it simply takes the possible values for each parameter and runs the code to try out all possible combinations, outputs the result for each combination, as well as outputs the combination which gives the best accuracy.
Random Search :
As the name implies, Random Search randomly selects a number of hyperparameters pairs from the domain and tests only those.
Random search resembles the Grid Search approach, but this is less time consuming and easily computable as this doesn’t examine every possible combination.
Here, chances are there to miss on a few combinations which could have been optimal ones as the model chooses random combinations and tries to fit the dataset and test the accuracy.
I hope this article gives you a basic understanding of Hyperparameter and its tuning techniques.
