Everything about Hyper parameter Tuning
What is Hyper parameter tuning
Parameter refers to as a variable, so it can have a wide range of values which internally works on a model, and hyper parameters are parameters which are explicitly altered to increase the performance of a model.
In simple words, this hyper parameters can be used to improve your accuracy of your model, it is also important to tune the parameters with correct values to avoid over fitting and to improve the generalization of the model on new data.
Examples of Hyper parameters
Let’s consider a deep neural network, there are certain parameters that alter the performance of the model, like learning rate and number of neurons in a model, so these parameters value can be altered explicitly, and if you based upon your range of values, you can see your performance of your model have improved for certain value that you have specified, and this is how hyper parameter tuning works.
Note: It’s not certain that your hyper parameter tuning always results in improving your accuracy, sometimes it may also the other side, it’s always based upon your values.
Types of Hyper parameters tuning techniques
There are several types of hyper parameter tuning techniques, and it is getting developed day by day, I have listed some commonly known types:
- Grid Search: a commonly used method that involves specifying a set of possible values for each hyper parameter, and then training and evaluating a model for each combination of hyper parameter values.
- Random Search: as the name suggests this technique will randomly make a sample from your distribution, and use it to evaluate the model.
- Bayesian Optimization: methods uses a probabilistic way to map the hyper parameter values to increase the performance.
- Gradient-Based Optimization: method optimizes the parameters by taking the gradient of the performance with respect to the hyper parameters.
- Evolutionary algorithms: method uses evolutionary algorithms to optimize the hyper parameters.
- Ensemble methods: combines several models with different hyper parameters to improve performance.
- Optuna: is an automatic hyper parameter tuning library which allows user to optimize machine learning models with minimal code changes.
As there are several techniques to perform hyper parameter tuning, there are certain techniques that are majorly used among the mentioned, those techniques are, Random Search, Grid Search, Bayesian Optimization and Gradient Based Optimization.
Which Hyper parameter technique to use
Among these you might be confused which one to use, select the one that matches with your complexity of your model.
For example, let’s take that you have a small dataset and also a small set of hyper parameters, and you want to perform hyper parameter tuning, then my suggestion would be Grid Search, Bayesian optimization and Gradient Based optimization, since all these techniques will take every value specified in the distribution of hyper parameters and runs it with each data, since the data is small this will be a major help.
In case, if you are having a large dataset with wide range of hyper parameters, then I would prefer going with Random Search, Evolutionary optimization and Ensemble optimization, since they perform very well on large datasets with large range of hyper parameters.
So, this is everything that you need to know about hyper parameter tuning, since it plays a vital role in building machine learning models. Try to implement hyper parameter yourself, and learn more.