Hyperparameter Tuning: Grid Search and Random Search
Hyperparameters are parameters that are not learned during training but need to be set beforehand.
For example, while training an ANN model, we firstly need to assign the hyperparameters such as: number of layers, number of neurons and, learning rate. To optimize the model, we must obtain the best combination of hyperparameters for the model to start training.
So, how to obtain the hyperparameters?
Grid Search
Tries all out the possible combinations of hyperparameters in a predefined grid. We will define the grid for the number of layers, the number of neurons and learning rate such:
Then, grid search systematically evaluates each combination of hyperparameters and selects the one gives the best performance on a validation set.
Random Search
Randomly samples combinations of hyperparameters from the hyperparameter space. This method is less computationally expensive because it explores a random subset of the hyperparameters space. Nevertheless while random search seems less systematic it often performs pretty well and it can be more efficient than grid search especially in high dimensional spaces.
Conclusion
Grid search guarantees that you will find the optimal combation of hyperparameters within the specified grid but it can also totally miss the space where our model best performs as is depicted below. Also, it can be computationally expensive especially if the hyperparameter space is large while on the other hand random search is less computationally demanding it can often find good hyperparameters configuration within fewer evaluations but there is no guarantee that you will find the optimal solution.
