Grid Search and Bayesian Optimization simply explained

An Introduction to Hyperparameter Tuning and two of the most popular Techniques

Image by the author

Table of content

- Introduction

- Grid Search vs. Bayesian Optimization

• Support Vector Regression — How it works
• Model Performance Evaluation
• Finding the optimal hyperparameter settings
• Grid Search
• From Grid Search to Bayesian Optimization

-Summary

-References

As a small remark in advance — The article focuses on explaining the process behind Bayesian hyperparameter optimization as intuitively as possible. It does not cover the application of existing libraries nor the direct performance comparison between different types of hyperparameter optimization methods for specific use cases.

Introduction

Hyperparameters are parameters that are set before the actual training to control the learning process. The decision tree requires a limit for the maximum number of nodes of the tree; the polynomial regression the polynomial degree of the trained model; the support vector regression the kernel, the regularization parameter C and the margin of tolerance ϵ. All…