Hyperparameter Optimization Methods

4 min readOct 10, 2021

It is a challenging task to choose the best hyperparameters for our model. We can accomplish that using GridSearchCV or RandomizedSearchCV. Those are very useful methods that can be used for hyperparameter optimization.

In this tutorial, you will learn,

What is Hyperparameter optimization?

To customized a machine learning model, we can use hyperparameters. Hyperparameters are set manually to guide the learning process. Values taken as hyperparameters can vary from dataset to dataset. So, selecting the best hyperparameters for a particular dataset is called hyperparameter optimization or hyperparameter tuning.

E.g.: There are many hyperparameters in support vector machine algorithm. C, Kernel, gamma are some of them. Then in hyperparameter optimization, we find out the most suitable C value, kernel, gamma as hyperparameters.

What is GridSearchCV?

GridSearchCV is a technique that can use for hyperparameter optimization. We can use GridSearchCV from sklearn.model_selection package. Behind the scenes, it gives the best hyperparameters from the grid of parameters. Basically, it is a cross-validation method.

For example, in the case of two hyperparameters,

According to the example above, check all pairs of hyperparameters and gives the most suitable pair of values.

What is RandomizedSearchCV?

RandomizedSearchCV is also a technique that can use for hyperparameter optimization. We can use RandomizedSearchCV from sklearn.model_selection package. In RandomizedSearchCv does not test all parameters. Only one hyperparameter combination test in one iteration. Then according to the number of iterations, the same number of combinations will test and will give the most suitable combination.

For example, in the case of two hyperparameters,

According to the example above, randomly search hyperparameter combinations.

Implement Hyperparameter Tuning

Let’s try to work with the iris dataset to classify three iris species (Versicolor, Virginica, Setosa) using its features (Sepal length, Sepal width, Petal Length, and Petal width). Click here to download the iris dataset. Here we will use the SVM algorithm and then we will select the most suitable hyperparameters for that using GridSearchCV and RandomizedSearchCV.

Import necessary libraries and load dataset.

import pandas as pd
import numpy as np
import seaborn as sns
data = pd.read_csv("iris.csv")
print(data.sample(5))

Drop id column

data = data.drop("Id",axis=1)
print(data.head())

Let’s see the distribution of the dataset

sns.pairplot(data, hue='Species')

Divide dataset into x and y

x = data.drop('Species',axis=1)
y = data['Species']

Divide dataset into train and test set

from sklearn.model_selection import train_test_split
x_train,x_test,y_train,y_test = train_test_split(x,y,test_size=0.2)

Learn more about train_test_split.

Build basic SVM model.

from sklearn.svm import SVC
model = SVC()

But SVM has many different hyperparameters. To make it easier, let’s consider only “C” and “Kernel”. By default, use 1 as “C” and use “rbf” as “Kernel”. But they are not always the most suitable parameters. Then, let’s select the most suitable hyperparameters.

Hyperparameter optimization using GridSearchCV

from sklearn.model_selection import GridSearchCV# define different values for hyper-parameters
param_grid = {'C':[0.1,1,10],
'kernel':['rbf','linear','poly']}grid_search = GridSearchCV(estimator=model, param_grid=param_grid, cv=5)
grid_search.fit(x_train,y_train)OUTPUT:GridSearchCV(cv=5, estimator=SVC(),
             param_grid={'C': [0.1, 1, 10],
                         'kernel': ['rbf', 'linear', 'poly']})# we can best combination of values for C and Kernel
print(grid_search.best_params_)OUTPUT:{'C': 0.1, 'kernel': 'linear'}# Accuracy for the best hyper-parameters
grid_search.score(x_test, y_test)OUTPUT:0.9333333333333333

Hyperparameter optimization using RandomizedSearchCV

from sklearn.model_selection import RandomizedSearchCV# define different values for hyper-parameters
param_dist = {'C':[0.1,1,10],
'kernel':['rbf','linear','poly']}randomized_search = RandomizedSearchCV(estimator=model, param_distributions=param_dist, cv=5,n_iter=5)
randomized_search.fit(x_train,y_train)OUTPUT:RandomizedSearchCV(cv=5, estimator=SVC(), n_iter=5,
                   param_distributions={'C': [0.1, 1, 10],
                                        'kernel': ['rbf', 'linear', 'poly']})# we can best combination of values for C and Kernel
print(randomized_search.best_params_)OUTPUT:{'kernel': 'linear', 'C': 1}# Accuracy for the best hyper-parameters
randomized_search.score(x_test, y_test)OUTPUT:0.9666666666666667

Note: You know, in this case, the dataset is divided randomly. So the answer you get may be different than the answer you get.

Conclusion

Hyperparameter optimization is very useful to develop the performance of the Machine Learning model. In this tutorial, we learned two common approaches we used to the hyperparameter tunning process. That is GridSearchCV and RandomizedSearchCV. Both are more effective and will give the best results.

Learn more:

Books:

Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow

AI and Machine Learning for Coders

Tutorials:

Machine Learning

Deep Learning

Natural Language Processing