HyperParameter Tuning — Hyperopt Bayesian Optimization for (Xgboost and Neural Network)

Hyperparameters: These are certain values/weights that determine the learning process of an algorithm.

Certain parameters for an Machine Learning model: learning-rate, alpha, max-depth, col-samples , weights, gamma and so on.

Certain parameters for an Deep Learning model: units(no of units), layer(no of layers), dropout ratio, kernel regularizers, activation function and so on.

Hyperparameter optimization is the selection of optimum or best parameter for a machine learning / deep learning algorithm. Often, we end up tuning or training the model manually with various possible range of parameters until a best fit model is obtained. Hyperparameter tuning helps in determining the optimal tuned parameters and return the best fit model, which is the best practice to follow while building an ML/DL model.

In this section let's discuss on one of the most accurate and successful hyperparameter method, which is HYPEROPT and algorithm to apply

Optimization is nothing but finding a minimum of cost function , that determines an overall better performance of a model on both train-set and test-set.

HYPEROPT: It is a powerful python library that search through an hyperparameter space of values . It implements three functions for minimizing the cost function,

1. Random Search
2. TPE (Tree Parzen Estimators)
3. Adaptive TPE

Importing required packages:

import hyperopt
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials

Hyperopt functions for optimization:

• hp.choice(label, options) — Returns one of the options, which should be a list or tuple.
• hp.randint(label, upper) — Returns a random integer between the range [0, upper).
• hp.uniform(label, low, high) — Returns a value uniformly between low and high.
• hp.quniform(label, low, high, q) — Returns a value round(uniform(low, high) / q) * q, i.e it rounds the decimal values and returns an integer
• hp.normal(label, mean, std) — Returns a real value that’s normally-distributed with mean and standard deviation sigma.

Steps involved in hyperopt for a Machine learning algorithm-XGBOOST:

Step 1: Initialize space or a required range of values:

Step 2: Define objective function:

Step 3: Run Hyperopt function:

Here, ‘best’ gives you the optimal parameters of the best fitted model and its loss function value.

Trials: It is an object that contains or stores all the statistical and diagnostic information such as hyperparameter, loss-functions for each set of parameters that the model has been trained.

fmin: It is an optimization function that minimizes the loss and takes in 4 inputs. Algorithm used is ‘tpe.suggest’ , other algorithm that can be used is ‘tpe.rand.suggest’.

Step 4: Load/obtain the best model:

Here, the object ‘model’ is the best tuned model for the given data. Use this model to evaluate against the test-set or use it for the prediction.

Steps involved in hyperopt for a Deep learning algorithm/neural networks:

Step 1: Initialize space or a required range of values:

Step 2: Define objective function:

Note: Use appropriate regularization function and batch size that is suitable for your concerned data.

Step 3: Run Hyperopt function:

Step 4: Load/obtain the best model:

Here, the object ‘model’ is the best tuned model for the given data. Use this model to evaluate against the test-set or use it for the prediction.

Another popular parameter tuner is Optuna. Look into the below article link for the comparison of Optuna v/s Hyperopt.

Optuna vs Hyperopt: Which Hyperparameter Optimization Library Should You Choose? - neptune.ai

Conclusion:

We have discussed on how to use sklearn python library ‘hyperopt’ which is widely preferred in the field of Data Science. Hyperparameter tuning is an important step in building a learning algorithm model and it needs to be well scrutinized. Thanks for the time on reading this article, do appreciate!