Hyper-parameter Optimization
If you were to count all the possible classification algorithms and their parameters available just within the sklearn API, you would end up with something like 1.2 duodecillion combinations (okay, I don’t actually know, but a s*** ton). Each combination is something that we might want to try in our effort to find the best performing model for our problem. There is no free lunch after all. Data Scientists call this search hyperparameter tuning or hyperparameter optimization.
Many of you have employed GridSearchCV and RandomSearchCV to help you narrow the search space. Let take a quick review of these two methods:
Grid Search Cross Validation (GridSearchCV)
Grid search works by trying every possible combination of parameters you want to try in your model. Those parameters are each tried in a series of cross-validation passes. This technique has been in vogue for the past several years as a way to tune your models.
Let’s take a quick look at the process in python with an SVM:
from sklearn.model_selection import GridSearchCV
from sklearn import dataset, svmiris = dataset.load_iris()# Paramater Grid for a Support Vector Machine Classifier
parameters = {'kernel' :('linear', 'rbf'),
'C': [1,10]}# Instaniate SVM Classifier
svc = svm.SVC(gamma="scale")# Instantiate our models with each combo of paramters
clf = GridSearchCV(svc, parameters, cv=5)# Fit each model - automatically picks the best one
clf.fit(iris.data, iris.target)
We are trying only twenty models with the grid above. Given the size of our dataset and the number of models, the run time for this grid will be trivial. Imagine though, our dataset is an order of magnitudes larger, and we decided to tweak many more parameters in our model. The runtime then would be considerably larger. Days or weeks longer if you are tuning neural networks.
Random Search Cross Validation (RandomizedSearchCV)
Enter randomized search. Consider trying every possible combination takes a lot of brute force computation. Data Scientists are an impatient bunch, so they adopted a faster technique: randomly sample from a range of parameters. The idea is that you will cover on the near-optimal set of parameters faster than grid
search. This technique, however, is naive. It doesn’t know or remember anything from its previous runs.
from scipy.stats import randint sp_randintfrom sklearn.model_selection import RandomizedSearchCV
from sklearn.datasets import load_digits
from sklearn.ensemble import RandomForestClassifier# Data
digits = load_digits()
X, y = digits.data, digits.target# Instantiate a classifier
clf = RandomForestClassifier(n_estimators=20)# Specify parameters and distributions to sample from
param_dist = {"max_depth": [3,None],
"max_features": sp_randint(1,11),
"min_samples_split": sp_randint(2,11),
"bootstrap": [True, False],
"criterion": ["gini", "entropy"]}# random search
n_iter_search = 20
random_search = RandomizedSearch(clf, param_distributions=param_dist,
n_iter=n_iter_search, cv=5)random_search.fit(X,y)
Bayesian Hyperparameter Optimization
Both GridSearchCV and RandomizedSearchCV are both naïve approaches; each model run is uninformed by a previous model.
Build a probability model of the object function and use it to select the most promising hyperparameters to evaluate in the true objective function.
Bayesian approaches, in contrast to random or grid search, keep track of past evaluation results which they use to form a probabilistic model mapping hyperparameters to a probability of a score on the objective function.
This is a surrogate function for the objective function ( p(y|x). Using a surrogate function limits calls to the object function making optimizing the objective easier by selecting the next hyperparameters with Bayesian methods.
- Build a surrogate probability model of the object function (our algorithm)
- Find the hyperparameters that perform best on the surrogate
- Apply these hyperparameters to the true objective function
- Update the surrogate model incorporating the new results
- Repeat steps 2–4 until max iterations or time is reached
At a high-level, Bayesian optimization methods are efficient, because they choose the next hyperparameters in an informed manner. The basic idea: spend a little more time selecting the next hyperparameters in order to make fewer calls to the objective function. By evaluating hyperparameters that appear more promising from past results, Bayesian methods can find better model settings than random search in fewer iterations.
Luckily for us, we do not have to implement these procedures by hand. The Python ecosystem has several popular implementations: Spearmint, MOE (developed by Yelp), SMAC, and Hyperopt. We will focus on Hyperopt. It seems to be the most popular implementation. It also has a nice wrapper for sklearn aptly called hyperopt-sklearn.
Installing hyperopt-sklearn:
git clone <https://github.com/hyperopt/hyperopt-sklearn.git>
cd hyperopt
pip install -e .Sample search for a classification algorithm using the hyperopt-sklearn package. The package implements sklearn classification models in its searches. The package is still in the early stages.
from hpsklearn import HyperoptEstimator, any_sparse_classifier, tfidf
from sklearn.datasets import fetch_20newsgroups
from sklearn import metrics
from hyperopt import tpe
import numpy as np# Download the data and split into training and test setstrain = fetch_20newsgroups( subset='train' )
test = fetch_20newsgroups( subset='test' )
X_train = train.data
y_train = train.target
X_test = test.data
y_test = test.targetestim = HyperoptEstimator( classifier=any_sparse_classifier('clf'),
preprocessing=[tfidf('tfidf')],
algo=tpe.suggest, trial_timeout=300)estim.fit( X_train, y_train )print( estim.score( X_test, y_test ) )
# <<show score here>>
print( estim.best_model() )
# <<show model here>>
The data science community is quickly adopting Bayesian hyperparameter optimization for deep learning. The run-time for model evaluation makes these methods preferable to manual or grid-based methods. There is a hyperopt wrapper for Keras called hyperas which simplifies bayesian optimization for keras models.
