Using MLFlow with HyperOpt for Automated Machine Learning

source: databrick

At Fasal we train and deploy machine learning models very fast and efficiently and for this, we have set up an AutoML approach to select the best performing model. In this article, I will discuss the approach that we use to automate our training and logging of machine learning models.

Hyperparameter optimization is one of the most important steps in a machine learning task to get the right set of hyper-parameters for obtaining the best performing model. We use the HyperOpt library along with MLFlow to track the performance of machine learning models developed.

HyperOpt is an open-source Python library that works on Bayesian optimization principles to achieve the right set of hyperparameters for your model.

Hyperopt can be installed using the below command —

pip install hyperopt

Follow the below steps to run HyperOpt —

1. Define a search space: Search space is the domain of hyperparameters on which optimization processes will operate. In the below code snippet we have defined space for hyperparameter optimization of the Gradient Boosting model.
2. Define an objective function: The objective function is the loss function that has to be minimized during optimization.
3. Selection for search algorithm: In the code snippet below, I have used the Tree Parzen estimator for selecting the best hyperparameters.
4. Maximum number of evaluations
5. Selecting a trial object(optional) to keep the history of all hyperparameters, loss, and other information for future reference.

from hyperopt import tpe, hp, fmin, STATUS_OK,Trials
##Search Space
space = { 
    'boosting_type': hp.choice('boosting_type', ['gbdt','goss']),
    'metric': hp.choice('metric',['rmse']),
    'max_depth':scope.int(hp.quniform('max_depth', 2, 16, 1)),
    'min_data_in_leaf': scope.int(hp.quniform('min_data_in_leaf', 30, 150, 1)),
    'num_leaves': scope.int(hp.quniform('num_leaves', 30, 150, 1)),
    'learning_rate': hp.loguniform('learning_rate', np.log(0.01), np.log(0.2)),
    'min_child_samples': hp.quniform('min_child_samples', 20, 500, 5),
    'reg_alpha': hp.uniform('reg_alpha', 0.0, 10),
    'reg_lambda': hp.uniform('reg_lambda', 0.0, 10),
    'colsample_bytree': hp.uniform('colsample_by_tree', 0.6, 1.0),
    "feature_pre_filter": hp.choice("feature_pre_filter",[False])
}

In the below snippet, the Objective function is defined.

Objective
def eval_metrics(actual, pred):
    rmse = np.sqrt(mean_squared_error(actual, pred))
    mae = mean_absolute_error(actual, pred)
    r2 = r2_score(actual, pred)
    return rmse, mae, r2
def objective(params, n_folds = N_FOLDS):
    """Objective function for Light Gradient Boosting Machine Hyperparameter Tuning"""
    
    # Perform n_fold cross validation with hyperparameters
    cv_results = lgb.cv(params, train_set, nfold = n_folds,num_boost_round=400,stratified=False, 
                        early_stopping_rounds = 30, seed = 50)
    print(params)
    print(cv_results)
    # Extract the best score
    best_score = min(cv_results['rmse-mean'])
# Dictionary with information for evaluation
    return {'loss': best_score, 'params': params, 'status': STATUS_OK}

Once objective function and search space are defined, the optimization process starts with the Tree Parzen estimator as below to select the best hyper-parameters “ params”.

MAX_EVALS = 10
bayes_trials = Trials()
print('optimization starting')
# Optimize
best = fmin(fn = objective, space = space, algo = tpe.suggest, 
                max_evals = MAX_EVALS, trials = bayes_trials)
print('optimization complete')
best_model = bayes_trials.results[np.argmin([r['loss'] for r in 
        bayes_trials.results])]
params=best_model['params']

After getting best hyperparameters, these are used to train LightGBM model and accuracy metrics of test set is stored and tracked using mlflow.

MLFlow is the open-source platform used to track the accuracy and saving of machine learning models. It has its separate tracking URI by default operating at 5000 port.

MLFlow server can be started by typing mlflow UI in the terminal. Once the server is started, we can see mlflow UI at https://127.0.0.1:5000 where logs of each experiment can be stored and compared with other experiments.

Installing mlflow:

pip install mlflow

Setting up mlflow experiment:

import mlflow
mlflow.set_experiment(‘experiment_name’)

Training and storing logs:

MlFlow run is started using the command mlflow.start_run() and is ended with mlflow.end_run(). Mlflow log can be stored by using mlflow.log_param() which is a key-value parameter where key and value both can be string and mlflow.log_metric () which is key value metric where value must be a number. Here, I have stored my best hyperparameters “params” obtained via hyperopt, features after doing feature engineering, rmse, mae and r2 score to compare my multiple runs

with mlflow.start_run() as run: 
        # Training LightGBM model
         gbm = lgb.train(params,
                        lgb_train,
                        num_boost_round=500,
                        valid_sets=lgb_eval,
                        early_stopping_rounds=30)
        y_test['pred'] = gbm.predict(x_test, num_iteration=gbm.best_iteration)
        (rmse, mae, r2) = eval_metrics(y_test.iloc[:,3], y_test['pred'])
y_test['mae']=abs(y_test['target']-y_test['pred'])
        # tracking run parameters
        mlflow.log_param("hyper-parameters", params)
        mlflow.log_param("features", x_train.columns)
        mlflow.log_metric("rmse", rmse)
        mlflow.log_metric("r2", r2)
        mlflow.log_metric("mae", mae)
        mlflow.sklearn.log_model(gbm, "model")
        modelpath = "micro_{}_{}".format(mod,valdate)
        mlflow.lightgbm.save_model(gbm, modelpath)
mlflow.end_run()

Once logs are stored, they can be visualized in MLflow UI which has metadata as a source that represents a link to your code, here it is ipython notebook but one can store git repo also. Model type(sklearn here), Parameters which can have any key-value pair in string, here these are hyper parameters and features used druing modelling while Metrics is for storing accuracy and eval scores. MLflow UI has an option to compare different runs in an experiment with each other by selecting them and clicking on comparing tab on the top.

Mlflow UI showing different runs and metrics for an experiment

Also read —

Background Job Processing at Scale Using AWS Lambda and SQS