Using Optuna to Optimize Gluon Hyperparameters
Gluon + Optuna!
Optuna is a hyperparameter optimization framework applicable to machine learning frameworks and black-box optimization solvers. The Gluon library in Apache MXNet (incubating) provides a clear, concise, and simple API for deep learning. It makes it easy to prototype, build, and train deep learning models without sacrificing training speed. Combining the two of them allows for automatic tuning of hyperparameters to find the best performing models.
Creating the Objective Function
Optuna is a black-box optimizer, which means it needs an objective function, which returns a numerical value to evaluate the performance of the hyperparameters, and decide where to sample in upcoming trials.
In our example, we will be doing this for identifying MNIST characters from the Optuna GitHub examples folder. In this case, the objective function frame looks like this:
Notice that the objective function is passed an Optuna specific argument of trial. This object is passed to the objective function to be used to specify which hyperparameters should be tuned. This returns the accuracy of the model as val_acc, which is used by Optuna as feedback on the performance of the trial.
Defining the hyperparameters to be tuned
Optuna allows you to define the types and ranges of hyperparameters you want to tune directly within your code using the trial object. This saves the effort of learning specialized syntax for hyperparameters, and also means you can use normal Python code for looping through or defining your hyperparameters.
Optuna supports a variety of hyperparameter settings, which can be used to optimize floats, integers, or discrete categorical values. Numerical values can be suggested from a logarithmic continuum as well. In our MNIST example, we optimize the model optimizer hyperparameters within the objective function:
Which optimizer is used is selected from a list of possible choices by optimizer_name = trial.suggest_categorical("optimizer", ["Adam", "RMSprop", "SGD"]). The learning rate should vary by orders of magnitude, so log=True is used in the setting of lr = trial.suggest_uniform("lr", 1e-5, 1e-1, log=True), which will vary the values logarithmically from .00001 to 0.1.
For the definition of the model itself, Optuna leverages eager mode to allow normal Python looping to determine the number of layers and nodes in each layer with trial.suggest_int(“n_layers”, 1, 3)for the layers and trial.suggest_int(“n_units_l{}”.format(i), 4, 128) for the number of nodes in each layer, such as n_units_l1 or n_units_l2.
Running the Trials
The default sampler in Optuna Tree-structured Parzen Estimator (TPE), which is a form of Bayesian Optimization. Optuna uses TPE to search more efficiently than a random search, by choosing points closer to previous good results.
To run the trials, create a study object, which sets the direction of optimization ("maximize" or "minimize"), along with other settings. Then, the study object run with optimize(objective, n_trials=100, timeout=600), to do one hundred trials with a timeout after ten minutes.
Each trial is chosen after evaluating all the trials that have been previously done, using a sampler to make a smart guess where the best values hyperparameters can be found. Optuna provides Tree-structured Parzen Estimator (TPE) samplers, which is a kind of bayesian optimization, as the default sampler.
The best values from the trials can be accessed through study.best_trial, and other methods of viewing the trials, such as formatting in a dataframe, are available.
Pruning — Early Stopping of Poor Trials
Pruning trials is a form of early-stopping which terminates unpromising trials, so that computing time can be used for trials that show more potential. In order to do pruning, it’s necessary to open up the black-box of the Objective function some more to provide intermittent feedback on how the trial is going to Optuna, so it can compare the progress with the progress of other trials, and decide whether to stop the trial early, and provide a method to receive a method from Optuna when the trial should be terminated, and also allow the trial in session to terminate cleanly after recording the results.
trial.report is used to communicate with Optuna about the progress of the trial. In this example, the objective function communicates the current epoch and the accuracy. trial.should_prune() is how Optuna communicates to the objective function if it should terminate early.
To the Future, and Beyond!
For those interested, Optuna has many other features, including visualizations, alternative samplers, optimizers, and pruning algorithms, as well as the ability to create user-defined versions as well. If you have more computing resources available, Optuna provides an easy interface for parallel trials to increase tuning speed.
Give Optuna a try today. See Installation and Optuna Github.
This post uses Gluon with Apache MXNet 1.7.0and optuna 2.3.0.
