Announcing Optuna 2.0

hvy
hvy
Jul 29 · 8 min read
Image for post
Image for post

New Features

Hyperparameter Importances

study.optimize(...)importances = optuna.importance.get_param_importances(study)
importances.optuna.importance.get_param_importances(
study, evaluator=optuna.importance.FanovaImportanceEvaluator()
)
fig = optuna.visualization.plot_param_importances(study)fig.show()
Image for post
Image for post
Hyperparameter importances assessed using mean decrease impurity. Different colors are used to distinguish between integers, floats and categorical hyperparameters.

Performance Improvements

Image for post
Image for post
Comparisons of time taken for optimization with version 1.0, with simple objectives function that suggests floats using the Tree-structured Parzen Estimator (`TPESampler`). Version 2.0 is significantly faster when the number of hyperparameters and trials increase. Note that the `TPESampler` must fetch all previous trials in each suggestion, and that the y-axis is logarithmic. MySQL database storages are used in all experiments.

Hyperband Pruning

study = optuna.create_study(
pruner=optuna.pruners.HyperbandPruner(max_resource=”auto”)
)
study.optimize(...)
Image for post
Image for post
Hyperband (`tpe-hyperband`) converges faster than previous pruners including the median pruner (`tpe-median`) and is more stable when executed multiple times (see the shaded variance). 1 budget corresponds to 100 training epochs. `tpe-nop` means no pruning.

New CMA-ES Sampling

study = optuna.create_study(
sampler=optuna.integration.CmaEsSampler()
)
study = optuna.create_study(
sampler=optuna.samplers.CmaEsSampler()
)
study = optuna.create_study(
sampler=optuna.integration.PyCmaSampler()
)
Image for post
Image for post
The new CMA-ES converges faster when considering pruned trials during its optimization.

Integrations

Documentation

PyTorch Collaborations

AllenNLP

(Update) Optuna for PyTorch Reinforcement Learning

What’s Ahead

Contributors

Optuna

A hyperparameter optimization framework

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store