Optuna 4.0: What’s New in the Major Release
We are pleased to announce the fourth major version release of our hyperparameter optimization framework Optuna. Please check out the release notes!
Since its release in 2018, Optuna has developed many features and grown into software used by many users. As for the user community, we have achieved 10,000+ GitHub stars, 3M+ monthly downloads, used by 16,000+ repositories, 5,000+ paper citations, and 18,000+ mentions on Kaggle. It has become one of the most popular hyperparameter optimization OSS libraries.
The goals of v4.0 development are as follows:
Easy feature share among users
- The feature-sharing platform OptunaHub allows anyone to easily share new sampler and visualization algorithms.
Enhancements of features useful for optimizing generative AI and diverse computing environments
- We have started the official support for Artifact Store, which manages files such as generated images and trained models on Optuna.
- We have stabilized
JournalStoragewith NFS, which enables distributed optimization via NFS, and improved support for environments where RDBs such as MySQL cannot be used.
Delivery of important features and algorithms to all users
- We have enhanced features, such as the speedup of the multi-objective
TPESamplerand the addition of a newTerminatoralgorithm.
Easy Feature Share among Users
Official Release of Feature-Sharing Platform OptunaHub
We have released the official version of OptunaHub, a feature-sharing platform for Optuna. A large number of optimization and visualization algorithms are available in OptunaHub. Contributors can easily register their methods and deliver them to Optuna users around the world.
OptunaHub provides state-of-the-art and domain-specific optimization methods. OptunaHub makes it easy for users to share these methods. By developing a feature-sharing platform, feature development will become more active. Users can use Optuna more conveniently with more third-party features.
Please also read the OptunaHub release blog post.
Enhancements of Features Useful for Optimizing Generative AI and Diverse Computing Environments
Enhanced Experiment Management Feature: Official Support of Artifact Store
Artifact Store is a file management feature for files generated during optimization, dubbed artifacts. For example, files such as text, images, and audio output by generative AI, or files such as snapshots of deep learning models, which are too large to be handled efficiently in an RDB, may be generated during hyperparameter optimization. Artifact Store makes it easy to manage these files and allows users to view them on Optuna Dashboard.
In Optuna v4.0, we stabilized the existing file upload API and further enhanced the usability of Artifact Store by adding some APIs such as an artifact download API. We also added features to show JSONL and CSV files on Optuna Dashboard in addition to the existing support for images, audio, and video. With this official support, the API backward compatibility will be guaranteed.
For more details, please check the blog post.
JournalStorage: Official Support of Distributed Optimization via Network File System
JournalStorage is a new Optuna storage experimentally introduced in Optuna v3.1 (see the blog post for details). Unlike conventional storages, which store “values”, JournalStorage stores “operation logs”. JournalStorage makes it easier to implement custom storage backends.
Optuna has JournalFileBackend, a storage backend for various file systems. It can be used on NFS, allowing Optuna to scale to multiple nodes. This can be especially useful for users who want to use Optuna in environments where it is difficult to set up database servers often required for distributed optimization such as MySQL and PostgreSQL.
In Optuna v4.0, the API for JournalStorage has been reorganized, and JournalStorage is officially supported. This official support guarantees its backward compatibility from v4.0. For details on the API changes, please refer to the Optuna v4.0 Migration Guide.
import optuna
from optuna.storages import JournalStorage
from optuna.storages.journal import JournalFileBackend
def objective(trial: optuna.Trial) -> float:
...
storage = JournalStorage(JournalFileBackend("./journal.log"))
study = optuna.create_study(storage=storage)
study.optimize(objective)Delivery of Important Features and Algorithms to All Users
Significant Speedup of Multi-Objective TPESampler
Multi-objective optimization generally requires more trials than single-objective optimization because it is harder to optimize multiple objective functions simultaneously. Hence, it is necessary to develop more sample-efficient algorithms. By default, NSGAIISampler is used in Optuna for multi-objective optimization, but a recent paper has reported that TPESampler achieves better sample efficiency for about 1,000–10,000 trials [1]. In addition, TPESampler supports features not implemented in NSGAIISampler, such as handling of dynamic search space and user-defined categorical distance; therefore, TPESampler is expected to handle more comprehensive problem setups than NSGAIISampler.
Before v4.0, the multi-objective TPESampler sometimes limits the number of trials during optimization due to the sampler bottleneck after a few hundred trials. Optuna v4.0 drastically improves the sampling speed, e.g., 300 times faster for three objectives with 200 trials, and enables users to handle much more trials.
Please check the blog post for details.
Introduction of a New Terminator Algorithm
Although hyperparameter optimization of machine learning algorithms is essential for better performance, it is known that it causes the overfitting of hyperparameters to the validation dataset for too many trials. To address this problem, Optuna supports Terminator, which terminates the optimization process itself before the hyperparameter overfitting. Users can also opt to run optimization with a fixed budget and then visualize at which trial the overfitting begins based on the Terminator outputs. By doing so, users can gain insight into when trials start to overfit based on the visualization backed by some theoretical foundation.
Optuna Terminator was originally introduced for hyperparameter optimization of machine learning algorithms using cross-validation. To accept broader use cases, Optuna v4.0 introduced the Expected Minimum Model Regret (EMMR) algorithm [2]. Please refer to the EMMREvaluator document for details.
Enhancements of Constrained Optimization
Constrained optimization is a feature supported by Optuna’s core algorithms, such as TPESampler (v3.0 and later) and NSGAIISampler (v2.5 and later), and visualization has been enhanced since v3.3. In v4.0, we have further enhanced features related to constrained optimization.
In v4.0, we added support for constraint optimization of study.best_trial and study.best_trials. They are guaranteed to satisfy the constraints, which was not the case previously.
Removing Deprecated Features
Optuna removes deprecated features in major releases. To prevent users’ code from suddenly breaking, we take a long interval between when a feature is deprecated and when it is removed. By default, features are removed when the major version has increased by two since the feature was deprecated. For this reason, the main target features for removal in v4.0 were deprecated at v2.X. Please refer to the migration guide for the removed features list.
Here, we will introduce the migration of the multi-objective optimization API. Multi-objective optimization, which was introduced as an experimental feature in v1.4, was made stable in v2.4 by unifying the API with single-objective optimization. In v4.0, the experimental feature API introduced in v1.4 has been removed.
Code using the removed API
sampler = optuna.samplers.MOTPESampler()
study = optuna.multi_objective.create_study(
directions=["minimize", "minimize"], sampler=sampler)Code using the current API
# Use TPESampler, which supports multi-objective optimization,
# instead of MOTPESampler.
sampler = optuna.samplers.TPESampler()
# The optuna.multi_objective submodule has been deleted.
# We use the same API as for single-objective optimization.
study = optuna.create_study(
directions=["minimize", "minimize"], sampler=sampler)What’s Ahead
By adding new features and improving stability, Optuna has been developed to make efficient hyperparameter optimization algorithms practical. In v4.0, we added new algorithms, covered a wider range of problem settings, and supported more diverse computational environments. Optuna will still continue to evolve. We will make Optuna applicable to more and more problem settings based on user demands and support new algorithms through OptunaHub. We will continue these efforts in the next v4.1 release.
Contributors and Sponsors
As with any other release, this one would not have been possible without the feedback, code, and comments from many contributors.
@47aamir, @Alnusjaponica, @HideakiImamura, @Obliquedbishop, @RektPunk, @TTRh, @aaravm, @aisha-partha, @alxhslm, @c-bata, @caleb-kaiser, @contramundum53, @eukaryo, @gen740, @kAIto47802, @karthikkurella, @keisuke-umezawa, @keita-sa, @kz4killua, @nabenabe0928, @neel04, @not522, @nzw0301, @porink0424, @sgerloff, @toshihikoyanase, @virendrapatil24, @y0z
Optuna is sponsored by the following sponsors on GitHub.
@AlphaImpact, @dec1costello, @dubovikmaster, @shu65, @raquelhortab
Thanks to those who have followed the projects from the very early days and those who have joined along the way.
Reference
[1] Yoshihiko Ozaki, Yuki Tanigaki, Shuhei Watanabe, Masahiro Nomura, and Masaki Onishi (2022). Multiobjective Tree-Structured Parzen Estimator. Journal of Artificial Intelligence Research (JAIR) 2022. Available from https://doi.org/10.1613/jair.1.13188.
[2] Ishibashi, H., Karasuyama, M., Takeuchi, I. & Hino, H.. (2023). A Stopping Criterion for Bayesian Optimization by the Gap of Expected Minimum Simple Regrets. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:6463–6497. Available from https://proceedings.mlr.press/v206/ishibashi23a.html.
