Announcing Optuna 4.2

c-bata
Optuna
Published in
7 min readJan 20, 2025

We are pleased to announce the release of Optuna 4.2! Optuna 4.2 now supports several new optimization algorithms, a gRPC storage proxy for large-scale hyperparameter optimization, and benchmark functions. This blog will walk you through these updates and explain how to utilize the new features. Let’s take a closer look at these new and improved features.

Please try upgrading to the latest versions of Optuna and OptunaHub!

$ pip install -U optuna optunahub

gRPC Storage Proxy for RDBStorage

We’ve made performance improvements for large-scale distributed optimization. For example, our previous release, v4.1, delivered significant performance improvements, as shown in Table 1.

Table 1: Comparison of the RDBStorage’s speed benchmark between v4.0 and v4.1.

However, in cases where the RDB server becomes heavily loaded due to more large-scale distributed optimization that involves a large number of workers, overcoming this problem through query tuning or RDB server parameter tuning can be challenging. The gRPC storage proxy is a feature designed to support such large-scale distributed optimization. The conceptual diagram is shown in Figure 1.

Figure 1: The Conceptual Diagram for Using the gRPC Storage Proxy

As shown in Figure 1, the gRPC storage proxy sits between the optimization workers and the database server, proxying the calls of Optuna’s storage APIs. In large-scale distributed optimization settings where hundreds to thousands of workers are operating, placing a gRPC storage proxy for every few tens can significantly reduce the load on the RDB server which would otherwise be a single point of failure. The gRPC storage proxy enables sharing the cache about Optuna studies and trials, which can further mitigate load.

We conducted performance comparisons with the following setup. We prepared a MySQL server, placed a proxy server every ten for 300 workers to relay the storage API calls, conducted a distributed optimization of 10,000 Trials using the RandomSampler. Table 2 shows the average time taken per worker. In this scenario, we were able to reduce the processing time by about 60%.

Table 2: Speed Comparison of the gRPC Proxy where 300 Workers Are Running

It’s important to note that the gRPC storage proxy could be slower in cases where the number of workers are not so huge as the load on MySQL is not too heavy. Indeed, we confirmed a slowdown of about 1.2–1.3 times with a single worker. Please be aware that the gRPC Storage Proxy is specifically designed to streamline large-scale distributed optimization.

By using the gRPC storage proxy, it is possible to reduce the load on the database server when conducting large-scale distributed optimization. Please refer to the official documentation for further details on how to utilize the gRPC storage proxy.

SMAC3: Random Forest-Based Bayesian Optimization Developed by AutoML.org

SMAC3 [1] is a hyperparameter optimization framework developed by AutoML.org, one of the most influential AutoML research groups. The Optuna-compatible SMAC3 sampler is now available thanks to the contribution to OptunaHub by Difan Deng (@dengdifan), one of the core members of AutoML.org. We can now use the method widely used in AutoML research and real-world applications from Optuna, hopefully reducing the implementational effort even if it is a little. We show the basic usage of SMACSampler from Optuna below:

$ pip install optunahub smac==2.2.0
import optuna
import optunahub
from optuna.distributions import FloatDistribution


def objective(trial: optuna.Trial) -> float:
x = trial.suggest_float("x", -10, 10)
y = trial.suggest_float("y", -10, 10)
return x**2 + y**2


smac_mod = optunahub.load_module("samplers/smac_sampler")
n_trials = 100
sampler = smac_mod.SMACSampler(
{"x": FloatDistribution(-10, 10), "y": FloatDistribution(-10, 10)},
n_trials=n_trials,
)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=n_trials)

Please check the OptunaHub SMAC3 page and try it out!

[1] M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. Benjamins, R. Sass, and F. Hutter. SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization. Journal of Machine Learning Research, 23(54), 1–9.

OptunaHub Now Supports Benchmark Functions

Benchmarking the performance of optimization algorithms is an essential process indispensable to the research and development of algorithms. The newly added benchmark in OptunaHub v0.2.0, released in conjunction with Optuna v4.2, is a new feature for users to conduct benchmarks conveniently.

In the following sample code, we compare and display the performance of the four kinds of samplers using a two-dimensional Sphere function, which is part of a group of benchmark functions widely used in the black-box optimization research community known as Blackbox Optimization Benchmarking (BBOB). The output image is shown in Figure 2.

import optuna
import optunahub

bbob_mod = optunahub.load_module("benchmarks/bbob")
smac_mod = optunahub.load_module("samplers/smac_sampler")
sphere2d = bbob_mod.Problem(function_id=1, dimension=2)

n_trials = 100
studies = []
for study_name, sampler in [
("random", optuna.samplers.RandomSampler(seed=1)),
("tpe", optuna.samplers.TPESampler(seed=1)),
("cmaes", optuna.samplers.CmaEsSampler(seed=1)),
("smac", smac_mod.SMACSampler(sphere2d.search_space, n_trials, seed=1)),
]:
study = optuna.create_study(directions=sphere2d.directions,
sampler=sampler, study_name=study_name)
study.optimize(sphere2d, n_trials=n_trials)
studies.append(study)

optuna.visualization.plot_optimization_history(studies).show()
Figure 2: Result of benchmarking

By using OptunaHub Benchmarks, you can easily benchmark algorithms using various problems. Besides the purposes of research and development of algorithms, this feature can be used for preliminary experiments before tackling real-world problems and as toy problems during algorithm learning, among other use cases. We hope you will find it very useful!

Gaussian Process-Based Bayesian Optimization with Inequality Constraints

Optuna has supported Gaussian process-based Bayesian optimization sampler, i.e., GPSampler, since Optuna 3.6. Please check the past article for more details. Since Gaussian process-based Bayesian optimization is a very popular method in various research fields such as aircraft engineering and materials science, we worked on its extension and adapted GPSampler to constrained optimization [1,2] in Optuna 4.2. We show the basic usage below:

$ pip install optuna==4.2.0
# GPSampler requires scipy and torch along with the Optuna dependencies.
$ pip install scipy
$ pip install torch - extra-index-url https://download.pytorch.org/whl/cpu
import numpy as np
import optuna


def objective(trial: optuna.Trial) -> float:
x = trial.suggest_float("x", 0.0, 2 * np.pi)
y = trial.suggest_float("y", 0.0, 2 * np.pi)
c = float(np.sin(x) * np.sin(y) + 0.95)
trial.set_user_attr("c", c)
return float(np.sin(x) + y)


def constraints(trial: optuna.trial.FrozenTrial) -> tuple[float]:
return (trial.user_attrs["c"],)


sampler = optuna.samplers.GPSampler(constraints_func=constraints)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=50)

We compared the optimization performance between the constrained GPSamplerand Optuna’s default sampler for constrained optimization (TPESampler). The comparison result is shown in Figure 3. Since this problem is with a low budget and a numerical search space, GPSampler outperforms TPESampler. Please try out GPSampler for constrained optimization especially when only a small number of trials are available!

Figure 3: Performance comparison between a constrained GPSampler and the Optuna default sampler for constrained optimization (TPESampler). We used Simulation 2 proposed by J. Gardner et al. [1] and ran the experiments over 10 random seeds. The horizontal axis represents the number of trials and the vertical axis represents the feasible objective value. Since this problem is with a low budget and a numerical search space, GPSampler outperforms TPESampler.

[1] J. Gardner, M. Kusner, ZE. Xu, K. Weinberger, and J. Cunningham. Bayesian optimization with inequality constraints. In International Conference on Machine Learning, 2014.
[2] M. Gelbart, J. Snoek, and R. Adams. Bayesian optimization with unknown constraints. arXiv:1403.5607, 2014.

c-TPE: Support Constrained TPESampler

Although Optuna has supported constrained optimization for TPESampler, which is the default Optuna sampler, since v3.0.0, its algorithm design and performance comparison have not been verified academically. OptunaHub now supports c-TPE [1], which is another constrained optimization method for TPESampler. Importantly, the algorithm design and its performance comparison are publicly reviewed to be accepted to IJCAI, a top-tier AI international conference. Here is the usage example of c-TPE.

$ pip install optunahub
import numpy as np
import optuna
import optunahub


def objective(trial: optuna.Trial) -> float:
x = trial.suggest_float("x", 0.0, 2 * np.pi)
y = trial.suggest_float("y", 0.0, 2 * np.pi)
c = float(np.sin(x) * np.sin(y) + 0.95)
trial.set_user_attr("c", c)
return float(np.sin(x) + y)


def constraints(trial: optuna.trial.FrozenTrial) -> tuple[float]:
return (trial.user_attrs["c"],)


sampler = optunahub.load_module("samplers/ctpe").cTPESampler(
constraints_func=constraints
)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=50)

As shown in Table 3, cTPESampleroutperforms TPESampler. One of the advantages of cTPESampler over, for example, GPSampler is a light-weight sampling, enabling more n_trials. If you need a performant, yet light-weight, sampler for constrained optimization, please use cTPESampler.

Table 3: The comparison between the default constrained TPESampler supported by Optuna and cTPESampler. Each sampler optimized 54 types of 5D constrained optimization problems available in bbob-constrained with the optimization budget of 100 Trials. Each row shows the number of problems where each sampler outperformed the other w.r.t. best_trial.value with the budget of N trials. Note that we compared the medians of best_trial.value over 10 different studies. For example, cTPESampler could outperform TPESampler for 46 problems with a budget of 100 Trials. Please notice that since TPESampler and cTPESampler tied for some problems, the summations of each row do not become 54.

[1] S. Watanabe and F. Hutter. c-TPE: Tree-Structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization. International Joint Conference on Artificial Intelligence (IJCAI), 2023.

What’s Ahead

With the release of Optuna 4.2, we’ve utilized OptunaHub, our feature-sharing platform, to introduce new state-of-the-art algorithms. In addition, we’ve introduced a new feature specifically for users who require large-scale distributed optimization, such as gRPC storage proxy.

As we consistently roll out new features and improvements, we are also starting preparations for our next major release, v5. Please note that v5.0 is still a while away (at least a year). At Optuna, we believe that feedback and feature requests from users are very important. If you have any requests or ideas for v5, we would be delighted to get your feedback on the GitHub Discussion linked below.

https://github.com/optuna/optuna/discussions/5930

Contributors

As with any other release, this one would not have been possible without the feedback, code, and comments from many contributors.

@HideakiImamura, @JLX0, @KannanShilen, @SimonPop, @boringbyte, @c-bata, @fusawa-yugo, @gen740, @himkt, @iamarunbrahma, @kAIto47802, @ktns, @mist714, @nabenabe0928, @not522, @nzw0301, @porink0424, @sawa3030, @sulan, @unKnownNG, @willdavidson05, @y0z

Thanks to those who have followed the projects from the very early days and those who have joined along the way.

Next Step

Check out the release notes for more information. To get the latest news from Optuna, follow us on X.com. For feedback, please file an issue or create a pull request on GitHub.

--

--

c-bata
c-bata

Written by c-bata

Creator of go-prompt and kube-prompt. Optuna core-dev. GitHub: c-bata

No responses yet