Introduction
Hello everyone. I am Hideaki Imamura, a committer to Optuna.
Optuna is a software being developed as an open-source software for black-box optimization. As our new challenge towards the Optuna ecosystem, we recently released the beta version of OptunaHub, a feature sharing platform for Optuna. Despite being newly public, OptunaHub already has many features registered by a number of contributors, thanks to our supportive community. Today, I would like to pick up and introduce one of these features, the CatCmaSampler, which is CMA-ES for mixed search spaces with continuous, discrete, and categorical variables.
TL;DR
- The cutting-edge evolutionary computation algorithm CatCMA has been published on OptunaHub. Check it out on the individual page.
- CatCMA is an algorithm that excels in mixed search spaces with continuous and discrete variables.
- We have confirmed that it outperforms CMA-ES in real-world hyperparameter optimization of neural networks.
CMA-ES and mixed search space
Have you heard about the Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES)? Introduced in 1996 by Nikolaus Hansen and Andreas Ostermeier, CMA-ES is a black-box optimization method in the field of evolutionary computation. While it is limited to a continuous search space, it is renowned for its fast operation and good performance. We won’t be explaining CMA-ES in detail in this blog post, but if you’re interested, please check out this survey paper.
However, the limitation of CMA-ES to continuous search spaces was a significant drawback. Many real-world black-box optimization tasks involve search spaces including integer or categorical variables (discrete values without order). Typical examples include hyperparameter optimization of machine learning models. The bagging_freq parameter of the LightGBM model, which determines how often bagging is performed, is an integer, and the parameter determining whether to use conventional Gradient Boosting Decision Tree or DART as a booster in the XGBoost model is categorical. Applying CMA-ES directly to hyperparameter optimization with such search spaces is admittedly challenging.
In 2022, CMA-ES with Margin, a method capable of handling both continuous and integer variables simultaneously, was proposed to apply CMA-ES to objective functions with mixed search spaces. However, it was not designed to handle categorical variables. Then, in July 2024, at the international conference GECCO, CatCMA was proposed as a method applicable to search spaces including both continuous and categorical variables [1].
Optuna’s CMA-ES is implemented using the Python library developed by CyberAgent. This CMA-ES implementation is a lightweight and fast library, making it ideal for CatCMA. Using this implementation, with the help of Masahiro Nomura, one of the authors of the paper, CatCMA was quickly added to OptunaHub! Thanks to this contribution, Optuna can now apply CMA-ES to any mixed search space, and it can handle a wide range of real-world black box optimization applications in combination with various existing features in Optuna.
To use it, it’s as simple as instantiating it through OptunaHub and passing it to Optuna’s Study creation process, as shown below:
# Install OptunaHub and cmaes library if not already installed.
# pip install optunahub "cmaes>=0.11.0"
import optuna
import optunahub
mod = optunahub.load_module(package="samplers/catcma")
sampler = mod.CatCmaSampler()
study = optuna.create_study(sampler=sampler)The power of CatCMA
We have tested whether the CatCMA introduced in OptunaHub can effectively perform hyperparameter optimization of machine learning that includes continuous variables and categorical variables.
In this experiment, we will optimize hyperparameters such as the type of activation function, the number of hidden layer units, and the initial learning rate (categorical variables, integer variables, and continuous variables, respectively) for the training of a 3-layer neural network.
For the benchmark problem, we used a problem related to tabular data included in the HPOBench published by the AutoML.org group. Specifically, we used the NavalPropulsion problem from the tabular_benchmarks(+) listed here. This problem has the following nine optimizable hyperparameters:
For this problem, we performed optimization using random search, the regular CMA-ES (the method of rounding off after sampling for discrete values, and random searching for categorical variables), and the newly introduced CatCmaSampler. We performed optimization for 1000 trials 10 times, and the graph plotting the mean and variance is as follows.
The x-axis represents the number of trials (marked as Budget), and the y-axis denotes the best (minimum) mean square error among the trials up to that point. In the 10 optimizations performed, it is evident that CatCMA demonstrates higher performance than regular CMA-ES.
The following graph presents the progress of the execution time of each trial. The x-axis represents the number of trials, and the y-axis shows the execution time of that trial in seconds. For this experiment, we used a benchmark dataset assuming that the time taken to evaluate the objective function is nearly zero, so these execution times represent how long CMA-ES and CatCMA take to sample parameters for each trial. As can be seen, the sampling speed of CatCMA hardly changes compared to CMA-ES.
Conclusion
We have shown that the CatCmaSampler, which was just recently introduced to OptunaHub, Optuna’s feature sharing platform for Optuna, can deliver very high performance even in actual hyperparameter optimization of machine learning. Please feel free to use the CatCmaSampler and try to solve real-world black box optimization application problems!
OptunaHub, the feature sharing platform for Optuna, is looking for OSS contributors to add new features. We look forward to collaborating with researchers who want to spread their methods to the world and developers who want to share the algorithms they have implemented with the Optuna community. For more details, please refer to this beta release introduction blog.
References
[1] Ryoki Hamano, Shota Saito, Masahiro Nomura, Kento Uchida, and Shinichi Shirakawa. CatCMA: Stochastic Optimization for Mixed-Category Problems. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO ’24), 2024. https://arxiv.org/abs/2405.09962
