OptKeras: a wrapper around Keras and Optuna

Note: The same content is available at https://github.com/Minyus/optkeras

A Python package designed to optimize hyperparameters of Keras Deep Learning models using Optuna. Supported features include pruning, logging, and saving models.

What is Keras?

Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.

What is Optuna?

Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning.

What are the advantages of OptKeras?

• Optuna supports pruning option which can terminate the trial (training) early based on the interim objective values (loss, accuracy, etc.). Please see Optuna’s key features. OptKeras can leverage Optuna’s pruning option. If enable_pruning is set to True and the performance in early epochs is not good, OptKeras can terminate training (after the first epoch at the earliest) and try another parameter set.
• Optuna manages logs in database using SQLAlchemy and can resume trials after interruption, even after the machine is rebooted (after 90 minutes of inactivity or 12 hours of runtime of Google Colab) if the database is saved as a storage file. OptKeras can leverage this feature.
• More epochs do not necessarily improve the performance of Deep Neural Network. OptKeras keeps the best value though epochs so it can be used as the final value.
• OptKeras can log metrics (loss, accuracy, etc. for train and test datasets) with trial id and timestamp (begin and end) for each epoch to a CSV file.
• OptKeras can save the best Keras models (only the best Keras model overall or all of the best models for each parameter set) with trial id in its file name so you can link to the log.
• OptKeras supports randomized grid search (randomized search by sampling parameter sets without replacement; grid search in a randomized order) useful if your primary purpose is benchmarking/comparison rather than optimization.

How to install OptKeras?

Option 1: install from the PyPI

pip install optkeras

Option 2: install from the GitHub repository

pip install git+https://github.com/Minyus/optkeras.git

Option 3: clone the GitHub repository, cd into the downloaded repository, and run:

python setup.py install

How to use OptKeras?

Please see the OptKeras example available in Google Colab (free cloud GPU) environment.

To run the code, navigate to “Runtime” >> “Run all”.

To download the notebook file, navigate to “File” >> “Download .ipynb”.

Here are the basic steps to use.

Will OptKeras limit features of Keras or Optuna?

Not at all! You can access the full feaures of Keras and Optuna even if OptKeras is used.

What parameaters are available for OptKeras?

• monitor: The metric to optimize by Optuna. ‘val_loss’ in default.
• enable_pruning: Enable pruning by Optuna. False in default. Reference: https://optuna.readthedocs.io/en/latest/tutorial/pruning.html
• enable_keras_log: Enable logging by Keras CSVLogger callback. rue in default. Reference: https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/CSVLogger
• keras_log_file_suffix: Suffix of the file if enable_keras_log is True. ‘_Keras.csv’ in default.
• enable_optuna_log: Enable generating a log file by Optuna study.trials_dataframe(). True in default. Reference: https://optuna.readthedocs.io/en/latest/reference/study.html#optuna.study.Study.trials_dataframe
• optuna_log_file_suffix: Suffix of the file if enable_optuna_log is True. ‘Optuna.csv’ in default.
• models_to_keep: The number of models to keep. Either 1 , 0, or -1 (save all models). 1 in default.
• ckpt_period: Period to save model check points. 1 in default. Reference: https://keras.io/callbacks/#modelcheckpoint
• save_weights_only: if True, then only the model’s weights will be saved (model.save_weights(filepath)), else the full model is saved (model.save(filepath)). False in default. Reference: https://keras.io/callbacks/#modelcheckpoint
• save_best_only: if save_best_only=True, the latest best model according to the quantity monitored will not be overwritten. True in default. Reference: https://keras.io/callbacks/#modelcheckpoint
• model_file_prefix: Prefix of the model file path if models_to_keep is not 0. ‘model_’ in default.
• model_file_suffix: Suffix of the model file path if models_to_keep is not 0. ‘.h5’ in default.
• directory_path: The path of the directory for the files. Current working directory in default.
• verbose: How much info to print onto the screen. Either 0 (no messages), 1 , or 2 (troubleshooting) 1 in default.
• random_grid_search_mode: Run randomized grid search instead of optimization. False in default.
• **kwargs: parameters for optuna.study.create_study(): study_name, storage, sampler=None, pruner=None, direction=’minimize’ Reference: https://optuna.readthedocs.io/en/latest/reference/study.html#optuna.study.create_study

Which version of Python is supported?

Python 3.5 or later

What was the tested environment for OptKeras?

• Google Colaboratory with GPU enabled
• NVIDIA Tesla T4
• Driver Version: 410.79
• CUDA Version: 10.0
• Ubuntu 18.04.1 LTS
• Python 3.6.7
• Keras 2.2.4
• TensorFlow 1.13.1
• Optuna 0.10.0
• OptKeras 0.0.5

About author

Yusuke Minami

• https://github.com/Minyus
• https://www.linkedin.com/in/yusukeminami/
• https://twitter.com/Minyus86

License

MIT License (see https://github.com/Minyus/optkeras/blob/master/LICENSE).