Genetic algorithms and hyperparameters — Weekend of a Data Scientist

Alexander Osipenko
Aug 24, 2018 · 4 min read

Weekend of a Data Scientist is series of articles with some cool stuff I care about. Idea is to spend weekend by learning something new, reading and coding.

Hi! Do you like genetic algorithms as much as I do? I hope so!

This week I wanna share my experience with searching best hyperparameters for Neural Networks and how you can use auto-ML (to some extent) right now!

What are genetic algorithms?

So a genetic algorithm is a solution to the optimization problem, for example, if you need to find the best set of parameters to minimize some loss function. Genetic algorithms are part of the bigger group of evolutionary algorithms. The idea is inspired by nature and natural selection.

In nature you need at least three requirements for evolution to take place:

  1. Inheritance — organisms must be able to transfer some specific characteristics to the offspring;
  2. Variation — organisms within the population must have different characteristics;
  3. Competition — more offspring are produced than can survive.

Now imagine that instead of organisms you have some objects coded in a specific programming environment. So how would it work if, for example, you will have a population of ML models?

  1. Firstly you generate your initial population of ML models and randomly choose hyperparameters.
  2. You calculate your loss function for each model, for example, log-loss.
  3. Then you choose some amount of models with the lowest error.
  4. Now you need to create offspring, so you create a population of new ML models based on the best models from the previous generation and slightly change their hyperparameters. Your new population will be contained from models from the previous population and freshly generated models in some proportion, for example, 50/50.
  5. You calculate your loss function, sort your models and repeat the whole process again.

Genetic algorithms are not perfect, you still need to specify you loss-function, your population size, a ratio of offspring with changed parameters and so on.

How can you use genetic algorithms for hyperparameter tuning?

Hyperparameters are very important, they can have a crucial effect on model performance. It is not easy to find the best set of hyperparameters, because it often can be computationally expensive. Let’s see how we can apply genetic algorithms for hyperparameters tuning. Fortunately, like almost everything else in Python, you don’t need to write this from scratch.

Probably the most user-friendly library for genetic hyperparameter tuning.

GitHub: https://github.com/EpistasisLab/tpot

Image for post
Image for post
animation from TPOT GitHub

First time I used it, it was not obvious to me how to extract the best hyperparameters and write them to some variable. So below is my way of extracting best hyperparameters from TPOT.

How to tune Neural Network?

In a case with neural networks, you can also use genetic algorithms and they work much faster then grid-search based approach.

I found this GitHub repository-Evolve a neural network with a genetic algorithm and really liked this approach. So I made a wrapper from that and a python package, so you can simply install it with pip and use with whatever data you want (the initial example uses only several hardcoded datasets).

Example of usage:

from evolution import NeuroEvolution

params = {
"epochs": [10, 20, 35],
"batch_size": [10, 20, 40],
"n_layers": [1, 2, 3, 4],
"n_neurons": [20, 40, 60],
"dropout": [0.1, 0.2, 0.5],
"optimizers": ["nadam", "adam"],
"activations": ["relu", "sigmoid"],
"last_layer_activations": ["sigmoid"],
"losses": ["binary_crossentropy"],
"metrics": ["accuracy"]
}
# x_train, y_train, x_test, y_test - prepared data

search = NeuroEvolution(generations = 10, population = 10, params=params)

search.evolve(x_train, y_train, x_test, y_test)


100%|██████████| 10/10 [05:37<00:00, 29.58s/it]
100%|██████████| 10/10 [03:55<00:00, 25.55s/it]
100%|██████████| 10/10 [02:05<00:00, 15.05s/it]
100%|██████████| 10/10 [01:37<00:00, 14.03s/it]
100%|██████████| 10/10 [02:49<00:00, 22.53s/it]
100%|██████████| 10/10 [02:37<00:00, 23.14s/it]
100%|██████████| 10/10 [02:36<00:00, 21.37s/it]
100%|██████████| 10/10 [01:57<00:00, 18.56s/it]
100%|██████████| 10/10 [02:42<00:00, 25.29s/it]

"best accuracy: 0.79, best params: {'epochs': 35, 'batch_size': 40, 'n_layers': 2, 'n_neurons': 20, 'dropout': 0.1, 'optimizers': 'nadam', 'activations': 'relu', 'last_layer_activations': 'sigmoid', 'losses': 'binary_crossentropy', 'metrics': 'accuracy'}"

# or you can call it with

search.best_params

So it can also search for the best number of layers and neurons and wherever hyperparameters you defined.

Genetic algorithms are cool!

I’m a little bit biased for genetic algorithms, I think that they are very intuitive and yet powerful enough to solve complex computational tasks. But again it is not a silver bullet for every optimization task you might need to solve, in the future I will try to explore this topic in depth.

References:

TPOT: https://github.com/EpistasisLab/tpot

NeuroEvolution: https://github.com/subpath/neuro-evolution

And how do you tune your hyperparameters?

Cindicator

Cindicator is a fintech company that enables effective…

By Cindicator

News from Cindicator and its Hybrid Intelligence ecosystem. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Alexander Osipenko

Written by

Leading/Coaching/Building Data Science teams from scratch

Cindicator

Cindicator is a fintech company that enables effective asset management through predictive analytics based on Hybrid Intelligence. Here we share our news & views on token economy, smart money, Black Swans, data analysis, AI, Machine Learning, and other topics.

Alexander Osipenko

Written by

Leading/Coaching/Building Data Science teams from scratch

Cindicator

Cindicator is a fintech company that enables effective asset management through predictive analytics based on Hybrid Intelligence. Here we share our news & views on token economy, smart money, Black Swans, data analysis, AI, Machine Learning, and other topics.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store