# Genetic algorithms and hyperparameters — Weekend of a Data Scientist

Aug 24, 2018 · 4 min read

Weekend of a Data Scientist is series of articles with some cool stuff I care about. Idea is to spend weekend by learning something new, reading and coding.

Hi! Do you like genetic algorithms as much as I do? I hope so!

This week I wanna share my experience with searching best hyperparameters for Neural Networks and how you can use auto-ML (to some extent) right now!

# What are genetic algorithms?

So a genetic algorithm is a solution to the optimization problem, for example, if you need to find the best set of parameters to minimize some loss function. Genetic algorithms are part of the bigger group of evolutionary algorithms. The idea is inspired by nature and natural selection.

In nature you need at least three requirements for evolution to take place:

1. Inheritance — organisms must be able to transfer some specific characteristics to the offspring;
2. Variation — organisms within the population must have different characteristics;
3. Competition — more offspring are produced than can survive.

Now imagine that instead of organisms you have some objects coded in a specific programming environment. So how would it work if, for example, you will have a population of ML models?

1. Firstly you generate your initial population of ML models and randomly choose hyperparameters.
2. You calculate your loss function for each model, for example, log-loss.
3. Then you choose some amount of models with the lowest error.
4. Now you need to create offspring, so you create a population of new ML models based on the best models from the previous generation and slightly change their hyperparameters. Your new population will be contained from models from the previous population and freshly generated models in some proportion, for example, 50/50.
5. You calculate your loss function, sort your models and repeat the whole process again.

Genetic algorithms are not perfect, you still need to specify you loss-function, your population size, a ratio of offspring with changed parameters and so on.

# How can you use genetic algorithms for hyperparameter tuning?

Hyperparameters are very important, they can have a crucial effect on model performance. It is not easy to find the best set of hyperparameters, because it often can be computationally expensive. Let’s see how we can apply genetic algorithms for hyperparameters tuning. Fortunately, like almost everything else in Python, you don’t need to write this from scratch.

## TPOT

Probably the most user-friendly library for genetic hyperparameter tuning.

First time I used it, it was not obvious to me how to extract the best hyperparameters and write them to some variable. So below is my way of extracting best hyperparameters from TPOT.

# How to tune Neural Network?

In a case with neural networks, you can also use genetic algorithms and they work much faster then grid-search based approach.

I found this GitHub repository-Evolve a neural network with a genetic algorithm and really liked this approach. So I made a wrapper from that and a python package, so you can simply install it with pip and use with whatever data you want (the initial example uses only several hardcoded datasets).

Example of usage:

`from evolution import NeuroEvolutionparams = {    "epochs": [10, 20, 35],    "batch_size": [10, 20, 40],    "n_layers": [1, 2, 3, 4],    "n_neurons": [20, 40, 60],    "dropout": [0.1, 0.2, 0.5],    "optimizers": ["nadam", "adam"],    "activations": ["relu", "sigmoid"],    "last_layer_activations": ["sigmoid"],    "losses": ["binary_crossentropy"],    "metrics": ["accuracy"]}`

`# x_train, y_train, x_test, y_test - prepared datasearch = NeuroEvolution(generations = 10, population = 10, params=params)search.evolve(x_train, y_train, x_test, y_test)100%|██████████| 10/10 [05:37<00:00, 29.58s/it]100%|██████████| 10/10 [03:55<00:00, 25.55s/it]100%|██████████| 10/10 [02:05<00:00, 15.05s/it]100%|██████████| 10/10 [01:37<00:00, 14.03s/it]100%|██████████| 10/10 [02:49<00:00, 22.53s/it]100%|██████████| 10/10 [02:37<00:00, 23.14s/it]100%|██████████| 10/10 [02:36<00:00, 21.37s/it]100%|██████████| 10/10 [01:57<00:00, 18.56s/it]100%|██████████| 10/10 [02:42<00:00, 25.29s/it]"best accuracy: 0.79, best params: {'epochs': 35, 'batch_size': 40, 'n_layers': 2, 'n_neurons': 20, 'dropout': 0.1, 'optimizers': 'nadam', 'activations': 'relu', 'last_layer_activations': 'sigmoid', 'losses': 'binary_crossentropy', 'metrics': 'accuracy'}"# or you can call it withsearch.best_params`

So it can also search for the best number of layers and neurons and wherever hyperparameters you defined.

# Genetic algorithms are cool!

I’m a little bit biased for genetic algorithms, I think that they are very intuitive and yet powerful enough to solve complex computational tasks. But again it is not a silver bullet for every optimization task you might need to solve, in the future I will try to explore this topic in depth.

# References:

NeuroEvolution: https://github.com/subpath/neuro-evolution

# And how do you tune your hyperparameters?

Written by

## Cindicator

#### Cindicator is a fintech company that enables effective asset management through predictive analytics based on Hybrid Intelligence. Here we share our news & views on token economy, smart money, Black Swans, data analysis, AI, Machine Learning, and other topics.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just \$5/month. Upgrade