Genetic Algorithms: Biologically-Inspired Deep Learning Optimization

Recently, there have been significant research advancements in the field of neuroscience, biocomputation, and psychology related to how human-mind process information. In recent years, there has been significant progress in the research to apply bio-inspired techniques in various areas of computer science. The biological explanation of different phenomena has become the centre attraction in building various computational model. Nobel Laureate and Caltech President David Baltimore commented:

“Biology is today an information science. The output of the system, the mechanics of life, are encoded in a digital medium and read out by a series of reading heads. Biology is no longer solely the province of the small laboratory. Contributions come from many directions.”

With the use of bio-inspired computing in the field of Artificial Intelligence, the definition of computer science is changed. Computation is everywhere. Peter J. Denning commented this in one of his papers:

The old definition of computer science — the study of phenomena surrounding computers — is now obsolete. Computing is the study of natural and artificial information processes.

In the domain of Artificial Intelligence, evolutionary algorithms are playing a vital role in optimization problems. Particularly, genetic algorithms are making optimization easy. You can read more about optimization and local search here.

This blog will give you the basic intuitive ideas about genetic algorithms as an optimization technique in the context of deep learning. Here, we presumed that reader has a good background in Machine or Deep Learning domain.

Optimization Problem

A general idea of optimization comes from mathematics that selecting the best element from a set of available alternatives. Given a function f(x), optimization algorithms try to minimize or maximize this function. In the context of deep learning, we can use cost function as f(x) and try to optimize this cost function with well-known algorithms like gradient descent or adam optimization. The basic idea of these algorithms is to find a set of parameters (i.e., weights and bias) and hyperparameters for minimum or maximum of the cost function. However, optimization is an iterative, time-consuming, and laborious task. Machine Learning or Deep Learning mechanisms are a disaster without better optimization.

Genetic Algorithms

The core idea behind the genetic algorithm is “Evolution”. Evolution is a biological phenomena observed in nature. Based on the evolution of biological species, this theory leads us to one fascinating optimization algorithm called evolutionary algorithms, and genetic algorithms (GAs) are part of this.

Before understanding GAs, we need to clear some basic terminology used in explaining the process of GAs.

Basic Terms

• Individual — Any possible solution.
• Population — Group of all individuals.
• Generation — different populations over iterations
• Chromosome — Blueprint for an individual.
• Search Space — All possible solutions to the problem.
• Offspring — generated child which inherit the characteristics of the parents
• Fitness — how fit an individual is (the ability of an individual to compete with other individuals)

The basic idea of GA is to produce the best offspring over different generations via minimizing or maximizing fitness function. GA automatically drives stable solution for equations through evolution, which takes place over a series of discrete generations. Each generation consists of multiple individuals. The GA consists of three major steps to select the best offspring: 1) Variation, 2) Evaluation, and 3) Selection.

Variation

This process is similar to sexual reproduction. Given an individual in the population, we utilize the variation operators to produce its offspring. Specifically, several copies of each individual or parent are created, each of these is modified by different crossover or mutations. In a crossover, more than one parent is selected, and one or more off-springs are produced using the genetic material of the parents. In simple terms, mutation may be defined as a small random tweak in the chromosome, to get a new solution. You can read in detail about crossover and mutation here. Then, each modified copy is regarded as one child.

Evaluation

In a GA, evaluation is the operation of measuring the quality (performance) of individuals. For each child, its performance is evaluated by a fitness function. The definition of fitness function varies from problem to problem.

Selection

This is a very crucial part of the process of GA. The key idea behind the selection is to allow only the fittest offspring for the next production (variation). All children will be selected according to their fitness value, and the worst part is removed means they are killed. The rest remain alive offspring and they evolve to the next iteration (generation).

The overall idea of GA is to evolve the population of individual solutions towards better solutions to an optimization problem. This idea is helpful in solving the traditional deep learning optimization problems. Here, I will provide an intuitive approach to this.

GA for Deep Learning Optimization

The idea behind deep learning optimization is to find the maximum or minimum of the cost function for the best selection of parameters and hyperparameters. To solve this optimization problem, we can devise GA. In this GA, randomly generated weights and bias are considered as an individual. Via using randomization, we will generate different individuals and create a population. For variation, we will use the mutation operator, which consists of different cost functions such as Binary Cross-Entropy (BCE), Mean Square Error (MSE), etc. We will now produce different offspring using these mutation operators. After this, we calculate fitness corresponding to each child. In the deep learning problems, we always minimize the cost function; hence, we will select the offspring with maximum fitness that helps to reduce cost function. We iterate this GA for a number of generation and select the best offspring that can give the optimized output for deep learning architecture.

Remember that the effectiveness of your GA depends on the selection of fitness function.

My next blog will give you some ideas about how to select fitness function, and also provide the implementation of GA for one optimization problem.

Please check out my previous blog on “Machine Learning: Voice assistants can start responding to Whisper”.

If you would like to follow my work on Computer Science and Intelligent Systems, you can check out my Medium and GitHub, as well as other research contributions at my website. You can also tweet at me on Twitter, email me directly, or find me on LinkedIn.