# History of Neural Network

**Biological Neuron -Reticular Theory (1871–1873)****Joseph von Gerlach** proposed that the nervous system is a single continuous network as opposed to a network of many discrete cells .

**Neuron Doctrine(1888–1891)**

Santiago Ram ́on y Cajal used Golgi’s technique to study the nervous system and proposed that it is actually made up of discrete individual cells formimg a network(as opposed to a single continuous network)

The term(spelledneuronneuronein British English) was itself coined byWaldeyer-Hartzas a way of identifying the cells in question around1891.

**The Final Word**

In 1950s electron microscopy finally confirmed the neuron doctrine by unambiguously demonstrated that nerve cells were individual cells interconnected through synapses (a network of many individual neurons).

**Era of Artificial Neural Network.**

**McCulloch Pitts Neuron(1943)**

McCulloch (neuroscientist) and Pitts (logician) proposed a highly simplified model of the neuron.

#### Perceptrons (1958)

Perceptron was a form of neural network introduced in **1958** by **Frank Rosenblatt.**

**Preceptrons network introduced Weight and Bias in ANN** . These discovery leads to the stepping stone for upcoming advancement in AI Research.

**Multilayer Perceptrons**

MLP introduced by Ivakhnenko in **1965.** This is perceptrons with hidden layer.

Winter of Neural Network (1969–1986)

Famous book **“Perceptrons”** by **Minsky and Papert** outlined the limits of

what perceptrons could do which leads to **abandonment of connectionist AI**.

Backpropagation (1986) -Rebirth of Neural Network

Basics of continuous backpropagation were derived in the **context of control theory** by Henry J. Kelley in 1960.

Arthur E. Bryson in 1961 used principles of **dynamic programming.**

In 1970 Linnainmaa published the general method for automatic differentiation (AD) of discrete connected networks of **nested differentiable functions** (Base for current Backpropagation algorithm).

In **1986 Rumelhart, Hinton and Williams** showed experimentally that this method can generate useful internal representations of incoming data in hidden layers of neural networks.

**Universal Approximation Theorem(1989)**

The theorem states that simple neural networks can *represent* a wide variety of interesting functions when given appropriate parameters; however, it does not touch upon the algorithmic learnability of those parameters.

First versions of the UAT was proved by **George Cybenko** in 1989 for sigmoid activation functions.

**Convolutional Neural Network(1989)**Handwriting digit recognition using backpropagation over a Convolutional NeuralNetwork (LeCun et. al.)

**CNN** design follows **vision processing** in living organisms

Work by Hubel and Wiesel in the 1950s and 1960s showed that cat and monkey visual cortexes contain neurons that individually respond to small regions of the visual field. Provided the eyes are not moving, the region of visual space within which visual stimuli affect the firing of a single neuron is known as its **receptive field.**

*DEEPER AND DEEPER*

New ERA of Neural Network

2006

**Unsupervised Pre-Training(2006)**

An **autoencoder** is a type of **ANN **used to learn efficient data codings in an unsupervised manner.

The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for the purpose of **dimensionality reduction**.

**Hinton and Salakhutdinov** described an effective way of initializing the weights that allows deep autoencoder networks to learn a low-dimensional representation of data.

New record on MNIST(2011)Ciresan et. al. set a new record on the MNIST dataset using good old back-

propagation on GPUs(GPUs enter the scene)

Winning more visual recognition challenges(2012–2016)