Artificial Neural Networks (Deep Learning)

ANNs (artificial neural networks, which contemporarily are known as “Deep Learning” — DL) are essentially computer programs loosely inspired by the structure of the biological brain.

The brain is made up of billions of cells called neurons connected via synapses. New observations and experiences not only alter the strength of those synaptic connections, but their accumulation drives convergence of the connections strength — resulting in “learning”.

ANNs simulate these structures in software, pretty much by leverage ‘digital versions’ of neurons, synapses, and connection strengths. By feeding training examples (“experience”) to an ANN, and by adjusting the weights accordingly, an ANN learns complex functions much like a biological brain.

The original ANN built on such biological foundations was the Perceptron. It used two layers of connected (software) neurons and could be taught to perform simple image recognition tasks. Improved understanding of the visual cortex (the portion of the brain focused onto visual processing) led to the development of the Neocognitron, an ANN comprised of stacks of smaller/ simpler layers. The use of multiple layers makes the network “deep” and allows it to perceive the world across multiple levels of abstraction. As a result, the Neocognitron was able to recognize characters in different positions and of various sizes.

Deep ANNs excelled at certain perception tasks, but they were difficult to train. In response, computer scientists developed backpropagation, a technique that trains deep networks by applying calculus to labeled data sets.

By combining deep neural networks and backpropagation started to drive towards very powerful results. In the early nineties, a team led by Yann LeCun (then at AT&T Labs, currently doing research at NYU and Facebook) developed a so-called Convolutional neural network (CNN) trained by backpropagation that was able to read handwritten numbers with 99% accuracy at a 9% rejection rate.

Subsequently, this system found its way into the banking system and processed more than 10% of all the checks in the United States. Despite the early success of ANNs, the AI community in general was still a bit skeptical. Researchers generally preferred other ML approaches and algorithms that were simpler to implement and train (also less demanding from computation standpoint), such as SVMs for vision tasks and HMMs (Hidden Markov Models) for voice, still in the nineties.

In 2012, neural networks began to deliver superior performance results that eclipsed other ML algorithms in both visual and audio applications:

• image classification → DL reduced error rates from 26% to 3%.
• voice recognition → DL reduced error rates from 16% to 6%. (within recent years, DL has surpassed human performance)

Gradually, Google, Facebook, Microsoft, Apple, Baidu, IBM, and other big titans in the tech industry started to gravitate to DL for image and voice recognition. Likewise, AI startups have focused on DL as key enabling technology.

3 factors have contributed to the recent breakthroughs in DL:

• faster processor performance,
• larger data sets, and
• more sophisticated neural nets