Dropout | one minute summary

A regularization strategy motivated by a theory of the role of sex in evolution

Image modified from Srivastava, Nitish, et al. “Dropout: a simple way to prevent neural networks from overfitting.” The journal of machine learning research 15.1 (2014): 1929–1958.

The purpose of dropout is alluded to in the title of its 2014 paper by Srivastava et al: “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”. It has since become one of the most popular regularization techniques.

Prerequisites: Regularization

1. Why? Theoretically, a great way to make predictions with a given model is to train many versions of it separately and then average the outputs. However, this is a really computationally expensive idea.
2. What? Dropout is a regularization technique that prevents overfitting by approximating training multiple different neural network architectures in parallel.
3. How? During training, some hidden units are randomly ignored (“dropped out”). For each training pass, different hidden units are randomly dropped, yielding a different “thinned” network each time. This makes the network more robust by preventing units from co-adapting (correcting each other’s mistakes), and therefore helps prevent overfitting. There is no dropout during testing, and thus the weights are scaled down to compensate for more nodes being active.