BitPost : A brief history of machine learning through time
Late 19th and early 20th century : Modern statistics and decision theory emerged. The technical repertoire in hypothesis testing and binary classification like true positive and false positive were formulated.
1940s : A decade of active research on artificial neural networks, often called connectionism. A 1943 paper by McCulloch and Pitts formalized artificial neurons and provided theoretical results about the universality of artificial neural networks as computing devices. A 1949 book by Donald Hebb pursued the central idea that neural networks might learn by constructing internal representations of concepts.
1950s : Dynamic programming formulated by Bellman in 1954 emerged for optimization problems, where at every time step, we observe data, take an action, and pay a cost. Around the mid 1950s, Rosenblatt had devised a machine for image classification. Equipped with 400 photosensors the machine could read an image composed of 20 by 20 pixels and sort it into one of two possible classes. Rosenblatt developed the Perceptron in 1957 and continued to publish on the topic in the years that followed.
1960s : The research in the decade that followed Rosenblatt’s work had essentially all the ingredients of what is now called machine learning, specifically, supervised learning. The ideas from perceptrons had also solidified into a broader subject called pattern recognition that knew most of the concepts we consider core to machine learning today. The connections with mathematical optimization including gradient descent and linear programming also took shape during the 1960s.
1970s : During this time, artificial intelligence research led to a revolution in expert systems, logic and rule based models that had significant industrial impact were being developed.
1980s : In the late 1980s, the first widely used benchmarks emerged. The dataset-as-benchmark paradigm caught on and became core to applied machine learning research for decades to come. Graduate student David Aha created the UCI machine learning repository that made several datasets widely available via FTP. Aiming to better quantify the performance of AI systems, the Defense Advanced Research Projects Agency (DARPA) funded a research program on speech recognition that led to the creation of the influential TIMIT speech recognition benchmark. By the end of 80s, gradient descent was being used to backpropagate and learn convolutional kernel coefficients for CNNs using images of handwritten letters.
1990s : The start of an era where computers began playing Go and winning the game of chess and powering dexterous manipulation in robotic systems. Researchers realised that reinforcement learning methods were approximation schemes for dynamic programming. Powered by this connection, a mix of researchers from AI and operations research applied neural nets and function approximation to simplify the approximate solution of dynamic programming problems such as infrastructure planning, supply chain management, and the landing of SpaceX rockets. LSTMs were introduced for NLP by Hochreiter and Schmidhuber.
Jump to 00s : The ImageNet moment arrives. Organized from 2010 until 2017, the competition became a striking showcase for performance of deep learning methods for image classification. 1 million images belonging to 1000 different object classes was the basis of the ImageNet Large Scale Visual Recognition Challenge.
Fast forward to today : From ResNets, YOLO, GANs to RNN, Transformers, BERT, GPT like countless amazing architectures are being researched and developed by tech industry giants like Google and Facebook in a frenzy. How? Simple, unlimited data and unlimited resources to train on, which certainly wasn’t a case back then and yet it is only this resourceful history which has brought us here. I was amazed to read about the beginning of what we know as ML today and thought of sharing it with you all. Hope you enjoyed it just as much as I did. Read more in detail at mlstory.org. Do add more timeline in comments if you’re aware of it, and I’ll update this post. Thanks!
You can follow me on Twitter, my work on Github, and connect professionally with me on LinkedIn.
