“Deep Learning est mort. Vive Differentiable Programming!”
Recently, Tesla’s Director of AI, Andrej Karpathy, wrote a blog post making the case for a new kind of software that represented a fundamental shift in how we see and develop technology. Challenging the common view that neural networks are just another tool in the machine learning toolbox, Karpathy asserts that “neural networks are not just another classifier, they represent the beginning of a fundamental shift in how we write software. They are Software 2.0.”
Yann LeCun, Director of Facebook AI Research and the inventor of convolutional neural networks, took it a step further in a Facebook post over the weekend. He argues that Deep Learning has out-lived his usefulness as a buzz-phrase, and that it’s time for a rebranding to reflect the development of “a new kind of software” that is differentiable and optimizable. “Deep Learning est mort. Vive Differentiable Programming!”, LeCun proclaims resoundingly.
Yeah, Differentiable Programming is little more than a rebranding of the modern collection Deep Learning techniques, the same way Deep Learning was a rebranding of the modern incarnations of neural nets with more than two layers.
The important point is that people are now building a new kind of software by assembling networks of parameterized functional blocks and by training them from examples using some form of gradient-based optimization….It’s really very much like a regular program, except it’s parameterized, automatically differentiated, and trainable/optimizable.
Semantics, semantics, semantics, right? It’s just a word. That was my first impression as well. But it’s not the word that matters; rather, it’s the broadening of research and engineering understanding from deep neural networks to an entire field of technology built on top of our existing software stack that is parameterized, differentiable, and optimizable. The term “Deep Learning” is fuzzy, limiting, and not particularly useful. When every neural network becomes a deep neural network, what exactly does “deep learning” convey? Hype, and nothing else.
One commenter on his Facebook post argued that it should just be called “machine learning”, to which LeCun retorted, “Why don’t we just write F(x)=0 and declare that every theory is merely a special case of that?” In other words, machine learning is too general of a word, and does not fully capture the essence of the computational innovations on the horizon.
It’s likely the term “differentiable programming” won’t catch on, especially given the potential confusion with “differential programming” (the word is so long, I couldn’t even list it as a Medium tag for this post). But it’s not the specific word that’s important; rather, it’s the fundamental shift in mentality. It’s the understanding that machine learning is no longer just one more tool in the toolbox — it is the new technological frontier that will expand how we develop technology, as well as what technology will accomplish, in the coming decades.
