Conditional renormalization is an oft-unsung technique powering many recent ML successes; how does it work and where did the idea come… — Batch Normalization, and the zoo of related normalization strategies that have grown up around it, have played an interesting array of roles in recent deep learning research: as a wunderkind optimization trick, a focal point for discussions about theoretical rigor and, importantly, but somewhat more in the sidelines, as a…

This post gives an intuitive explanation of an approach called Natural Gradient, an elegant way to dynamically adjust gradient step size. — I’m going to tell a story: one you’ve almost certainly heard before, but with a different emphasis than you’re used to. To a first (order) approximation, all modern deep learning models are trained using gradient descent. At each step of gradient descent, your parameter values begin at some starting point…

“Graph” is a one of those terms that’s fallen prey to natural language’s tendency to be less precise than its mathematical counterpart. — “Graph” is a one of those terms that’s fallen prey to natural language’s tendency to be less precise than its mathematical counterpart: in everyday parlance, a graph can often be used to mean a plot, a chart, a data visualization more generally. However, from a strictly quantitative perspective, “graph” refers…

In the world of modern machine learning, the convolution operator occupies the strange position: it’s both trivially familiar to anyone… — In the world of modern machine learning, the convolution operator occupies the strange position: it’s both trivially familiar to anyone who’s read a neural network paper since 2012, and simultaneously an object whose deeper mathematical foundations are often poorly understood. In audio processing contexts, you might hear it described as…

Reinforcement Learning strikes me as the wild west of machine learning right now: a place full of drama and progress, with dreams of grand… — Reinforcement Learning strikes me as the wild west of machine learning right now: a place full of drama and progress, with dreams of grand futures hovering on everyone’s horizons. But, also, a place a bit removed from the rules and assumptions governing the worlds of supervised and unsupervised learning. One…

(If you haven’t done so yet, I recommend going back and reading Part 1 of this series on VAE failure modes; I spent more time there… — (If you haven’t done so yet, I recommend going back and reading Part 1 of this series on VAE failure modes; I spent more time there explaining the basics of generative models in general and VAEs in particular, and, since this post will pretty much jump right in where it…

It’s a truth universally acknowledged: that data not in possession of labels must be in want of unsupervised learning. — It’s a truth universally acknowledged: that data not in possession of labels must be in want of unsupervised learning. Glibness aside, it’s commonly understood that supervised learning has meaningful downsides: labels are costly, noisy, and direct your problem towards the achievement of a somewhat artificial goal, rather than simply learning…

The premise of meta learning was an intoxicating one to me, when I first of heard it: the project of building machines that are not only… — The premise of meta learning was an intoxicating one to me, when I first of heard it: the project of building machines that are not only able to learn, but are able to learn how to learn. The dreamed-of aspiration of meta learning is algorithms able to modify fundamental aspects…