Batch Normalization, and the zoo of related normalization strategies that have grown up around it, have played an interesting array of roles in recent deep learning research: as a wunderkind optimization trick, a focal point for discussions about theoretical rigor and, importantly, but somewhat more in the sidelines, as a…

I’m going to tell a story: one you’ve almost certainly heard before, but with a different emphasis than you’re used to. To a first (order) approximation, all modern deep learning models are trained using gradient descent. At each step of gradient descent, your parameter values begin at some starting point…

If you followed any machine learning news towards the end of last year, you probably saw images like this circulating, the results of a new Generative Adversarial Network (GAN) architecture specified by a recent paper by a team at NVIDIA: Hearing that jaw-dropping results are being produced by some novel…

“Graph” is a one of those terms that’s fallen prey to natural language’s tendency to be less precise than its mathematical counterpart: in everyday parlance, a graph can often be used to mean a plot, a chart, a data visualization more generally. However, from a strictly quantitative perspective, “graph” refers…

In the world of modern machine learning, the convolution operator occupies the strange position: it’s both trivially familiar to anyone who’s read a neural network paper since 2012, and simultaneously an object whose deeper mathematical foundations are often poorly understood. In audio processing contexts, you might hear it described as…

Reinforcement Learning strikes me as the wild west of machine learning right now: a place full of drama and progress, with dreams of grand futures hovering on everyone’s horizons. But, also, a place a bit removed from the rules and assumptions governing the worlds of supervised and unsupervised learning. One…

(If you haven’t done so yet, I recommend going back and reading Part 1 of this series on VAE failure modes; I spent more time there explaining the basics of generative models in general and VAEs in particular, and, since this post will pretty much jump right in where it…

It’s a truth universally acknowledged: that data not in possession of labels must be in want of unsupervised learning. Glibness aside, it’s commonly understood that supervised learning has meaningful downsides: labels are costly, noisy, and direct your problem towards the achievement of a somewhat artificial goal, rather than simply learning…

The premise of meta learning was an intoxicating one to me, when I first of heard it: the project of building machines that are not only able to learn, but are able to learn how to learn. The dreamed-of aspiration of meta learning is algorithms able to modify fundamental aspects…