Neural networks are one of the greatest innovations in modern machine learning, with demonstrated abilities to produce mind-boggling imagery, meaningful text, decent translations, and numerous other feats that were inconceivable even a decade ago. Inasmuch as neural network practitioners care about theory — theoretical understanding of neural networks is notoriously…