A Physicist’s View: The Thermodynamics of Machine Learning

Complex systems are ubiquitous in nature, and physicists have found great success using thermodynamics to study these system. Machine learning can be very complex, so can we use thermodynamics to understand it?

What does water going downstream and machine learning have in common? Both are minimizing something, and the analogies run deeper than you might think (Credit: Appolinary Kalashnikova)

As a theoretical physicist turned data scientist, people often ask me how relevant my academic training was. While it is true that my ability to calculate particle interactions and understand the structures of our Universe have no direct relevance in my daily work, the physics intuitions that I learned are of immeasurable value.

Probably the most relatable areas of physics to data science is statistical physics. Below, I’ll share some thoughts on how I connect the dots, and draw inspirations from physics to help me understand an important part of data science — machine learning (ML).

While some of these thoughts below are definitely not fully mathematically rigorous, I do believe some of them are of profound importance in helping us understand the why/how of ML.

Models as Dynamical Systems

One of the key problems of data science is to to predict/describe some quantities using some…