Latent Variables Probabilistic Modeling

Over the past year, I have taken more and more interest in Bayesian statistics and probabilistic modeling. Along this journey, I have encountered the latent probabilistic models.

We will start by explaining the concept of a latent variable. And to properly understand its benefits we need to make sure that you are familiar with the Bayesian framework for linear regression and this is what this first article is about.

As a quick note, I am no expert in Bayesian statistics. But I wanted to share the current state of my knowledge because I know it can help some people (myself included). Let’s dive in, shall we!

This post was inspired by this excellent course on Coursera: Bayesian Methods for machine learning. If you are into machine learning I definitely recommend this course.

The Bayesian framework for linear regression

In a typical regression problem, a response variable y is modeled as a weighted linear combination of predictors and can be expressed for example with 7 predictors using the following formula:

Or in plain matrix notation:

In the frequentist paradigm we find the vectors of weights w with maximum likelihood estimation (MLE) by computing the total sum of squares and resolving the following optimization problem:

There exist a closed-form solution to this problem which can be expressed as: