EXPLAINABLE AI

Theoretical Deep Dive Into Linear Regression

Learn about why linear regression is how it is, and how to naturally extend it in various ways

Photo by Erik van Dijk on Unsplash

Most aspiring data science bloggers do it: write an introductory article about linear regression — and it is a natural choice since this is one of the first models we learn when entering the field. While these articles are great for beginners, most do not go deep enough to satisfy senior data scientists.

So, let me guide you through some unsung, yet refreshing details about linear regression that will make you a better data scientist (and give you bonus points during interviews).

This article is quite math-heavy, so in order to follow, it is beneficial to have some solid foundation with probabilities and calculus.

The Data Generation Process

I’m a big fan of thinking about the data generation process when modeling. People who dealt with Bayesian modeling know what I mean, but for the others: Imagine you have a dataset (X, y) consisting of samples (x, y). Given x, how to get to a target y?

Let us assume that we have n data points and that each x has k components/features.