Normal Equation for Linear Regression
Frequently Asked FAMGA Applied Scientist Interview Question
Kyle is interviewing Cartman for a data scientist position. Cartman is a graduate student studying computer science at Stanford.
Kyle: Ok, let’s start with an easy question. Can you derive the linear regression from scratch? Let’s say we have about 30 data points on the price of houses (y) and the size of houses (x).
Cartman: Sure! I will do this… (Starts writing on the whiteboard)
Kyle: Yes, so far so good. Now, how are you going to solve this?
Cartman: I’ll use the Gradient Descent!
Kyle: Hmm… Why? 🤔
Cartman: Because that’s how we solve the optimization problem.
Kyle: Yes, you are right, but can you think of any other way to solve this problem besides using Gradient Descent? For Linear Regression, personally I would never use Gradient Descent. Can you guess the reason why?
Cartman: …… 😶
In this case, you don’t have to use the gradient descent method because there is a closed-form solution for Linear Regression, also called the Normal Equation.
When asked how they would optimize anything, some applicants will immediately…
