Why Logistic Regression?
Embarking on the journey of understanding logistic regression, I found myself pondering a series of questions that lingered persistently in my mind. This time, I decided to delve into these inquiries, seeking clarity and unraveling the complexities surrounding logistic regression.
In summary, this article covers:
- Why linear regression can’t be used for classification problems?
- What is a logistic function?
- How does Logistic regression model the relationship between a response variable and predictor variables?
Why Linear regression can’t be used in classification?
The linear regression model assumes that the response variable is continuous or quantitative. Suppose we want to classify a customer into class1, class2, or class3. Now for linear regression to work, we build a mapping as below:
Y = 1 if class1; 2 if class2 ;3 if class3
Using the above mapping, we build a linear regression model. Unfortunately, the above mapping assumes that the difference in class1 and class2 is equal to class2 and class3 and has inherent ordering in classes. In case someone has chosen a different mapping and produced results that would have resulted in totally different results.
However, if the situation is a binary outcome i.e. Y = 0 or 1, then the above mapping would have worked. The only challenge, in that case, would have been that some of our estimates for the outcome variable, Y, might been outside that range of [0,1] interval, making them hard to interpret as probabilities.
As you can see in Figure 1, some estimated probabilities are negative.
Why Logit Function?
To avoid the above problem shown on the left side of Figure 1, we must use a transformation such that estimated probabilities are in the range of [0,1] for all values of X. Many functions meet the description. However, for logistic regression, we use the logistic or sigmoid function.
How does Logistic Regression model the relationship between outcome and predictor variables?
The logistic model relates the probability that an outcome i.e. pi to the predictors x1, i,x2, i….,xk, i much like that of multiple regression. k here is the number of predictor variables.
If we model outcome variables directly, without any transformation, some of the observations might have pi <0 and pi >1.
To avoid this problem we are using the logit function as shown below.
By replacing the LHS of Equation 1 with Equation 2, we get Equation 3.
If we rearrange equation 3, we will get equation 4.
Now different coefficients will be calculated using the maximum likelihood function which will be covered in the next article.
