# Understanding Logistic Regression using Log Odds

I had to adjust my thinking when it comes to logistic regression because it models a probability rather than a mean and it involves the non-linear transformation. In this article, I will explain the log odds interpretation of logistical regression in math, and also run a simple logistical regression model with real data.

The odds of an event are the probability of an event that it happens over the probability that it doesnโt. For example, if the P (success) = 0.8, and P (failure) = 0.2, the odds of success will be 0.8/0.2=4.

We use logistic regression to model a binary outcome variable (y is either 0 or 1). Similar to a Bernoulli random variable, we want to consider the ๐ (๐ฆ=1|๐ฅ) and ๐ (๐ฆ=0|๐ฅ). Also, we know that the general linear model specification: ๐ธ(๐ฆ|๐ฅ)=๐(๐ฅโฒ๐ฝ), we can derive the conditional mean in the case of linear regression to be:

**๐ธ(๐ฆ|๐ฅ)=๐ (๐ฆ=1|๐ฅ)ร1+๐ (๐ฆ=0|๐ฅ)ร0=๐ (๐ฆ=1|๐ฅ)**

Therefore, the expectation we are modeling is a probability: **๐ (๐ฆ=1|๐ฅ). **However, if we model it with a linear combination of the independent variable and parameters:๐(๐ฆ=1|๐ฅ)=๐ฅโฒ๐ฝ, it does not work because probability should be bounded between 0 and 1.

Therefore, we choose a link function ๐(๐ฅโฒ๐ฝ) to give values between zero and one. And for logistic regression, the link function we use is a ** logit **link function:

Because the underlying model is a probability, we use maximum likelihood estimation for logistic regression. To construct a likelihood function, it is the same form as the Bernoulli random variable.

Even though logistic regression is mainly used for classification and prediction in machine learning, for the sake of completing this article about using the log odds to interpret logistic regression, I ran a simple logistic regression in Python to get a sense of what the results look like. I used a dataset that contains 4,000+ emails. It includes 57 variables which are features indicating indicators whether an email is spam or not, for example, *word_free* variable indicates the email contains the keyword โfreeโ. The *spam* variable is a binary variable showing each email has been tagged as spam or not. I ran the simple code below and printed out the results:

The coefficient of world_free is 1.55. Thus, the odds that an email is spam increase almost exp(1.55) ~= 5 times if that email contains the word free. Itโs worth noting that the summary table above also nicely provides p-value and confidence interval (95%). I would like to dig deeper into how Pythonโs statsmodels library computes the standard error and p-values for my curiosity. Perhaps I will write another article about it later :).

Lastly, I will quickly go over the key assumptions of logistic regression as I did for the OLS linear regression in this article. It is needless to say that knowing the key assumptions underlying the method is important. Also, โassumptions about logistic and linear regressionโ has been one of the top questions in the data scientist interview.

- The outcome is a binary or dichotomous variable
- Linearity of independent variables and log odds (we already derived log(P/1โP)) = ๐ฅโฒ๐ฝ in the section above)
- No perfect collinearity among covariates
- the observations are independent of each other

reference: Taddy, Matt. *Business Data Science: Combining Machine Learning and Economics to Optimize, Automate, and Accelerate Business Decisions, August 21, 2019.*