Logistic regression for Machine Learning

Logistic regression is basically a supervised classification algorithm. In a classification problem, the output can take only discrete values (such as pass/fail, win/lose, healthy/sick )for given set of inputs.

It is one of the simplest ML algorithms that can be used for various classification problems such as spam detection, Diabetes prediction, cancer detection, heart attack prediction etc.

Types of logistic regression

• Binary or binomial — In this kind of classification output or dependent variable will have only 2 possible values . For example dependent variable may represent success or failure, yes or no etc.
• Ordinal — In this kind of classification output or dependent variable can have 3 or more than 3 possible ordered types which mean the types have a quantitative significance. For example these variables may represent ‘poor’, ‘good' and ‘excellent’ and each category can have scores like 0,1,2.
• Multinomial — In this kind of classification output or dependent variable can have 3 or more than 3 possible unordered types which mean the types doesn’t have a quantitative significance. For example these variables may represent ‘type 1’ or ‘type 2’ or ‘type 3’.

Logistic Function

The logistic function, also called the sigmoid function it is a S-shaped curve that can take any real-valued number and map it into a value between 0 and 1, but never exactly at those limits.

Logistic function: 1 / (1 + e^-value)

Cost function

We can not use MSE in logistic regression because our prediction function is non-linear (due to sigmoid transform). In logistic regression we use a logarithmic loss function to calculate the cost for misclassifying.

Gradient descent

To minimize our cost, we use Gradient Descent. There are other more sophisticated optimization algorithms out there such as conjugate gradient like BFGS, but you don’t have to worry about these. Machine learning libraries like Scikit-learn hide their implementations so you can focus on more interesting things!

Representation for logistic regression

Logistic regression uses an equation as the representation, very much like linear regression.

Below is an example logistic regression equation:

y = e^(b0 + b1*x) / (1 + e^(b0 + b1*x))

Where y is the predicted output, b0 is the bias or intercept term and b1 is the coefficient for the single input value (x). Each column in your input data has an associated b coefficient (a constant real value) that must be learned from your training data.

Code

Before moving to the implementation of logistic regression we have to preprocess the data which includes cleaning, Instance selection, normalisation, transformation, feature extraction and selection, etc.

Splitting dataset into training data and testing data.

Training logistic regression model and predicting on test data.

Thank you for reading.