Naive Bayes Algorithm

This blog is a deep dive into understanding Naive Bayes classifier algorithm ,its applications and intricasies about the type of data that this algorithm suitable for.

Before getting into the Naive Bayes there are two types of models one needs to understand to get a comprehensive view of different models in general.

They are deterministic and probabilistic models

Deterministic Models : Deterministic models are mathematical models that have no random or probabilistic components. In a deterministic model, the set of variables describes a system where the current state fully determines the next state through known relationships. That is, there is no inherent randomness or unpredictability in the model

Probabilistic Models: Probabilistic models are mathematical models that incorporate some degree of randomness or uncertainty. They are based on the mathematical theories of probability and statistics.

In a probabilistic model, variables take on probability distributions rather than fixed values.

Naive bayes classifier is one such algorithm where a statistical technique us used to take into account the impact of random events or actions in predicting the potential occurrence of future outcomes.The Naive Bayes algorithm is a classification algorithm that is based on the Bayesian theorem. It uses the probabilities of features to make predictions about the class a particular sample belongs to.

How Naive Bayes Works :

The Naive Bayes algorithm makes the assumption that the presence or absence of a particular feature is unrelated to the presence or absence of other features. This is called the conditional independence assumption. Even though this is often an unrealistic assumption, the Naive Bayes classifier has been found to perform surprisingly well on real-world data

he Naive Bayes algorithm makes the assumption that the presence or absence of a particular feature is unrelated to the presence or absence of other features. This is called the conditional independence assumption. Even though this is often an unrealistic assumption, the Naive Bayes classifier has been found to perform surprisingly well on real-world data

Bayes theorem provides a way of calculating posterior probability P(c|x) from P(c), P(x) and P(x|c). Look at the equation below:

• P(c|x) is the posterior probability of class (c, target) given predictor (x, attributes).
• P(c) is the prior probability of class.
• P(x|c) is the likelihood which is the probability of the predictor given class.
• P(x) is the prior probability of the predictor.

Here is how the Naive Bayes algorithm works for classification:

1. Let D be a dataset containing samples belonging to classes C1, C2, …, Cn
2. For each class, calculate the prior probability P(Ck) by getting the percentage of samples belonging to that class
3. For each feature/attribute f, calculate P(f|Ck) which is the probability of feature f given class Ck
4. To classify a new sample x with features f1, f2,…fn, use Bayes theorem to calculate the posterior probability for each class:

P(Ck|x) = P(Ck) * P(f1|Ck) * P(f2|Ck) * … * P(fn|Ck)

Predict x as belonging to the class Cj for which P(Cj|x) is maximum

The major advantage of Naive Bayes is that it only requires a small amount of training data to estimate the parameters necessary for classification. Since independent variables are assumed, only the variances of the variables for each class need to be determined and not the entire covariance matrix.

Below is a pseudocode snippet for training and testing a Naive Bayes classifier:

The Naive Bayes algorithm has the advantage of being simple to understand and implement while performing classification efficiently. It scales well to large datasets and performs well in multi-class prediction problems. The independence assumption it makes rarely impacts accuracy significantly in practical applications.