Introduction to Basic Machine Learning Algorithms
Machine learning has revolutionized the way we solve complex problems by enabling computers to learn from data and make predictions or decisions without explicit programming. It encompasses a wide range of algorithms that can be categorized into different types based on their functionality and application. In this blog, we will explore some of the basic machine learning algorithms and their use cases.
Traditional Programming vs. Machine Learning
Imagine teaching a child to identify dogs. You show pictures, point out features like floppy ears and wagging tails. This is traditional programming — a set of instructions for specific tasks.
Now, imagine showing the child hundreds of dog and cat pictures without labels. They start noticing patterns — furry creatures with four legs. Slowly, they can distinguish dogs based on these patterns. This is machine learning — the computer learns from data to perform tasks.
The Two Main Learning Styles: Supervised and Unsupervised
Think of machine learning algorithms as having different learning styles:
Supervised Learning: Like a student with a teacher, the algorithm is trained on labeled data. Imagine showing the child pictures labeled “dog” and “cat.” The algorithm learns the patterns that differentiate them and uses this knowledge to classify new, unseen images.
Unsupervised Learning: The algorithm explores unlabeled data on its own, like a child encountering a new toy. The goal is to uncover hidden patterns and group similar data points together.
Let’s see all basic ML algorithm and start our ML journey.
- Linear Regression:
- Type: Supervised learning (Regression)
- Use: Predicts a continuous outcome based on one or more predictor features. Assumes a linear relationship between predictors and the target variable.
class LinearRegression:
def __init__(self,L=0.001,n_iters=1000):
self.L = L
self.n_iters = n_iters
self.weights = None
self.bias = None
def fit(self,X,y):
n_samples,n_features = X.shape
self.weights = np.zeros(n_features)
self.bias = 0
for _ in range(self.n_iters):
y_pred = np.matmul(X,self.weights) + self.bias
dw = (1 / n_samples) * np.matmul(X.T, (y_pred - y))
db = (1 / n_samples) * np.sum(y_pred - y)
self.weights -= self.L * dw
self.bias -= self.L * db
def predict(self,X):
y_approx = np.matmul(X,self.weights) + self.bias
return y_approx2. Linear Kernel Regression:
- Type: Supervised learning (Regression)
- Use: Extends linear regression by using the kernel trick to map input data into a higher-dimensional space, allowing for more complex relationships to be captured.
3. K-NN (K-Nearest Neighbors) Classifier:
- Type: Supervised learning (Classification)
- Use: Classifies data points based on the majority class of their k-nearest neighbors in the feature space.
class KNN:
def __init__(self,K=3):
self.k = k
def fit(self,X,y):
self.X = X
self.y = y
def predict(self,X_test):
return np.array([self._predict(x_test) for x_test in X_test])
def _predict(self,x_test):
distances = [euclidean_distance(x,x_test) for x in self.X]
idx = np.argsort(distances)[:self.k]
labels = [self.y[i] for i in idx]
return Counter(labels).most_common(1)[0][0]4.Logistic Regression:
- Type: Supervised learning (Classification)
- Use: Predicts the probability of a binary outcome. Despite its name, logistic regression is used for classification problems.
class LogisticRegression:
def __init__(self,L=0.01,n_iters=1000):
self.L = L
self.n_iters = n_iters
self.weights = None
self.bias = None
def fit(self,X,y):
n_samples, n_features = X.shape
self.weights = np.zeros(n_features)
self.bias = 0
for _ in range(self.n_iters):
linear_model = np.dot(X,self.weights) + self.bias
y_pred = sigmoid(linear_model)
dw = np.dot(X.T,(y_pred - y))
db = np.sum(y_pred - y)
self.weights -= self.L * dw
self.bias -= self.L * db
def predict(self,x_test):
linear_model = np.dot(x_test,self.weights) + self.bias
y_pred = sigmoid(linear_model)
return [1 if i > 0.5 else 0 for i in y_pred]5.SVM (Support Vector Machine):
- Type: Supervised learning (Classification)
- Use: Classifies data points by finding the hyperplane that best separates different classes. Effective in high-dimensional spaces.
6.K-SVM (Kernel Support Vector Machine):
- Type: Supervised learning (Classification)
- Use: Extends SVM by using the kernel trick to handle non-linear relationships between input features.
7.Naive-Bayes classification:
- Type: Supervised learning (Classification)
- Use: Applies Bayes ’s theorem with “naive” assumption of feature independence to classify data points.
8.Decision Tree:
- Type: Supervised learning (Classification)
- Use: Builds a tree like model by recursively splitting data based on the most significant features.
class DecisionTree:
def __init__(self,min_samples_split=2,max_depth=100,n_feats=None):
self.min_samples_split = min_samples_split
self.max_depth = max_depth
self.n_feats = n_feats
self.root = None
def fit(self,X,y):
self.n_feats = X.shape[1] if not self.n_feats else min(X.shape[1],self.n_feats)
self.root = self._grow_tree(X,y)
def _grow_tree(self,X,y,depth=0):
n_samples, n_features = X.shape
n_labels = len(np.unique(y))
# stopping criteria
if(depth >= self.max_depth or n_labels == 1 or n_features < self.min_samples_split):
leaf_value = Counter(y).most_common(1)[0][0]
return Node(value = leaf_value)
feat_idxs = np.random.choice(n_features,self.n_feats,replace=False)
# greedily select the best split according to information gain
best_feat, best_thresh = self._best_criteria(X, y, feat_idxs)
# grow the children that result from the split
left_idxs, right_idxs = self._split(X[:, best_feat], best_thresh)
left = self._grow_tree(X[left_idxs, :], y[left_idxs], depth + 1)
right = self._grow_tree(X[right_idxs, :], y[right_idxs], depth + 1)
return Node(best_feat, best_thresh, left, right)
def predict(self,X_test):
return np.array([self._traverse_tree(x_test,self.root) for x_test in X_test])
def _traverse_tree(self,x_test,node):
if node.is_leaf_node():
return node.value
if x_test[node.feature] <= node.threshold:
return self._traverse_tree(x_test,node.left)
return self._traverse_tree(x_test,node.right)
def _best_criteria(self,X,y,feat_idxs):
best_gain = -1
split_idx,split_thresh = None, None
for feat_idx in feat_idxs:
X_column = X[:,feat_idx]
thresholds = np.unique(X_column)
for threshold in thresholds:
gain = self._information_gain(y,X_column,threshold)
if gain > best_gain:
best_gain = gain
split_idx = feat_idx
split_thresh = threshold
return split_idx, split_thresh
def _information_gain(self,y,X_column,threshold):
parent_entropy = entropy(y)
left_idxs,right_idxs = self._split(X_column,threshold)
n = len(X_column)
n_l,n_r = len(left_idxs),len(right_idxs)
e_l,e_r = entropy(y[left_idxs]),entropy(y[right_idxs])
child_entropy = (n_l/n)*e_l + (n_r/n)*e_r
return (parent_entropy - child_entropy)
def _split(self, X_column, split_thresh):
left_idxs = np.argwhere(X_column <= split_thresh).flatten()
right_idxs = np.argwhere(X_column > split_thresh).flatten()
return left_idxs, right_idxs