Basic AI Algorithms Explained

7 min readJan 20, 2024

Introduction

The world of artificial intelligence (AI) can seem complex and intimidating, especially for beginners. However, at its core, AI is driven by algorithms — sets of rules or instructions given to a computer to help it learn from data. In this guide, I’ll explain some basic AI algorithms: decision trees, linear regression, and k-nearest neighbors. These algorithms lay the foundation for understanding more complex AI systems.

Decision Trees

A decision tree is a graphical representation of possible solutions to a decision based on certain conditions. It’s like a flowchart where each internal node represents a “test” or “question” on an attribute (e.g., ‘Is it raining?’), each branch represents the outcome of the test (e.g., ‘Yes’ or ‘No’), and each leaf node represents a class label or decision outcome. This structure helps in decision making by breaking down a complex decision into a series of simpler decisions, making it easier to visualize and interpret.

How Decision Trees Work

Selecting the Best Attribute: At each node in the tree, the algorithm selects the attribute that best splits the set of items. This is often done using criteria like Gini impurity or information gain in Classification Trees, or variance reduction in Regression Trees.
Creating Branches for the Attribute: Once an attribute is selected, the dataset is split into subsets that contain possible values for this attribute. This process is recursively repeated on each derived subset.
Terminating the Process: The recursion is terminated once either of the following conditions is met: all tuples in a derived subset belong to the same class, there are no more remaining attributes, or the subset of data points is too small.

Types of Decision Trees

Classification Trees: Used when the target variable is categorical. The outcome is the class to which the data point belongs.
Regression Trees: Used when the target variable is continuous. The outcome is a real number (e.g., price, temperature).

Advantages of Decision Trees

Interpretability: They are simple to understand and interpret, making them valuable in decision-making processes.
No Need for Data Preprocessing: They don’t require normalization of data.
Handle Both Numerical and Categorical Data: They can handle datasets with both types of data.

Disadvantages of Decision Trees

Overfitting: They can create over-complex trees that do not generalize well from the training data.
Instability: Small variations in the data can result in a completely different tree.
Bias: Trees that are deeper tend to be more biased to the training set.

Decision Tree in Action

Imagine you want to decide whether to play tennis based on various weather conditions like outlook (sunny, overcast, rainy), temperature (hot, mild, cool), humidity (high, normal), and windy (true, false).

Starting at the Root: The algorithm might first consider the outlook. If the outlook is overcast, it may lead to a ‘Yes’ decision to play tennis.
Branching Out: If the outlook is not overcast, the next question might be about humidity or wind. Each answer leads to a new branch.
Reaching a Conclusion: Following these branches, based on the combinations of these attributes, you’ll eventually reach a leaf node — the decision of whether to play tennis or not.

Practical Applications

Medical Diagnosis: Helping doctors to diagnose diseases based on symptoms and patient history.
Financial Analysis: Assessing loan applications by considering factors like credit history and income.
Customer Relationship Management: Predicting customer behavior based on past interactions and preferences.

By understanding decision trees, one can gain insights into how structured decision-making happens in machine learning. This method, while simple, lays the groundwork for more advanced techniques in AI and machine learning.

Linear Regression

Linear regression is one of the simplest and most widely used statistical techniques for predictive modeling. It aims to model the relationship between two variables by fitting a linear equation to observed data. The variable we want to predict is called the dependent variable (or sometimes, the outcome, target, or criterion variable), while the variable we are using for making predictions is called the independent variable (or predictor variable).

The Basics of Linear Regression

The Equation: Linear regression fits a line to the data points, which is represented by the equation y = a * x + b, where:

y is the dependent variable,
x is the independent variable,
a is the slope of the line, which represents the relationship between x and y,
b is the y-intercept.

Goal of the Model: The goal is to find the values of a and b that minimize the error in prediction.

Types of Linear Regression

Simple Linear Regression: Involves a single independent variable to predict a dependent variable.
Multiple Linear Regression: Involves two or more independent variables to predict a dependent variable.

Implementing Linear Regression

Let’s take a simple example to understand how linear regression can be implemented in Python using the scikit-learn library.

from sklearn.linear_model import LinearRegression
import numpy as np

# Example data
X = np.array([[1], [2], [3], [4], [5]])  # Independent variable
y = np.array([2, 4, 5, 4, 5])            # Dependent variable

# Creating the Linear Regression model
model = LinearRegression()

# Fitting the model
model.fit(X, y)

# Making predictions
predicted = model.predict(np.array([[6]]))
print("Predicted value for input 6 is:", predicted[0])

In this example, X is the independent variable, and y is the dependent variable. The model learns from this data and can then predict the value of y for a new X.

Benefits of Linear Regression

Simplicity and Interpretability: It’s straightforward to implement, understand, and interpret its results.
Basis for Many Methods: Many other more complex algorithms are built on top of linear regression.

Limitations of Linear Regression

Assumes Linear Relationship: It assumes a linear relationship between the independent and dependent variables, which isn’t always the case.
Sensitive to Outliers: Outliers can significantly impact the regression line and thus the prediction.
Multicollinearity: The model can be unreliable when independent variables are highly correlated with each other.

Applications of Linear Regression

Economics: Predicting GDP growth, unemployment rates, etc.
Business: Estimating sales and revenue projections based on market trends.

Linear regression is a fundamental tool in the data scientist’s toolbox. It’s a first-line approach for modeling relationships between variables and offers a good starting point for predicting outcomes.

K-Nearest Neighbors (KNN)

K-Nearest Neighbors (KNN) is a simple yet versatile algorithm used in statistical learning and machine learning for classification and regression. Unlike other machine learning algorithms, which require training before making a prediction, KNN makes predictions using the entire dataset as its training set, making it a type of ‘lazy learning’.

How KNN Works

Choose the Number of Neighbors (K): The first step in KNN is to choose the number of neighbors, denoted as ‘K’. This number determines how many nearest neighbors of a data point the algorithm will consider while making predictions.
Calculate Distance: For each point in the dataset, KNN calculates the distance between that point and the point we wish to classify. This can be done using various methods, such as Euclidean distance, Manhattan distance, or Hamming distance.
Identify Nearest Neighbors: After calculating distances, the algorithm sorts these distances and picks the top ‘K’ nearest neighbors.
Aggregate Neighbor Information: For classification tasks, KNN looks at the class of the ‘K’ closest points and assigns the class based on a majority vote. For regression tasks, it can take the average of these ‘K’ neighbors.

Choosing the Right Value for K

Too Low K: A very low value of K makes the algorithm sensitive to noise in the dataset.
Too High K: A high value of K makes it computationally expensive and may lead to underfitting.
Finding the Sweet Spot: Often, the square root of the number of data points is used as a starting point.

Advantages of KNN

Simplicity and Effectiveness: KNN is easy to understand and implement.
No Assumptions About Data: It doesn’t assume anything about the underlying data distribution.
Versatility: It can be used for both classification and regression.

Disadvantages of KNN

Computationally Intensive: As the dataset grows, the prediction gets slower.
Sensitive to Irrelevant Features: It can perform poorly if there are many irrelevant features in the data.
Sensitive to the Scale of Data: Features need to be normalized, or else the algorithm might weight some features more heavily than others.

Practical Implementation

Here’s a basic example of KNN implemented in Python using scikit-learn:

from sklearn.neighbors import KNeighborsClassifier
import numpy as np

# Example dataset
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])  # Features
y = np.array([0, 0, 1, 1])                      # Labels

# KNN model
knn = KNeighborsClassifier(n_neighbors=3)

# Training the model
knn.fit(X, y)

# Predicting a new data point
predicted = knn.predict(np.array([[5, 5]]))
print("Predicted class for [5, 5]:", predicted[0])

Applications of KNN

Medical Diagnosis: Classifying patient conditions based on symptoms and past patient data.
Recommendation Systems: Suggesting products or media based on customer similarity.
Financial Fraud Detection: Identifying unusual patterns that indicate fraudulent activities.

KNN’s simplicity and effectiveness make it a popular choice for many practical applications. Its ability to make decisions based on the entirety of the data available to it, without a rigorous training phase, offers a unique approach in the field of machine learning.

Conclusion

In this guide, we’ve explored essential AI algorithms: Decision Trees, Linear Regression, and K-Nearest Neighbors. Each algorithm offers unique insights into AI’s capabilities, from making complex decisions simpler with Decision Trees, predicting relationships through Linear Regression, to the straightforward yet effective approach of K-Nearest Neighbors.

Understanding these fundamental algorithms is crucial for anyone beginning their journey in AI. They are not just tools for data scientists but are stepping stones towards a broader comprehension of how AI shapes our world. As AI continues to evolve, these basics lay the groundwork for more advanced and innovative technologies in the future.