Building a Learning Vector Quantization (LVQ1) Network From Scratch with Python

9 min readApr 9, 2023

Learning Vector Quantization (LVQ)

Building a Learning Vector Quantization 1 (LVQ1) network for classification is a powerful machine learning technique that can be used to classify data into binary or multiple categories. LVQ1 is a supervised learning algorithm that is based on the concept of competitive learning similar to Self-Organizing Map, where the network learns to classify data by assigning each data point to the closest prototype vector, or codebook vector, that represents one of the target classes. In this way, LVQ1 is able to learn from labeled training data and can generalize to classify new, unseen data points.

In this tutorial, we will learn how to build an LVQ1 network from scratch using Python (numpy for calculations and Pandas for loading the data). We will begin by exploring the basic principles of LVQ1, and then move on to implementing the algorithm step-by-step in Python. By the end of this tutorial, you will have a working LVQ1 model that is capable of classifying data, and you will have gained valuable insight into the inner workings of this powerful machine learning algorithm.

LVQ1 Algorithm

The motivation for the algorithm of LVQ is to find the output unit that is closest to the input vector.

Step0: Initialize prototype vector and learning rate.

Step1: While stopping condition = false, repeat step2 to step6

Step2: For each training input vector x, do step3 and step4

Step3: Find the winner neuron J such that || x-w_i ||=min

Step4: Update w_j as follows:

If the class of the winner neuron J matches the class of x, update its weight vector:

w_j = w_j + α * (x — w_j)

Otherwise,

w_i = w_i — α * (x — w_i).

Step5: Reduce the learning rate (alpha)

Step6: Test stopping condition.

The condition may specify a fixed number of iterations or the learning rate reaching a sufficiently small value.

Building an LVQ1 Network

Import Libraries

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

Load The Data

I have loaded a small sample dataset consisting of feature1, feature2, feature3, and output. The output column is the target variable we aim to predict.

the dataset I used can be found here!

# load dataset
loaded_data = pd.read_csv('/content/data.csv')

# extract features
features = loaded_data.columns[loaded_data.columns != 'y']

# split the data set into sampels(features) and labels
samples =  loaded_data[features].values.tolist()
labels  =  loaded_data['y'].tolist()

Initiate Prototype Vectors

Create an empty array to store the weights. As we are performing binary classification (1 or 0), we only need to compute two prototype vectors. Each weight vector will correspond to one of the possible class values: one weight vector will have a value of 1, and the other will have a value of 0.

Then, choose only the feature columns from the DataFrame. For each prototype vector. Here we will take the features of the first two prototype vectors and remove them with their labels from the dataset.

Finally, we will append these two prototype vectors into the weights array.

# initiate prototype vectors
# create an array for weights
weights = []
# append the first two sampels into it 
# and remove them from the training samples
prototype1 = samples.pop(0)
weights.append(prototype1)
prototype2 = samples.pop(0)
weights.append(prototype2)
# remove thier labels from the training labels
labels.pop(0)
labels.pop(0)
# remove them from the data frame
train_data = loaded_data.iloc[2:]

Plot The Data In 3D Before Training

Now, Let’s take a look at the data. Since each sample has 3 features, we will plot the data in 3D.

# plot samples in 3D
# make a figure
fig=plt.figure(figsize=(8,8))
# make a 3D subplot 
ax=fig.add_subplot(111,projection='3d')
# give each class a color 
# class 1 ==> green
# class 0 ==> blue
c_colors = np.where(train_data['y']==1,'g','b')
# plot the training samples as circles
ax.scatter(train_data['x1'],train_data['x2'],train_data['x3'],color=c_colors, )
# plot the 1st sample as a blue triangle 
ax.scatter(weights[0][0], weights[0][1], weights[0][2], s=40, color='b', marker='^', label = 'class 0')
# plot the 1st sample as a green triangle
ax.scatter(weights[1][0], weights[1][1], weights[1][2], s=40, color='g', marker='^', label = 'class 1')
ax.legend(loc="best")

The data points corresponding to class 0 will be displayed in blue, while the ones corresponding to class 1 will be shown in green. Additionally, the two prototype vectors that we have selected will be represented as triangles.

LVQ1 Class

We have implemented the LVQ class as follows: each object of this class is implemented by passing the values of epochs and learning rate (alpha). This class has three functions, including:

The “winner” function, which takes the weights matrix (which contains the two prototype vectors) and a sample as inputs and calculates the Euclidean distance between the input sample and class 1 and class 2 using the “norm.linalg” function. Then, it returns the label of the closest class.
The “update” function, which takes the weight matrix, a sample, and the current label of the sample as inputs. If the obtained label is equal to the current label of the prototype class, this function updates the corresponding prototype weights using the following formula: w_j(new)=w_j(old)+α[x−w_(old)] Otherwise, it updates the prototype weights using the following formula: w_j(new)=w_j(old)−α[x−w_j(old)]
The “reduce_lr” function, which is responsible for decreasing the value of alpha by a very small amount using the following formula: α(new) = α(old) — (step_number * 0.000001)

# LVQ1 model

class LVQ:

  def __init__(self, epochs, alpha):
    self.epochs = epochs
    self.alpha = alpha

  # define a function to find the winning vector
  # by calculating euclidean distance  
  def winner(self, weights, sample):
      
    distance_0 = 0
    distance_1 = 0
      
    for i in range(len(sample)):
      # calculate euclidean distance between the given vector and the 1st prototype
      distance_0 = distance_0 + np.linalg.norm(sample[i] - weights[0][i])
      # calculate euclidean distance between the given vector and the 2nd prototype
      distance_1 = distance_1 + np.linalg.norm(sample[i] - weights[1][i])

    if distance_0 < distance_1:
        return 0 
    else: 
        return 1

  # define a function to update the winning prototype    
  def update( self, weights, sample, l_sample, winner_class) :
    # if the label of the nearest prototype == the label of the given vector
    # then add alpha
    if l_sample == winner_class:
      for i in range(len(weights[0])) :
        weights[winner_class][i] = weights[winner_class][i] + self.alpha * (sample[i] - weights[winner_class][i]) 
    # if the label of the nearest prototype != the label of the given vector
    # then subtract alpha
    else:
      for i in range(len(weights[0])) :
        weights[winner_class][i] = weights[winner_class][i] - self.alpha * (sample[i] - weights[winner_class][i])
  
  # reduce learning rate
  def reduce_lr(self, step):
    self.alpha = self.alpha - (step * 0.000001)

Train The Network

Initially, we instantiated an object named LVQ1 with parameters of 10 epochs and 0.5 alpha from the LVQ class. Next, using a for loop that iterates through the samples, we obtained the nearest class for each sample and obtained its current label from the list of labels. We then updated the weights using the update function and repeated this process for all samples. Finally, we decreased alpha.

This entire process was repeated for a certain number of epochs.

# train LVQ1 network

# create an object from LVQ class
LVQ1 = LVQ(10, 0.05) 

# repeate for all the epoches
for i in range(LVQ1.epochs):
  # for each sample
  for j in range(len(samples)):
    # take the sample
    sample = samples[j]
    # find the nearest class
    winner_class = LVQ1.winner(weights, sample)
    # take the label of the given sample
    l_sample = labels[j]
    # update prototypes (weights)
    LVQ1.update(weights, sample, l_sample, winner_class)
  # reduce learning rate
  LVQ1.reduce_lr(i)

Plot The Data In 3D After Training

After completing 10 epochs of training, we anticipate that the weight values of the two prototype vectors have been modified. Thus, we can visualize the data and inspect the result.

# plot samples after training
# make a figure
fig=plt.figure(figsize=(8,8))
# make a 3D subplot 
ax=fig.add_subplot(111,projection='3d')
# give each class a color 
# class 1 ==> green
# class 0 ==> blue
c_colors = np.where(train_data['y']==1,'g','b')
# plot the training samples as circles
ax.scatter(train_data['x1'],train_data['x2'],train_data['x3'],color=c_colors, )
# plot the 1st sample as a blue triangle 
ax.scatter(weights[0][0], weights[0][1], weights[0][2], s=40, color='b', marker='^', label = 'class 0')
# plot the 1st sample as a green triangle
ax.scatter(weights[1][0], weights[1][1], weights[1][2], s=40, color='g', marker='^', label = 'class 1')
ax.legend(loc="best")

As you can see, the trained prototype vector of class 0 is located in the center of class 0 samples and the trained prototype vector of class 1 is located in the center of class 1 samples.

Test The Network

How can we test the network?

Making predictions is simple. We just need to use the prototype vectors that have been trained!

We will pass in our test sample to the winner function from the trained network and predict the winner class.

test_sample = np.array([[0.304, 1.488, 1.408]])

# test The network with the given sample
# the test sample
test_sample = np.array([[0.304, 1.488, 1.408]])
# convert it into list
sample = test_sample[0].T.tolist()
# make prediction, find the corresponding class
w_class = LVQ1.winner(weights, sample )

print("The given sample belongs to class : ", w_class)

The output of the previous code can be seen below:

The given sample belongs to class : 1

Visualize The Result

To ensure the accuracy of our prediction, let’s visualize the data along with the provided test sample and determine which class it is closest to.

# see the test sample on the 3D plot
# make a figure
fig=plt.figure(figsize=(8,8))
# make a 3D subplot 
ax=fig.add_subplot(111,projection='3d')
# give each class a color 
# class 1 ==> green
# class 0 ==> blue
c_colors = np.where(train_data['y']==1,'g','b')
# plot the training samples as circles
ax.scatter(train_data['x1'],train_data['x2'],train_data['x3'],color=c_colors, )
# plot the 1st sample as a blue triangle 
ax.scatter(weights[0][0], weights[0][1], weights[0][2], s=40, color='b', marker='^', label = 'class 0')
# plot the 1st sample as a green triangle
ax.scatter(weights[1][0], weights[1][1], weights[1][2], s=40, color='g', marker='^', label = 'class 1')
# plot the test sample as a red triangle
ax.scatter(test_sample[0][0], test_sample[0][1], test_sample[0][2], s=40, color='r', marker='^', label = 'test sample')
ax.legend(loc="best")

The test sample will be represented as a red triangle.

As you can see, the network’s prediction was correct, as it can be observed that the given test sample is closer to class 1.

The colab notebook containing the python code for this implementation can be accessed through the following link.

Advantages Of LVQ

The model is trained significantly faster than other neural network techniques like MLP
It is able to summaries or reduce large datasets to a smaller number of prototype (codebook) vectors suitable for classification or visualization.
Able to generalize features in the dataset providing a level of robustness.
Can approximate just about any classification problem as long as the attributes can be compared using a meaningful distance measure.
Attractive feature of LVQ is that it can be easily applied to a multi-class problem.
Not limited in the number of dimensions in the prototype (codebook) vectors like nearest neighbour techniques.
Normalization of input data is not required (normalized may improve accuracy if attribute values vary greatly)
Can handle data with missing values.
The generated model can be updated incrementally.

Disadvantages Of LVQ

Need to be able to generate useful distance measures for all attributes (Euclidean is usually used for numeric attributes)
Model accuracy is highly dependent on the initialization of the model as well as the learning parameters used (learning rate, training iterations, etcetera)
Accuracy is also dependent on the class distribution in the training dataset, a good distribution of samples is needed to construct useful models.
It is difficult to determine a good number of prototype (codebook) vectors for a given problem.

Conclusion

We have successfully built a LVQ1 network to classify the given dataset into two categories. We trained the network with 10 epochs and 0.5 alpha, updated the weights of the prototype vectors, and made predictions using the trained weighting vectors. By visualizing the data, we could confirm the correctness of our predictions. The LVQ1 algorithm proved to be a simple yet effective way to classify data into distinct categories. With further optimization and tuning of the hyperparameters, it could potentially be used for more complex classification problems.

Support me

Hopefully this helped you, if you enjoyed it, you could follow me!