What are parametric and Non-Parametric Machine Learning Models?
Table of Contents ∘ Definition based on assumptions
∘ Definition based on parameters
∘ Properties of parametric models
∘ Properties of non-parametric models ∘ Examples of parametric models
∘ Examples of non-parametric models
Introduction
Machine Learning algorithms are basically mathematical functions that try to find a relationship between input and output variables.
If we have tabular data with columns ‘Experience’ (input) and ‘Salary’(target), We are trying to find a relationship between input and target. As experience changes, salary also changes. The function y = f(x) tries to find the relationship between the input feature x and the target y. But sometimes we may know or may not know the nature of the function.
Let us find out some of the properties, differences, and examples of parametric and non-parametric models in this blog.
Definition based on assumptions:
Whenever we make some assumptions about the nature of the function and distribution of the data, it is a parametric machine learning algorithm.
Linear Regression is a parametric machine learning algorithm since there are many assumptions. We know that the resulting function will be a line in 2 dimensions.
If we do not make any assumptions about the nature of the function, it is a non-parametric model.
A decision tree is a non-parametric model since there are no assumptions required to train a decision tree.
Definition based on parameters:
What are the parameters?
Simple Linear Regression with y = (m)(x)+c has 2 parameters. Slope m and intercept c. A simple neural network will have a fixed number of weights which are the parameters of a neural network.
A parametric machine learning algorithm has a fixed number of parameters and they do not change with the size of the data. The number of parameters will not grow with the data.
Eg, if we have about 1000 employees and we are trying to apply a linear regression algorithm, internally the algorithm is trying to find only 2 parameters m and c. Even if we add more employees, the number of parameters to find will not change.
A non-parametric machine learning model will not have a fixed number of parameters. The number of parameters will grow with the data.
Eg, The structure of a decision tree built with a set of 1000 rows will change when we add another 1000 rows to a training set.
Properties of parametric models:
- Since we already know the nature of the function and assumptions, the task of classification or regression becomes simple. The mathematical function will be a simple one.
- Since we know the assumption and nature of the function, less data is sufficient to build a model.
- Since we make many assumptions, there is a tendency to underfit. Eg, when we have non-linear data and we try to train a linear regression to it.
Properties of non-parametric models:
- Since we do not know the nature of the function and do not make any assumptions, it results in a complex mathematical function.
- Since we do not know the nature of the function and do not make any assumptions, more data is required in the training phase.
- Since we do not make any assumptions, there is a tendency to overfit because importance is given to every data point.
Examples of parametric models
Eg: Linear Regression, Logistic Regression, Linear SVM, Naive Bayes, simple neural networks, etc.
Examples of non-parametric models
Eg: Other SVM, KNN, Decision Tree, Bagging algorithms, and Boosting algorithms.
