Learning a function
Microsoft Scholarship Foundation course Nanodegree Program has introduce me to interesting concepts of machine learning which I was not familiar at the beginning .
Machine learning is a data science technique used to extract patterns from data, allowing computers to identify related data, and forecast future outcomes, behaviors, and trends. In this article I am going to share you an important concept under machine learning ; Learning a function. we can generally think of a machine learning algorithm as a process for learning, and models as specific representations that we train using data. In essence, machine learning algorithms aim to learn a target function (f) that describes the mapping between data input variables (X) and an output variable (Y).
Y = f(X)
The core goal is to learn a useful transformation of the input data that gets us closer to the expected output. Since the process extrapolates from a limited set of values, there will always be an error e which is independent of the input data (X) such that:
Y = f(X) + e
The variable e is called irreducible error because no matter how good we get at estimating the target function (f), we cannot reduce this error.
Note that the irreducible error is different from the model error. Irreducible error is caused by the data collection process — such as when we don’t have enough data or don’t have enough data features. In contrast, the model error measures how much the prediction made by the model is different from the true output. The model error is generated from the model and can be reduced during the model learning process.
Parametric vs. Non-parametric Machine Learning Algorithms
Based on the assumptions about the shape and structure of the function they try to learn, machine learning algorithms can be divided into two categories: parametric and non-parametric.
Parametric Machine Learning Algorithms
Parametric machine learning algorithms make assumptions about the mapping function and have a fixed number of parameters. No matter how much data is used to learn the model, this will not change how many parameters the algorithm has. With a parametric algorithm, we are selecting the form of the function and then learning its coefficients using the training data.
An example of this would be the approach used in linear regression algorithms, where the simplified functional form can be something like:
B_0 + B_1 * X_1 + B_2 * X_2 = 0B0+B1∗X1+B2∗X2=0
This assumption greatly simplifies the learning process; after selecting the initial function, the remaining problem is simply to estimate the coefficients B0, B1, and B2 using different samples of input variables X1 and X2.
Benefits:
- Simpler and easier to understand; easier to interpret the results
- Faster when talking about learning from data
- Less training data required to learn the mapping function, working well even if the fit to data is not perfect
Limitations:
- Highly constrained to the specified form of the simplified function
- Limited complexity of the problems they are suitable for
- Poor fit in practice, unlikely to match the underlying mapping function.
Non-parametric Machine Learning Algorithms
Non-parametric algorithms do not make assumptions regarding the form of the mapping function between input data and output. Consequently, they are free to learn any functional form from the training data.
A simple example is the K-nearest neighbors (KNN) algorithm. KNN does not make any assumptions about the functional form, but instead uses the pattern that points have similar output when they are close.
Benefits:
- High flexibility, in the sense that they are capable of fitting a large number of functional forms
- Power by making weak or no assumptions on the underlying function
- High performance in the prediction models that are produced
Limitations:
- More training data is required to estimate the mapping function
- Slower to train, generally having far more parameters to train
- Overfitting the training data is a risk; overfitting makes it harder to explain the resulting predictions
Conclusion
This article summarizes the differences between machine learning algorithms and models , the basics to categorize parametric and non parametric machine learning algorithms as well as the benefits and limitations of these two categories .
Reference
