ML Series: Day 5 — Nonlinear regression

Figure 1. Different types of regression

Non-linear regression refers to a model in which there is a non-linear relationship between the dependent variable and the independent variables. This model is more accurate and flexible than the linear model and can obtain various curves for complex relationships between two or more variables. When the relationship between the data cannot be predicted with a straight line and is in the form of a curve, we should use a non-linear model. Unlike linear regression, there is no assumption of linearity of data in a non-linear model. Figure 2 shows linear and non-linear data.

Figure 2. Linear data (left side) and non-linear data (right side)

In Figure 2, the data related to the job position (Position level) on the horizontal axis and salary (Salary) on the vertical axis show that these data have non-linear properties and linear regression shows the relationship between the data.

Figure 3. Data (left side) and fitted model (right side) with linear regression

As it is clear, linear regression could not fit a good line to the data and we need to train a model that can discover more complex relationships from the data. Therefore, we introduce polynomial regression to overcome this problem, which helps to identify the curvilinear relationship between independent and dependent variables.

Polynomial regression is a form of linear regression that, due to the non-linear relationship between dependent and independent variables, we add a number of terms to linear regression to convert it to polynomial regression. This method is such that before training the model, in the pre-processing stage, using a method called feature transformation, we convert the linear regression into a polynomial. Figure 4 shows a polynomial from the first degree (simple linear regression) to the fourth degree.

Figure 4. polynomial regression with different degrees

In Figure 4, “Degree” refers to the polynomial degree, and as you can see, the first degree polynomial is the simple linear regression, and the higher the polynomial degree, the more complex curves can be fitted to the data.

Model parameters are values learned during training by machine learning algorithms, such as slope and width from the origin in linear regression; Whereas, hyperparameters are determined by humans before training. Polynomial degree is a hyperparameter and we should determine it according to the complexity of the data. A high degree of polynomial can lead to overfitting and a low degree of polynomial may cause underfitting in the learning process, and therefore we must choose the optimal value of the degree for polynomial (I will explain the precise and scientific meaning of the underfitting and overfitting).

In Part 5, we talked about the Nonlinear regression. In Part 6: Machine Learning Series: Day 6 — Preprocessing (Part 1), we discuss preprocessing in machine learning in multiple parts, and after that, I will explain important methods and algorithms in the field of AI.

If you like the article and would like to support me make sure to:

👏 Clap for the story (as much as you liked it) and follow me 👉
📰 View more content on my medium profile
🔔 Follow Me: LinkedIn | Medium | GitHub | Twitter