Support Vector Regression (SVR) Model: A Regression-Based Machine Learning Approach
In this article, we will briefly discuss the SVR model. We will discuss three types of SVR namely, S-SVR (Scaling-SVR), Z-SVR (Z-score-SVR), and R-SVR (Range-SVR). Afterward, we will discuss its application in predicting the Average Localisation Error (ALE) in node localisation process in Wireless Sensor Network (WSNs).
You can download our paper for more details. You can write to me (abhilash.singh@ieee.org)if you have any question or visit my web page for more updates. Also, check the suggested reading section at the end of this article for recent research articles. Don’t forget to subscribe to my YouTube channel.
Introduction
Broadly, learning algorithms are divided into supervised and unsupervised learning. Further, supervised learning is classified into classification and regression learning, whereas unsupervised learning is classified into clustering and dimension reduction techniques.
The key objective of regression-based machine learning algorithms is to predict the predictand based on a mapping function. This mapping function is modelled by feeding a set of features and predictand data known as training data set. SVR is used in many applications such as image processing, remote sensing, and blockchain. It has superb generalisation competence along with high accuracy. Also, the computational complexity is independent of the input feature data set.
Let us see the governing equations of SVR in the next section;
Support Vector Regression
SVR was initially proposed by Drucker et al., which is a supervised learning technique, based on the concept of Vapnik’s support vectors. SVR aims at reducing the error by determining the hyperplane and minimising the range between the predicted and the observed values. Minimising the value of w in the Equation given below is similar to the value defined to maximise the margin, as shown in Fig. 1.
where the summation part represents an empirical error. Hence, to minimise this error, we use the following equation.
where the alpha term represents the Lagrange multiplier and its value is greater than equal to 1. K represents the kernel function and B represents the bias term. In this study, we have used the Polynomial kernel given by:
Where d is the polynomial degree and γ is the polynomial constant.
SVR performs better performance prediction than other algorithms like Linear Regression, KNN and Elastic Net, due to the improved optimisation strategies for a broad set of variables. Moreover, it is also flexible in dealing with geometry, transmission, data generalisation and additional functionality of kernel. This additional functionality enhances the model capacity for predictions by considering the quality of features.
The training samples influence the SVR model’s fitting performance since the SVR algorithm is sensitive to the interference in the training data. Besides, SVR is useful in resolving high dimensional features regression problem, and well-function if the feature metrics is larger than the size of the sample. In this study, we have extracted four features, namely anchor ratio, transmission range, node density and the number of iterations from modified CS algorithm simulation.
Feature scaling is essential for SVR because, when one function has greater magnitudes than others, the other features will dominate while measuring the distance. To avoid this, we have used various standardisation approaches. Based on this, we have proposed three methods, as shown in Fig. 2.
The method I is S-SVR (Scaling SVR). In this method, we first standardised the features using the Equation given below;
Where x is the feature vector, xs is the standardised data, and σ is the standard deviation of the feature vector. The method II is Z-SVR (Z-score SVR). In this method, we have standardised the features using the Equation given below;
Where x bar is the mean of the feature vector. The method III is the R-SVR (Range SVR). In this method, we have standardised the features using the Equation given below;
Afterward, we trained and tested the SVR models in a 70:30 ratio, as shown in Fig. 2. In this study, the dimension of the features vector is 107 × 1. Hence, we have used 75 data for training and the remaining 32 for testing.
Application of SVR for predicting ALE in node localisation in WSNs.
For simulating the SVR model, we performed the hyperparameter tuning through the grid search algorithm. In doing so, we fixed one of the hyper-parameter (i.e., epsilon at 0.01) and applied the grid search algorithm to find the value of the other hyper-parameter (i.e., C). We created a 100 × 100 grid for the penalty factor, C. Each grid represents a specific value of C. On simulating the grid search algorithm, it finds an optimal grid that corresponds to the minimum value of the MSE. The range of optimal C for all three methods along with the other simulation parameter value is given in the table below.
Results
In this section, we have presented the results of the method I, II and III for ALE prediction in the respective subsections. We have plotted a linear regression curve between the predicted ALE and the simulated ALE for comparison.
Performance of method I
We have compared the predicted ALE results, thus obtained by method I with the simulated results of the modified CS algorithm. We found that predicted results accorded well with the simulated results and gathered along the straight regression line with mild scattering (Fig. 3). The shaded grey region corresponds to the 95% Confidence Interval (CI) of the regression line and suggests that the predicted result has a strong positive correlation with R = 0.80 and RMSE = 0.23m.
Once we calculated the predicted ALE through method II, we have evaluated its performance with the simulated results of the modified CS algorithm. In doing so, we found a good agreement between both with R = 0.81 and RMSE = 0.20m (Fig. 4). However, some observed values lie outside the CI of the regression line due to the overestimation of the ALE value by the SVR model. The overestimation probably occurs due to the positive bias. This type of error comes under systematic error which is mainly due to the model or approach used.
We have compared the predicted ALE of method III with the simulated ALE obtained through a modified CS algorithm. In this case, also, we found a strong correlation between the variables (Fig. 5). Here, we found a pragmatic correlation of R = 0.82 with RMSE = 0.15m.
Conclusion
In this article, we discussed and investigated three SVR-based machine learning models for ALE prediction. These methods are defined based on the standardisation method used. In method I, II and III, we have used scaling, Z-score, and range standardisation methods respectively. Afterward, we trained the SVR model with the polynomial kernel using the standardised data and evaluated its performance using the correlation of coefficient and RMSE metrics. In doing so, we found that range standardisation of the features (i.e., method III) results in lower RMSE in ALE prediction. Also, the coefficient of correlation is highest in method III.
Video (do like the video and subscribe to the channel for more such videos)
References
- Singh, Abhilash, Vaibhav Kotiyal, Sandeep Sharma, Jaiprakash Nagar, and Cheng-Chi Lee. “A Machine Learning Approach to Predict the Average Localization Error With Applications to Wireless Sensor Networks.” IEEE Access 8 (2020): 208253–208263.
- Singh, Abhilash, Jaiprakash Nagar, Sandeep Sharma, and Vaibhav Kotiyal. “A Gaussian process regression approach to predict the k-barrier coverage probability for intrusion detection in wireless sensor networks.” Expert Systems with Applications 172 (2021): 114603.
Datasets availability
Download the datasets from UCI Machine Learning Repository
Code availability
Download the code from my MATLAB repository
Suggested Readings
