Prevent Losses by Implementing Simple Price Anomaly Detection
Nowadays, a lot of companies often execute a very large number of price updates in a short period of time, even in a minute. In this case, applying wrong updates may potentially lead to a serious negative impact for business (e.g., revenue losses). Early detection of anomalies is an important part of such a pricing system, thus we can prevent wrong update before it results to huge losses.
Anomaly detection is a method of identification of any deviations from the “norm” in your data or products. Anomaly detection systems help reveal whatever is out of the ordinary in your dataset: points, patterns or tendencies. The anomaly detection technique was created to help solve the problems by using certain method. In this case, we use standard deviation technique to measure the anomaly patterns.
Anomaly Detection Technique — Standard Deviation
In statistics, if a data distribution is approximately normal then about 68% of the data values lie within one standard deviation of the mean and about 95% are within two standard deviations, and about 99.7% lie within three standard deviations
Therefore, if the data point is more than 3 times the standard deviation, then those points are very likely to be anomalous or outliers.
But it is not limited only to 3 times of the standard deviation. We could set another number as a multiplier depends on the accuracy of the rules. Let’s see the illustration below:
We could see if the multiplier x = 4.5 and x = 5 are almost have the same values as the real anomalies data.
By testing several types of multiplier will be beneficial to validate which multiplier will be used for final threshold of anomalies. Based on the graph above, we see that 4.5 standard deviation gives the best result by only sending less than 5% false alarm. Thus, 4.5 standard deviation will be set as a threshold of the anomalies.
