MAPE vs MAE: Which Metric is Better?
Introduction
Searching the web reveals many results for what MAE and MAPE are and when to use them. This article will differ by taking the reader through numerous examples that highlight the advantages and disadvantages for each metric, their differences and the contexts to which they are best suited.
What are MAPE and MAE?
MAPE and MAE are both performance metrics that can be used to see how well regression models are performing, as well as other machine-learning models like time-series forecasts. They are specifically used to monitor accuracy (not precision). A good visual to demonstrate the difference between accuracy and precision is shown below. The MAPE and MAE metrics will show how close the actual values and predicted values are to each other but ignore what the ideal values should be for both. For example, there might be a range that we want values to fall between; accuracy metrics will not take this into consideration.
MAPE stands for Mean Absolute Percentage Error and MAE stands for Mean Absolute Error. From their names we can see they are similar, except one is a percentage. Both require two data points: the predicted values and the actual values and both quantify the average deviation (error) between these two values.
MAPE and MAE Equations with a Simple Example
MAPE Equation:
MAE Equation:
Example 1: If we have an actual value of 58 and a predicted value of 52, what is the MAPE and MAE?
The MAE is the easiest to work out, and the MAPE uses the MAE to obtain its value.
The MAE is simply the absolute difference between the actual value and the predicted value. So: |58- 52| = 6
The MAPE is the same but divided by the actual value. So: |58–52|/58 = 0.103. The MAPE value is a percentage, so can be multiplied by 100. For this example, the difference is just greater than 10%.
From this we can see that the lower the value, the better the performance is for both metrics.
For this example, there is only one predicted value and one actual value; these metrics usually take in numerous values and are aggregated, so the MAE or MAPE values are summed up then divided by how many there are (hence, ‘mean’ values).
We can also see that the MAE is an easier metric to understand as it has the same units as the actual and predicted values but doesn’t take in the magnitude of the difference like MAPE. The MAPE can be more beneficial because it shows the percentage difference. The fact that MAE does not show the percentage difference can be a disadvantage when comparing the accuracy of different models, because the range of the predicted and actual values may differ.
Further Examples Demonstrating the Pros and Cons of MAPE and MAE
Example 2
This following example shows a downside of MAPE: that the values are asymmetric, meaning that MAPE reports higher errors if the predicted value is higher than the actual value and reports lower errors if the predicted value is lower than the actual, even if the MAE is the same.
In the example table above, we can see that the MAE for both dates is 300, however, the MAPE is very different between them, depending on if it is the predicted or actual value that is the largest. Without the addition of the MAE, the MAPE score does not give as much insight.
Example 3
This next example shows that MAPE is more susceptible to extreme values than MAE.
I used a dataset that contained the actual and predicted values for different models over time. I created a MAPE and MAE value for each pair of actual and predicted values, to show the accuracy of the different regression models on different days. I had over 11.5k rows with only 453 having MAPE values greater than 100 (3.9% of the data). Some of these 453 values were extreme, from when the actual value was very low, causing the MAPE to be >1000. The range of the MAE was always between 0–100, because with the chosen models, the lowest a value could be was 0 and the highest was 100.
Using this data, I created two visualisations that showed the spread of the MAPE or MAE values per market. As we can see below with the spread of the MAPE values, despite there only being 3.9% of values greater than 100, they skew the data significantly so that we cannot see the spread of the interquartile range (IQR). From MAPE, we can see which markets have the largest number of extreme values and how high these values go, but gain little insight into the most common values. Whilst MAPE can have extreme values, these are only in a minority results (it is unlikely that a predicted value will be so low or high and the actual will be the absolute opposite, unless a model is significantly under-performing).
Whereas because MAE values can only lie between a much smaller range, the spread of data is more evenly distributed. Just like MAPE, we can see that Romania, Russia and the US have a lot of high values, but they don’t dominate over the other values and markets. For all markets, we can see the IQR and roughly where the lowest and highest values lie.
Example 4:
This next example will highlight the significance of MAPE dividing by the actual value, so when MAPE is 1x, 10x and 100x the MAE value and when MAPE results in an error.
Note: unlike the example above, the MAPE value has not been multiplied by 100, so this part matches the formulae at the top of the page.
Because the MAPE score is divided by the actual value, if the actual value is 1, 10 or 100 then the MAPE score will be 1, 10, 100 times smaller than the MAE (respectfully). This point indicates that the scale of MAPE values is different to MAE’s, which is constant.
Secondly, because the MAPE is divided by the actual value, if the actual value is zero, then a division error occurs. One way to avoid this is to not divide by the zero, hence make it return the MAE values for these instances. This is a disadvantage of MAPE, especially if the actual values are likely to be zero.
This means MAPE in these situations does not provide any information, whereas MAE can because no division is involved.
It is important to understand if this is a good or bad thing for your use case, the MAPE is normalized so works on the same scale for models with different ranges of actual and predicted values whereas the MAE uses the same units.
Example 5
This final example will highlight if the values of the metrics are consistent in their order. I will plot countries ordered by MAE from lowest to highest and see if the order of countries is the same with MAPE also ordered from lowest to highest.
We can see below that Romania and Russia have the lowest MAE values, and Switzerland has the highest. However, this not the case in the right visual, that displays the ranking for the MAPE values per market, because now Romania and Russia have the largest MAPE values and Switzerland performs averagely, in position 6 out of 10 markets.
Therefore, we can’t assume the value of the MAPE score with the MAE score, and vice versa. This shows that it can be beneficial to report both the MAPE and MAE scores.
The best metric to use though when comparing different markets that have different ranges for actual and predicted values is the MAPE, because if the MAE is 13 for one market and 53 for another, comparing these does not make sense if the second market has a different range of values, the average percentage error gives more insight.
Conclusion
In conclusion, both MAPE and MAE have advantages and disadvantages. MAE is great when focusing on one model or comparing models where the range of actual and predicted values are the same. Although MAE treats extreme values the same as normal values and can’t be used to compare models with different ranges. MAPE is best when comparing models with different ranges and understanding the percentage difference. However, MAPE is asymmetric, is more prone to skewing the data with extreme values and is affected by a division error if the actual value is zero.
Therefore, it is important to understand the use case to see which metric is best to use or use both, to avoid the limitation of the other metric.
