Mean absolute percentage error

The mean absolute percentage error (MAPE), also known as mean absolute percentage deviation (MAPD), is a measure of prediction accuracy of a forecasting method in statistics. It usually expresses the accuracy as a ratio defined by the formula:

{\mbox{MAPE}}=100{\frac {1}{n}}\sum _{t=1}^{n}\left|{\frac {A_{t}-F_{t}}{A_{t}}}\right|

where $A t$ is the actual value and $F t$ is the forecast value. Their difference is divided by the actual value $A t$ . The absolute value of this ratio is summed for every forecasted point in time and divided by the number of fitted points $n$ .

MAPE in regression problems

Mean absolute percentage error is commonly used as a loss function for regression problems and in model evaluation, because of its very intuitive interpretation in terms of relative error.

Definition

Consider a standard regression setting in which the data are fully described by a random pair $Z=(X,Y)$ with values in $\mathbb {R} ^{d}\times \mathbb {R}$ , and $n$ i.i.d. copies $(X_{1},Y_{1}),...,(X_{n},Y_{n})$ of $(X,Y)$ . Regression models aim at finding a good model for the pair, that is a measurable function $g$ from $\mathbb {R} ^{d}$ to $\mathbb {R}$ such that $g(X)$ is close to $Y$ .

In the classical regression setting, the closeness of $g(X)$ to $Y$ is measured via the $L 2$ risk, also called the mean squared error (MSE). In the MAPE regression context,^[1] the closeness of $g(X)$ to $Y$ is measured via the MAPE, and the aim of MAPE regressions is to find a model $g_{\text{MAPE}}$ such that:

$g_{\mathrm {MAPE} }(x)=\arg \min _{g\in {\mathcal {G}}}\mathbb {E} {\Biggl [}\left|{\frac {g(X)-Y}{Y}}\right||X=x{\Biggr ]}$

where ${\mathcal {G}}$ is the class of models considered (e.g. linear models).

In practice

In practice $g_{\text{MAPE}}(x)$ can be estimated by the empirical risk minimization strategy, leading to

${\widehat {g}}_{\text{MAPE}}(x)=\arg \min _{g\in {\mathcal {G}}}\sum _{i=1}^{n}\left|{\frac {g(X_{i})-Y_{i}}{Y_{i}}}\right|$

From a practical point of view, the use of the MAPE as a quality function for regression model is equivalent to doing weighted mean absolute error (MAE) regression, also known as quantile regression. This property is trivial since

${\widehat {g}}_{\text{MAPE}}(x)=\arg \min _{g\in {\mathcal {G}}}\sum _{i=1}^{n}\omega (Y_{i})\left|g(X_{i})-Y_{i}\right|{\mbox{ with }}\omega (Y_{i})=\left|{\frac {1}{Y_{i}}}\right|$

As a consequence, the use of the MAPE is very easy in practice, for example using existing libraries for quantile regression allowing weights.

Consistency

The use of the MAPE as a loss function for regression analysis is feasible both on a practical point of view and on a theoretical one, since the existence of an optimal model and the consistency of the empirical risk minimization can be proved.^[1]

WMAPE

WMAPE (sometimes spelled wMAPE) stands for weighted mean absolute percentage error.^[2] It is a measure used to evaluate the performance of regression or forecasting models. It is a variant of MAPE in which the mean absolute percent errors is treated as a weighted arithmetic mean. Most commonly the absolute percent errors are weighted by the actuals (e.g. in case of sales forecasting, errors are weighted by sales volume).^[3] Effectively, this overcomes the 'infinite error' issue.^[4] Its formula is:^[4] ${\mbox{wMAPE}}={\frac {\displaystyle \sum _{i=1}^{n}\left(w_{i}\cdot {\tfrac {\left|A_{i}-F_{i}\right|}{|A_{i}|}}\right)}{\displaystyle \sum _{i=1}^{n}w_{i}}}={\frac {\displaystyle \sum _{i=1}^{n}\left(|A_{i}|\cdot {\tfrac {\left|A_{i}-F_{i}\right|}{|A_{i}|}}\right)}{\displaystyle \sum _{i=1}^{n}\left|A_{i}\right|}}$

Where $w_{i}$ is the weight, $A$ is a vector of the actual data and $F$ is the forecast or prediction. However, this effectively simplifies to a much simpler formula: ${\mbox{wMAPE}}={\frac {\displaystyle \sum _{i=1}^{n}\left|A_{i}-F_{i}\right|}{\displaystyle \sum _{i=1}^{n}\left|A_{i}\right|}}$

Confusingly, sometimes when people refer to wMAPE they are talking about a different model in which the numerator and denominator of the wMAPE formula above are weighted again by another set of custom weights $w_{i}$ . Perhaps it would be more accurate to call this the double weighted MAPE (wwMAPE). Its formula is: ${\mbox{wwMAPE}}={\frac {\displaystyle \sum _{i=1}^{n}w_{i}\left|A_{i}-F_{i}\right|}{\displaystyle \sum _{i=1}^{n}w_{i}\left|A_{i}\right|}}$

Issues

Although the concept of MAPE sounds very simple and convincing, it has major drawbacks in practical application,^[5] and there are many studies on shortcomings and misleading results from MAPE.^[6]^[7]

It cannot be used if there are zero or close-to-zero values (which sometimes happens, for example in demand data) because there would be a division by zero or values of MAPE tending to infinity.^[8]
For forecasts which are too low the percentage error cannot exceed 100%, but for forecasts which are too high there is no upper limit to the percentage error.
MAPE puts a heavier penalty on negative errors, $A_{t}<F_{t}$ than on positive errors.^[9] As a consequence, when MAPE is used to compare the accuracy of prediction methods it is biased in that it will systematically select a method whose forecasts are too low. This little-known but serious issue can be overcome by using an accuracy measure based on the logarithm of the accuracy ratio (the ratio of the predicted to actual value), given by ${\textstyle \log \left({\frac {\text{predicted}}{\text{actual}}}\right)}$ . This approach leads to superior statistical properties and also leads to predictions which can be interpreted in terms of the geometric mean.^[5]
People often think the MAPE will be optimized at the median. But for example, a log normal has a median of $e^{\mu }$ where as its MAPE is optimized at $e^{\mu -\sigma ^{2}}$ .

To overcome these issues with MAPE, there are some other measures proposed in literature:

Mean Absolute Scaled Error (MASE)
Symmetric Mean Absolute Percentage Error (sMAPE)
Mean Directional Accuracy (MDA)
Mean Arctangent Absolute Percentage Error (MAAPE): MAAPE can be considered a slope as an angle, while MAPE is a slope as a ratio.^[7]

External links

References

^ ^a ^b de Myttenaere, B Golden, B Le Grand, F Rossi (2015). "Mean absolute percentage error for regression models", Neurocomputing 2016 arXiv:1605.02541
^ "Understanding Forecast Accuracy: MAPE, WAPE, WMAPE".
^ "WMAPE: Weighted Mean Absolute Percentage Error".
^ ^a ^b "Statistical Forecast Errors".
^ ^a ^b Tofallis (2015). "A Better Measure of Relative Prediction Accuracy for Model Selection and Model Estimation", Journal of the Operational Research Society, 66(8):1352-1362. archived preprint
^ Hyndman, Rob J., and Anne B. Koehler (2006). "Another look at measures of forecast accuracy." International Journal of Forecasting, 22(4):679-688 doi:10.1016/j.ijforecast.2006.03.001.
^ ^a ^b Kim, Sungil and Heeyoung Kim (2016). "A new metric of absolute percentage error for intermittent demand forecasts." International Journal of Forecasting, 32(3):669-679 doi:10.1016/j.ijforecast.2015.12.003.
^ Kim, Sungil; Kim, Heeyoung (1 July 2016). "A new metric of absolute percentage error for intermittent demand forecasts". International Journal of Forecasting. 32 (3): 669–679. doi:10.1016/j.ijforecast.2015.12.003.
^ Makridakis, Spyros (1993) "Accuracy measures: theoretical and practical concerns." International Journal of Forecasting, 9(4):527-529 doi:10.1016/0169-2070(93)90079-3

[demyttenaere2016-1] Myttenaere, B Golden, B Le Grand, F Rossi (2015). "Mean absolute percentage error for regression models", Neurocomputing 2016 arXiv:1605.02541

[baeldungdef-2] "Understanding Forecast Accuracy: MAPE, WAPE, WMAPE".

[ibfdef-3] "WMAPE: Weighted Mean Absolute Percentage Error".

[statisticalforecast-4] "Statistical Forecast Errors".

[tofallis2015-5] Tofallis (2015). "A Better Measure of Relative Prediction Accuracy for Model Selection and Model Estimation", Journal of the Operational Research Society, 66(8):1352-1362. archived preprint

[6] Hyndman, Rob J., and Anne B. Koehler (2006). "Another look at measures of forecast accuracy." International Journal of Forecasting, 22(4):679-688 doi:10.1016/j.ijforecast.2006.03.001.

[Kim2016-7] Kim, Sungil and Heeyoung Kim (2016). "A new metric of absolute percentage error for intermittent demand forecasts." International Journal of Forecasting, 32(3):669-679 doi:10.1016/j.ijforecast.2015.12.003.

[8] Kim, Sungil; Kim, Heeyoung (1 July 2016). "A new metric of absolute percentage error for intermittent demand forecasts". International Journal of Forecasting. 32 (3): 669–679. doi:10.1016/j.ijforecast.2015.12.003.

[9] Makridakis, Spyros (1993) "Accuracy measures: theoretical and practical concerns." International Journal of Forecasting, 9(4):527-529 doi:10.1016/0169-2070(93)90079-3

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]