VII. Evaluation¶
Mean Absolute Percentage Error (MAPE)¶
The Mean Absolute Percentage Error (MAPE) is a standard metric used to evaluate the accuracy of forecasting models. It quantifies the average absolute difference between actual and predicted values as a percentage of the actual values. The formula for MAPE is:
MAPE = (100 / n) × Σ |(y_t − ŷ_t) / y_t
where y_t is the actual value at time t, ŷ_t is the predicted value at time t, and n is the total number of observations. A lower MAPE value indicates a more accurate forecasting model.
from darts.metrics import mape
# Define models in desired order
models = {
"AR(1)": "/Users/omarabdesslem/Documents/Projects/Thesis-Website/public/AR/ar_mape.txt",
"ARMA(1,0,1)": "/Users/omarabdesslem/Documents/Projects/Thesis-Website/public/ARMA/arma_mape.txt",
"ARIMA(1,1,0)": "/Users/omarabdesslem/Documents/Projects/Thesis-Website/public/ARIMA/arima_mape.txt",
"SARIMA(1,0,0)(1,0,0,52)": "/Users/omarabdesslem/Documents/Projects/Thesis-Website/public/SARIMA/sarima_mape.txt",
"SARIMAX(1,0,0)(1,0,0,52) + Temp": "/Users/omarabdesslem/Documents/Projects/Thesis-Website/public/SARIMAX/sarimax_mape.txt",
}
# Read MAPE values
mape_scores = {}
for model_name, path in models.items():
try:
with open(path, "r") as f:
mape_scores[model_name] = float(f.read().strip())
except Exception as e:
mape_scores[model_name] = None
print(f"⚠️ Could not read MAPE for {model_name}: {e}")
# Filter out any models with missing values
mape_scores = {k: v for k, v in mape_scores.items() if v is not None}
# Create DataFrame in order
df = pd.DataFrame({
"Model": list(mape_scores.keys()),
"MAPE": list(mape_scores.values())
})
# Plot settings
colors = ['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728', '#9467bd']
plt.figure(figsize=(10, 6))
bars = plt.bar(df["Model"], df["MAPE"], color=colors[:len(df)])
# Annotate bars with MAPE %
for bar in bars:
height = bar.get_height()
plt.text(bar.get_x() + bar.get_width()/2, height + 0.2,
f"{height:.2f}%", ha='center', va='bottom', fontsize=11)
plt.ylabel("MAPE (%)", fontsize=12)
plt.title("Forecasting Model Comparison – 8-Week Horizon", fontsize=14)
plt.xticks(rotation=15)
plt.ylim(0, max(df["MAPE"]) + 2.5)
plt.grid(axis='y', linestyle='--', alpha=0.4)
plt.tight_layout()
plt.show()
