Model Performance

Final evaluation results on held-out test set (2022–2023)

Static results. The metrics on this page reflect the final evaluation of the six trained models against the held-out test set (2022–2023). They are computed once at the end of training and do not change in production. Live forecasts for the current week are shown on the Map view.

Production Model

SARIMAX

SARIMAX — chosen for interpretability

RMSE (weighted)

25.28

Production model, all horizons

Average R²

0.907

Across all departments

Departments Covered

Excluding Bogotá (COD 11)

Model Comparison

RMSE / MAE averaged across 4 horizons


1	SARIMA	Native	25.17	0.908	13.63
2	SARIMAXProduction	Native	25.28	0.907	13.70
3	NeuralNetwork	Direct	26.76	0.914	12.63
4	ElasticNet	Direct	27.54	0.900	14.91
5	RandomForest	Direct	31.37	0.866	14.68
6	XGBoost	Direct	33.83	0.847	14.84

SARIMA ranks first by RMSE, but SARIMAX was selected for production: the marginal 0.46% accuracy gap is outweighed by its ability to attribute forecasts to climate variables — a capability SARIMA structurally lacks.

RMSE Degradation by Forecast Horizon

Key Findings

SARIMAX chosen for interpretability, not raw accuracy

SARIMA marginally outperforms SARIMAX in weighted RMSE (25.17 vs 25.28, a 0.46% gap). SARIMAX was deployed regardless because it attributes each forecast to climate variables (temperature, precipitation, humidity, ONI), which SARIMA cannot do. For a public-health tool, the question "why is this week elevated?" matters more than the last decimal of RMSE.

Time-series models beat tree ensembles on outbreak peaks

SARIMA (R² ≈ 0.91) and SARIMAX (R² ≈ 0.91) outperform Random Forest (0.866) and XGBoost (0.847) because tree-based predictions are bounded by the training range. They cannot extrapolate during outbreak peaks beyond historical maxima — a structural disqualifier for early warning systems.

Recursive ElasticNet diverges catastrophically

Under the iterative (recursive) strategy where t+1 predictions feed back as lag inputs for t+2, ElasticNet collapsed from R² = 0.94 at t+1 to R² = −2,764.74 at t+4. The compounding interaction between log1p targets, linear coefficients, and expm1 inverse exponential transform amplifies any residual error. The Direct Strategy was adopted for all tabular models as a result.

Climate variability is locally distinct

Climate-influence attribution differs across departments: Valle del Cauca's forecasts are dominated by temperature, Boyacá responds primarily to ONI (ENSO state), and Chocó shows no single dominant driver. This per-department heterogeneity justified training 32 separate SARIMAX models rather than a single global model.