Model Transparency Panel

We don't ask you to trust the model — we show you the backtest. Every number below is reproducible from the source scripts linked at the bottom of this page.

NFL — held-out 2025 season (never used to tune the model)

Blend weights were selected on 2024 games only, then locked and applied unchanged to 285 games in 2025. This is the honest test: performance on games the tuning process never saw.

Spread error (MAE, points)

8.353model

9.670Vegas closing line

13.6% better than Vegas

Total error (MAE, points)

9.461model

10.423Vegas closing line

9.2% better than Vegas

Win probability calibration (Brier)

0.1636model

0.2112Vegas closing line

22.5% better than Vegas

Statistical significance (spread, paired bootstrap, 5,000 resamples)

Model beats Vegas by 1.32 points on average, 95% CI [-2.03, -0.62] — statistically significant, not noise.

Full 13-season sample (2013–2025, 3,562 games)

Unblended heuristic simulation only (no supervised ML, no live market blend) -- shows the base model already had a slight edge before the ML layer was added.

Spread error (MAE, points)

9.622model

9.918Vegas closing line

3.0% better than Vegas

Total error (MAE, points)

10.283model

10.505Vegas closing line

2.1% better than Vegas

Methodology & honest caveats

Supervised gradient-boosted model trained on 2013-2023 NFL games, blended with a Monte Carlo matchup simulation and live market lines. Blend weights were grid-searched on the 2024 season only, then applied unchanged to the 2025 season, which the tuning process never saw.

This is a backtest on completed historical games using nflverse's free closing lines (spread_line/total_line), not a live prospective track record yet -- live CLV tracking requires a working Odds API key and several weeks of accumulated open-to-close snapshots, which is in progress.
The validated blend weights only apply once a season's rolling features contain real, week-varying data. Before games are played (e.g. the 2026 season as of this report), the model automatically falls back to more conservative, lower-trust blend weights -- see detect_real_rolling_features in nfl_supervised_retrain.py.

Report generated 2026-07-17 · model nfl-v1.5-matchup-sim · source: scripts/nfl/historical_market_backtest.py, scripts/nfl/tune_blend_weights.py, scripts/nfl/full_pipeline_holdout_backtest.py

Other sports & live tracking

Live closing-line-value tracking (open vs. close, in real time as the season plays out) is being built out now that the odds pipeline is fully wired — it will appear here once enough games have accumulated real open-to-close data.