We Backtested Our Own Numbers
Here's what held up and what didn't. This page is the model shop's audited financials — four studies, run on data the models never trained on, with the losses printed as large as the wins.
Backtested over 16 season pairs (2008-2025, 8,676 player-season transitions), SLATE's WAR proxy predicts next-season 5v5 on-ice xG impact better than raw scoring in every single pair (pooled Spearman 0.33 vs 0.22) and is highly stable year over year (0.70); on pure next-season scoring, raw points/60 remains the better predictor (0.80 vs 0.67) — the WAR proxy buys two-way signal, not a better crystal ball for points. The xG shot model, scored on a fully unseen season (119,870 shots from 2024-25), is decile-calibrated and beats a league-average baseline on Brier (0.0599 vs 0.0654, AUC 0.77). The win-probability model shows modest but real held-out skill (Brier 0.2453 vs naive 0.2491, AUC 0.61 on 2023-24). Rankings are robust to +/-20% blend-weight changes (worst top-100 rank correlation 0.99). Honest limits: the WAR backtest sample carries survivorship bias (both-season qualifiers only), win-prob skill is small in absolute terms, and none of this proves the specific blend weights are optimal — only that the system's signal is real and reproducible.
Generated July 14, 2026 · static, no live backend · every number below is committed JSON
Does WAR predict the future?
We computed our WAR proxy for every qualified skater in each season from 2008–2024, then asked a simple question: does this season's number predict nextseason's? We race it against two dead-simple baselines — raw points/60 and time on ice — on three different targets, pooled over 16 season pairs and 8,676 player-season transitions.
| Predicting → | WAR proxy | Points/60 | TOI/game |
|---|---|---|---|
| Next-season points/60 | 0.671 | 0.799best | 0.049 |
| Next-season 5v5 on-ice xG impact | 0.332best | 0.216 | 0.152 |
| Next-season WAR proxy (repeatability) | 0.700best | 0.650 | 0.304 |
Predicting next-season 5v5 xG impact: WAR proxy vs points/60
▸ How this was measured
Replicated server.py's warProxy formula exactly (verbatim component construction from MoneyPuck skaters.csv, percentile baselines, blend weights, 15th-percentile replacement, all-situations minutes scaling) in validation/warproxy.py. Computed it for every qualified skater in each season 2008-2024 from archived MoneyPuck season summaries, then correlated season-N values against three season-N+1 outcomes: all-situations points/60, 5v5 on-ice xGF/60 minus xGA/60, and next-season warProxy itself (repeatability). Two baselines predict the same targets: raw points/60 and TOI/game. Pearson and Spearman reported per season pair and pooled via Fisher-z weighted average; forwards and defensemen also split out. Sample = players qualified in both seasons of a pair.
Across 16 season pairs (2008-2025), warProxy in season N predicts next-season 5v5 on-ice xG impact better than raw points/60 in 16/16 pairs (pooled Spearman 0.332 vs 0.216) and is more repeatable year-over-year (0.700). But it LOSES to raw points/60 at predicting next-season scoring in 16/16 pairs (0.671 vs 0.799) — points predict points, and blending in defensive/on-ice signal dilutes pure scoring prediction. Even the xG-impact correlation is modest in absolute terms (~0.33). Caveats: the sample only includes players who qualified (20+ GP, 8+ min/gm) in BOTH seasons (survivorship bias), warProxy's minutes-scaling partly bakes in coach usage, part of its self-correlation reflects shared construction rather than pure skill signal, and pooled all-skater correlations are inflated by forward-vs-defense level differences — the F/D splits in the results are the cleaner read (e.g. pts60->pts60 drops to 0.69 within forwards).
Is the xG model calibrated?
A shot model earns trust by being honest about probability: when it says a shot is worth 10%, ten in a hundred should go in. We scored every shot of the 2024-25 (MoneyPuck shots_2024.csv) season — 119,870 shots the model never trained on— and bucketed them into deciles of predicted goal probability.
Predicted vs actual goal rate, by decile
▸ How this was measured
Loaded models/xg_model.joblib exactly as server.py does and scored every shot of the 2024-25 MoneyPuck season (downloaded archive, identical schema to training data, never used in training — training used 2022-23 and 2023-24). Deciles of predicted xG vs actual goal rate, plus Brier/log-loss against a constant league-average baseline. Provenance was verified by reconstructing the original 80/20 stratified split (random_state=42) and matching the training-time log-loss. Playoff shots included (training included them).
On a fully out-of-sample season (2024-25, 119,870 shots), the xG model clearly beats a league-average-rate baseline (Brier 0.0599 vs 0.0654; log-loss 0.2215 vs 0.2544; AUC 0.766) and its aggregate calibration is tight (mean predicted 0.0695 vs actual 0.0703). Weaknesses: it is mildly overconfident on the most dangerous shots (top decile predicted 0.237 vs actual 0.221) and slightly under elsewhere; shots_2023 WAS in the training set, so the training-time 'test' metrics were a random within-season split — the 2024-25 numbers here are the honest ones; and live NHL-API shots lack the rebound/rush flags the model was trained with, so deployed in-game xG runs slightly degraded relative to this benchmark.
Is the win-probability model reliable?
Same calibration test, harder problem. Single-game NHL outcomes are close to a coin flip, so the bar is honesty, not clairvoyance. We recovered the model's true held-out season (2023-24, 1,315 games, trained on 2022-23) and binned predicted home-win probability into ten bins.
Predicted vs actual home-win rate, by probability bin
▸ How this was measured
Loaded models/game_outcome_model.joblib (the v2 model trained by improve_models.py on leak-free running-average team features) and reproduced its original temporal split: trained on season 2022-23, tested on 2023-24. Row counts match the bundle's recorded n_train/n_test exactly, so this is the model's true held-out set. Reliability computed in 10 equal-width probability bins; Brier and log-loss compared against always predicting the training home-win rate.
On the recovered held-out season (2023-24, n=1315), the win-probability model edges out the naive home-win-rate baseline only modestly (Brier 0.2453 vs 0.2491) with AUC 0.608 — real but small signal, in line with public single-game NHL models (~0.60 AUC). Weaknesses are real: the model is overconfident at the extremes (games it prices at 0.70-0.90 home win probability landed near 0.65-0.70, and its rare sub-0.30 calls landed near 0.39-0.43), this same season was used for model selection at training time (so the improvement estimate is slightly optimistic), and the features are season-to-date team aggregates with no goalie, injury, or rest information.
Is the ranking fragile to its weights?
The WAR proxy blends six components with hand-set weights. A fair worry: are the rankings an artifact of those exact numbers? We re-ran the full 2025-26 skater ranking 12 times, each time nudging one weight by ±20%.
A ±20% swing on any single weight moves the top-100 ordering by a rank correlation of at worst 0.993 — effectively identical. The points/60 coefficient moves things most, which is expected since it carries the largest weight.
▸ How this was measured
Recomputed the full skater ranking on the most recent completed season (2024-25) twelve times, each time multiplying exactly one of the six _rate60 blend coefficients (on-ice 0.20, individual-xG 0.38, pts/60 0.66, GAx 0.45, rel on/off 1.0, adjImpact 0.02) by 0.8 or 1.2. For the baseline top-100 players we report the Spearman correlation of their baseline vs perturbed ranks, and how many remain in the perturbed top 100.
The top-100 ordering is weight-robust: across all twelve +/-20% single-weight perturbations on the 2024-25 season, the worst top-100 rank correlation vs the baseline is 0.993 and at least 98 of the top-100 players stay in the top 100. The pts/60 coefficient (0.66) moves rankings the most, which is expected since scoring carries the largest weight. Caveat: this tests one-at-a-time perturbations only — joint perturbations of several weights would move rankings more — and robustness to weights is not evidence the chosen weights are optimal.
Limitations
Credibility is mostly a list of things you refuse to pretend. Every weakness surfaced by every study above, collected in one place — this section is the point of the page, not an appendix.
- WAR proxy: does it predict next season?
Across 16 season pairs (2008-2025), warProxy in season N predicts next-season 5v5 on-ice xG impact better than raw points/60 in 16/16 pairs (pooled Spearman 0.332 vs 0.216) and is more repeatable year-over-year (0.700). But it LOSES to raw points/60 at predicting next-season scoring in 16/16 pairs (0.671 vs 0.799) — points predict points, and blending in defensive/on-ice signal dilutes pure scoring prediction. Even the xG-impact correlation is modest in absolute terms (~0.33). Caveats: the sample only includes players who qualified (20+ GP, 8+ min/gm) in BOTH seasons (survivorship bias), warProxy's minutes-scaling partly bakes in coach usage, part of its self-correlation reflects shared construction rather than pure skill signal, and pooled all-skater correlations are inflated by forward-vs-defense level differences — the F/D splits in the results are the cleaner read (e.g. pts60->pts60 drops to 0.69 within forwards).
- xG shot model: calibration on an unseen season
On a fully out-of-sample season (2024-25, 119,870 shots), the xG model clearly beats a league-average-rate baseline (Brier 0.0599 vs 0.0654; log-loss 0.2215 vs 0.2544; AUC 0.766) and its aggregate calibration is tight (mean predicted 0.0695 vs actual 0.0703). Weaknesses: it is mildly overconfident on the most dangerous shots (top decile predicted 0.237 vs actual 0.221) and slightly under elsewhere; shots_2023 WAS in the training set, so the training-time 'test' metrics were a random within-season split — the 2024-25 numbers here are the honest ones; and live NHL-API shots lack the rebound/rush flags the model was trained with, so deployed in-game xG runs slightly degraded relative to this benchmark.
- Win-probability model: reliability on a held-out season
On the recovered held-out season (2023-24, n=1315), the win-probability model edges out the naive home-win-rate baseline only modestly (Brier 0.2453 vs 0.2491) with AUC 0.608 — real but small signal, in line with public single-game NHL models (~0.60 AUC). Weaknesses are real: the model is overconfident at the extremes (games it prices at 0.70-0.90 home win probability landed near 0.65-0.70, and its rare sub-0.30 calls landed near 0.39-0.43), this same season was used for model selection at training time (so the improvement estimate is slightly optimistic), and the features are season-to-date team aggregates with no goalie, injury, or rest information.
- warProxy: is the ranking fragile to its blend weights?
The top-100 ordering is weight-robust: across all twelve +/-20% single-weight perturbations on the 2024-25 season, the worst top-100 rank correlation vs the baseline is 0.993 and at least 98 of the top-100 players stay in the top 100. The pts/60 coefficient (0.66) moves rankings the most, which is expected since scoring carries the largest weight. Caveat: this tests one-at-a-time perturbations only — joint perturbations of several weights would move rankings more — and robustness to weights is not evidence the chosen weights are optimal.
These studies validate the same models described on the Method page. Nothing here was cherry-picked: the WAR proxy loses to raw points at predicting points, the win-probability model is barely clairvoyant, and none of this proves the chosen blend weights are optimal — only that the signal is real and reproducible.