We Backtested Our Own Numbers
Here's what held up and what didn't. This page is the model shop's audited financials — four studies, run on our own data the models never trained on, with the losses printed as large as the wins.
Backtested over 3 season pairs of our own on-ice xG (2022-23 to 2025-26, 1,784 player-season transitions), Backcheck's WAR proxy predicts next-season 5v5 on-ice xG impact better than raw scoring (3/3 pairs; pooled Spearman 0.39 vs 0.28) with year-over-year stability of 0.75; on pure next-season scoring, raw points/60 remains the better predictor (0.82 vs 0.72) — the WAR proxy buys two-way signal, not a better crystal ball for points. The xG shot model, scored on a season it never trained on (119,954 of our own collected shots from 2024-25), is decile-calibrated and beats a league-average baseline on Brier (0.0602 vs 0.0654, AUC 0.76). The win-probability model shows modest but real held-out skill (Brier 0.2469 vs naive 0.2499, AUC 0.57 on 2025-26). Rankings are robust to +/-20% blend-weight changes (worst top-100 rank correlation 0.99). Honest limits: every study here runs on the seasons we have collected ourselves, so the WAR backtest is short and carries survivorship bias (both-season qualifiers only), win-prob skill is small in absolute terms, and none of this proves the specific blend weights are optimal — only that the system's signal is real and reproducible from our own data.
Generated September 24, 2026 · our own NHL play-by-play collection, shift charts and xG model · static, no live backend · every number below is committed JSON
Does WAR predict the future?
We computed our WAR proxy for every qualified skater in each season we have our own on-ice xG for (2022-23 to 2025-26), then asked a simple question: does this season's number predict next season's? We race it against two dead-simple baselines — raw points/60 and time on ice — on three different targets, pooled over 3 season pairs and 1,784 player-season transitions. The backtest grows a pair every time an earlier season is collected.
| Predicting → | WAR proxy | Points/60 | TOI/game |
|---|---|---|---|
| Next-season points/60 | 0.720 | 0.816best | 0.154 |
| Next-season 5v5 on-ice xG impact | 0.391best | 0.275 | 0.218 |
| Next-season WAR proxy (repeatability) | 0.755best | 0.707 | 0.392 |
Predicting next-season 5v5 xG impact: WAR proxy vs points/60
▸ How this was measured
Replicated server.py's warProxy formula exactly (component construction, percentile baselines, blend weights, 15th-percentile replacement, all-situations minutes scaling) in validation/warproxy.py and ran it on the same player frame the sidecar ranks from: NHL stats-API counting stats joined to our own on-ice xG (data/onice_{season}.json, built from the shots we collect and the NHL shift charts). Computed it for every qualified skater in each season (2022-23 to 2025-26), then correlated season-N values against three season-N+1 outcomes: all-situations points/60, 5v5 on-ice xGF/60 minus xGA/60, and next-season warProxy itself (repeatability). Two baselines predict the same targets: raw points/60 and TOI/game. Pearson and Spearman reported per season pair and pooled via Fisher-z weighted average; forwards and defensemen also split out. Sample = players qualified in both seasons of a pair.
Across 3 season pairs (2022-23 to 2025-26), warProxy in season N predicts next-season 5v5 on-ice xG impact better than raw points/60 (3/3 pairs; pooled Spearman 0.391 vs 0.275) and its year-over-year repeatability is 0.755. On next-season scoring, raw points/60 remains the better predictor (0.816 vs 0.720; warProxy wins 0/3 pairs) — points predict points, and blending in defensive/on-ice signal dilutes pure scoring prediction. Caveats: only 3 season pairs so far — the seasons we have collected shots and shift charts for ourselves; more pairs land as earlier seasons are collected. The sample includes only players who qualified (20+ GP, 8+ min/gm) in BOTH seasons (survivorship bias), warProxy's minutes-scaling partly bakes in coach usage, part of its self-correlation reflects shared construction rather than pure skill signal, and pooled all-skater correlations are inflated by forward-vs-defense level differences — the F/D splits are the cleaner read (pts60->pts60 is 0.74 within forwards).
Is the xG model calibrated?
A shot model earns trust by being honest about probability: when it says a shot is worth 10%, ten in a hundred should go in. We scored every shot we collected for the 2024-25season — 119,954 shots the model never trained on (it learned from 2025-26) — exactly as the site scores them, calibrator included, and bucketed them into deciles of predicted goal probability.
Predicted vs actual goal rate, by decile
▸ How this was measured
Loaded models/xg_model.joblib and models/xg_calibrator.joblib exactly as server.py does and scored every unblocked shot attempt we collected for 2024-25 from the NHL play-by-play (119,954 shots, shootout attempts excluded) — a season the model never saw (training used 2025-26). Deciles of predicted xG vs actual goal rate, plus Brier/log-loss against a constant league-average baseline, for the deployed pipeline (model + isotonic calibrator) with the raw model reported alongside. Provenance was verified by rebuilding the training frame with the same loader and reconstructing the original 80/20 stratified split (random_state=42) to match the training-time log-loss.
On a fully out-of-sample season (2024-25, 119,954 shots; the model trained on 2025-26), the deployed xG pipeline beats a league-average-rate baseline (Brier 0.0602 vs 0.0654; log-loss 0.2256 vs 0.2545; AUC 0.765) and its aggregate level is off (mean predicted 0.0678 vs actual 0.0703, 0.964x). The most dangerous decile is overconfident (predicted 0.245 vs actual 0.220); the largest decile miss is D10 (0.245 vs 0.220). Weaknesses: the calibrator was fitted on the training season, so the level correction is itself in-sample; our play-by-play shots carry no rebound or rush flags (both features sit at zero in training and in production alike); and the eval season is EARLIER than the training season, so rule and tracking drift, if any, cuts against the model here.
Is the win-probability model reliable?
Same calibration test, harder problem. Single-game NHL outcomes are close to a coin flip, so the bar is honesty, not clairvoyance. We recovered the model's true held-out season (2025-26, 1,205 games, trained on 2022-23, 2023-24, 2024-25) and binned predicted home-win probability into ten bins.
Predicted vs actual home-win rate, by probability bin
▸ How this was measured
Loaded models/game_outcome_model.joblib (trained by improve_models.py on leak-free running-average team features built from our own scored shots and the NHL schedule results) and reproduced its temporal split: trained on 2022-23, 2023-24, 2024-25, tested on 2025-26. Row counts match the bundle's recorded n_train/n_test, so this is the model's true held-out set. Reliability computed in 10 equal-width probability bins; Brier and log-loss compared against always predicting the training home-win rate.
On the recovered held-out season (2025-26, n=1205), the win-probability model edges out the naive home-win-rate baseline (Brier 0.2469 vs 0.2499) with AUC 0.572 — real but small signal, in line with public single-game NHL models (~0.60 AUC). The largest reliability miss among bins with 20+ games is the 0.7-0.8 bin: priced at 0.728, landed at 0.667 (n=69). Weaknesses are real: the model was trained on 3 seasons of our own data (2022-23, 2023-24, 2024-25), the held-out season was also used for model selection at training time (so the improvement estimate is slightly optimistic), and the features are season-to-date team aggregates with no goalie, injury, or rest information.
Is the ranking fragile to its weights?
The WAR proxy blends six components with hand-set weights. A fair worry: are the rankings an artifact of those exact numbers? We re-ran the full 2025-26 skater ranking 12 times, each time nudging one weight by ±20%.
A ±20% swing on any single weight moves the top-100 ordering by a rank correlation of at worst 0.995 — effectively identical. The verdict names the coefficient that moves things most.
▸ How this was measured
Recomputed the full skater ranking on the most recent season we have our own on-ice xG for (2025-26) twelve times, each time multiplying exactly one of the six _rate60 blend coefficients (on-ice 0.20, individual-xG 0.38, pts/60 0.66, GAx 0.45, rel on/off 1.0, adjImpact 0.02) by 0.8 or 1.2. For the baseline top-100 players we report the Spearman correlation of their baseline vs perturbed ranks, and how many remain in the perturbed top 100.
The top-100 ordering is weight-robust: across all twelve +/-20% single-weight perturbations on the 2025-26 season, the worst top-100 rank correlation vs the baseline is 0.995 and at least 98 of the top-100 players stay in the top 100. The adj coefficient (0.02) moves rankings the most. Caveat: this tests one-at-a-time perturbations only — joint perturbations of several weights would move rankings more — and robustness to weights is not evidence the chosen weights are optimal.
Limitations
Credibility is mostly a list of things you refuse to pretend. Every weakness surfaced by every study above, collected in one place — this section is the point of the page, not an appendix.
- WAR proxy: does it predict next season?
Across 3 season pairs (2022-23 to 2025-26), warProxy in season N predicts next-season 5v5 on-ice xG impact better than raw points/60 (3/3 pairs; pooled Spearman 0.391 vs 0.275) and its year-over-year repeatability is 0.755. On next-season scoring, raw points/60 remains the better predictor (0.816 vs 0.720; warProxy wins 0/3 pairs) — points predict points, and blending in defensive/on-ice signal dilutes pure scoring prediction. Caveats: only 3 season pairs so far — the seasons we have collected shots and shift charts for ourselves; more pairs land as earlier seasons are collected. The sample includes only players who qualified (20+ GP, 8+ min/gm) in BOTH seasons (survivorship bias), warProxy's minutes-scaling partly bakes in coach usage, part of its self-correlation reflects shared construction rather than pure skill signal, and pooled all-skater correlations are inflated by forward-vs-defense level differences — the F/D splits are the cleaner read (pts60->pts60 is 0.74 within forwards).
- xG shot model: calibration on an unseen season
On a fully out-of-sample season (2024-25, 119,954 shots; the model trained on 2025-26), the deployed xG pipeline beats a league-average-rate baseline (Brier 0.0602 vs 0.0654; log-loss 0.2256 vs 0.2545; AUC 0.765) and its aggregate level is off (mean predicted 0.0678 vs actual 0.0703, 0.964x). The most dangerous decile is overconfident (predicted 0.245 vs actual 0.220); the largest decile miss is D10 (0.245 vs 0.220). Weaknesses: the calibrator was fitted on the training season, so the level correction is itself in-sample; our play-by-play shots carry no rebound or rush flags (both features sit at zero in training and in production alike); and the eval season is EARLIER than the training season, so rule and tracking drift, if any, cuts against the model here.
- Win-probability model: reliability on a held-out season
On the recovered held-out season (2025-26, n=1205), the win-probability model edges out the naive home-win-rate baseline (Brier 0.2469 vs 0.2499) with AUC 0.572 — real but small signal, in line with public single-game NHL models (~0.60 AUC). The largest reliability miss among bins with 20+ games is the 0.7-0.8 bin: priced at 0.728, landed at 0.667 (n=69). Weaknesses are real: the model was trained on 3 seasons of our own data (2022-23, 2023-24, 2024-25), the held-out season was also used for model selection at training time (so the improvement estimate is slightly optimistic), and the features are season-to-date team aggregates with no goalie, injury, or rest information.
- warProxy: is the ranking fragile to its blend weights?
The top-100 ordering is weight-robust: across all twelve +/-20% single-weight perturbations on the 2025-26 season, the worst top-100 rank correlation vs the baseline is 0.995 and at least 98 of the top-100 players stay in the top 100. The adj coefficient (0.02) moves rankings the most. Caveat: this tests one-at-a-time perturbations only — joint perturbations of several weights would move rankings more — and robustness to weights is not evidence the chosen weights are optimal.
These studies validate the same models described on the Method page. Nothing here was cherry-picked, and nothing here leans on another publication's numbers: every study runs on the shots, shifts and results we collect from the NHL ourselves. The verdicts above print the losses next to the wins, and none of this proves the chosen blend weights are optimal — only that the signal is real and reproducible from our own data.