Every prediction we've ever made, graded
Our win-probability model logs a prediction for every game — pre-game, at the closing line, and at each half-inning while it's live — and grades itself once the game is final. No windows are cherry-picked; losses are shown with the same prominence as wins. Updated daily.
Calibration — Predicted Probability vs Actual Win Rate
A perfectly calibrated model sits on the diagonal: when we say 70%, the home team should win about 70% of the time. Dots are sized by how many games fall in each 10-point bucket.
| Predicted | Games | Avg pred | Actual win |
|---|---|---|---|
| 30–40% | 1 | 40% | 100% |
| 40–50% | 31 | 48% | 35% |
| 50–60% | 57 | 55% | 61% |
| 60–70% | 6 | 63% | 33% |
| 70–80% | 1 | 71% | 0% |
Brier Score vs Baselines — Lower Is Better
Head to Head vs the Market Closing Line
Historical Validation (Backtest) — Retrospective Research
Methodology. An as-of-date replay: each game is re-predicted using only data that existed before first pitch (records, probable starters, PitchIQ as of that day). The v2 pitcher adjustment and calibration were fit on 2022–2024 (7,289 games); the 2025 season (2,430 games) was held out from all tuning, so the holdout numbers are the honest out-of-sample test.
| Metric | v1 (records-only) | v2 (deployed) |
|---|---|---|
| Brier score · lower is better | 0.2498 | 0.2452 |
| Calibration error (ECE) · lower is better | 0.0592 | 0.0226 |
| Log loss · lower is better | 0.6936 | 0.6832 |
| Accuracy · picks vs actual | 55.5% | 55.3% |
Methodology & Honesty
The model (v2). DiamondIQ estimates home win probability from team records and home-field advantage (a log5 prior), the starting-pitcher quality gap (PitchIQ), and a calibration shrink — with live game state once a game is underway. The pitcher adjustment and calibration were fit on 2022–2024 and validated on the untouched 2025 season (v2 beat v1 on both Brier and calibration). It does not yet model bullpens, lineups, or weather.
Every prediction is version-tagged. Rows are labeled with the model version that produced them (v1 → v2), so this track record shows the transition honestly rather than silently swapping models under a single number.
Why the market comparison is still generous to the market. The closing line also prices in bullpens, lineups, and late scratches that v2 does not. We show the comparison for accountability, not to claim an edge.
How grading works. Every prediction is scored with the Brier score (p − outcome)² and log loss once the game is final. Calibration buckets predictions by predicted probability and compares each bucket to the actual home win rate. Nothing is re-stated or re-scored after the fact.
No cherry-picking. The numbers above are the cumulative record over every graded game — no date windows, no dropped losses. The full log is downloadable as CSV.
Get the numbers in your inbox
Double opt-in. Unsubscribe any time. We never share your email.