Calibration
When the model says 30%, does it happen 30% of the time?
Accuracy on its own is a poor test — always backing the favourite scores well
and tells you nothing about what a 30% call is worth. The question that matters
is whether the numbers mean what they say. Every point below is a band of
claimed probability plotted against how often those calls actually landed,
out of sample. On the diagonal is right.
Picks evaluated
5,128
Claimed
35.7%
Landed
36.3%
Mean error
1.68pp
Marker size is how many picks sit in that band. The straight line is perfect calibration; above it the model is too cautious, below it too confident.
Band by band
| Claimed | Picks | Said | Landed | Difference | |
|---|---|---|---|---|---|
| 0.00-0.05 | 638 | 2.3% | 3.4% | +1.2pp | |
| 0.05-0.10 | 503 | 7.2% | 7.8% | +0.6pp | |
| 0.10-0.15 | 371 | 12.0% | 12.9% | +0.9pp | |
| 0.15-0.20 | 425 | 18.0% | 20.9% | +2.9pp | |
| 0.20-0.25 | 390 | 22.8% | 22.3% | -0.5pp | |
| 0.25-0.30 | 258 | 27.5% | 27.9% | +0.4pp | |
| 0.30-0.35 | 261 | 32.5% | 35.6% | +3.1pp | |
| 0.35-0.40 | 228 | 37.2% | 41.2% | +4.0pp | |
| 0.40-0.45 | 201 | 42.6% | 43.8% | +1.1pp | |
| 0.45-0.50 | 285 | 47.7% | 46.0% | -1.7pp | |
| 0.50-0.55 | 250 | 52.1% | 54.8% | +2.7pp | |
| 0.55-0.60 | 210 | 57.7% | 59.5% | +1.9pp | |
| 0.60-0.65 | 188 | 62.7% | 62.2% | -0.4pp | |
| 0.65-0.70 | 141 | 67.5% | 63.8% | -3.7pp | |
| 0.70-0.75 | 161 | 72.5% | 69.6% | -2.9pp | |
| 0.75-0.80 | 156 | 77.7% | 75.6% | -2.1pp | |
| 0.80-0.85 | 193 | 82.0% | 82.4% | +0.4pp | |
| 0.85-0.90 | 104 | 88.1% | 81.7% | -6.4pp | |
| 0.90-0.95 | 95 | 93.0% | 92.6% | -0.3pp | |
| 0.95-1.00 | 70 | 97.2% | 97.1% | 0.0pp |
Evaluated out of sample: each prediction was made by a model fitted only on matches before it, so nothing here saw its own answer. The sealed record of live picks is the forward-looking counterpart to this.