Calibration
When the model says 30%, does it happen 30% of the time?
Accuracy on its own is a poor test — always backing the favourite scores well
and tells you nothing about what a 30% call is worth. The question that matters
is whether the numbers mean what they say. Every point below is a band of
claimed probability plotted against how often those calls actually landed,
out of sample. On the diagonal is right.
Picks evaluated
95,636
Claimed
36.6%
Landed
36.5%
Mean error
0.39pp
Marker size is how many picks sit in that band. The straight line is perfect calibration; above it the model is too cautious, below it too confident.
Band by band
| Claimed | Picks | Said | Landed | Difference | |
|---|---|---|---|---|---|
| 0.00-0.05 | 10,846 | 2.4% | 2.8% | +0.4pp | |
| 0.05-0.10 | 9,063 | 7.4% | 7.0% | -0.4pp | |
| 0.10-0.15 | 6,933 | 12.1% | 12.0% | -0.1pp | |
| 0.15-0.20 | 7,557 | 17.8% | 17.6% | -0.2pp | |
| 0.20-0.25 | 7,057 | 22.6% | 22.8% | +0.2pp | |
| 0.25-0.30 | 5,314 | 27.2% | 27.7% | +0.5pp | |
| 0.30-0.35 | 5,189 | 32.5% | 32.1% | -0.4pp | |
| 0.35-0.40 | 4,283 | 37.5% | 38.3% | +0.8pp | |
| 0.40-0.45 | 4,079 | 42.6% | 42.2% | -0.3pp | |
| 0.45-0.50 | 5,362 | 47.6% | 47.4% | -0.3pp | |
| 0.50-0.55 | 4,759 | 52.2% | 52.3% | +0.1pp | |
| 0.55-0.60 | 3,711 | 57.6% | 57.6% | -0.1pp | |
| 0.60-0.65 | 3,476 | 62.5% | 62.4% | -0.1pp | |
| 0.65-0.70 | 2,813 | 67.5% | 66.1% | -1.4pp | |
| 0.70-0.75 | 3,389 | 72.4% | 72.3% | -0.2pp | |
| 0.75-0.80 | 3,169 | 77.6% | 76.6% | -1.0pp | |
| 0.80-0.85 | 3,095 | 82.2% | 82.9% | +0.7pp | |
| 0.85-0.90 | 2,311 | 87.8% | 87.0% | -0.8pp | |
| 0.90-0.95 | 1,857 | 92.6% | 92.9% | +0.3pp | |
| 0.95-1.00 | 1,373 | 97.0% | 95.8% | -1.2pp |
Evaluated out of sample: each prediction was made by a model fitted only on matches before it, so nothing here saw its own answer. The sealed record of live picks is the forward-looking counterpart to this.