Calibration
When the model says 30%, does it happen 30% of the time?
Accuracy on its own is a poor test — always backing the favourite scores well
and tells you nothing about what a 30% call is worth. The question that matters
is whether the numbers mean what they say. Every point below is a band of
claimed probability plotted against how often those calls actually landed,
out of sample. On the diagonal is right.
Picks evaluated
99,874
Claimed
36.6%
Landed
36.5%
Mean error
0.38pp
Marker size is how many picks sit in that band. The straight line is perfect calibration; above it the model is too cautious, below it too confident.
Band by band
| Claimed | Picks | Said | Landed | Difference | |
|---|---|---|---|---|---|
| 0.00-0.05 | 11,285 | 2.4% | 2.8% | +0.4pp | |
| 0.05-0.10 | 9,431 | 7.4% | 7.0% | -0.4pp | |
| 0.10-0.15 | 7,252 | 12.1% | 12.0% | -0.1pp | |
| 0.15-0.20 | 7,875 | 17.8% | 17.5% | -0.3pp | |
| 0.20-0.25 | 7,336 | 22.6% | 22.7% | +0.1pp | |
| 0.25-0.30 | 5,565 | 27.2% | 27.7% | +0.5pp | |
| 0.30-0.35 | 5,456 | 32.5% | 32.3% | -0.2pp | |
| 0.35-0.40 | 4,486 | 37.5% | 38.1% | +0.6pp | |
| 0.40-0.45 | 4,293 | 42.6% | 42.1% | -0.5pp | |
| 0.45-0.50 | 5,576 | 47.6% | 47.4% | -0.3pp | |
| 0.50-0.55 | 4,965 | 52.2% | 52.4% | +0.2pp | |
| 0.55-0.60 | 3,887 | 57.6% | 57.7% | +0.0pp | |
| 0.60-0.65 | 3,643 | 62.5% | 62.3% | -0.1pp | |
| 0.65-0.70 | 2,949 | 67.5% | 65.8% | -1.8pp | |
| 0.70-0.75 | 3,545 | 72.4% | 72.4% | 0.0pp | |
| 0.75-0.80 | 3,318 | 77.6% | 76.8% | -0.8pp | |
| 0.80-0.85 | 3,237 | 82.2% | 82.8% | +0.6pp | |
| 0.85-0.90 | 2,430 | 87.8% | 87.2% | -0.6pp | |
| 0.90-0.95 | 1,918 | 92.6% | 92.9% | +0.3pp | |
| 0.95-1.00 | 1,427 | 97.0% | 95.7% | -1.2pp |
Evaluated out of sample: each prediction was made by a model fitted only on matches before it, so nothing here saw its own answer. The sealed record of live picks is the forward-looking counterpart to this.