Chapter 23 · Worked solutions
Chapter 23 workings: Backtesting Without Fooling Yourself
Chapter twenty-three worked: the three leaks in a too-good Championship backtest, why each one flatters the result, and what the honest version does.
The system in the exercise reports £1,000 growing to £1,847 over three seasons. Before asking whether that is skill, ask whether the test could have produced it without any. Each of the three leaks below lets information from the future, or prices from the past, into the result.
The exercises
Three leaks in one paragraph
A backtest summary describes how team strength was measured, how matches were split into training and test sets, and which prices were used. Find the three leaks and the honest fix for each.
Leak one: the ratings know the future
A rating built from every goal of the season includes goals scored after the match being priced. A side that collapsed in spring looks weak in October, so the model appears to have spotted a decline it could not have seen. The fix is walk-forward: on the morning of each round, rebuild ratings from matches already played, price the round, then add its results.
Leak two: the split shuffles time
Drawing rows at random puts some March matches in the training set and some August matches in the test set. The model then learns from later results and is tested on earlier ones, which no punter ever gets to do. The fix is a cut by date: fit on earlier seasons, test on the one that follows, and never let a test match predate a training match.
Leak three: prices nobody could take
A Friday average across many firms is not a price you were offered, and by kick-off it may have moved a long way, often for a reason. Winning against a stale number says nothing about winning against the market. The fix is to settle each bet at a price that was genuinely available when the model fired, and to record the fair closing price alongside it. If the edge only exists against Friday's snapshot, it does not exist.
What to do with the £1,847
Nothing, yet. Rerun the test with all three fixed. In my experience, most of a headline return like this disappears with the first fix alone.
| The step | The leak | The honest version |
|---|---|---|
| Team strength from full-season averages | Lookahead | Ratings rebuilt after every round from earlier matches only |
| Random 70/30 split of rows | Chronological leakage | Train on earlier dates, test only on later ones |
| Friday average prices, no closing check | Stale odds | Settle at prices you could have taken, and measure against the close |
The answer
Lookahead in the ratings, time shuffled by a random split, and stale prices with no closing-line check; fix them with walk-forward ratings, a date-based split, and available prices benchmarked against the close.
These are the workings. The argument for them, and the exercises themselves, are in chapter 23 of the book, Backtesting Without Fooling Yourself.
Out now · free to read on Kindle Unlimited