What a backtest can and cannot tell you

By Otto Research 3 min read

A backtest runs a strategy over historical prices as if they were arriving live, records the trades it would have taken, and summarises them. Used for what it is, it is a useful tool. Used as evidence that a strategy makes money, it is the most common way traders are misled.

It can tell you whether a strategy's logic does what it claims: places stops, sizes correctly, trades the markets and hours it should. It can show the shape of a strategy, how often it trades, how long it holds, how its drawdowns relate to its returns. Run over a long period it shows how the rules behaved through conditions the live record has not yet met. And it can show how sensitive the strategy is to costs, by running it at different spread assumptions and watching the result.

It cannot tell you what the strategy will earn. The data is cleaner than any market a trader dealt in; the fills are perfect; the spread does not widen; slippage is absent unless modelled; swap and commission are included only if asked. Above all, the strategy's parameters were almost always chosen on the same data the test shows, so the test is a description of the fit rather than a forecast of the future. The strategy shown is also the survivor of every version the developer discarded.

Every one of those limits pushes the result the same way, upward. A backtest is therefore an upper bound on what the strategy might do, not an estimate of it.

The honest use is as context beside a live record: a forecast with a range, against which the live results can be compared. A live record that runs modestly below its backtest confirms that the backtest was honest. A live record that falls far below it, or a strategy with no live record at all, has been shown nothing by the backtest except what its author hoped.

From theory to live

See it put to work.

The concepts on these pages are what Otto is built on. Watch it trade them live, on real accounts.