Backtest against live: the gap as a lie detector

By Otto Research 3 min read

A backtest is a forecast with a range. It says what a strategy would have done in the past under stated assumptions, and, by implication, roughly what it should do in the future if the assumptions hold and the strategy is what it appears to be. A live record is what actually happened. The gap between them is the most useful figure a strategy can produce.

Otto's backtest page shows that comparison. The backtest's expected path, with a band around it drawn from the backtest's own variability, and the live fleet's equity plotted against it from the day the accounts started. If live results sit inside the band, the strategy is behaving as its backtest implied, allowing for the costs and fills a backtest cannot model. If they sit below it for a sustained period, something the backtest assumed is not true, and the page will say so rather than quietly redrawing the band.

The reason this matters is that it catches the failure that every other measure misses. A strategy whose backtest was fitted to its data starts below the band from the first month and stays there, because the edge it showed was an artefact of the fitting. A strategy with a real edge runs modestly below its backtest, by the size of the real-world costs, and stays there. A strategy whose edge is decaying drifts down out of the band over time. Drawdown alone cannot distinguish these; the comparison can, months before the drawdown would.

Live below the backtest by a modest, stable margin is the healthy case, and it is what to expect. Live above the backtest is a good period, not a better strategy. Live far below is a warning, and this site will treat it as one.

From theory to live

See it put to work.

The concepts on these pages are what Otto is built on. Watch it trade them live, on real accounts.