Survivorship bias

By Otto Research 3 min read Article 61 of 95

Survivorship bias is the error of judging a population by the members that remain. The EAs on sale are the ones whose sellers kept selling; the signals published are the ones that were not deleted; the strategy a developer shows is the one that worked out of the many they tried. Everything visible has been filtered by success, and its apparent quality is partly that filter.

Where it appears

In signal listings, where accounts that blew up are removed and the average of the rest looks better than the average of all. In EA marketplaces, where a seller with five failed products and one that ran well shows the one. In backtests, where the strategy tested is the survivor of a search and its results include the luck of having been chosen. In the instruments themselves, where a stock index is composed of the companies that did not fail, and its history is smoother than the history of investing in it would have been. And in a trader's own memory, which keeps the strategies that worked and forgets the abandoned ones.

Why it matters

If a hundred developers each fit a random strategy to the last five years, some will produce excellent backtests by chance, and those are the ones who publish. A buyer sees only the top of a distribution that was mostly noise, and the visible results overstate what the method can do by an amount that depends on how many were tried, which the buyer cannot know.

Correcting for it

Prefer evidence that survivorship cannot touch: a forward test from a fixed start date, where nothing was selected afterwards. Ask how many strategies or configurations were tried before the published one, and treat a vague answer as "many". Look for records that include a losing period, since a seller who shows only winners has selected the window. And treat a seller's other products, discontinued or renamed, as part of their record.

Worked example

A marketplace lists 300 gold EAs. A buyer sorts by twelve-month return and looks at the top ten. All ten show returns above 80 per cent. Of the 300, 190 were released in the last year, and of those, the 60 that lost money in their first months have been withdrawn or renamed. The top ten are the best of a set that has already had its failures removed, over a period chosen because it was good. Their average future return will be far below 80 per cent, and several will be withdrawn in turn.

What this means when an EA is trading

Every EA you can see has survived something. The question is whether it survived a fair test, a forward period it could not have been fitted to, or merely survived being noticed. Only the first is evidence. A live verified record from a sealed start is the only form that removes the selection, and even then, the length of the record and the number of the seller's other products tell you how much selection happened before the start.

From theory to live

See it put to work.

The concepts on these pages are what Otto is built on. Watch it trade them live, on real accounts.