What a backtest can tell you, and what it cannot

The share strategies are tested over five years of daily prices for ten of the largest companies on the NSE, the same ten every strategy trades. Five years is long enough to contain a long rise, a sharp fall and a couple of sideways stretches, which matters because a rule that only ever saw a rising market has not been tested at all.

The intraday strategies are tested on five-minute bars, where five years would be too much data to hold, so the window is shorter. The options strategies cannot be tested yet, because we do not hold historical option prices, and they are not listed for exactly that reason. An article on each says so.

Costs are charged. Every simulated trade pays brokerage and the usual taxes, and fills are nudged against the strategy the way a real order is. A strategy that trades often pays for it in the test, as it would in life. Several of ours lost on that alone, and the articles on them say so too.

The thing that makes the test worth anything is that the code which ran it is the code that runs the strategy with pretend money against live prices, and the code that will place real orders with your broker. A strategy is not rewritten between the rehearsal and the performance. It is the same rule, fed by a different source of prices.

That is also why the test can be re-run. When we fix something in the engine, every listed strategy is tested again from scratch, and one that no longer clears the bar is taken down. It has happened. In October we found that a rule to sell one company could, on a particular day, sell a different one, and every multi-company number we had was void. We re-ran every one. A listing that no longer clears the bar is withdrawn by the same gate that admitted it.

Before a strategy can be listed, its test has to show at least twenty trades over at least ninety days, with the money won across the winning trades larger than the money lost across the losing ones, and a score from a review that looks at how the result was produced, not only what it was. A strategy that made one lucky trade does not pass. A strategy that made a hundred small careful ones and came out a little ahead does.

The bar is deliberately not 'made a lot'. It is 'made something, repeatably, after costs, with a record long enough to mean anything'. Plenty of strategies on this site are not listed because they cannot clear it, and the articles on them are here anyway, because understanding why a rule does not work is as useful as understanding why one does.

• Reading the return as a forecast. A strategy that returned a certain amount over five particular years is telling you what it did, not what it will do. The next five years contain different weeks.
• Forgetting the market did some of the work. A share strategy that is fully invested through a long rise earns the rise. Part of its return is the market's, and the honest question is what it did in the stretches when the market was not helping, which is why the worst dip is shown next to the return on every listing.
• Trusting a short record. A strategy that trades six times a year has thirty trades in five years. Thirty is enough to pass the bar and nowhere near enough to be sure. The slower a strategy, the wider the error bars around its number.

Try it with pretend money. Every listed strategy can run against live prices with a simulated account for as long as you like, and that run is the test the backtest cannot be: it shows you how the rule behaves on days nobody chose, and whether you can live with watching it. A strategy that loses three trades in a row in the test is a line on a chart. The same three trades, live, are three evenings. Find out which kind of person you are before a rupee is at stake.

A backtest tells you whether the rule has an edge. Pretend money tells you whether you can keep it.