Testing without lying to yourself

A test asks whether a written method would have been tolerable on data you did not use to invent it. This chapter is about the ordinary ways a private test flatters its author, and a small design that avoids them.

The flattery is usually accidental

You notice that buying above the 30-week average worked on a chart you have been staring at, so you adopt it, and then you ‘test’ it on that same chart. The test cannot fail. You chose the rule because of the chart. The remedy is chronological. Write the rule. Freeze it. Run it on a period that ended before you wrote it, and then, more importantly, obey it on the period that begins after you wrote it. The second part is the only test that includes your own behaviour. A backtest does not click your mouse.

A small design that fits a private method

1. Freeze the text, including size, exits, and the refusal list. Date the freeze.
2. Pick a past window you have not been browsing for examples. Two or three years is enough to see several bad months. It is not enough to prove a lifetime.
3. Record every signal the text would have taken, including the dull ones. Skipping a signal because you ‘would not have taken that one’ is editing the method mid-test.
4. Charge something for each round trip. A test with zero costs flatters turnover. Use a simple rupee cost you can defend, and the same cost on every trade.
5. Report the ugly lines: the worst stretch from peak to trough, the longest time to get back, the number of losses in a row, and the fraction of months that lost money. The average return is the line people quote. The ugly lines are the lines they have to live through.
6. Do not change the text because of one ugly line and then re-run the same window. That second run is a new invention, not a confirmation. If you change the text, the window you just used is now contaminated, and only a later window can speak.

What a tolerable result looks like

Suppose the frozen method, over thirty months, would have made a modest gain, with a worst decline of about 18% of the capital you assigned to it, and with six losing months in a row in the middle. You then ask a household question, not a statistical one: if that 18% arrives next year, in real money, do I stop? If the answer is yes, the method does not fit you, even if the full-period number is positive. Fit is the ability to still be following the text at the bottom of the ugly stretch. A method you abandon at the bottom has a live result worse than the test, because you eat the decline and miss the recovery that made the test look acceptable.

Paper results and live results

A paper test fills at the price you circled. A live order fills at the price someone will actually trade, a minute later, with charges. Expect the live version to be a little worse. If it is dramatically worse, the usual reasons are overrides, size that changes after wins, and signals you skip when they feel uncomfortable. Those are behaviour gaps. They will not be fixed by adding a filter to the text so that the uncomfortable signals disappear from the backtest. The uncomfortable signals are part of what you tested.

What to remember

• Freeze the rules before you look at the result. A rule chosen from the chart cannot be tested on that chart.
• Include every signal. A skipped signal is a different method.
• Put a cost on every round trip.
• The worst decline and the longest losing streak decide whether you can follow it. The average does not.
• Changing the text contaminates the window you learned the change from.