Backtesting and Automated Execution - Chan, Ernie (Algorithmic Trading, 2013)
Backtesting is the process of feeding historical data to a trading strategy to see how it would have performed. Once every detail of a strategy has been implemented as a backtest program, we can then look for pitfalls in the backtesting process. A long list of pitfalls will be examined. In any backtest, we also face the problem of finite sample size: whatever statistical measures we compute are subject to randomness. We will address this issue using a general methodology called “hypothesis testing.” Even if a backtest is done correctly without pitfalls and with high statistical significance, it doesn't necessarily mean that it is predictive of future returns. Regime shifts can spoil everything, and a few important historical examples will be highlighted. The choice of a software platform for backtesting is also an important consideration, and criteria for this choice will be discussed.