Backtesting Pitfalls

Almost every backtest overstates the returns the strategy would have delivered. This is not usually the result of dishonesty. It is the default outcome of careful work on imperfect data, because the errors that inflate results are numerous and the errors that deflate them are rare. Knowing the specific mechanisms is what allows you to discount a published result by roughly the right amount, and to read the factor data on this site with appropriate scepticism.

Survivorship bias

If your database contains the companies that exist today and you run history backwards, every firm that went bankrupt, was acquired on bad terms, or was delisted has quietly disappeared from the sample. The strategy is then tested only on survivors. The effect is largest exactly where it does most damage: among small, distressed, and deep-value companies, which are the ones most likely to fail and also the ones factor strategies deliberately overweight.

A related and subtler problem is the handling of delisting returns. When a company is delisted after collapsing, the correct treatment is to record the terminal loss. Some datasets simply end the series, which silently converts a total loss into a non-event.

Look-ahead bias

Look-ahead bias is the use of information that would not have been available at the moment the trade was supposedly made. The most common form involves accounting data. A company's fiscal-year figures are not public on the last day of the fiscal year; they appear in a filing weeks or months later. Sorting portfolios on December's book value in January assumes knowledge nobody had.

Fama and French handle this deliberately by imposing a lag between the accounting date and the portfolio formation date, which is one reason their published series are a defensible research baseline. Restated financials are a harder version of the same problem: databases often store the corrected figure rather than the one originally reported, and no amount of lagging recovers what investors actually saw.

Multiple testing and the replication problem

If enough strategies are tested against the same history, some will look excellent by chance alone. This is not a hypothetical concern. Hundreds of factors have been published in the academic literature, and researchers examining that body of work have argued that conventional statistical thresholds are far too lenient given how many hypotheses have been tried against the same few decades of data. Harvey, Liu and Zhu (2016) made the case that the bar for declaring a new factor real needs to be substantially higher than the usual one.

The problem compounds when the same dataset is reused. A strategy refined over many iterations against one history is fitted to that history whether or not any single step looked like curve-fitting. The honest defences are out-of-sample testing on data genuinely held back, testing in other countries and asset classes, and requiring an economic rationale that was specified before the test rather than constructed afterwards to explain the result.

Transaction costs, capacity, and shorting

Paper portfolios rebalance instantly at the closing price. Real ones pay spreads, move the market when they trade in size, and sometimes cannot fill at all. These costs fall hardest on high-turnover strategies and on small, illiquid securities, which is to say on precisely the strategies that tend to look best on paper. Momentum is the clearest case: it is a high-turnover signal by construction, so it is unusually sensitive to what you assume about trading costs.

Long-short factor returns carry a further complication. The academic series assume the short leg can be established and held at no cost. In practice, borrowing small, distressed, or heavily shorted stocks can be expensive or impossible, and borrow costs are highest exactly when the trade is most crowded. A long-only implementation avoids this but captures only part of the factor.

How this affects the data on this site

The series published in the Kenneth R. French Data Library and displayed throughout this dashboard are theoretical long-short portfolio returns, gross of transaction costs, taxes, borrow costs, and management fees. They are a research baseline and are widely used as such, but they are not the returns an investor would have received. The gap between the two is the entire subject of the article on ETFs and theoretical indexes in this section.

Reading a published or self-run backtest well mostly comes down to a few questions: does the sample include companies that no longer exist, is there a lag between accounting data and portfolio formation, how many variations were tried before this one was reported, what turnover does the strategy require, and what would happen to the result if realistic costs were charged against it. A backtest that survives those questions is worth attention. Most do not.

Harvey, C. R., Liu, Y., & Zhu, H. (2016). "... and the Cross-Section of Expected Returns." Review of Financial Studies, 29(1), 5-68.