Factor Backtests: The Pitfalls of Simulation
"Past performance is not indicative of future results." It is the most ignored disclaimer in finance. In the world of factor investing, backtests (simulating how a strategy would have performed historically) are ubiquitous. However, live results almost always degrade compared to the theoretical simulation.
1. The Transaction Cost Reality
Most academic data (including the Fama-French library data used here) ignores transaction costs. They assume you can buy and sell thousands of stocks instantly at the closing price.
The Momentum Problem: Momentum strategies require high turnover (often 100-200% per year). In the real world, the bid-ask spreads and commissions to execute these trades can consume 2-3% of the alpha annually. A backtest showing a 5% premium might actually deliver 2% net of fees.
2. The Micro-Cap Illusion
Many factor premiums are strongest in "micro-cap" stocks (the smallest of the small).
- Backtest: Buying the cheapest 10% of micro-caps generates massive returns.
- Reality: These stocks are illiquid. You might not be able to deploy even $1 million without moving the price 10% against yourself.
3. P-Hacking and Data Mining
With enough computing power, you can find a pattern in any dataset."Buying stocks with tickers starting with 'A' on Tuesdays when it rains in London"might show statistical significance over a 20-year period purely by chance.
Harvey, Liu, and Zhu (2016) argued that there is a "Zoo of Factors" - hundreds of supposed factors published in academia, most of which are statistical mirages. They suggest that for a factor to be credible today, it needs a much higher statistical hurdle (t-stat greater than 3.0) and a sensible economic rationale.
4. The Definition Game
How do you define "Value"?
Is it Price-to-Book? Price-to-Earnings? Enterprise Value-to-EBITDA?
Do you rebalance monthly, quarterly, or annually?
A researcher might run 1,000 variations of a strategy and publish the one that worked best. This "overfitting" ensures the backtest looks perfect, but the strategy is tuned to the noise of the past, not the signal of the future.
How to Evaluate a Factor Strategy
When looking at a factor fund or strategy, ask:
- Is it robust? Does it work across different time periods, different geographies (e.g., Japan, Europe), and slight definition changes?
- Is it implementable? Does the fund manager account for trading costs and liquidity?
- Is there a reason? Is there a risk-based or behavioral explanation for why this premium exists?
Harvey, Campbell R., Yan Liu, and Heqing Zhu. "…and the Cross-Section of Expected Returns." The Review of Financial Studies, 2016.