Almost any strategy can be made to show a great backtest. The skill isn't running the backtest — it's knowing which results to distrust.
The five ways backtests lie
1. Look-ahead bias. The strategy uses information that wouldn't have been available at the time of the trade — a common bug when regime labels, fundamentals, or "next day's close" leak into a signal computed on today's bar.
2. Overfitting to the sample. Enough parameters, tuned on enough historical data, will fit noise. If a strategy needed 12 tunable thresholds to look good over one 2-year window, it's memorized that window, not found an edge.
3. Survivorship bias. Backtesting only on assets that still exist today silently excludes the ones that went to zero or got delisted — inflating results for anything long-biased.
4. No transaction costs, or unrealistic ones. Slippage and fees erase a large fraction of high-frequency mean-reversion edges specifically. A backtest with 0% costs is a backtest of a strategy nobody can trade.
5. Regime cherry-picking. Reporting the result only over the window where the strategy happened to work, without disclosing how it did in other regimes.
What a trustworthy backtest actually needs
- The exact code that would trade live, not a simplified reimplementation for the backtest. If the backtest and the live strategy are different code paths, a bug fix to one doesn't reach the other, and the two silently drift.
- Out-of-sample and walk-forward validation, not just an in-sample fit.
- A breakdown by market regime (trending, ranging, high-volatility) — a strategy's edge is rarely regime-independent, and the honest version of a backtest says so.
- Realistic fee and slippage assumptions stated explicitly, not omitted.
- Enough trades for the win rate to mean anything. Twelve trades is an anecdote, not a track record.
A minimal pre-trust checklist
Before acting on any backtest — including a vendor's, including your own:
- Does it reuse the live signal code, or a reimplementation?
- Is there a stated fee/slippage model, and is it realistic for the asset class?
- Does performance hold up out-of-sample, not just on the fitted window?
- Is there a per-regime breakdown, and does the strategy have a real edge in more than one regime?
- How many independent trades does the result rest on?
If a backtest can't answer these, the honest response isn't "prove it's wrong" — it's "it hasn't earned trust yet." That's the bar we hold our own strategy backtests to, which is why every backtest run in Atradian's Strategy Lab replays the same signal functions the bot trades live, reports a per-regime breakdown, and lets you queue your own hypothesis against it rather than take our word for the result.