One thousand strategy variants. A measured Sharpe of 2.0 for the winner. A 5% false-positive rate. Those numbers sound like a strong result until you ask how many attempts went into finding it. With enough tries, a convincing backtest can emerge from noise alone.
The surprising number is not the Sharpe. It is the trial count. Every indicator window, entry threshold, universe choice, and discarded idea is another chance to select a lucky result. If your research process forgets those attempts, your confidence calculation does too.
Why does trying more strategies make the winner look better?
Imagine testing 1,000 strategies that have no real edge. Each has a 5% chance of appearing statistically significant under a conventional test. Those trials are not perfectly independent in real research, but the basic intuition holds: the more candidates you inspect, the more likely one will look exceptional by chance.
Even if every candidate were independent, the probability that at least one clears that 5% threshold is about 99.4%. In practice, nearby parameter settings often behave similarly, so 1,000 variants are not 1,000 independent coin flips. But the effective number of attempts is rarely one, either.
Here’s a small trap I’ve seen in notebooks: a researcher tests lookbacks from 5 to 100, plots the Sharpe surface, then reports the clean-looking peak. The surface is useful for understanding sensitivity. The peak is also the product of a search, and it deserves less credit than a threshold chosen before the experiment.
What counts as a research trial?
Count decisions that were influenced by observed results. A trial can be a coded backtest, but it can also be a manual change made after seeing the equity curve. If you widened a stop because the old setting missed a rally, that revision used information from the test period.
| Research action | Usually counts as a trial? | Why |
|---|---|---|
| Testing another signal threshold | Yes | The result helps select a candidate |
| Changing the date range after seeing a drawdown | Yes | The sample was chosen in response to performance |
| Fixing a documented data bug | Usually no | The change restores the intended experiment |
| Running the same frozen strategy on a new holdout period | No new selection trial yet | It becomes one if you tune against that result |
There’s judgment in the boundary. A typo fix is different from changing a feature after noticing where it failed. Keep a short trial log with the hypothesis, change, data period, and outcome. It needn’t be elegant; a dated CSV is better than reconstructing your search from memory three months later.
Can I correct my Sharpe for multiple testing?
Methods such as the Deflated Sharpe Ratio estimate how much apparent performance could arise from selection among many candidates, while accounting for factors such as return distribution and trial dependence. They are useful diagnostics, not a machine that converts a messy research process into certainty. Their outputs depend on assumptions and on whether your trial count is credible.
For a rough feel, suppose a researcher tested 400 variants, found a Sharpe of 1.8, and estimates that their returns have substantial skew and fat tails. That result should face a tougher hurdle than a Sharpe of 1.8 from one pre-registered test. The correction may weaken the evidence considerably; it cannot tell you that the edge is real.
Record the search before you need to defend it: candidate count, parameter ranges, data periods viewed, and any manual revisions. If those details are unknown, describe the backtest as exploratory.
What should happen before paper trading?
Freeze one candidate and define the next evaluation before you run it. A useful sequence is to choose the strategy using training data, inspect it on validation data, then reserve a final period or paper-trading window that does not feed back into design. Each stage answers a different question; repeatedly adjusting the strategy on the final stage turns it into more training data.
Also inspect whether nearby settings tell the same story. A broad patch of modestly positive results is usually more credible than one needle-like peak, though robustness does not rescue a flawed execution model. Fees, funding, impact, and instrument availability still need to match what the strategy could have faced.
Paper trading adds evidence about operational parity: timing, order handling, and whether the live research pipeline behaves as expected. It cannot erase the selection that already happened. A thousand lucky backtests stay a thousand attempts when the first paper order goes out.
So when a backtest reports a Sharpe of 2, ask for the research history beside the equity curve. How many ideas were tried? Which results changed the next experiment? The answer may be the most important statistic in the report.
← All posts
