We tested 240 parameter sets. The best showed a Sharpe ratio of 2.1; the median was 0.1; and only 17 cleared 1.0. Those numbers sound like a discovery until you learn that, even when every strategy has zero true edge, a search this wide can produce a flattering winner.
The surprising number is 240. A backtest report usually treats the selected strategy as though it arrived alone. Research does not work that way. Every parameter tweak, feature swap, universe change, and rerun after an ugly result gives noise another chance to look like a signal.
Why does trying more strategies inflate the best Sharpe?
Imagine 240 independent strategies with no real edge and noisy estimated Sharpes. Most estimates land near zero, but the maximum tends to be positive. The more candidates you inspect, the farther that maximum can wander from the truth.
As a rough illustration, suppose each candidate's estimated Sharpe has a standard error of 0.5. With 240 independent draws, the largest estimate can land around 1.4 standard errors above the mean, or roughly 0.7 Sharpe, just from selection. Real strategy candidates are correlated, so this calculation is not a correction you can plug into a report. It shows why “the winner beat zero” is weak evidence after a broad search.
And the search is rarely just the parameter grid. You may have tried three signal families, discarded two after seeing their curves, changed the trading universe, and then rerun the survivor with a different rebalance schedule. If those choices were influenced by performance, they belong in the research history.
What counts as a backtest trial?
Count a decision as a trial when its result could have affected which strategy you chose. That includes runs you did not publish and variants you stopped early because the equity curve looked bad. A compile failure is not a performance trial; a run that produced a disappointing Sharpe usually is.
| Research action | Usually count it? | Why |
|---|---|---|
| Change a lookback from 20 to 30 after reviewing results | Yes | The first result informed the next candidate |
| Run the same fixed strategy on a predeclared new date range | Usually no new selection trial | The rule was fixed before seeing that evidence |
| Change the universe after seeing which assets won | Yes | Universe selection can improve the apparent result |
| Fix a syntax error and rerun the unchanged code | No | No research choice depended on performance |
The dividing line is whether you used an outcome to make a choice. That can be hard to reconstruct from memory, which is why an experiment log should record the hypothesis, code or configuration version, data window, cost assumptions, and result for every run.
Can I correct for multiple testing?
There are methods that estimate selection bias or adjust significance for many trials. They can be useful, but they need assumptions about how candidates relate and how returns behave. A nominal p-value or a single deflated Sharpe estimate cannot rescue a search whose real history you have lost.
Start with the parts you control. Freeze a final specification before evaluating a genuinely untouched period. Keep the whole candidate set visible. When testing many related variants, group them by the decision that produced them instead of pretending each was an independent discovery. Report the search alongside the winner: number of trials, selection rule, and whether the final evaluation influenced further edits.
A useful test: If the holdout result disappoints, will you change the strategy and call the same period a holdout again? If yes, it is now part of research. You need new evidence for the next evaluation.
What should an AI research agent log?
An agent can generate candidates faster than a person can keep them in working memory. That makes trial accounting part of the research system, not a reporting chore. Store each candidate before it runs, including its parent experiment and the reason it exists. Keep failed and discarded runs. Record when a human or agent inspected performance and what decision followed.
Then make the selection process reproducible: given the same candidate set and rule, it should pick the same winner. If an agent keeps changing its prompt, objective, or constraints after seeing results, those changes are part of the search too.
A Sharpe of 2.1 may be worth investigating. Its meaning depends on the path that produced it. When the research log says 240 candidates competed for the headline, you can ask the right next question: what evidence remains that none of those choices has already consumed?
← All posts
