Adding more markets to a strategy search can make the best backtest look stronger while making the evidence weaker. A higher Sharpe after adding symbols is often treated as diversification; if you searched those symbols for the winner, you also increased the number of chances to get lucky.
That distinction matters whether an agent proposes hundreds of signals or a researcher tests a handful by hand. The search space includes every market, timeframe, feature, and parameter combination whose result influenced the final choice. “We only shipped one strategy” doesn’t tell you how many candidates auditioned.
Why can a larger market universe inflate the best Sharpe?
Imagine testing the same strategy template on 40 unrelated markets. Assume, for simplicity, that every market’s true Sharpe is zero and each estimated Sharpe has a standard error of 0.5. The estimates bounce around zero. Pick the highest one, though, and it will usually be positive simply because you picked a maximum.
Under a rough normal approximation, the largest of 40 independent noise estimates is expected to land around 1.3 standard errors above zero: roughly 0.65 Sharpe in this example. The exact number depends on dependence, sample length, and the distribution of returns. The point is that selection creates optimism even when no strategy has an edge.
Keep the search log. Record every market and candidate that was inspected, including failed runs and variants that were abandoned. The final strategy’s trial count is about what informed the choice, not how many files survived cleanup.
Markets aren’t independent coin flips. BTC and ETH often move together; two equity indices may share most of their risk. That reduces the effective number of independent trials, but it doesn’t make the problem disappear. And correlated markets can still differ in fees, funding, liquidity, and the hours when a signal works.
Does testing more markets count as multiple testing?
It does when you use the results to decide what to keep. If the original hypothesis was “this rule captures a funding effect across major perpetuals,” looking across markets is part of testing that hypothesis. If you test 50 markets, find the best, then write the hypothesis around its behavior, universe selection has become part of the optimization.
There’s a useful difference between breadth chosen in advance and breadth chosen after seeing results. A fixed basket can reveal whether an idea is robust across a stated domain. A basket expanded until one symbol produces a striking equity curve is a search procedure. Both can be legitimate research, but they answer different questions.
| Research choice | What it can support | What it cannot establish alone |
|---|---|---|
| Predeclared basket, equal treatment | Whether the rule behaves across the chosen markets | That the basket represents every tradable market |
| Best-performing market selected after tests | Which tested market had the strongest historical result | That its selected Sharpe is an unbiased estimate |
| Markets held out before search | Whether a frozen rule transfers to untouched markets | That future conditions will match the holdout |
How do I test whether an edge transfers to new markets?
Freeze the research choices before opening the holdout. That means the signal definition, parameters, universe rules, rebalance schedule, and execution assumptions. Then evaluate the unchanged strategy on markets that did not influence those choices.
For example, suppose a crypto momentum rule is developed on BTC and ETH perpetuals through 2023. You could reserve a set of other liquid perpetuals, chosen by a rule that uses only information available before the test period, and keep them out of all tuning. Apply the frozen rule to those contracts in 2024 and 2025. Include each venue’s actual fee tier, funding payments, contract specifications, and a defensible impact assumption; a transfer test without market-specific trading costs can mistake a cost difference for a signal failure—or hide one.
Report the distribution across held-out markets, not only the pooled Sharpe. A pooled portfolio can conceal that one market supplied nearly all the return. Show each market’s net result, trade count, drawdown, and the share of total P&L contributed by the largest one. If the rule works only on one contract, call it a contract-specific result.
Keep the time split honest, too. A market can be untouched while its data period overlaps the same regime that drove development performance. A cross-market holdout asks whether the rule travels across instruments; a later-period holdout asks whether it travels through time. Those are separate tests, and a strong result on one doesn’t substitute for the other.
What if the strategy really should use more markets?
Then include them, but define how. A portfolio rule that trades every eligible market according to a fixed signal is different from a contest in which the best historical market wins. Eligibility should be determined without looking at future performance: for example, by a point-in-time liquidity threshold and a minimum history requirement. Membership changes need timestamps, too, or a current list of “liquid markets” can smuggle today’s survivors into an old backtest.
Critics are right that a narrow universe can produce a brittle strategy and that broad testing is a valuable robustness check. The answer isn’t to avoid testing markets. It’s to separate markets used to discover an idea from markets used to challenge the frozen idea, and to count the discovery work as part of the evidence.
A bigger universe can help a strategy survive outside the lab. It can also help a lucky backtest win the lab’s selection contest. The search log and the holdout tell you which story your Sharpe supports.
← All posts
