Assessing Factor Reliability When Backtest Trade Counts Differ
Summary
The document raises a question about comparing backtested factors when they produce different numbers of trades. Its example contrasts one factor with 100 trades and a 2:1 profit-loss ratio against another with 1,000 trades and a 1.5:1 ratio, asking whether the higher observed ratio or the larger sample should count more when judging performance.
It notes the intuition that more observations may make results more representative of underlying probabilities, but does not provide an answer or statistical method. The post contains no data beyond the illustrative comparison and no discussion of return distributions, dependence among trades, uncertainty intervals, or multiple testing. It is useful as a framing of the sample-size and reliability problem, not as a validated comparison procedure.
Key ideas
- The post asks how trade count should affect confidence in a factor’s backtested performance.
- Its example compares a higher profit-loss ratio from fewer trades with a lower ratio from more trades.
- It offers no statistical method for quantifying uncertainty or deciding which factor is superior.
- Trade count alone is insufficient to establish that a backtest result will generalize.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.