Training-Set Size Can Change Backtest Results

Summary

This brief troubleshooting exchange addresses why two backtests can report different returns even when their factors and backtest dates match. The response attributes the mismatch primarily to a difference in training-set data volume. It adds that using the same amount of training data can produce the same result, pointing to a shared experiment as an illustration.

The takeaway is that matching the factor definitions and evaluation period may not be enough to reproduce a run when model training is involved; the amount of training data can also affect the result. The exchange is very short and gives no detail about the underlying model, data selection, validation setup, or other settings that might contribute to discrepancies. It offers no reported returns or controlled comparison, so the explanation is a practical diagnostic rather than a general proof that training-set size is the sole cause.

Key ideas

• Backtests with matching factors and evaluation dates may still differ when their training data volumes differ.
• The response identifies training-set size as the main cause of the discrepancy in this case.
• It says matching the training data amount can reproduce the same result.
• The exchange does not describe the model or rule out other differences in backtest configuration.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.