Comparing Online Learning Speeds with Test-Then-Train Backtests
Summary
The article compares four logistic-regression update schedules for predicting the direction of the next bar: a fixed model, per-bar online updates, periodic rolling retraining, and updates triggered by deteriorating recent loss. The models share the same features, initialization, trading thresholds, return calculation, and cost assumption, and run as synchronized shadow accounts. The design emphasizes timing discipline: generate a forecast after one bar closes, evaluate it against the next bar, then train only after its label becomes available. It also describes checks for completed bars and explains why the model’s residual predictive quality, parameter movement, turnover, and simulated equity all matter.
In the reported default-parameter backtest, frequent online and rolling updates moved parameters without reducing out-of-sample Log Loss. Error gating reduced turnover and direct reversals but did not produce a tradable edge. The author treats this as one comparison run, not a general result. The simulation uses simplified open-to-close returns and standardized costs; actual fills, slippage, and market impact may differ. Further tests across instruments, timeframes, regimes, and nearby settings are needed, and the error trigger is not a formal drift detector.
Key ideas
- The experiment isolates update frequency by keeping model structure, features, trading rules, and cost assumptions constant.
- Predictions are evaluated before the newly observed sample is used for training.
- Faster updates can increase parameter movement and turnover without improving out-of-sample loss.
- The reported single backtest found no tradable edge for the tested update methods.
- Simplified execution costs and a single test setting limit the conclusions.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.