Speeding Up Machine Learning Strategy Backtests with Parallelism and Chunked Data

Summary

This Chinese-language post describes an optimization of a machine-learning strategy template for the BigQuant platform. The author reports reducing a two-year backtest from 65 minutes to 26 seconds by using parallel computation and delayed execution. The post also describes loading data in segments to control memory use, and says a two-core, 16 GB server completed a ten-year run in a little over two minutes. These examples are presented as the author’s own experience rather than as a controlled benchmark.

The template is also designed to accept added factors without manually changing a factor list, and it uses a platform trading component intended to provide fuller backtest reporting. The motivation is to make repeated strategy testing practical without relying on a high-memory local computer or slow data retrieval. The post provides no details here about the exact implementation, hardware comparison, strategy returns, or validation of results. The author warns that the hand-built program may still contain bugs, so the reported speed gains do not establish strategy quality or backtest reliability.

Key ideas

• Parallel computation and delayed execution are used to shorten the reported machine-learning backtest time.
• Segmented data loading is intended to reduce memory pressure during long historical runs.
• The example reports large speed improvements, but it does not provide a controlled benchmark methodology.
• The template is designed to accept additional factors without editing a fixed factor list.
• The author cautions that the hand-built program may contain bugs, and speed alone says nothing about strategy performance.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.