Reducing Memory Use and Runtime in ETF Rotation Backtests

Summary

This note describes ways to reduce memory use and speed up parameter tuning for an ETF rotation strategy. It recommends moving suitable calculations into SQL, loading only needed data, reusing cached data, releasing large objects, precomputing values during initialization rather than recalculating them for each bar, and using parallel computation for long-running tasks. The author also proposes combining multiprocessing on individual machines with distributed cluster execution.

The document reports a comparison in which an unoptimized run exceeded available memory and crashed, while a tuned parallel run completed parameter search with less memory and in less time. These are platform-specific observations; the note does not provide the strategy rules, code-level profiling, or enough experimental detail to establish general performance gains. Precomputing ETF weights and using a two-level parallel setup are presented as ideas to investigate, not completed optimizations with measured effects.

Key ideas

• The note recommends limiting data loads, reusing cached inputs, and releasing large objects to manage memory.
• Calculations can be batched during initialization instead of repeated for every time step.
• SQL and parallel computation may reduce Python workload and parameter-tuning time.
• The author reports a platform-specific memory and runtime comparison, while proposed precomputation and cluster scaling remain untested ideas.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.