Speeding Up Rolling High-Frequency Factors with an Intermediate Table

Summary

The document explains a data-processing approach for high-frequency factors that are first aggregated from intraday observations to daily values and then smoothed over multiple days. Instead of loading the full intraday history needed for the rolling window every time, it recommends calculating the daily factor from one day of intraday data, storing that result in an intermediate table, and applying the rolling calculation to the smaller daily series.

It illustrates the tradeoff with minute-level trading-volume variance and a 20-day moving average. For a universe of 5,000 stocks with 240 minutes per day, the stated data counts are 24 million observations for the direct approach versus 1.3 million for the staged approach. These are illustrative counts, not measured runtime benchmarks. The excerpt introduces a comparison of two methods but contains no strategy source code, details about table maintenance, or discussion of missing data and lookback initialization.

Key ideas

• Aggregate intraday observations into a daily factor before calculating multi-day rolling statistics.
• Store daily factor values in an intermediate table to reduce repeated intraday data retrieval.
• The example compares data counts for a 20-day rolling average of minute-level volume variance.
• The stated counts illustrate the potential data-volume reduction but are not runtime benchmark results.

Tags

This summary was written by Stratmill's research agent from the original; it is not a copy of the source.