Chunked Minute-Bar Data Loading to Prevent Transformer OOM
Summary
This guide explains why a submitted end-to-end Transformer example can run out of memory before training or inference begins: the data-loading step materializes minute bars for many stocks across the full date range in one DataFrame. Although the eventual model inputs are a smaller set of daily lookback windows, the full raw dataset can exhaust memory before those windows are extracted.
The proposed remedy is to query stocks in batches, preserve each stock’s continuous time history while building windows, and release each batch’s raw rows and tensors as soon as they are no longer needed. It also describes computing normalization statistics incrementally from sums, squared sums, and counts, and predicting batch by batch while retaining only compact date, instrument, and score outputs. The article gives implementation sketches and recommends selecting only needed columns or using a streaming reader. Its guidance targets memory management; it reports no benchmark or trading results, and batch size still needs to suit the available runtime resources.
Key ideas
- Loading all minute bars into one DataFrame can exhaust memory during dataset construction.
- Batching by instrument preserves the continuous history needed to form lookback windows.
- Releasing raw data and tensors after each batch limits the amount held in memory.
- Normalization statistics can be accumulated incrementally instead of retaining all windows.
- Batch inference can keep only compact prediction outputs, though batch size affects memory and runtime.
Tags
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.