Reusing Processed Qlib Data to Reduce Repeated Training Setup
Summary
This Qlib demonstration explains how to reuse a processed data handler across repeated model training runs. It first trains the same configured task more than once without explicitly reusing the handler, then constructs the configured data handler in memory and supplies that object to copied task configurations. The intended benefit is avoiding repeated loading and preprocessing of data already held by the handler.
The example also changes the training segment while reusing the handler, illustrating that task settings can be adjusted independently of the cached processed data. Its evidence is procedural: it logs elapsed time for the two approaches, but reports no actual timings or measured speedup. The comments specify that reusing processed data can save reload and preprocessing time, while backtesting can still take substantial time. The document focuses on data preparation workflow rather than a trading method, and does not discuss cache invalidation, memory costs, or whether reuse is appropriate when underlying data or handler settings change.
Key ideas
- Qlib data handlers can be initialized once and reused across training tasks.
- Reusing processed data is intended to reduce repeated disk loading and preprocessing.
- The demonstration compares elapsed-time logging for runs with and without handler reuse but gives no timing results.
- A training segment can be changed while keeping the processed handler in the task configuration.
- The example notes that reuse does not remove the time required by the backtest phase.
Tags
Full text
# data_mem_resuse_demo.py
```py
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
"""
The motivation of this demo
- To show the data modules of Qlib is Serializable, users can dump processed data to disk to avoid duplicated data preprocessing
"""
from copy import deepcopy
from pathlib import Path
import pickle
from pprint import pprint
from ruamel.yaml import YAML
import subprocess
from qlib import init
from qlib.data.dataset.handler import DataHandlerLP
from qlib.log import TimeInspector
from qlib.model.trainer import task_train
from qlib.utils import init_instance_by_config
# For general purpose, we use relative path
DIRNAME = Path(__file__).absolute().resolve().parent
if __name__ == "__main__":
init()
repeat = 2
exp_name = "data_mem_reuse_demo"
config_path = DIRNAME.parent / "benchmarks/LightGBM/workflow_config_lightgbm_Alpha158.yaml"
yaml = YAML(typ="safe", pure=True)
task_config = yaml.load(config_path.open())
# 1) without using processed data in memory
with TimeInspector.logt("The original time without reusing processed data in memory:"):
for i in range(repeat):
task_train(task_config["task"], experiment_name=exp_name)
# 2) prepare processed data in memory.
hd_conf = task_config["task"]["dataset"]["kwargs"]["handler"]
pprint(hd_conf)
hd: DataHandlerLP = init_instance_by_config(hd_conf)
# 3) with reusing processed data in memory
new_task = deepcopy(task_config["task"])
new_task["dataset"]["kwargs"]["handler"] = hd
print(new_task)
with TimeInspector.logt("The time with reusing processed data in memory:"):
# this will save the time to reload and process data from disk(in `DataHandlerLP`)
# It still takes a lot of time in the backtest phase
for i in range(repeat):
task_train(new_task, experiment_name=exp_name)
# 4) User can change other parts exclude processed data in memory(handler)
new_task = deepcopy(task_config["task"])
new_task["dataset"]["kwargs"]["segments"]["train"] = ("20100101", "20131231")
with TimeInspector.logt("The time with reusing processed data in memory:"):
task_train(new_task, experiment_name=exp_name)
```Shown in full with attribution under the source's licence. Licence: MIT
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.