Configuring High-Frequency Qlib Data Splits and Normalization
Summary
This configuration describes a Qlib data pipeline for five-minute CSI 300 constituent data over a specified two-year period. It defines training, validation, and test segments, with the training interval also used to fit feature processing. The feature dataset uses a high-frequency handler with open, high, low, and close fields, while the backtest dataset uses close and volume.
The setup specifies a 240-minute trading day, price and volume normalization groups, and a time-range filter. It also points to local paths for generated data and normalization statistics. This is an example configuration rather than a complete methodology: it does not explain how the instruments are selected, what model is trained, how trading decisions are made, or how performance is evaluated. The date ranges and settings are specific to this dataset and may need adjustment for other data sources or market calendars.
Key ideas
- The configuration sets five-minute frequency data for a CSI 300 instrument set.
- Training, validation, and test intervals are defined separately.
- Feature processing uses OHLC fields and grouped normalization for price and volume.
- The backtest dataset uses close and volume data with the same time segmentation.
- The file specifies data pipeline settings but does not describe a strategy or its results.
Tags
Full text
# pickle_data_config.yml
```yml
# start & end time for training/validation/test datasets
start_time: !!str &start 2020-01-01
end_time: !!str &end 2021-12-31
train_end_time: !!str &tend 2021-06-30
valid_start_time: !!str &vstart 2021-07-01
valid_end_time: !!str &vend 2021-09-30
test_start_time: !!str &tstart 2021-10-01
# the instrument set
instruments: &ins csi300s19_22
# qlib related configuration
qlib_conf:
provider_uri:
5min: ./data/bin # path to generated qlib bin
redis_port: 233
feature_conf:
path: ./data/pickle/feature.pkl # output path of feature
class: DatasetH
module_path: qlib.data.dataset
kwargs:
handler:
class: HighFreqGeneralHandler
module_path: qlib.contrib.data.highfreq_handler
kwargs:
start_time: *start
end_time: *end
fit_start_time: *start
fit_end_time: *tend
instruments: *ins
day_length: 240 # how many minutes in one trading day
freq: 5min
columns: ["$open", "$high", "$low", "$close"]
infer_processors:
- class: HighFreqNorm
module_path: qlib.contrib.data.highfreq_processor
kwargs:
feature_save_dir: ./stat/ # output path of statistics of features (for feature normalization)
norm_groups:
price: 8
volume: 2
inst_processors:
- class: TimeRangeFlt
module_path: qlib.data.dataset.processor
kwargs:
start_time: "2020-01-01"
end_time: "2021-12-31"
freq: 5min
segments:
train: !!python/tuple [*start, *tend]
valid: !!python/tuple [*vstart, *vend]
test: !!python/tuple [*tstart, *end]
backtest_conf:
path: ./data/pickle/backtest.pkl # output path of backtest
class: DatasetH
module_path: qlib.data.dataset
kwargs:
handler:
class: HighFreqGeneralBacktestHandler
module_path: qlib.contrib.data.highfreq_handler
kwargs:
start_time: *start
end_time: *end
instruments: *ins
day_length: 240
freq: 5min
columns: ["$close", "$volume"]
inst_processors:
- class: TimeRangeFlt
module_path: qlib.data.dataset.processor
kwargs:
start_time: "2020-01-01"
end_time: "2021-12-31"
freq: 5min
segments:
train: !!python/tuple [*start, *tend]
valid: !!python/tuple [*vstart, *vend]
test: !!python/tuple [*tstart, *end]
freq: 5min
```Shown in full with attribution under the source's licence. Licence: MIT
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.