Saving and Reloading Qlib Data and Model Objects
Summary
The document explains how Qlib serializes objects such as data handlers, datasets, processors, and models to disk using pickle-compatible formats. A Serializable object saves public attributes by default, with options to configure what is included or select a different backend. It gives a dataset save-and-load example to illustrate the workflow.
A key limitation is that serialization preserves object state, not the underlying market data. After loading a dataset, users may need to reinitialize it with current instruments, date ranges, or data segments so it can generate data again. The document offers implementation guidance for managing reusable research objects, but it presents no trading strategy, performance evidence, or evaluation of how serialized state affects results.
Key ideas
- Qlib Serializable objects can save their state to disk and be loaded again.
- By default, public attributes are saved, with configuration options to change that behavior.
- The pickle backend can be changed to support additional object types.
- Dataset state may preserve normalization statistics, while market data itself must be regenerated.
- Reloaded datasets may need initialization with updated instruments, dates, or segments.
Tags
Full text
# serial
.. _serial:
=============
Serialization
=============
.. currentmodule:: qlib
Introduction
============
``Qlib`` supports dumping the state of ``DataHandler``, ``DataSet``, ``Processor`` and ``Model``, etc. into a disk and reloading them.
Serializable Class
==================
``Qlib`` provides a base class ``qlib.utils.serial.Serializable``, whose state can be dumped into or loaded from disk in `pickle` format.
When users dump the state of a ``Serializable`` instance, the attributes of the instance whose name **does not** start with `_` will be saved on the disk.
However, users can use ``config`` method or override ``default_dump_all`` attribute to prevent this feature.
Users can also override ``pickle_backend`` attribute to choose a pickle backend. The supported value is "pickle" (default and common) and "dill" (dump more things such as function, more information in `here <https://pypi.org/project/dill/>`_).
Example
=======
``Qlib``'s serializable class includes ``DataHandler``, ``DataSet``, ``Processor`` and ``Model``, etc., which are subclass of ``qlib.utils.serial.Serializable``.
Specifically, ``qlib.data.dataset.DatasetH`` is one of them. Users can serialize ``DatasetH`` as follows.
.. warning::
The direct local ``pickle.load`` below can execute code and must only read a
file from an independently trusted source and store. Recorder/workflow
``trusted`` settings do not authorize or protect this API. For the separate
recorder loading policy and upgrade steps, see :ref:`artifact_loading_migration`.
.. code-block:: Python
##=============dump dataset=============
dataset.to_pickle(path="dataset.pkl") # dataset is an instance of qlib.data.dataset.DatasetH
##=============reload dataset=============
with open("dataset.pkl", "rb") as file_dataset:
dataset = pickle.load(file_dataset)
.. note::
Only state of ``DatasetH`` should be saved on the disk, such as some `mean` and `variance` used for data normalization, etc.
After reloading the ``DatasetH``, users need to reinitialize it. It means that users can reset some states of ``DatasetH`` or ``QlibDataHandler`` such as `instruments`, `start_time`, `end_time` and `segments`, etc., and generate new data according to the states (data is not state and should not be saved on the disk).
A more detailed example is in this `link <https://github.com/microsoft/qlib/tree/main/examples/highfreq>`_.
API
===
Please refer to `Serializable API <../reference/api.html#module-qlib.utils.serial.Serializable>`_.Shown in full with attribution under the source's licence. Licence: MIT
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.