Qlib Client-Server Data Access and Local Offline Configuration
Article Qlib
Summary
This Qlib documentation explains how a client can access market data managed on a central server. The client configuration points to a provider location, a local mount path, and a data service endpoint; NFS mounts the shared files, while a Flask service supports client-server communication. Examples show initializing Qlib and requesting closing prices for a Chinese stock. The document also describes Windows NFS setup and notes that mount paths must follow its stated drive-letter format.
For offline use, it shows a separate configuration that points Qlib at a local data directory and selects local calendar, instrument, feature, expression, and dataset providers. The examples demonstrate data retrieval but do not compare performance or validate a trading strategy. The stated limitations are slower performance for some calendar and instrument-list APIs than the older offline API, and incorrect updates for rolling expressions with parameter zero. The server addresses and setup details are specific to the documented environment and may not apply elsewhere.
Key ideas
- The client-server framework centralizes Qlib data and allows remote access through a mounted data directory and service connection.
- Qlib initialization requires configuration for the provider location, mount path, and service connection.
- The document provides separate setup examples for Linux, PAI, and Windows environments.
- A local configuration can use Qlib data without the client-server connection.
- Some client-server APIs may be slower, and rolling expressions with parameter zero have a stated update limitation.
Tags
Full text
# client
.. _client:
Qlib Client-Server Framework
============================
.. currentmodule:: qlib
Introduction
------------
Client-Server is designed to solve following problems
- Manage the data in a centralized way. Users don't have to manage data of different versions.
- Reduce the amount of cache to be generated.
- Make the data can be accessed in a remote way.
Therefore, we designed the client-server framework to solve these problems.
We will maintain a server and provide the data.
You have to initialize you qlib with specific config for using the client-server framework.
Here is a typical initialization process.
qlib ``init`` commonly used parameters; ``nfs-common`` must be installed on the server where the client is located, execute: ``sudo apt install nfs-common``:
- ``provider_uri``: nfs-server path; the format is ``host: data_dir``, for example: ``172.23.233.89:/data2/gaochao/sync_qlib/qlib``. If using offline, it can be a local data directory
- ``mount_path``: local data directory, ``provider_uri`` will be mounted to this directory
- ``auto_mount``: whether to automatically mount ``provider_uri`` to ``mount_path`` during qlib ``init``; You can also mount it manually: sudo mount.nfs ``provider_uri`` ``mount_path``. If on PAI, it is recommended to set ``auto_mount=True``
- ``flask_server``: data service host; if you are on the intranet, you can use the default host: 172.23.233.89
- ``flask_port``: data service port
If running on 10.150.144.153 or 10.150.144.154 server, it's recommended to use the following code to ``init`` qlib:
.. code-block:: python
>>> import qlib
>>> qlib.init(auto_mount=False, mount_path='/data/csdesign/qlib')
>>> from qlib.data import D
>>> D.features(['SH600000'], ['$close'], start_time='20080101', end_time='20090101').head()
[39336:MainThread](2019-05-28 21:35:42,800) INFO - Initialization - [__init__.py:16] - default_conf: client.
[39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:54] - qlib successfully initialized based on client settings.
[39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:56] - provider_uri=172.23.233.89:/data2/gaochao/sync_qlib/qlib
[39336:Thread-68](2019-05-28 21:35:42,809) INFO - Client - [client.py:28] - Connect to server ws://172.23.233.89:9710
[39336:Thread-72](2019-05-28 21:35:43,489) INFO - Client - [client.py:31] - Disconnect from server!
Opening /data/csdesign/qlib/cache/d239a3b191daa9a5b1b19a59beb47b33 in read-only mode
Out[5]:
$close
instrument datetime
SH600000 2008-01-02 119.079704
2008-01-03 113.120125
2008-01-04 117.878860
2008-01-07 124.505539
2008-01-08 125.395004
If running on PAI, it's recommended to use the following code to ``init`` qlib:
.. code-block:: python
>>> import qlib
>>> qlib.init(auto_mount=True, mount_path='/data/csdesign/qlib', provider_uri='172.23.233.89:/data2/gaochao/sync_qlib/qlib')
>>> from qlib.data import D
>>> D.features(['SH600000'], ['$close'], start_time='20080101', end_time='20090101').head()
[39336:MainThread](2019-05-28 21:35:42,800) INFO - Initialization - [__init__.py:16] - default_conf: client.
[39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:54] - qlib successfully initialized based on client settings.
[39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:56] - provider_uri=172.23.233.89:/data2/gaochao/sync_qlib/qlib
[39336:Thread-68](2019-05-28 21:35:42,809) INFO - Client - [client.py:28] - Connect to server ws://172.23.233.89:9710
[39336:Thread-72](2019-05-28 21:35:43,489) INFO - Client - [client.py:31] - Disconnect from server!
Opening /data/csdesign/qlib/cache/d239a3b191daa9a5b1b19a59beb47b33 in read-only mode
Out[5]:
$close
instrument datetime
SH600000 2008-01-02 119.079704
2008-01-03 113.120125
2008-01-04 117.878860
2008-01-07 124.505539
2008-01-08 125.395004
If running on Windows, open **NFS** features and write correct **mount_path**, it's recommended to use the following code to ``init`` qlib:
1.windows System open NFS Features
* Open ``Programs and Features``.
* Click ``Turn Windows features on or off``.
* Scroll down and check the option ``Services for NFS``, then click OK
Reference address: https://graspingtech.com/mount-nfs-share-windows-10/
2.config correct mount_path
* In windows, mount path must be not exist path and root path,
* correct format path eg: `H`, `i`...
* error format path eg: `C`, `C:/user/name`, `qlib_data`...
.. code-block:: python
>>> import qlib
>>> qlib.init(auto_mount=True, mount_path='H', provider_uri='172.23.233.89:/data2/gaochao/sync_qlib/qlib')
>>> from qlib.data import D
>>> D.features(['SH600000'], ['$close'], start_time='20080101', end_time='20090101').head()
[39336:MainThread](2019-05-28 21:35:42,800) INFO - Initialization - [__init__.py:16] - default_conf: client.
[39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:54] - qlib successfully initialized based on client settings.
[39336:MainThread](2019-05-28 21:35:42,801) INFO - Initialization - [__init__.py:56] - provider_uri=172.23.233.89:/data2/gaochao/sync_qlib/qlib
[39336:Thread-68](2019-05-28 21:35:42,809) INFO - Client - [client.py:28] - Connect to server ws://172.23.233.89:9710
[39336:Thread-72](2019-05-28 21:35:43,489) INFO - Client - [client.py:31] - Disconnect from server!
Opening /data/csdesign/qlib/cache/d239a3b191daa9a5b1b19a59beb47b33 in read-only mode
Out[5]:
$close
instrument datetime
SH600000 2008-01-02 119.079704
2008-01-03 113.120125
2008-01-04 117.878860
2008-01-07 124.505539
2008-01-08 125.395004
The client will mount the data in `provider_uri` on `mount_path`. Then the server and client will communicate with flask and transporting data with this NFS.
If you have a local qlib data files and want to use the qlib data offline instead of online with client server framework.
It is also possible with specific config.
You can created such a config. `client_config_local.yml`
.. code-block:: YAML
provider_uri: /data/csdesign/qlib
calendar_provider: 'LocalCalendarProvider'
instrument_provider: 'LocalInstrumentProvider'
feature_provider: 'LocalFeatureProvider'
expression_provider: 'LocalExpressionProvider'
dataset_provider: 'LocalDatasetProvider'
provider: 'LocalProvider'
dataset_cache: 'SimpleDatasetCache'
local_cache_path: '~/.cache/qlib/'
`provider_uri` is the directory of your local data.
.. code-block:: python
>>> import qlib
>>> qlib.init_from_yaml_conf('client_config_local.yml')
>>> from qlib.data import D
>>> D.features(['SH600001'], ['$close'], start_time='20180101', end_time='20190101').head()
21232:MainThread](2019-05-29 10:16:05,066) INFO - Initialization - [__init__.py:16] - default_conf: client.
[21232:MainThread](2019-05-29 10:16:05,066) INFO - Initialization - [__init__.py:54] - qlib successfully initialized based on client settings.
[21232:MainThread](2019-05-29 10:16:05,067) INFO - Initialization - [__init__.py:56] - provider_uri=/data/csdesign/qlib
Out[9]:
$close
instrument datetime
SH600001 2008-01-02 21.082111
2008-01-03 23.195362
2008-01-04 23.874615
2008-01-07 24.880930
2008-01-08 24.277143
Limitations
-----------
1. The following API under the client-server module may not be as fast as the older off-line API.
- Cal.calendar
- Inst.list_instruments
2. The rolling operation expression with parameter `0` can not be updated rightly under mechanism of the client-server framework.
API
***
The client is based on `python-socketio <https://python-socketio.readthedocs.io>`_ which is a framework that supports WebSocket client for Python language. The client can only propose requests and receive results, which do not include any calculating procedure.
Class
-----
.. automodule:: qlib.data.clientShown in full with attribution under the source's licence. Licence: MIT
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.