Training and Testing an EIIE Reinforcement Learning Portfolio
Notebook FinRL
Summary
This notebook demonstrates a portfolio optimization workflow using FinRL’s PortfolioOptimizationEnv and the EIIE policy architecture. It downloads data for ten Brazilian stocks, scales each stock’s series, trains a policy-gradient agent on an earlier period, and evaluates it on three later calendar-year periods. The environment uses a 50-day observation window, close, high, and low features, an initial cash amount, and a commission assumption. Results are plotted against a uniform buy-and-hold portfolio that allocates equal weights to each stock with no cash reserve.
The notebook reports qualitatively that the learned policy performs worse as the evaluation period moves further beyond its training data, and suggests retraining on more recent data to improve later-period results. It supplies plots rather than numerical performance statistics, and does not establish that the learned policy outperforms the baseline. The example uses a fixed stock universe and historical data, so its findings are limited by the chosen assets, periods, preprocessing, and simulation assumptions; the demonstration alone does not establish future performance.
Key ideas
- The notebook uses the EIIE architecture with a portfolio optimization environment and policy-gradient training.
- The example trains on historical Brazilian stock data and evaluates the policy in three subsequent calendar years.
- Its benchmark equally weights the selected assets and holds them without a cash allocation.
- The notebook reports weaker results in later evaluation periods and suggests using more recent training data.
- Plots are provided, but the document gives no numerical performance statistics or proof of persistent outperformance.
Tags
Full text
# A guide Portfolio Optimization Environment
# A guide Portfolio Optimization Environment
This notebook aims to provide an example of using PortfolioOptimizationEnv (or POE) to train a reinforcement learning model that learns to solve the portfolio optimization problem.
In this document, we will reproduce a famous architecture called EIIE (ensemble of identical independent evaluators), introduced in the following paper:
- Zhengyao Jiang, Dixing Xu, & Jinjun Liang. (2017). A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem. https://doi.org/10.48550/arXiv.1706.10059.
It's advisable to read it to understand the algorithm implemented in this notebook.
### Note
If you're using this environment, consider citing the following paper (in adittion to FinRL references):
- Caio Costa, & Anna Costa (2023). POE: A General Portfolio Optimization Environment for FinRL. In *Anais do II Brazilian Workshop on Artificial Intelligence in Finance* (pp. 132–143). SBC. https://doi.org/10.5753/bwaif.2023.231144.
```
@inproceedings{bwaif,
author = {Caio Costa and Anna Costa},
title = {POE: A General Portfolio Optimization Environment for FinRL},
booktitle = {Anais do II Brazilian Workshop on Artificial Intelligence in Finance},
location = {João Pessoa/PB},
year = {2023},
keywords = {},
issn = {0000-0000},
pages = {132--143},
publisher = {SBC},
address = {Porto Alegre, RS, Brasil},
doi = {10.5753/bwaif.2023.231144},
url = {https://sol.sbc.org.br/index.php/bwaif/article/view/24959}
}
```
## Installation and imports
To run this notebook in google colab, uncomment the cells below.
```python
## install finrl library
# !sudo apt install swig
# !pip install git+https://github.com/AI4Finance-Foundation/FinRL.git
```
```python
## We also need to install quantstats, because the environment uses it to plot graphs
# !pip install quantstats
```
```python
## Hide matplotlib warnings
# import warnings
# warnings.filterwarnings('ignore')
import logging
logging.getLogger('matplotlib.font_manager').disabled = True
```
#### Import the necessary code libraries
```python
import torch
from sklearn.preprocessing import MaxAbsScaler
from finrl.meta.preprocessor.yahoodownloader import YahooDownloader
from finrl.meta.preprocessor.preprocessors import GroupByScaler
from finrl.meta.env_portfolio_optimization.env_portfolio_optimization import PortfolioOptimizationEnv
from finrl.agents.portfolio_optimization.models import DRLAgent
from finrl.agents.portfolio_optimization.architectures import EIIE
device = 'cuda:0' if torch.cuda.is_available() else 'cpu'
```
## Fetch data
In his paper, *Jiang et al* creates a portfolio composed by the top-11 cryptocurrencies based on 30-days volume. Since it's not specified when this classification was done, it's difficult to reproduce, so we will use a similar approach in the Brazillian stock market:
- We select top-10 stocks from Brazillian stock market;
- For simplicity, we disconsider stocks that have missing data for the days in period 2011-01-01 to 2019-12-31 (9 years);
```python
TOP_BRL = [
"VALE3.SA", "PETR4.SA", "ITUB4.SA", "BBDC4.SA",
"BBAS3.SA", "RENT3.SA", "LREN3.SA", "PRIO3.SA",
"WEGE3.SA", "ABEV3.SA"
]
```
```python
print(len(TOP_BRL))
portfolio_raw_df = YahooDownloader(start_date = '2011-01-01',
end_date = '2022-12-31',
ticker_list = TOP_BRL).fetch_data()
portfolio_raw_df
```
```python
portfolio_raw_df.groupby("tic").count()
```
### Normalize Data
We normalize the data dividing the time series of each stock by its maximum value, so that the dataframe contains values between 0 and 1.
```python
portfolio_norm_df = GroupByScaler(by="tic", scaler=MaxAbsScaler).fit_transform(portfolio_raw_df)
portfolio_norm_df
```
```python
df_portfolio = portfolio_norm_df[["date", "tic", "close", "high", "low"]]
df_portfolio_train = df_portfolio[(df_portfolio["date"] >= "2011-01-01") & (df_portfolio["date"] < "2019-12-31")]
df_portfolio_2020 = df_portfolio[(df_portfolio["date"] >= "2020-01-01") & (df_portfolio["date"] < "2020-12-31")]
df_portfolio_2021 = df_portfolio[(df_portfolio["date"] >= "2021-01-01") & (df_portfolio["date"] < "2021-12-31")]
df_portfolio_2022 = df_portfolio[(df_portfolio["date"] >= "2022-01-01") & (df_portfolio["date"] < "2022-12-31")]
```
### Instantiate Environment
Using the `PortfolioOptimizationEnv`, it's easy to instantiate a portfolio optimization environment for reinforcement learning agents. In the example below, we use the dataframe created before to start an environment.
```python
environment = PortfolioOptimizationEnv(
df_portfolio_train,
initial_amount=100000,
comission_fee_pct=0.0025,
time_window=50,
features=["close", "high", "low"],
normalize_df=None
)
```
### Instantiate Model
Now, we can instantiate the model using FinRL API. In this example, we are going to use the EIIE architecture introduced by Jiang et. al.
:exclamation: **Note:** Remember to set the architecture's `time_window` parameter with the same value of the environment's `time_window`.
```python
# set PolicyGradient parameters
model_kwargs = {
"lr": 0.01,
"policy": EIIE,
}
# here, we can set EIIE's parameters
policy_kwargs = {
"k_size": 3,
"time_window": 50,
}
model = DRLAgent(environment).get_model("pg", device, model_kwargs, policy_kwargs)
```
### Train Model
```python
DRLAgent.train_model(model, episodes=40)
```
### Save Model
```python
torch.save(model.train_policy.state_dict(), "policy_EIIE.pt")
```
## Test Model
### Instantiate different environments
Since we have three different periods of time, we need three different environments instantiated to simulate them.
```python
environment_2020 = PortfolioOptimizationEnv(
df_portfolio_2020,
initial_amount=100000,
comission_fee_pct=0.0025,
time_window=50,
features=["close", "high", "low"],
normalize_df=None
)
environment_2021 = PortfolioOptimizationEnv(
df_portfolio_2021,
initial_amount=100000,
comission_fee_pct=0.0025,
time_window=50,
features=["close", "high", "low"],
normalize_df=None
)
environment_2022 = PortfolioOptimizationEnv(
df_portfolio_2022,
initial_amount=100000,
comission_fee_pct=0.0025,
time_window=50,
features=["close", "high", "low"],
normalize_df=None
)
```
### Test EIIE architecture
Now, we can test the EIIE architecture in the three different test periods. It's important no note that, in this code, we load the saved policy even though it's not necessary just to show how to save and load your model.
```python
EIIE_results = {
"training": environment._asset_memory["final"],
"2020": {},
"2021": {},
"2022": {}
}
# instantiate an architecture with the same arguments used in training
# and load with load_state_dict.
policy = EIIE(time_window=50, device=device)
policy.load_state_dict(torch.load("policy_EIIE.pt"))
# 2020
DRLAgent.DRL_validation(model, environment_2020, policy=policy)
EIIE_results["2020"]["value"] = environment_2020._asset_memory["final"]
# 2021
DRLAgent.DRL_validation(model, environment_2021, policy=policy)
EIIE_results["2021"]["value"] = environment_2021._asset_memory["final"]
# 2022
DRLAgent.DRL_validation(model, environment_2022, policy=policy)
EIIE_results["2022"]["value"] = environment_2022._asset_memory["final"]
```
### Test Uniform Buy and Hold
For comparison, we will also test the performance of a uniform buy and hold strategy. In this strategy, the portfolio has no remaining cash and the same percentage of money is allocated in each asset.
```python
UBAH_results = {
"train": {},
"2020": {},
"2021": {},
"2022": {}
}
PORTFOLIO_SIZE = len(TOP_BRL)
# train period
terminated = False
environment.reset()
while not terminated:
action = [0] + [1/PORTFOLIO_SIZE] * PORTFOLIO_SIZE
_, _, terminated, _ = environment.step(action)
UBAH_results["train"]["value"] = environment._asset_memory["final"]
# 2020
terminated = False
environment_2020.reset()
while not terminated:
action = [0] + [1/PORTFOLIO_SIZE] * PORTFOLIO_SIZE
_, _, terminated, _ = environment_2020.step(action)
UBAH_results["2020"]["value"] = environment_2020._asset_memory["final"]
# 2021
terminated = False
environment_2021.reset()
while not terminated:
action = [0] + [1/PORTFOLIO_SIZE] * PORTFOLIO_SIZE
_, _, terminated, _ = environment_2021.step(action)
UBAH_results["2021"]["value"] = environment_2021._asset_memory["final"]
# 2022
terminated = False
environment_2022.reset()
while not terminated:
action = [0] + [1/PORTFOLIO_SIZE] * PORTFOLIO_SIZE
_, _, terminated, _ = environment_2022.step(action)
UBAH_results["2022"]["value"] = environment_2022._asset_memory["final"]
```
### Plot graphics
```python
import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(UBAH_results["train"]["value"], label="Buy and Hold")
plt.plot(EIIE_results["training"], label="EIIE")
plt.xlabel("Days")
plt.ylabel("Portfolio Value")
plt.title("Performance in training period")
plt.legend()
plt.show()
```
```python
plt.plot(UBAH_results["2020"]["value"], label="Buy and Hold")
plt.plot(EIIE_results["2020"]["value"], label="EIIE")
plt.xlabel("Days")
plt.ylabel("Portfolio Value")
plt.title("Performance in 2020")
plt.legend()
plt.show()
```
```python
plt.plot(UBAH_results["2021"]["value"], label="Buy and Hold")
plt.plot(EIIE_results["2021"]["value"], label="EIIE")
plt.xlabel("Days")
plt.ylabel("Portfolio Value")
plt.title("Performance in 2021")
plt.legend()
plt.show()
```
```python
plt.plot(UBAH_results["2022"]["value"], label="Buy and Hold")
plt.plot(EIIE_results["2022"]["value"], label="EIIE")
plt.xlabel("Days")
plt.ylabel("Portfolio Value")
plt.title("Performance in 2022")
plt.legend()
plt.show()
```
We can see that the agent is able to learn a good policy but its performance is worse the more the test period advances into the future. To get a better performance in 2022, for example, the agent should probably be trained again using more recent data.
Shown in full with attribution under the source's licence. Licence: MIT
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.