Choosing a Stopping Point for Reinforcement Learning Portfolio Training
Summary
The document discusses how to choose a training duration for a deep reinforcement learning portfolio agent when training loss keeps falling but out-of-sample performance has flattened. It frames the number of training epochs as a hyperparameter and recommends selecting it using validation data, then retraining and evaluating once on a separate test set. Because financial observations are time ordered, it advises splitting the history into contiguous blocks and suggests rotating validation blocks through a form of cross-validation.
Other proposed checks include early stopping on validation reward, using a lower-capacity policy less prone to overfitting, and checkpointing training to inspect measures such as Sharpe ratio and turnover. The responses caution that daily market histories may be too short to support separate validation and test samples, and that visual inspection is qualitative rather than a definitive stopping rule. They also recommend considering walk-forward evaluation. The discussion offers practical options, not comparative experimental evidence that establishes one method as best.
Key ideas
- Training loss alone does not show when a portfolio agent will generalize best.
- Treat training duration as a hyperparameter and choose it using validation data.
- Use contiguous time blocks to reduce leakage when splitting financial histories.
- Checkpoint training and inspect portfolio reward, Sharpe ratio, and turnover as supplementary diagnostics.
- Walk-forward evaluation can add another perspective, though short histories limit validation choices.
Tags
Full text
# When to stop training? # When to stop training? I have built a deep reinforcement learning based portfolio optimisation agent. At a high level it is using macro economic data, valuations of the assets and a few technical indicators as the features. The policy network is a temporal convolution network with attention. The output of the policy network is the portfolio allocation. The agent is episodic with 365 days as the episode length. The episode terminates either at the end of of 365 days or when the drawdown of the portfolio hits 30%. With these conditions I have trained the agent on 10 years of data. The problem I am facing is of figuring when to stop training the agent. All the metrics like loss continue to improve with each iteration. The out of sample performance fluctuates in a narrow band after some 100 iterations indicating that it is possibly the best result the model can provide. But the loss metric keep decreasing. As one can see from these graphs that one cannot make a decision on when to stop the training based on the training loss. And any decision cannot be made based on Out of Sample Validation data set. Any suggestions ? ## Answer by lehalle (score 3) https://quant.stackexchange.com/a/48695 The standard approach for cross-validation is to work with three datasets - the training set, - the validation set, - the test set. You train on the training set (no surprise), and you use the validation set to decide for your hyper-parameters. Then you retrain, test on the test set (no surprise again), and it is done. My advice here is to consider that the number of epochs to train on is a hyper-parameter: thus choose it to generalize well on your training set. Then retrain with it and it is done. One more detail: splitting the history in sets is not trivial when we talk about time-series. You should take "contiguous blocks"; if you have 10 years (this is short, moreover you do not have the 2008 crisis in your history!) - 6 years for training, - 1.5 years for validation - the last 1.5 years for the test (this is very short...) You can use K-fold cross-validation to try to obtain more robust results: k-fold your 7.5 years by rotating the 1.5 years you keep for validation. Thanks to this you will have 5 validation sets to look at. ## Answer by Elrond (score 2) https://quant.stackexchange.com/a/47127 One option could be to use an early stopping mechanism, e.g. stop when the average reward per episode on a validation set stops increasing. In practice, I found this to be a bad idea as you generally don't have enough data to be able to afford a validation set when using daily data (in addition to the test set). If you are dealing with higher frequency data then daily, the story might be different. A second option is to use a simple policy network (e.g. a linear regression) which does not have enough capacity to overfit, and so you can train until when the reward on the training set stops to increase. A third option is to train for long enough (i.e. until when the reward stops increasing or until when you are confident to be in the overfitting region) while checkpointing the agent during training. Then use visualizations to inspect the behavior of the agent during training (e.g. Sharpe ratio or turnover as a function of the number of training steps). This is qualitative which doesn't tell you when to stop, but it does provide insights /let a human judge what could be a decent stopping point (e.g. turnover will probably become too high after a certain number of iterations) In addition, this problem should be evaluated in the context of walk-forward optimization, just to add another dimension to this already difficult problem.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.