Recurrent Reinforcement Learning for Trading with Latent Regimes
Summary
This paper studies optimal trading when a signal follows a mean-reverting Ornstein-Uhlenbeck process whose parameters change according to hidden regimes. It combines recurrent neural networks with reinforcement learning to infer information about the signal’s latent mean, reversion speed, and volatility from observations, then uses that information to guide trades.
The authors propose three DDPG-based approaches: one passes recurrent hidden states to the trading agent, while two others supply estimated regime probabilities or forecasts of the next signal value. Simulations with increasingly complex regime dynamics and an empirical equity pair-trading application favor the probability-based approach in cumulative reward and interpretability. The forecast-based method adds limited benefit, and the hidden-state method performs between the two. These results suggest that the representation of inferred information matters, but the document’s evidence comes from the specified simulations and application; it does not establish performance across other markets or deployment conditions.
Key ideas
- The trading signal is modeled as mean-reverting with parameters that shift across hidden regimes.
- A recurrent network extracts temporal information from observations of the signal.
- Three DDPG designs provide the agent with hidden states, regime probabilities, or signal forecasts.
- The regime-probability approach performs best in the reported simulations and pair-trading application.
- The information supplied to the agent affects both reported rewards and strategy interpretability.
Tags
Full text
# Deep reinforcement learning for optimal trading with partial information # Deep reinforcement learning for optimal trading with partial information Reinforcement Learning (RL) applied to financial problems has been the subject of a lively area of research. The use of RL for optimal trading strategies that exploit latent information in the market is, to the best of our knowledge, not widely tackled. In this paper we study an optimal trading problem, where a trading signal follows an Ornstein-Uhlenbeck process with regime-switching dynamics. We employ a blend of RL and Recurrent Neural Networks (RNN) in order to make the most at extracting underlying information from the trading signal with latent parameters. The latent parameters driving mean reversion, speed, and volatility are filtered from observations of the signal, and trading strategies are derived via RL. To address this problem, we propose three Deep Deterministic Policy Gradient (DDPG)-based algorithms that integrate Gated Recurrent Unit (GRU) networks to capture temporal dependencies in the signal. The first, a one -step approach (hid-DDPG), directly encodes hidden states from the GRU into the RL trader. The second and third are two-step methods: one (prob-DDPG) makes use of posterior regime probability estimates, while the other (reg-DDPG) relies on forecasts of the next signal value. Through extensive simulations with increasingly complex Markovian regime dynamics for the trading signal's parameters, as well as an empirical application to equity pair trading, we find that prob-DDPG achieves superior cumulative rewards and exhibits more interpretable strategies. By contrast, reg-DDPG provides limited benefits, while hid-DDPG offers intermediate performance with less interpretable strategies. Our results show that the quality and structure of the information supplied to the agent are crucial: embedding probabilistic insights into latent regimes substantially improves both profitability and robustness of reinforcement learning-based trading strategies.
Shown in full with attribution under the source's licence. Licence: abstract CC0
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.