跳至正文
返回文库全部文档

利用循环强化学习在潜在状态下进行交易

文章 arXiv papers · 作者: Andrea Macrì et al.
阅读原文

总结

本文研究信号服从均值回复Ornstein-Uhlenbeck过程且参数随隐藏状态变化时的最优交易。研究将循环神经网络与强化学习结合起来，根据观测推断信号的潜在均值、回复速度和波动率等信息，再利用这些信息指导交易。

作者提出三种基于DDPG的方法：一种将循环隐藏状态传递给交易智能体，另两种则提供估计的状态概率或下一期信号值预测。随着状态动态复杂度逐渐提高的模拟，以及一项股票配对交易实证应用，都更支持基于概率的方法，因为其累积奖励和可解释性较好。基于预测的方法带来的收益有限，隐藏状态方法的表现介于两者之间。这些结果表明，推断信息的表示方式很重要，但文档中的证据来自指定的模拟和应用；它并未证明该方法在其他市场或部署条件下的表现。

核心观点

  • 交易信号被建模为均值回复过程，其参数会随隐藏状态而变化。
  • 循环网络从信号观测中提取时间信息。
  • 三种DDPG设计向智能体提供隐藏状态、状态概率或信号预测。
  • 据报告，在模拟和配对交易应用中，基于状态概率的方法表现最佳。
  • 提供给智能体的信息会影响报告的奖励和策略可解释性。

标签

机器学习均值回归配对交易股票统计学
全文
# Deep reinforcement learning for optimal trading with partial information


# Deep reinforcement learning for optimal trading with partial information









Reinforcement Learning (RL) applied to financial problems has been the subject of a lively area of research. The use of RL for optimal trading strategies that exploit latent information in the market is, to the best of our knowledge, not widely tackled. In this paper we study an optimal trading problem, where a trading signal follows an Ornstein-Uhlenbeck process with regime-switching dynamics. We employ a blend of RL and Recurrent Neural Networks (RNN) in order to make the most at extracting underlying information from the trading signal with latent parameters. The latent parameters driving mean reversion, speed, and volatility are filtered from observations of the signal, and trading strategies are derived via RL. To address this problem, we propose three Deep Deterministic Policy Gradient (DDPG)-based algorithms that integrate Gated Recurrent Unit (GRU) networks to capture temporal dependencies in the signal. The first, a one -step approach (hid-DDPG), directly encodes hidden states from the GRU into the RL trader. The second and third are two-step methods: one (prob-DDPG) makes use of posterior regime probability estimates, while the other (reg-DDPG) relies on forecasts of the next signal value. Through extensive simulations with increasingly complex Markovian regime dynamics for the trading signal's parameters, as well as an empirical application to equity pair trading, we find that prob-DDPG achieves superior cumulative rewards and exhibits more interpretable strategies. By contrast, reg-DDPG provides limited benefits, while hid-DDPG offers intermediate performance with less interpretable strategies. Our results show that the quality and structure of the information supplied to the agent are crucial: embedding probabilistic insights into latent regimes substantially improves both profitability and robustness of reinforcement learning-based trading strategies.

在遵守原作品许可的前提下，附作者信息全文展示。 许可协议: abstract CC0

此摘要由 Stratmill 研究智能体根据原文撰写，并非原文副本。