Solving Stochastic HJB Equations with Value–Policy Iteration
Summary
The document explains a practical approach to solving a stochastic Hamilton–Jacobi–Bellman equation for a trading control problem. It separates the equation into an optimization over the control and the remaining terms that describe the value function’s dynamics. In an inventory-management example, the control is trading speed, with quadratic trading costs and a penalty for holding exposure. The optimal speed can be derived in closed form for the simplified setup.
The proposed procedure alternates between choosing a policy from a current value-function estimate and solving the resulting partial differential equation backward from its terminal condition. Repeating these steps is presented as Howard’s value–policy iteration. The answer describes the sequence conceptually and gives the simplified equations, but does not provide an implementation, runtime benchmarks, or a demonstration of real-time suitability. Its convergence comments apply to the stated iteration framing; they do not establish low latency for high-frequency trading or general performance for more complex models.
Key ideas
- An HJB combines optimization over a control with differential terms governing the value function.
- For a fixed value-function estimate, optimize the trading control and substitute the resulting policy into the equation.
- Solve the resulting PDE backward from its terminal condition, then repeat the policy and value updates.
- In the example, trading speed is the control and quadratic costs allow a closed-form optimum.
- The explanation does not provide code or evidence about real-time computational performance.
Tags
Full text
# How do you actually solve a stochastic HJB equation in practice?
# How do you actually solve a stochastic HJB equation in practice?
I've read a number of recent papers on market making. Nearly all of the more recent papers focus on defining the problem in terms of a state and action space, deriving the relevant HJB equations and constraints, and going from there.
I realize this could be viewed as a broad question, but I think that it's worthy from the perspective of a programmer. In practice, what are the general methods to attack the problem of solving an optimal policy in real time? I understand the theoretical underpinnings of the DPP, but there are few libraries or frameworks designed to help with this, and those that exist are extremely light on the documentation. I'm confused as to how viable methods like policy and value iteration are when solving online (in terms of computational cost and resulting added latency in the HFT world). Are there any good examples of someone taking a set of HJB equations for optimal market making (inventory management) or execution and going all the way (preferably with code or pseudocode) to a working mvp?
## Answer by lehalle (score 2)
https://quant.stackexchange.com/a/77780
An HJB is made of two components:
- a "core component" that corresponds to applying an optimal control
- the "dynamics" of the value function (that is surrounding this optimizattion).
Take the one dimension version of the HJB (2.3) of L, C. A., & Mouzouni, C. (2019). A mean field game of portfolio trading and its consequences on perceived correlations. that I know well: $${\gamma\over 2} q^2 =\partial_t u + a \mu\; \partial_s u+\sup_\nu \left\{ \nu\partial_q u - \ell \;\nu^2/V) \right\}$$ with the terminal condition $u_T=-Aq^2$ (here I took $L(x)=\ell x^2$ in the orginal equality to make it more simple).
- The part $\sup_\nu \left\{ \nu\partial_q u - \ell \;\nu^2/V) \right\}$ corresponds to applying an optimal control $\nu$ that is a speed of trading $\nu$. You may recognize the instantaneous market impact cost in $\nu^2$ and the "bleeding" part $\nu\partial_q u$ (it says that trading somehow corresponds to changing your inventory).
- The other terms are the cost ${\gamma\over 2} q^2$ of keeping an open exposure the natural decay of the value function with time $\partial_t u$ and the permanent market impact term $a \mu\; \partial_s u$ that changes the value of your position.
To solve it a simple way is to apply a sequence of value-policy iterations:
- you can start with a arbirary (but not too stupid) guess for the value $u_n(t,q)$ to get a position of size $q$ at $t$.
- thanks to that you can solve the optimal control part and find on optimal trading speed $\nu^* $ that solves the supremum (for each $(t,q)$). You can do it numerically in general but here it can be done in close-form: $\nu^*=\partial_q u_n\cdot V / (2\ell)$.
- thanks to that you can plug it into the HJB and find a PDE. For us it will be $${\gamma\over 2} q^2 =\partial_t u_n + a \mu\; \partial_s u+ {v\over 4\ell}(\partial_q u_n)^2.$$
- you can solve it a backward way since we have the terminal condition. It will give you $v_{n+1}$.
Now you can iterate. It is often talk the Howard method since it was explained in Howard, R.: Dynamic Programming and Markov Processes; MIT Press, Cambridge (1960).
What are the take-away:
- an HJB is a combination of a supremum (corresponding to solving the optimal control) and some PDE terms.
- once you solve the supremum, you get a PDE that is backward (that is natural: in control optimality comes from a backward reasoning).
- you can iterate and it will converge (moreover Howard tells you you converge a monotonic way: at each iteration you are closer to the true value function and the true optimal policy).
You can make the relation with Reinforcement Learning but it is not that true, since in RL the convergence is not monotonic. Nevertheless you can use RL to solve optimal trading problems, cf Section 5.4 of Mounjid, Othmane, and Charles‐Albert Lehalle. "Improving reinforcement learning algorithms: towards optimal learning rate policies." Mathematical Finance (2019).Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.