Choosing State Variables for a Bellman Equation
Summary
The document explains state variables as the information needed to determine a value function and the decisions available in a dynamic problem. Examples mentioned include wealth, capital, dividends, physical position, speed, and fuel. The central practical question is what information about the current situation affects the decision being modeled. A state vector can begin with a broad set of relevant quantities, then omit variables that do not affect the value or choice.
The answer connects state selection to the Markov decision problem: a sufficient state should summarize the past needed to describe current decisions and future outcomes. The balloon-control example illustrates how physical laws may require adding variables such as speed, alongside position and fuel, to represent the system’s dynamics. This is a conceptual explanation, not a formal test for state sufficiency; the appropriate variables depend on the specific model and its assumptions.
Key ideas
- State variables are the inputs on which a dynamic problem’s value function depends.
- Choose variables that influence the decision and the evolution of future outcomes.
- A sufficient state summarizes relevant history so the problem can be represented as Markovian.
- State selection depends on the model, and may require adding variables such as speed or resources.
Tags
Full text
# State Variables in a Bellman Equation # State Variables in a Bellman Equation Can anyone explain to me exactly what a state variable is in a Bellman equation?? $$ V(x,y)=max\space u(c)+\beta V(x',y')$$ In some models with capital savings it's the capital $k_t$ you walk into period $t$ in others where there are many future states its all aggregate $S$ in the past. In some models it's dividends $d_t$ of a stochastic asset. It's obviously relevant variables that influence your decision making but I can't seem to find a consistent way to be able to identify state equations in a lot of the problem I am solving. ## Answer by Louis. B (score 2) https://quant.stackexchange.com/a/22257 State variables are simply variables that matter for the value function, the variable on which the value function depends on if you want. Conceptually, you could think of a state vector containing all the variables and then removing the variables that don't matter from your state vector. In practice, the state variables choice really depends on the problem you're trying to solve. For physicist it could be the position of an object, the atmospheric pressure or whatever. For economist it could be the wealth of an agent, the level of capital of a firm, etc. It could be anything. If you're having hard time figuring out what could it be in your model, try to ask yourself question like : "On what does the decision that I'm trying to understand depend?" or "What could influence the choice of an agent in this economy?". You can provide more details on the problem you're trying to solve if you want a more detailed answer. ## Answer by M. Jeunesse (score 0) https://quant.stackexchange.com/a/31781 Louis's answer is good. In my understanding, state variables ensure your problem can be stated as a Markov decision problem. Think of a balloon landing control problem under fuel consumption constraint. Position and fuel level are obviously state variables. But Newton principle implies to take into account acceleration. It is often possible by integration of Newton principle, that only speed is needed. In the end, your state variables are position, speed, acceleration and fuel level. Your control is the fuel injection to balloon's engine.
Shown in full with attribution under the source's licence. Licence: CC BY-SA 4.0 (Stack Exchange)
This summary was written by Stratmill's research agent from the original; it is not a copy of the source.