Return, Value, and the Bellman Equation | Xingyu Yang

Two Words That Get Conflated

“Reward is the expected long-term reward” is a sentence that sounds almost right and is actually wrong in a way worth dwelling on.

Reward is one-step feedback: a single scalar handed back by the environment after one transition. Return is accumulated future reward from a given point onward, a sum, not a single number. The chain runs in one direction only: reward aggregates into return, and return in expectation becomes value. This chapter walks that chain end to end, and the payoff is that the Bellman equation, usually presented as a block to memorize, arrives as a consequence of one small lemma proved in the first section.

DefinitionDiscounted Return

The discounted return from time $t$ is

G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots

For a finite horizon ending at $T$ ,

G_t = R_{t+1} + \gamma R_{t+2} + \cdots + \gamma^{T-t-1}R_T.

If $\gamma = 0$ , the return collapses to the very next reward, $G_t = R_{t+1}$ .

The discount factor $\gamma \in [0,1]$ is not a minor tuning knob; it is what makes $G_t$ well-defined at all in the general case. An unweighted infinite sum of rewards has no reason to converge, and many environments genuinely admit unboundedly long or cyclic trajectories. Discounting also encodes something more substantive than convergence bookkeeping: it says that reward further in the future counts for less, which is a reasonable stance whenever the future is less certain than the present, or when a task simply prefers reward sooner rather than later. As $\gamma$ moves toward $0$ , the agent becomes almost entirely near-sighted, valuing only immediate reward; as $\gamma$ moves toward $1$ , it weighs the distant future almost as heavily as the present.

This also retires the loose end left open at the end of the first chapter. “Cumulative reward” over an unboundedly long future is now a convergent quantity, and it has a name.

The return $G_t$ has a structural property that everything downstream depends on: it can be split into what happens on the very next step, and what happens after that.

LemmaReturn Satisfies a One-Step Recursion

G_t = R_{t+1} + \gamma G_{t+1}

Proof

\begin{aligned} G_t &= R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots \\ &= R_{t+1} + \gamma\left(R_{t+2} + \gamma R_{t+3} + \cdots\right) \\ &= R_{t+1} + \gamma G_{t+1}. \end{aligned}

The middle step only factors $\gamma$ out of every term but the first; the result inside the parentheses is, by definition, $G_{t+1}$ .

This one-line identity is the entire mathematical content of the Bellman equation, before any expectation or environment structure gets involved. It is worth keeping in view for the rest of the chapter: every equation below is this lemma wearing progressively more clothing.

Two Value Functions

A value function estimates expected long-term cumulative reward, not just immediate reward. The Markov ladder already flagged why one such function will not be enough once actions exist: different actions available from the same state can lead to different futures. Here is that claim, made precise.

DefinitionState-Value and Action-Value Functions

Under a fixed policy $\pi$ , the state-value function is the expected return starting from state $s$ and following $\pi$ thereafter:

V^\pi(s) = \mathbb{E}_\pi[G_t \mid S_t = s]

The action-value function is the expected return starting from $s$ , taking action $a$ first, and following $\pi$ afterward:

Q^\pi(s,a) = \mathbb{E}_\pi[G_t \mid S_t = s, A_t = a]

Function	Evaluates	Includes first action?
$V^\pi(s)$	state	no
$Q^\pi(s,a)$	state-action pair	yes

The difference is exactly whether the first action is dictated by the policy or fixed by hand, and that difference is precisely what makes $Q^\pi$ useful for comparing candidate actions against each other, while $V^\pi$ only ever evaluates a state as a whole.

PropositionState-Value from Action-Value

V^\pi(s) = \sum_a \pi(a \mid s)\, Q^\pi(s,a)

Proof

$V^\pi(s)$ averages the return over trajectories starting at $s$ under $\pi$ ; the first action $A_t$ in such a trajectory is itself distributed as $A_t \sim \pi(\cdot \mid s)$ . Conditioning on the value of that first action and applying the tower property,

V^\pi(s) = \mathbb{E}_\pi[G_t \mid S_t=s] = \sum_a \pi(a\mid s)\, \mathbb{E}_\pi[G_t \mid S_t=s, A_t=a] = \sum_a \pi(a\mid s)\, Q^\pi(s,a).

Keep this proposition in hand. It is one of the two halves the Bellman expectation equation gets assembled from a few paragraphs below.

Value Now, Reward Plus Value Later

Now push the recursion lemma through the expectation. Take the MRP case first, where $V(s) = \mathbb{E}[G_t \mid S_t = s]$ and no actions are involved. Because the return satisfies $G_t = R_{t+1} + \gamma G_{t+1}$ , so does $V$ .

TheoremBellman Equation (Markov Reward Process)

V(s) = R(s) + \gamma \sum_{s'}P(s' \mid s)V(s')

Proof

Starting from the definition and applying the one-step recursion lemma:

\begin{aligned} V(s) &= \mathbb{E}[G_t \mid S_t = s] \\ &= \mathbb{E}[R_{t+1} + \gamma G_{t+1} \mid S_t = s] \\ &= \mathbb{E}[R_{t+1} \mid S_t = s] + \gamma\,\mathbb{E}[G_{t+1} \mid S_t = s]. \end{aligned}

The first term is exactly $R(s)$ , the expected immediate reward from $s$ , by definition of the reward function. The second term needs one more step: $\mathbb{E}[G_{t+1} \mid S_t = s]$ is an expectation over the next state $s'$ of the expectation of $G_{t+1}$ given that next state. This is the tower property (law of total expectation), conditioning on $S_{t+1}$ inside the outer expectation over $S_t$ :

\mathbb{E}[G_{t+1} \mid S_t = s] = \sum_{s'} P(s' \mid s)\, \mathbb{E}[G_{t+1} \mid S_{t+1} = s'] = \sum_{s'} P(s' \mid s)\, V(s').

The last equality holds because $\mathbb{E}[G_{t+1} \mid S_{t+1}=s']$ is, by definition, $V(s')$ , the process has no memory beyond the current state, by the Markov property. This step also silently uses that the dynamics are time-homogeneous: $V$ does not depend on $t$ itself, only on the state, so the value “one step into the future” and “at any other time” are computed by the same function $V$ . This is a standing assumption throughout this series, not something the Markov property alone guarantees. Substituting both pieces back in gives the claimed equation.

Read in words: value now equals immediate reward plus discounted expected value one step later. That sentence is the entire content of the Bellman equation; the sum over $s'$ is just what “expected value one step later” means once you write out the expectation explicitly. Every Bellman equation in reinforcement learning, expectation, optimality, MRP, MDP, tabular or function-approximated, is a variation on this same one-line idea, because it is a direct consequence of the return satisfying $G_t = R_{t+1} + \gamma G_{t+1}$ and nothing else.

From MRP to MDP: Two Halves of the Same Backup

The proposition above already proved $V^\pi(s) = \sum_a \pi(a\mid s)\,Q^\pi(s,a)$ , half of the relationship between $V^\pi$ and $Q^\pi$ . The other half is a direct MDP analogue of the MRP Bellman equation just derived, with the first action fixed.

PropositionAction-Value from State-Value

Q^\pi(s,a) = R(s,a) + \gamma \sum_{s'}P(s' \mid s,a)\,V^\pi(s')

The argument is identical to the MRP Bellman equation’s proof, with every transition and reward now conditioned on the fixed action $a$ as well as the state $s$ . Fixing $a$ turns the first step back into an ordinary MRP-style backup, which is why the proof is not repeated here.

CorollaryBellman Expectation Equations

Composing the two halves, substituting the action-value backup into the state-value average, gives the Bellman expectation equation:

V^\pi(s) = \sum_a \pi(a \mid s) \left[ R(s,a) + \gamma \sum_{s'}P(s' \mid s,a)V^\pi(s') \right]

This equation is usually presented as a single block to memorize. Built up this way, it decomposes into two independent, individually obvious facts: value-from-action-value is just the definition of expectation over the first action, and action-value-from-state-value is just the MRP Bellman equation with the action held fixed. The “expectation” in “Bellman expectation equation” refers to exactly one thing: the average over actions sampled from $\pi$ , layered on top of the same recursive structure derived from $G_t = R_{t+1} + \gamma G_{t+1}$ .

From Evaluating a Policy to Finding the Best One

Everything so far evaluates a fixed policy $\pi$ , this is the prediction problem. Control asks a different, harder question: what is the best possible value achievable at each state, over every policy at once?

RemarkBellman's Principle of Optimality

Richard Bellman’s original 1957 statement of the idea behind all of this is worth quoting directly, because it is the actual justification for replacing an average over actions with a maximum: an optimal policy has the property that, whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision. Optimality is self-similar across time: the tail of an optimal plan is itself an optimal plan for the sub-problem it starts. This is what licenses solving a problem with an unboundedly long horizon by a local, one-step recursive equation at all. Without it, there would be no reason to expect that a short-sighted, one-step-at-a-time backup could ever assemble into a globally optimal policy.

DefinitionOptimal Value Functions

V^*(s) = \max_\pi V^\pi(s) \qquad\qquad Q^*(s,a) = \max_\pi Q^\pi(s,a)

TheoremBellman Optimality Equations

V^*(s) = \max_a \left[ R(s,a) + \gamma \sum_{s'}P(s' \mid s,a)V^*(s') \right]

Q^*(s,a) = R(s,a) + \gamma \sum_{s'}P(s' \mid s,a)\max_{a'}Q^*(s',a')

RemarkWhy the Average Becomes a Maximum

These equations are not a separate derivation from the Bellman expectation equations above, they are what the expectation equations become once $\pi$ is required to be optimal. By the principle of optimality, an optimal policy must place all of its probability mass on whichever action maximizes $Q^*(s,a)$ at every state; averaging $Q^*(s,a)$ over a distribution that is entirely concentrated on its own maximizer is the same as just taking that maximum. The $\max$ is the trace left behind by optimizing over $\pi$ itself, not a new assumption bolted onto the earlier recursion.

Formula Ownership Rule

With three families of Bellman equation now on the table, a quick rule settles which one is in front of you:

Pattern in the equation	Which Bellman equation
No action term at all	Markov reward process
Action term, averaged with a fixed $\pi(a \mid s)$	Policy evaluation / prediction
Action term, combined with $\max$ or $\arg\max$	Control / optimality

References

Datawhale Easy RL, Chapter 2
OpenAI Spinning Up: Part 1, Key Concepts in RL
Bellman, R. (1957). Dynamic Programming. Princeton University Press.

Two Words That Get Conflated

Two Value Functions

Value Now, Reward Plus Value Later

From MRP to MDP: Two Halves of the Same Backup

From Evaluating a Policy to Finding the Best One

Formula Ownership Rule

Further Reading

References

Comments