Proximal Policy Optimization

The Waste in On-Policy Sampling

Policy Gradient ended with an estimator that works:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim p_\theta} \left[ \sum_{t=1}^{T} \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, \widehat A_t \right]$$

The whole of this chapter is contained in one detail of that line: the subscript. The expectation is taken over trajectories drawn from $p_\theta$, the distribution induced by the current parameters. Take a single gradient step, and $\theta$ is no longer the $\theta$ that generated the batch. Every sample in hand is now drawn from the wrong distribution, and the estimator that made the step has no license to make a second one.

So the loop is forced: collect a batch, take one step, discard the batch, collect again. Each trajectory funds exactly one update and is then thrown away. Nothing is recomputed and nothing is wasted twice; the waste is that expensive samples are used once.

How expensive depends on the problem, and for the domains this series is heading toward, the answer is: expensive enough that it dominates everything else. A single sample means generating a full token sequence, or running an entire denoising chain. The gradient step is cheap by comparison. Spending a whole round of generation to buy one step is the central inefficiency of on-policy policy gradient, and it is what Proximal Policy Optimization exists to fix.

The question this chapter answers is therefore narrow and practical:

Can one batch of trajectories fund many gradient steps?

The answer is yes, conditionally, and the rest of the chapter is about what the condition costs.

What We Are Estimating

Before touching the sampling machinery, it is worth being precise about the object being multiplied by the gradient, since every step below manipulates it and the final objective splits into cases according to its sign.

DefinitionAdvantage Function

For a policy $\pi$, the advantage function is

$$A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s).$$

Both terms come from Return, Value, and the Bellman Equation: $Q^\pi(s,a)$ is the expected return from taking $a$ in $s$ and following $\pi$ afterward, and $V^\pi(s)$ is the expected return from $s$ when the first action is also drawn from $\pi$. The difference is what taking $a$ specifically buys, relative to what the policy would have done on its own.

That “relative to” is not informal. It is exactly what the following says.

PropositionAdvantage Has Zero Mean Under Its Own Policy

For every state $s$,

$$\mathbb{E}_{a \sim \pi(\cdot \mid s)}\big[A^\pi(s,a)\big] = 0.$$

Proof

Expand the definition and split the expectation:

$$\mathbb{E}_{a \sim \pi(\cdot\mid s)}\big[A^\pi(s,a)\big] = \sum_a \pi(a\mid s)\big[Q^\pi(s,a) - V^\pi(s)\big] = \sum_a \pi(a\mid s)Q^\pi(s,a) - V^\pi(s)\sum_a \pi(a\mid s).$$

The first sum is $V^\pi(s)$, by the state-value-from-action-value proposition of the previous chapters. The second sum is $1$, since $\pi(\cdot\mid s)$ is a probability distribution. The two terms are therefore both $V^\pi(s)$ and cancel.

This fixes the meaning of the sign, which is the only property of $A^\pi$ the clipped objective will actually use:

• $A^\pi(s,a) > 0$: taking $a$ is better than what $\pi$ does on average from $s$. The update should make $a$ more likely.
• $A^\pi(s,a) < 0$: taking $a$ is worse than $\pi$‘s own average. The update should make $a$ less likely.

Because the mean is exactly zero, “better than average” is a statement about $\pi$ measuring itself. An advantage is not a score of an action in isolation; it is a comparison against the policy that is being improved, and it is recomputed every time that policy changes. That last clause is worth holding on to, because it is a second reason the samples in a batch go stale.

In practice $A^\pi$ is not known and must be estimated from the batch. Policy Gradient built the estimator this chapter inherits, $\widehat A_t = G_t - b(s_t)$ with $b(s_t) = V^\pi(s_t)$, and proved there that subtracting any action-independent baseline leaves the gradient direction unbiased. Nothing below depends on how $\widehat A_t$ is obtained, only on the fact that it estimates $A^{\pi_{\theta_{\text{old}}}}$ and carries a sign.

Importance Sampling

The obstacle is that the expectation is over the wrong distribution. There is a standard identity for exactly that situation.

LemmaImportance Sampling Identity

Let $p$ and $q$ be distributions over the same space, and let $f$ be a function such that $q(x) > 0$ for every $x$ with $p(x)f(x) \neq 0$. Then

$$\mathbb{E}_{x \sim p}\big[f(x)\big] = \mathbb{E}_{x \sim q}\left[\frac{p(x)}{q(x)}f(x)\right].$$

Proof

Write the left side as a sum and insert $q(x)/q(x)$, which is legitimate at every $x$ contributing a nonzero term, by the support condition:

$$\mathbb{E}_{x\sim p}[f(x)] = \sum_x p(x)f(x) = \sum_x q(x)\,\frac{p(x)}{q(x)}\,f(x) = \mathbb{E}_{x\sim q}\left[\frac{p(x)}{q(x)}f(x)\right].$$

Points where $p(x)f(x) = 0$ contribute zero to both sides and may be ignored regardless of $q(x)$. For continuous $x$ the sums become integrals and the argument is unchanged.

RemarkWhat the Support Condition Means Operationally

The condition is not decoration. Read it in the direction this chapter needs, with $q$ the old policy and $p$ the new one: any action the new policy would like to take must have had nonzero probability under the old policy. An action the old policy never proposes is invisible to the estimator, no matter how good the new policy thinks it is, because no sample of it exists to reweight. Reuse can only redistribute credit among behaviours already sampled; it cannot discover new ones. This is a real failure mode for a policy that has already collapsed onto a narrow set of actions, and it is a structural reason on-policy methods cannot be replaced by reuse indefinitely.

Applying the identity to trajectories exposes the piece of luck that makes any of this computable.

PropositionEnvironment Terms Cancel in the Trajectory Ratio

$$\frac{p_\theta(\tau)}{p_{\theta_{\text{old}}}(\tau)} = \prod_{t=1}^{T} \frac{\pi_\theta(a_t \mid s_t)} {\pi_{\theta_{\text{old}}}(a_t \mid s_t)}$$

Proof

The trajectory factorization from the previous chapter gives

$$p_\theta(\tau) = p(s_1)\prod_{t=1}^{T}\pi_\theta(a_t\mid s_t)\,p(s_{t+1}\mid s_t,a_t),$$

and the same expression with $\theta_{\text{old}}$ in place of $\theta$. Both the initial-state term $p(s_1)$ and every transition term $p(s_{t+1}\mid s_t,a_t)$ come from the environment and carry no dependence on the policy parameters, so they appear identically in numerator and denominator and cancel factor by factor, leaving only the policy terms.

This is the same cancellation that made the policy gradient computable without a model, appearing a second time for a different reason. There it removed $\nabla_\theta$ of terms the agent does not control; here it removes those terms from a ratio. In both cases the conclusion is that a quantity which appears to require knowing the environment’s dynamics does not.

DefinitionProbability Ratio

$$r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)} {\pi_{\theta_{\text{old}}}(a_t \mid s_t)}$$

DefinitionSurrogate Objective

$$L(\theta) = \mathbb{E}_{t}\big[r_t(\theta)\,\widehat A_t\big]$$

where the expectation is over timesteps of trajectories sampled with $\pi_{\theta_{\text{old}}}$, and $\widehat A_t$ estimates $A^{\pi_{\theta_{\text{old}}}}(s_t,a_t)$.

The reason $L$ is worth optimizing at all is that its gradient, at the point where the samples were collected, is the object the previous chapter derived.

PropositionThe Surrogate Is First-Order Correct at the Sampling Point

$$\nabla_\theta L(\theta)\big|_{\theta = \theta_{\text{old}}} = \nabla_\theta J(\theta)\big|_{\theta = \theta_{\text{old}}}$$

Proof

Differentiate the ratio and evaluate at $\theta = \theta_{\text{old}}$. The denominator does not depend on $\theta$, so

$$\nabla_\theta r_t(\theta)\Big|_{\theta_{\text{old}}} = \frac{\nabla_\theta \pi_\theta(a_t\mid s_t)\big|_{\theta_{\text{old}}}} {\pi_{\theta_{\text{old}}}(a_t\mid s_t)} = \nabla_\theta \log \pi_\theta(a_t\mid s_t)\Big|_{\theta_{\text{old}}},$$

the last step being the log-derivative identity $\nabla \log \pi = \nabla \pi / \pi$ from the previous chapter, read right to left. Since $\widehat A_t$ does not depend on $\theta$,

$$\nabla_\theta L(\theta)\big|_{\theta_{\text{old}}} = \mathbb{E}_t\big[\nabla_\theta \log \pi_\theta(a_t\mid s_t)\,\widehat A_t\big]\Big|_{\theta_{\text{old}}},$$

which is exactly the policy-gradient estimator.

The Surrogate Is Not the Objective

The proposition is easy to over-read. It does not say $L = J$, and it does not say $\nabla L = \nabla J$. It says the two gradients agree at one point, $\theta_{\text{old}}$, and says nothing whatsoever about any other $\theta$.

This distinction is the hinge of the entire chapter. $L$ is a local stand-in for $J$, accurate to first order at the place the batch was drawn, and its licence expires as $\theta$ moves. Treating $L$ as the thing to be maximized, rather than as a model that is trustworthy near $\theta_{\text{old}}$ and meaningless far from it, is precisely the error every mechanism in the rest of this chapter exists to prevent.

RemarkA Second Approximation Hiding in the Per-Timestep Form

Honesty requires flagging a gap between the trajectory identity and the surrogate defined above. The importance sampling lemma applied to whole trajectories produces the product $\prod_t r_t(\theta)$ as the weight on $R(\tau)$. The surrogate $L(\theta)$ instead attaches a single $r_t(\theta)$ to each timestep’s advantage.

These are not the same object. Passing from one to the other additionally assumes that the distribution over states visited does not shift much between $\pi_{\theta_{\text{old}}}$ and $\pi_\theta$: the per-timestep form reweights the action taken at $s_t$, but not the probability of having arrived at $s_t$ at all. That assumption is exactly as local as the first-order guarantee above, and it fails in the same regime, when the policies drift apart. The per-timestep surrogate is thus a second approximation layered on the first, and both are paid for by the same remedy. The chapter’s argument does not depend on quantifying this gap, only on noting that it points the same direction as everything else here: near is safe, far is not.

Unbiased Is Not Enough

The importance sampling identity is exact. It is tempting to conclude that reuse is therefore free, and the conclusion is wrong. The identity equates two expectations, and expectations are statements about infinitely many samples. What a finite batch delivers depends on variance.

PropositionVariance of the Importance Sampling Estimator

Under the conditions of the importance sampling lemma,

$$\operatorname{Var}_{x\sim q}\left[\frac{p(x)}{q(x)}f(x)\right] = \mathbb{E}_{x\sim p}\left[\frac{p(x)}{q(x)}f(x)^2\right] - \big(\mathbb{E}_{x\sim p}[f(x)]\big)^2,$$

which is to be compared against

$$\operatorname{Var}_{x\sim p}\big[f(x)\big] = \mathbb{E}_{x\sim p}\big[f(x)^2\big] - \big(\mathbb{E}_{x\sim p}[f(x)]\big)^2 .$$

Proof

Apply $\operatorname{Var}[X] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$ to $X = (p/q)f$ under $q$. The second term is $(\mathbb{E}_{x\sim q}[(p/q)f])^2 = (\mathbb{E}_{x\sim p}[f])^2$ by the lemma. For the first,

$$\mathbb{E}_{x\sim q}\left[\left(\frac{p(x)}{q(x)}f(x)\right)^2\right] = \sum_x q(x)\frac{p(x)^2}{q(x)^2}f(x)^2 = \sum_x p(x)\,\frac{p(x)}{q(x)}\,f(x)^2 = \mathbb{E}_{x\sim p}\left[\frac{p(x)}{q(x)}f(x)^2\right].$$

The two expressions differ in exactly one place: the reweighted estimator carries an extra factor of $p(x)/q(x)$ inside the second moment. Wherever the new policy places substantially more mass than the old one, that factor is large, and it enters the variance multiplicatively while contributing nothing to the mean.

RemarkThe Variance Can Be Infinite

The severity is worth stating plainly rather than as a caution about “large” variance. The extra factor is unbounded: $p/q$ can be arbitrarily large at points where $q$ is small but nonzero, so $\mathbb{E}_p[(p/q)f^2]$ can diverge even when $f$ is bounded and both $p$ and $q$ are perfectly ordinary distributions. An estimator can be unbiased and have infinite variance simultaneously. Unbiasedness constrains where the estimator points on average across infinitely many draws; it says nothing about what a batch of a few thousand will hand you. In that regime a handful of samples with enormous ratios dominate the average, and the estimate is noise wearing the shape of a gradient.

That is the first failure. The second is already on the table and is more fundamental.

The proposition of the previous section holds at $\theta_{\text{old}}$. As $\theta$ moves away, nothing has been established about the relationship between $L$ and $J$ at all. Push $\theta$ far enough and increasing $L$ carries no implication about $J$: the optimizer is diligently climbing a model whose only guarantee was local, and which it has left behind. The per-timestep approximation flagged above degrades on the same schedule, for the same reason.

The Hinge

Two independent failures, one conclusion.

The variance of the reweighted estimator grows with the divergence between $\pi_\theta$ and $\pi_{\theta_{\text{old}}}$. The validity of the surrogate as a model of $J$ is guaranteed only at $\theta_{\text{old}}$ and degrades as $\theta$ leaves it.

Reuse is legitimate, but only locally. Every remaining section of this chapter is a consequence of that sentence: first making “locally” precise, then finding a cheap way to enforce it.

How Far Is Far: KL Divergence

“Stay near $\theta_{\text{old}}$” is not yet a usable instruction, because it does not say near in what sense. The naive reading, near in parameter space, is the wrong one: $\lVert \theta - \theta_{\text{old}} \rVert$ has no fixed relationship to how much the policy’s behaviour changed. The same parameter displacement can leave a policy essentially unchanged in one region of parameter space and transform it completely in another. What the previous section’s failures actually depend on is the divergence between two distributions, so that is what has to be measured.

DefinitionKullback-Leibler Divergence

For distributions $p$ and $q$ over the same space, with $q(x) > 0$ wherever $p(x) > 0$,

$$\mathrm{KL}(p \,\Vert\, q) = \mathbb{E}_{x \sim p}\left[\log \frac{p(x)}{q(x)}\right] = \sum_x p(x) \log \frac{p(x)}{q(x)}.$$

PropositionNon-Negativity

$\mathrm{KL}(p \Vert q) \ge 0$, with equality if and only if $p = q$ almost everywhere under $p$.

Proof

Apply Jensen’s inequality to the concave function $\log$:

$$-\mathrm{KL}(p\Vert q) = \mathbb{E}_{x\sim p}\left[\log \frac{q(x)}{p(x)}\right] \le \log \mathbb{E}_{x\sim p}\left[\frac{q(x)}{p(x)}\right] = \log \sum_x p(x)\frac{q(x)}{p(x)} = \log \sum_x q(x) = \log 1 = 0,$$

so $\mathrm{KL}(p\Vert q) \ge 0$. Because $\log$ is strictly concave, Jensen holds with equality exactly when $q(x)/p(x)$ is constant $p$-almost everywhere; since both $p$ and $q$ sum to one, that constant must be $1$, giving $p = q$ almost everywhere.

RemarkIt Is Not a Distance

Calling $\mathrm{KL}$ a distance is an abuse worth refusing explicitly, because the habit causes real errors of intuition. It is not symmetric: $\mathrm{KL}(p\Vert q) \neq \mathrm{KL}(q\Vert p)$ in general, and the two differ in what they penalize. It does not satisfy the triangle inequality. It is a divergence: non-negative, zero exactly on the diagonal, and nothing more. When a direction is chosen below, it is chosen, not conventional.

Two facts connect this to the ratio, and they turn out to be the same fact.

PropositionThe Ratio Averages to One

For any state $s$, with expectation over the old policy’s own actions,

$$\mathbb{E}_{a \sim \pi_{\theta_{\text{old}}}(\cdot\mid s)}\big[r(\theta)\big] = 1 .$$

Proof

$$\mathbb{E}_{a\sim\pi_{\theta_{\text{old}}}(\cdot\mid s)}\left[\frac{\pi_\theta(a\mid s)}{\pi_{\theta_{\text{old}}}(a\mid s)}\right] = \sum_a \pi_{\theta_{\text{old}}}(a\mid s)\,\frac{\pi_\theta(a\mid s)}{\pi_{\theta_{\text{old}}}(a\mid s)} = \sum_a \pi_\theta(a\mid s) = 1,$$

using the support condition to cancel, and that $\pi_\theta(\cdot\mid s)$ is a distribution.

PropositionKL Is the Expected Negative Log Ratio

$$\mathrm{KL}\big(\pi_{\theta_{\text{old}}}(\cdot\mid s) \,\Vert\, \pi_\theta(\cdot\mid s)\big) = \mathbb{E}_{a \sim \pi_{\theta_{\text{old}}}(\cdot\mid s)}\big[-\log r(\theta)\big]$$

Proof

Directly from the definitions:

$$\mathrm{KL}\big(\pi_{\theta_{\text{old}}} \Vert \pi_\theta\big) = \mathbb{E}_{a\sim\pi_{\theta_{\text{old}}}}\left[\log \frac{\pi_{\theta_{\text{old}}}(a\mid s)}{\pi_\theta(a\mid s)}\right] = \mathbb{E}_{a\sim\pi_{\theta_{\text{old}}}}\left[-\log \frac{\pi_\theta(a\mid s)}{\pi_{\theta_{\text{old}}}(a\mid s)}\right] = \mathbb{E}_{a\sim\pi_{\theta_{\text{old}}}}\big[-\log r(\theta)\big].$$

CorollaryNon-Negativity, Again, in the Chapter's Own Notation

Combining the two propositions, and using that $-\log$ is convex:

$$\mathrm{KL}\big(\pi_{\theta_{\text{old}}} \Vert \pi_\theta\big) = \mathbb{E}\big[-\log r\big] \ \ge\ -\log \mathbb{E}\big[r\big] = -\log 1 = 0 .$$

The corollary is worth pausing on, because it says the divergence being controlled and the ratio being clipped are not two topics. The ratio averages to exactly one under the old policy; the divergence is the expected negative log of that same ratio; and Jensen’s gap between $\mathbb{E}[-\log r]$ and $-\log\mathbb{E}[r]$ is the divergence. A policy that has not moved has $r \equiv 1$ and zero divergence. A policy that has moved has ratios spread around one, and the divergence measures the spread on a log scale. Everything this chapter does to $r$ is, indirectly, being done to $\mathrm{KL}$.

RemarkA Note on Notation

Two abuses are common enough in the literature to name. The first is writing $\mathrm{KL}(\theta, \theta')$, a divergence between parameters. There is no such object: $\mathrm{KL}$ is a functional of two distributions, and the whole reason it is the right tool here is precisely that it does not reduce to a distance between the parameters that index them. The second is dropping the state. The definition above is per-state; a policy induces a different pair of action distributions at every $s$, and the quantity of interest is the average over the states actually visited,

$$\overline{\mathrm{KL}}(\theta_{\text{old}}, \theta) = \mathbb{E}_{s}\Big[\mathrm{KL}\big(\pi_{\theta_{\text{old}}}(\cdot\mid s) \Vert \pi_\theta(\cdot\mid s)\big)\Big].$$

This chapter fixes the direction $\mathrm{KL}(\pi_{\theta_{\text{old}}} \Vert \pi_\theta)$ throughout, with the old policy first. That is not arbitrary: it is the direction whose expectation is taken over the distribution the samples actually came from, and it is the one the ratio hands over for free in the proposition above.

Clipping

The remedy has to enforce “stay near” while remaining a first-order method: one gradient, no second derivatives, no constrained optimization, a few lines of code inside an ordinary training loop. Proximal Policy Optimization’s answer is to leave the optimizer alone and modify the objective, so that moving far simply stops paying.

DefinitionClipped Surrogate Objective

$$L^{\mathrm{CLIP}}(\theta) = \mathbb{E}_t \Big[ \min\big( r_t(\theta)\,\widehat A_t, \ \operatorname{clip}\big(r_t(\theta), 1-\epsilon, 1+\epsilon\big)\,\widehat A_t \big) \Big]$$

where $\operatorname{clip}(r, a, b) = \max(a, \min(r, b))$ and $\epsilon$ is a small constant, typically $0.2$.

The definition is compact and its behaviour is not obvious from reading it. The following makes it explicit.

PropositionClipping Is a One-Sided Gradient Mask

Write $L_t(\theta) = \min\big(r_t\widehat A_t, \operatorname{clip}(r_t, 1-\epsilon, 1+\epsilon)\widehat A_t\big)$ for a single sample, and treat it as a function of $r_t$. Then

$$\frac{\partial L_t}{\partial r_t} = \begin{cases} 0 & \text{if } \widehat A_t > 0 \text{ and } r_t > 1+\epsilon,\\ 0 & \text{if } \widehat A_t < 0 \text{ and } r_t < 1-\epsilon,\\ \widehat A_t & \text{otherwise,} \end{cases}$$

at every $r_t$ where the derivative exists.

Proof

Take the two signs of $\widehat A_t$ in turn, and within each, the three regions of $r_t$.

Suppose $\widehat A_t > 0$. For $r_t > 1+\epsilon$ the clip returns $1+\epsilon$, and since multiplying by a positive $\widehat A_t$ preserves order, $r_t \widehat A_t > (1+\epsilon)\widehat A_t$, so the $\min$ selects $(1+\epsilon)\widehat A_t$, a constant in $r_t$, with derivative $0$. For $r_t \in [1-\epsilon, 1+\epsilon]$ the clip returns $r_t$ and both arguments coincide at $r_t\widehat A_t$, with derivative $\widehat A_t$. For $r_t < 1-\epsilon$ the clip returns $1-\epsilon$, and $r_t \widehat A_t < (1-\epsilon)\widehat A_t$, so the $\min$ selects the unclipped $r_t\widehat A_t$, with derivative $\widehat A_t$.

Suppose $\widehat A_t < 0$. Multiplying by a negative number reverses order, so the selections swap. For $r_t < 1-\epsilon$, $r_t\widehat A_t > (1-\epsilon)\widehat A_t$, and the $\min$ selects $(1-\epsilon)\widehat A_t$, a constant, with derivative $0$. For $r_t \in [1-\epsilon,1+\epsilon]$ both arguments are $r_t\widehat A_t$, with derivative $\widehat A_t$. For $r_t > 1+\epsilon$, $r_t\widehat A_t < (1+\epsilon)\widehat A_t$, so the $\min$ selects the unclipped $r_t\widehat A_t$, with derivative $\widehat A_t$.

Collecting the six cases gives the claim; the derivative fails to exist only at the two kink points $r_t \in \{1-\epsilon, 1+\epsilon\}$.

Laid out as a table, the structure is easier to see than to read off the algebra:

	$r_t < 1-\epsilon$	$r_t \in [1-\epsilon,\, 1+\epsilon]$	$r_t > 1+\epsilon$
$\widehat A_t > 0$	$\widehat A_t$	$\widehat A_t$	$\mathbf{0}$
$\widehat A_t < 0$	$\mathbf{0}$	$\widehat A_t$	$\widehat A_t$

The gradient vanishes in exactly two of the six cells, and they are the same cell in different clothes. When $\widehat A_t > 0$ the update wants to raise $\pi_\theta(a_t\mid s_t)$, which raises $r_t$; the gradient switches off once $r_t$ has been raised past $1+\epsilon$. When $\widehat A_t < 0$ the update wants to lower the probability, lowering $r_t$; the gradient switches off once $r_t$ has fallen below $1-\epsilon$. In both, the objective stops rewarding motion once the sample has already been moved by more than a factor of $\epsilon$ in the direction it wanted to go.

The other four cells are the important ones for understanding why this works at all.

RemarkWhy the min Is There

The $\min$ does the asymmetry, and dropping it breaks the method in a way that is easy to miss.

Consider $\widehat A_t > 0$ with $r_t$ well below $1-\epsilon$: the probability of a good action has collapsed, perhaps pushed down by other samples sharing the parameters. A bare $\operatorname{clip}$, with no $\min$, would return $1-\epsilon$ here, a constant, and the gradient would be zero. The optimizer would be unable to repair the damage. With the $\min$, this cell selects the unclipped $r_t\widehat A_t$ and the gradient flows at full strength.

So the objective is permissive in one direction and indifferent in the other: it never blocks a sample from moving back toward $r_t = 1$, and it declines to push it further once it has gone past the band. The $\min$ is what makes the mask one-sided rather than a two-sided freeze.

RemarkWhy This Is Called Proximal

Note what the mechanism does not do. It does not project $r_t$ back into $[1-\epsilon, 1+\epsilon]$, it does not reject a step, and it does not solve any constrained problem. It removes the incentive to leave the band, by making the objective flat out there. Moving far is not forbidden; it is merely unprofitable.

That distinction sounds like a technicality and is in fact the whole subject of this chapter’s last section.

The Algorithm

Everything assembles into a loop with one extra layer compared to REINFORCE: an inner loop that spends the same batch repeatedly.

Algorithm 1 PPO with the Clipped Objective

Require: initial policy parameters $\theta$, clip width $\epsilon$, epochs $K$, step size $\alpha$

1:repeat

2:$\theta_{\text{old}} \gets \theta$ // freeze the distribution the batch comes from

3:collect batch $\mathcal{D}$ of trajectories by running $\pi_{\theta_{\text{old}}}$

4:compute $\widehat{A}_t$ for every timestep in $\mathcal{D}$

5:for $k \gets 1$ to $K$ do

6:for all minibatches $B \subseteq \mathcal{D}$ do

7:$\theta \gets \theta + \alpha \nabla_\theta L^{\mathrm{CLIP}}(\theta; B)$

8:end for

9:end for

10:until converged

Typical values are $K$ between $3$ and $10$, and $\epsilon = 0.2$.

Step 1 is where the chapter’s whole argument is discharged. Freezing $\theta_{\text{old}}$ fixes the distribution the batch is understood to come from, and it stays frozen for all $K$ epochs while $\theta$ moves away from it. At the start of the inner loop $\theta = \theta_{\text{old}}$, so $r_t = 1$ everywhere and the first update is exactly the policy-gradient step of the previous chapter. Every subsequent update is reuse, paid for with the ratio and policed by the clip. When the loop ends, $\theta_{\text{old}}$ is refreshed and the batch discarded.

The gain is the factor $K$. One round of generation now funds $K$ epochs of updates rather than one, and in a setting where generation dominates the cost, that is close to a factor of $K$ in wall-clock efficiency.

What Clipping Actually Guarantees

The story assembled so far is tidy: reuse needs locality, locality is measured by $\mathrm{KL}$, and clipping keeps the ratio in a band around one. It is worth being precise about how much of that last step is actually true, because the tidy version is false and widely believed.

Clipping Does Not Bound the Ratio

The clipped objective does not constrain $r_t$ to $[1-\epsilon, 1+\epsilon]$. It cannot: $L^{\mathrm{CLIP}}$ is an objective, not a constraint set, and the proposition above says only that its derivative with respect to that sample’s own ratio vanishes outside the band.

The parameters are shared. The gradient actually applied is a sum over the whole minibatch,

$$\nabla_\theta L^{\mathrm{CLIP}} = \mathbb{E}_t\left[\frac{\partial L_t}{\partial r_t}\,\nabla_\theta r_t(\theta)\right],$$

and a sample whose own $\partial L_t/\partial r_t$ has been masked to zero still has its $r_t$ moved, because every other sample in the minibatch is pulling on the same $\theta$. Masking a sample’s gradient stops the objective from pushing that sample further out; it does nothing to stop the sample from being carried further out by its neighbours. And nothing anywhere projects a stray ratio back: once $r_t$ is outside the band, the clip’s only response is to stop caring.

Across $K$ epochs of minibatch updates, ratios therefore drift out of the band routinely, and $\overline{\mathrm{KL}}(\theta_{\text{old}}, \theta)$ is bounded by no function of $\epsilon$.

The consequence for how this chapter should be read is worth stating without hedging. The trust-region reading of PPO, in which $\epsilon$ delimits a region the policy is kept inside, is a story about the objective’s shape, not a theorem about the algorithm’s behaviour. What is true is weaker and still useful: clipping removes the reward for large policy changes, so the optimizer has no incentive to seek them, and in practice the ratios mostly stay near one, mostly. The gap between “has no incentive to” and “cannot” is why implementations monitor $\overline{\mathrm{KL}}$ during training and stop the inner loop early when it grows too large, a safeguard that would be redundant if clipping did what it is usually described as doing.

RemarkThe Honest Summary

PPO takes an exact identity (importance sampling), which licenses reuse, and pays for it with a variance that grows as the policies separate, plus a surrogate whose accuracy is guaranteed at a single point. It then controls the separation with a per-sample, one-sided, first-order gradient mask that guarantees nothing, and it works well.

This is not a defect in the exposition. It is what PPO is: a heuristic with an excellent empirical record, wearing the vocabulary of a trust region it does not implement. A reader who can say precisely which parts are theorems (the identity, the variance formula, the first-order agreement, the gradient mask) and which parts are hope (that the mask keeps $\mathrm{KL}$ small) understands PPO better than one who has been told the tidy version.

Extension: The Penalty Variant

The paper that introduced PPO proposed two ways to keep the policy near $\pi_{\theta_{\text{old}}}$, and this chapter has developed only one of them. The other replaces the clip with an explicit penalty:

$$L^{\mathrm{KLPEN}}(\theta) = \mathbb{E}_t\big[r_t(\theta)\widehat A_t\big] - \beta\, \overline{\mathrm{KL}}(\theta_{\text{old}}, \theta),$$

with $\beta$ adapted between iterations: raised when the measured $\overline{\mathrm{KL}}$ overshoots a target, lowered when it undershoots. It is the more direct expression of the idea, since it penalizes exactly the quantity the previous sections identified, and it does not pretend that bounding a ratio bounds a divergence.

It is nonetheless not what the field settled on. The clipped variant performed better in the original comparison and is simpler to tune, having one constant instead of a controller with a target and an adaptation rule, and essentially all subsequent work means the clipped objective when it says PPO. The penalty variant is included here as a pointer rather than a topic: readers who want it will find it in the original paper and in Easy RL, Chapter 5.2.1.

Further Reading

This chapter assumed the policy-gradient identity, the log-derivative trick, the trajectory factorization, and the advantage estimator, all from Policy Gradient, and the value functions of Return, Value, and the Bellman Equation.

The $\mathrm{KL}$ divergence introduced here returns as a load-bearing object rather than a diagnostic in the chapters on preference-based methods, where a penalty against a fixed reference policy, rather than against the previous iterate, becomes part of the objective being optimized.

References

• Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347
• Datawhale Easy RL, Chapter 5
• OpenAI Spinning Up: Proximal Policy Optimization