Group Relative Policy Optimization

The Second Model

Proximal Policy Optimization closed on an objective that reuses a batch $K$ times, policed by a clip on the probability ratio:

$$L^{\mathrm{CLIP}}(\theta) = \mathbb{E}_t \Big[ \min\big( r_t(\theta)\,\widehat A_t, \ \operatorname{clip}\big(r_t(\theta), 1-\epsilon, 1+\epsilon\big)\,\widehat A_t \big) \Big].$$

Everything in that line was accounted for except one symbol: $\widehat A_t$. The chapter treated the advantage estimate as inherited machinery, $\widehat A_t = G_t - b(s_t)$ with $b(s_t) = V^\pi(s_t)$, and proved only that subtracting an action-independent baseline leaves the gradient unbiased. It never said where $V^\pi$ comes from. In the algorithm actually run, it comes from a second neural network.

RemarkWhat the Advantage Estimate Hides

In full PPO the advantage is computed by Generalized Advantage Estimation, which combines the observed rewards with a learned value function $V_\psi(s_t)$ trained alongside the policy by regression toward returns. That $V_\psi$ is not a small auxiliary. It is a network of roughly the policy’s own size, with its own forward and backward pass, its own optimizer state, and its own failure modes. PPO is an actor-critic method, and the critic is the half that this chapter’s predecessor quietly folded into the symbol $\widehat A_t$.

So the true cost of a PPO step is two models, not one. The policy is what you want; the critic exists only to supply the baseline that makes the policy’s gradient low-variance. It is pure overhead in the sense that it is discarded at the end of training, and it doubles the memory and roughly the compute of every update.

For the domains this series is heading toward, the overhead is compounded by a mismatch. Consider where the reward actually appears.

RemarkTerminal Reward Makes the Critic Ill-Posed

In the settings that motivate this chapter, a single trajectory is a full generated sequence, and the reward is one scalar delivered at the end: a completion is scored once it is complete, a proof once it is finished. There is no per-step reward, only a terminal one.

A per-token value function is then being asked to predict, at every intermediate token, the expected final score of a sequence that does not yet exist. It must learn to be accurate thousands of tokens before the only supervision signal arrives. This is a genuinely hard regression problem, and it is hard precisely in proportion to how long the generation is, which is to say most where it is needed most. The critic is not only expensive; in the terminal-reward regime it is estimating an awkward object.

The question this chapter answers is whether the baseline can be had without the second network at all.

Can the variance reduction of a value baseline be obtained without learning a value function?

The answer is yes, and it costs samples instead of a model. The rest of the chapter is what that trade buys and what it assumes.

A Baseline You Can Sample

The licence for what follows was proved two chapters ago and is worth restating exactly, because the entire method is one substitution into it.

PropositionAny Action-Independent Baseline Is Unbiased

For any function $b$ that depends on the state but not on the action,

$$\mathbb{E}_{a \sim \pi_\theta(\cdot\mid s)}\big[\nabla_\theta \log \pi_\theta(a\mid s)\, b(s)\big] = 0,$$

so subtracting $b(s)$ from the return inside the policy gradient changes neither the gradient’s expectation nor its direction.

Proof

Pull $b(s)$ out of the action-expectation, since it does not depend on $a$, and use the log-derivative identity in reverse:

$$\mathbb{E}_{a}\big[\nabla_\theta \log \pi_\theta(a\mid s)\big] = \sum_a \pi_\theta(a\mid s)\,\frac{\nabla_\theta \pi_\theta(a\mid s)}{\pi_\theta(a\mid s)} = \nabla_\theta \sum_a \pi_\theta(a\mid s) = \nabla_\theta 1 = 0.$$

Multiplying the zero vector by $b(s)$ leaves zero.

The best such baseline, the one that removes the most variance, is the state’s own value $V^\pi(s) = \mathbb{E}_{a\sim\pi}[Q^\pi(s,a)]$: it centers each return on what the policy expects from that state. PPO learns $V^\pi$ with the critic. But $V^\pi(s)$ is an expectation, and there is a second way to obtain an expectation, older than any network.

The Substitution

The value $V^\pi(s)$ is $\mathbb{E}_{a\sim\pi}[\text{return from }(s,a)]$. In the terminal-reward setting the return from $(s,a)$ is just the final reward of the trajectory that $a$ begins, so

$$V^\pi(s) = \mathbb{E}_{o \sim \pi(\cdot\mid s)}\big[r(s,o)\big].$$

This is an average over the policy’s own outputs from $s$. Rather than train a network to predict it, sample it. Draw a group of outputs $o_1,\dots,o_G$ from $s$, score each, and take the mean:

$$b(s) \;\approx\; \frac{1}{G}\sum_{j=1}^{G} r(s,o_j).$$

The same generations that the policy-gradient step consumes are, collectively, a Monte Carlo estimate of the baseline that step needs. The critic’s entire job is done by the batch, for free.

This is the whole idea of Group Relative Policy Optimization. Where PPO learns a baseline, GRPO samples one, using a group of completions per prompt precisely so the group has a mean to subtract.

RemarkOne Honest Dependence

The baseline theorem requires $b(s)$ to be independent of the action being scored. The group mean $\frac{1}{G}\sum_j r(s,o_j)$ includes the term $r(s,o_i)$ when it is used as the baseline for output $o_i$, so it is not strictly independent of $o_i$, and the estimator carries a bias of order $1/G$. A leave-one-out mean, excluding $o_i$, would be exactly unbiased. The method as published uses the full mean for simplicity and takes $G$ large enough that the distinction is immaterial. Say it plainly rather than bury it: dropping the learned baseline costs a small, controllable bias, plus $G$ generations per prompt instead of one.

The Group-Relative Advantage

Subtracting the group mean gives a centered reward. GRPO takes one further step and normalizes by the group’s spread.

DefinitionGroup-Relative Advantage (Outcome Supervision)

For a prompt $q$, let $\mathbf{r} = (r_1,\dots,r_G)$ be the rewards of a sampled group of outputs. The advantage assigned to every token of output $o_i$ is

$$\widehat A_{i,t} = \frac{r_i - \operatorname{mean}(\mathbf{r})}{\operatorname{std}(\mathbf{r})}.$$

Two things are happening in that ratio, and they have different standing.

The numerator, $r_i - \operatorname{mean}(\mathbf{r})$, is the baseline subtraction of the previous section: the centered reward, an estimate of $A^\pi = Q^\pi - V^\pi$ restricted to the terminal signal. Its meaning is exactly the advantage’s meaning from PPO, an output scored relative to the policy’s own average on this prompt, positive for above-average completions and negative for below. The sign is a comparison, not an absolute score, just as before.

The denominator, $\operatorname{std}(\mathbf{r})$, is not part of that argument.

RemarkWhat the Standard Deviation Is and Is Not

Dividing by $\operatorname{std}(\mathbf{r})$ rescales each prompt’s advantages to unit variance before they enter the gradient. Its purpose is practical: prompts differ in how spread out their rewards are, and without normalization an easy prompt with tightly clustered rewards and a hard prompt with widely scattered ones would contribute gradients of very different magnitude for no principled reason. The normalization puts them on a common scale, acting as a per-prompt adaptive step size.

It is not, however, part of the unbiased-baseline story. Multiplying an advantage by the positive scalar $1/\operatorname{std}(\mathbf{r})$ preserves its sign and the ordering within the group, but it does reweight how much each prompt pulls on $\theta$: prompts with low reward variance are amplified. This is a deliberate heuristic, not a theorem, and later analyses have noted it introduces a mild bias toward such prompts. For this chapter it is enough to keep the two parts separate: the numerator is the baseline the previous section justified; the denominator is a scaling choice layered on top.

Every token of $o_i$ receives the same $\widehat A_{i,t}$, because the reward is terminal: there is one scalar for the whole output, and it is broadcast to each token that helped produce it. This is credit assignment at its bluntest, every token in a good completion is reinforced equally, and it is exactly what the terminal-reward structure permits and the per-token critic tried to refine.

RemarkProcess Supervision as a Refinement

When a reward can be given at the end of each reasoning step rather than only at the end of the whole output, the same construction runs with finer granularity: normalize the step rewards across the group, and set each token’s advantage to the sum of normalized rewards from that token onward. This recovers some of the within-trajectory credit assignment the terminal signal throws away, without reintroducing a learned value function. It is a refinement of the advantage, not a change to anything else in the chapter, and outcome supervision is the version to hold in mind.

The Reference Policy

The critic is gone. One term of the objective is left to build, and it discharges a promise the previous chapter made explicitly.

PPO’s KL divergence measured the gap between $\pi_\theta$ and $\pi_{\theta_{\text{old}}}$, the previous iterate. It was a diagnostic: a quantity watched to decide whether reuse had drifted too far, controlling the locality that importance sampling required. It regularized nothing directly; the clip did the work, and the KL was a monitor.

The KL that enters here is a different object with a different job.

DefinitionKL-Regularized Objective

Let $\pi_{\text{ref}}$ be a fixed reference policy, typically the model at the start of fine-tuning. The objective is to maximize the reward while staying close to that reference:

$$\mathcal{J}(\theta) = \mathbb{E}_{o \sim \pi_\theta}\big[r(q,o)\big] - \beta\,\mathrm{KL}\big(\pi_\theta \,\Vert\, \pi_{\text{ref}}\big).$$

Three differences from PPO’s use of KL matter here, and they are easy to conflate:

• The second argument is fixed. $\pi_{\text{ref}}$ does not move as training proceeds; it is a frozen anchor, not the last step’s parameters. The policy is kept near where it started, not near where it was.
• It is part of the objective, not a monitor. The $\beta\,\mathrm{KL}$ term is optimized, its gradient flows into $\theta$. It is load-bearing in a way PPO’s diagnostic KL never was.
• Its purpose is different. PPO’s KL guarded the validity of sample reuse. This one guards against the policy exploiting the reward: a learned reward $r$ is an imperfect proxy, and a policy free to maximize it will find its blind spots and produce degenerate, high-scoring, useless outputs. Anchoring to $\pi_{\text{ref}}$ keeps the policy in the region where the reward is trustworthy.

This Is the Objective Both Chapters Share

$\mathcal{J}(\theta)$ is the destination of the whole series’ second half. GRPO reaches its maximum by the reinforcement-learning loop this chapter is assembling: sample, score, estimate advantage, step. The next chapter, on Direct Preference Optimization, will take this exact objective and solve it in closed form, discovering that when the reward comes from preferences the loop can be removed entirely. Everything downstream is a statement about how to optimize this one line. Introduced here, it is inherited there.

One smaller design choice is deliberate in the original work and easy to miss. The reward-model literature that PPO fine-tuning grew out of folds the KL penalty into the reward, subtracting a per-token $\beta \log \frac{\pi_\theta}{\pi_{\text{ref}}}$ from the score before advantages are computed. GRPO instead adds the KL directly to the loss, leaving the advantage computation to depend only on the reward model’s output. The two are not equivalent bookkeeping: keeping KL out of the reward means the group normalization above acts on the clean reward signal, and the divergence is regularized as its own term rather than laundered through the advantage.

Estimating the Divergence

The objective needs $\mathrm{KL}(\pi_\theta \Vert \pi_{\text{ref}})$, but the expectation defining it ranges over the whole vocabulary at every token and is not computed exactly. It is estimated from the sampled tokens, and the choice of estimator is not obvious, because the naive one misbehaves.

The definition is $\mathrm{KL}(\pi_\theta \Vert \pi_{\text{ref}}) = \mathbb{E}_{o\sim\pi_\theta}\big[\log \frac{\pi_\theta(o)}{\pi_{\text{ref}}(o)}\big]$. Estimating it by its integrand, a single sample of $\log \frac{\pi_\theta}{\pi_{\text{ref}}}$, is unbiased but has a defect: it is negative whenever a sampled token is more likely under $\pi_{\text{ref}}$ than under $\pi_\theta$, even though the quantity it estimates is non-negative. A regularizer that is supposed to penalize divergence but frequently reports a negative value injects noise of the wrong sign. GRPO uses a better estimator.

DefinitionThe $k_3$ KL Estimator

Writing $\rho = \dfrac{\pi_{\text{ref}}(o_t \mid q, o_{<t})}{\pi_\theta(o_t \mid q, o_{<t})}$ for a sampled token, the per-token divergence estimate is

$$\widehat{\mathrm{KL}}_t = \rho - \log \rho - 1.$$

It has exactly the two properties the naive estimator lacks and keeps, together, the one it had.

PropositionThe Estimator Is Non-Negative and Unbiased

For every sampled token, $\widehat{\mathrm{KL}}_t \ge 0$, with equality iff $\pi_\theta = \pi_{\text{ref}}$ at that token. And taking the expectation over $o_t \sim \pi_\theta$,

$$\mathbb{E}\big[\widehat{\mathrm{KL}}_t\big] = \mathrm{KL}\big(\pi_\theta(\cdot\mid q,o_{<t}) \,\Vert\, \pi_{\text{ref}}(\cdot\mid q,o_{<t})\big).$$

Proof

Non-negativity. For any $\rho > 0$ the inequality $\log \rho \le \rho - 1$ holds, with equality iff $\rho = 1$ (it is the tangent line to the concave $\log$ at $\rho=1$). Rearranged, $\rho - \log \rho - 1 \ge 0$, and $\rho = 1$ means the two policies agree on that token.

Unbiasedness. Split the expectation into three terms. The first is

$$\mathbb{E}_{o_t\sim\pi_\theta}[\rho] = \sum_{o_t} \pi_\theta(o_t)\,\frac{\pi_{\text{ref}}(o_t)}{\pi_\theta(o_t)} = \sum_{o_t} \pi_{\text{ref}}(o_t) = 1,$$

the same “the ratio averages to one” identity used in the PPO chapter, now with the reference in the numerator. The second is

$$\mathbb{E}_{o_t\sim\pi_\theta}[-\log\rho] = \mathbb{E}_{o_t\sim\pi_\theta}\left[\log\frac{\pi_\theta(o_t)}{\pi_{\text{ref}}(o_t)}\right] = \mathrm{KL}(\pi_\theta \Vert \pi_{\text{ref}}),$$

by the definition of KL. The third is the constant $-1$. Summing, $\mathbb{E}[\widehat{\mathrm{KL}}_t] = 1 + \mathrm{KL} - 1 = \mathrm{KL}$.

So $\widehat{\mathrm{KL}}_t$ estimates the right thing on average and never reports a negative penalty. That it is both unbiased and non-negative is the reason it is preferred over the integrand; the extra $\rho - 1$, which averages to zero, is a control variate that removes the sign problem without touching the mean.

The Objective and the Algorithm

Assembling the pieces, the clipped surrogate of PPO, the group advantage in place of the critic’s, and the reference-KL as an explicit term, gives the GRPO objective.

DefinitionGRPO Objective

$$\mathcal{J}^{\mathrm{GRPO}}(\theta) = \mathbb{E}\Bigg[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\Big( r_{i,t}(\theta)\,\widehat A_{i,t},\ \operatorname{clip}\big(r_{i,t}(\theta), 1-\epsilon, 1+\epsilon\big)\,\widehat A_{i,t} \Big) - \beta\,\widehat{\mathrm{KL}}_{i,t} \Bigg]$$

where $r_{i,t}(\theta) = \dfrac{\pi_\theta(o_{i,t}\mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q, o_{i,<t})}$ is the per-token probability ratio, $\widehat A_{i,t}$ is the group-relative advantage, and the expectation is over prompts $q$ and groups $\{o_i\}_{i=1}^G$ sampled from $\pi_{\theta_{\text{old}}}$.

Read against PPO’s $L^{\mathrm{CLIP}}$, the objective is the same clipped surrogate with two edits. The advantage $\widehat A_{i,t}$ is now the group-relative quantity, no critic anywhere in it. And the $-\beta\,\widehat{\mathrm{KL}}_{i,t}$ term is appended, the reference anchor made explicit. The clipping machinery, and everything the previous chapter proved about what it does and does not guarantee, carries over unchanged: it still only masks each sample’s own gradient outside the band, still bounds no divergence by itself. That analysis is not repeated here; it applies verbatim.

The loop is PPO’s with the critic’s training deleted and a group sample inserted.

Algorithm 1 GRPO with Outcome Supervision

Require: initial policy $\pi_\theta$, reward model $r_\varphi$, prompts $\mathcal{D}$, group size $G$, clip width $\epsilon$, KL weight $\beta$, inner epochs $\mu$, step size $\alpha$

1:$\pi_{\text{ref}} \gets \pi_\theta$ // freeze the anchor for the KL term

2:repeat

3:$\pi_{\theta_{\text{old}}} \gets \pi_\theta$

4:sample a batch of prompts $q \sim \mathcal{D}$

5:for each $q$, sample a group $\{o_i\}_{i=1}^{G} \sim \pi_{\theta_{\text{old}}}(\cdot \mid q)$

6:score every output: $r_i \gets r_\varphi(q, o_i)$

7:$\widehat A_{i,t} \gets \dfrac{r_i - \operatorname{mean}(\mathbf{r})}{\operatorname{std}(\mathbf{r})}$ for all tokens $t$ of $o_i$

8:for $k \gets 1$ to $\mu$ do

9:$\theta \gets \theta + \alpha \nabla_\theta \mathcal{J}^{\mathrm{GRPO}}(\theta)$

10:end for

11:until converged

What was deleted is the entire actor-critic apparatus for the value function: no value network, no value forward pass, no value loss, no value optimizer state. In a regime where the models are large enough that holding two of them dominates the memory budget, dropping one is not a marginal saving. What was added is $G$ generations per prompt instead of one, which is more sampling but of the cheap kind relative to holding and training a second network, and it is sampling the group approach turns to double duty: the same completions are both what the policy gradient reinforces and what estimates the baseline.

RemarkRefreshing the Anchor

Freezing $\pi_{\text{ref}}$ once at the start is the simplest version and the one to understand first. In practice the reference is sometimes refreshed to the current policy every so often, so the anchor tracks the policy at a slow cadence while $\pi_{\theta_{\text{old}}}$ tracks it at every step. This is a knob on how much total drift from the starting model is permitted, and it does not change anything in the objective above; it only says which frozen copy plays $\pi_{\text{ref}}$.

Application: Reinforcement Learning on a Language Model

The chapter so far is agent-agnostic. Nothing above mentions language: it needs a state, a policy over actions, a terminal scalar reward, and a reference policy, and a robot controller with a sparse end-of-episode reward would fit the same objective. But the setting where GRPO is usually first met, and the one that motivated its terminal-reward framing, is post-training a language model, and it is worth spending a section reading the abstractions back onto that case, because the mapping is where most of the confusion lives.

Four questions carry the translation.

How does the Markov decision process transfer? The correspondence is exact once the pieces are named:

RL object	Language-model instantiation
initial state $s_1$	the prompt $q$
action $a_t$	the next token $o_t$
state $s_t$	the prompt plus tokens so far, $(q, o_{<t})$
transition $p(s_{t+1}\mid s_t,a_t)$	deterministic: append $o_t$ to the context
trajectory $\tau$	the full completion $o = (o_1,\dots,o_T),\ T=\lvert o\rvert$
terminal reward	one score $r_\varphi(q, o)$ for the finished completion

The transition is the special feature: appending a token to a context is deterministic, so all the environment’s stochasticity that the PPO chapter carried, $p(s_{t+1}\mid s_t,a_t)$, collapses. The only randomness in a rollout is the policy’s own sampling of tokens. The generation is the trajectory.

What is the action space, and what supplies the reward? The action space is the vocabulary, tens of thousands of tokens, and the policy $\pi_\theta(o_t \mid q, o_{<t})$ is literally the language model’s next-token distribution, the softmax over that vocabulary. There is no separate policy network to build; the model being fine-tuned is the policy. The reward is not part of the environment, as it would be in a game. It is supplied by a separate learned model $r_\varphi$, and the humans whose judgment it encodes are not in the loop.

How are tokens treated inside the objective? Every mechanism in the objective is per-token: a ratio $r_{i,t}$, a clip, a KL estimate, one for each token position. But the reward is per-sequence, delivered once. Outcome supervision resolves the mismatch by broadcasting the single normalized reward to every token of the completion, so $\widehat A_{i,t}$ is constant across $t$ within an output. This is the same fact that made the per-token critic awkward, seen from the other side: because the signal is terminal, a constant-per-sequence advantage is what there is to work with, and the group baseline supplies it without pretending to per-token resolution the reward does not have.

How does human preference enter? Not through the loop, but through $r_\varphi$, which is trained once, beforehand, on human comparisons. This is the piece that turns “reinforcement learning” into “reinforcement learning from human feedback,” and it is worth making precise, because the reward model built here is exactly the object the next chapter dismantles.

DefinitionThe Preference Reward Model

Humans are shown a prompt $q$ and two completions and asked which they prefer, yielding a dataset of triples $(q, o_w, o_l)$ where $o_w$ is the preferred (“winning”) completion and $o_l$ the rejected (“losing”) one. Under the Bradley-Terry model, the probability that $o_w$ is preferred is assumed to be

$$P(o_w \succ o_l \mid q) = \sigma\big(r_\varphi(q, o_w) - r_\varphi(q, o_l)\big),$$

where $\sigma$ is the logistic function. The reward model $r_\varphi$ is trained to maximize the likelihood of the observed preferences, i.e.\ to minimize

$$\mathcal{L}(\varphi) = -\,\mathbb{E}_{(q,o_w,o_l)}\big[\log \sigma\big(r_\varphi(q,o_w) - r_\varphi(q,o_l)\big)\big].$$

The model learns a scalar score whose differences reproduce human choices. Only differences are constrained: the Bradley-Terry likelihood is unchanged if a constant is added to every score at a given prompt, so $r_\varphi$ is identified only up to a per-prompt shift, a fact that looks like a technicality here and becomes the load-bearing cancellation of the next chapter.

With the reward model in hand, the three stages and the flow of data among the models look like this:

The diagram makes the model economy visible. One supervised model is trained and then copied three ways: it seeds the reward model, it becomes the initial policy, and a frozen copy of it is the reference. During GRPO only the policy trains; the reward model and the reference are frozen. The preference dataset touches the pipeline exactly once, at reward-model training, and thereafter human judgment reaches the policy only through $r_\varphi$‘s scores.

And a single rollout, the trajectory the RL objective is defined over, is one left-to-right generation with the score arriving only at the end:

This is the concrete shape of everything the abstract sections built: tokens are actions, the growing context is the state, the completion is the trajectory, and the one reward at the end is spread back across the tokens that produced it.

One Gradient, Many Methods

There is a vantage point from which GRPO, PPO, and the method of the next chapter are visibly the same construction, and reaching it is the cleanest way to set up what comes next. Write the gradient of every method in this family in one form:

$$\nabla_\theta \mathcal{J}(\theta) = \mathbb{E}_{(q,o)\sim \mathcal{D}} \left[ \frac{1}{|o|}\sum_{t=1}^{|o|} \underbrace{GC(q,o,t)}_{\text{gradient coefficient}}\, \nabla_\theta \log \pi_\theta(o_t \mid q, o_{<t}) \right].$$

Every method here is a weighted maximum-likelihood objective on generated sequences: it pushes up the log-probability of tokens, and the only thing that differs between methods is the weight $GC$ on each token and where the data $\mathcal{D}$ comes from.

Method	Data source	Gradient coefficient $GC$
Supervised fine-tuning	fixed demonstrations	$1$
Rejection sampling	model’s correct samples	$\mathbb{1}[o \text{ correct}]$
PPO	on-policy samples	$A_t$ (critic advantage)
GRPO	on-policy groups	$\widehat A_{i,t} + \beta(\cdots)$ (group advantage + KL)
DPO	preference pairs	a preference weight (next chapter)

Seen this way, “reinforcement learning fine-tuning” is reweighted imitation. Supervised fine-tuning imitates every demonstration equally; rejection sampling imitates only the correct samples; PPO and GRPO imitate on-policy samples weighted by how much better than average they were. The reinforcement-learning machinery, the sampling loop, the reward model, the advantage, the clip, all of it, exists to compute the weight $GC$ for each token.

The Door to the Next Chapter

If the machinery only exists to compute a weight, the natural question is whether the weight can be computed without the machinery. For preference data the answer is yes. The next chapter takes the KL-regularized objective introduced above, solves it exactly, and finds that the optimal policy’s own log-probabilities encode the reward, so the weight can be written in closed form from a pair of completions with no sampling loop, no reward model, and no advantage. GRPO removed the critic; Direct Preference Optimization removes the loop.

Further Reading

This chapter assumed the clipped surrogate, the probability ratio, and the analysis of what clipping guarantees from Proximal Policy Optimization, and the baseline theorem and advantage from Policy Gradient.

The KL-regularized objective built in the middle of the chapter is the shared starting point of the next one, where it is not optimized by sampling but solved in closed form. The Bradley-Terry reward model defined in the application section is the object that chapter shows can be bypassed: its per-prompt shift ambiguity, noted above as a technicality, is exactly what cancels.

References

• Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. Introduces GRPO (§4) and the unified-paradigm view (§5.2.1).
• Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training Language Models to Follow Instructions with Human Feedback. arXiv:2203.02155. The RLHF pipeline and the per-token KL-in-reward formulation.
• Schulman, J. (2020). Approximating KL Divergence. joschu.net/blog/kl-approx.html. The $k_3$ estimator used for the divergence term.
• Bradley, R. A., & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4), 324-345.