04 July 2026In progress

Reinforcement Learning, from First Principles

A bilingual series building reinforcement learning from the ground up in nine chapters: from what makes it a distinct learning problem, through the Markov ladder and the Bellman machinery, to a full derivation of policy gradient, the sample-reuse machinery of PPO, GRPO's removal of the critic, and DPO's direct formulation of preference learning as policy optimization.

Reinforcement Learning

Reading guide

This nine-chapter series develops reinforcement learning from first principles. Begin with the learning problem and Markov models, use returns and value functions to derive the Bellman equations, and turn those equations into planning algorithms. Policies as probability distributions then lead to policy gradients, PPO, GRPO, and DPO.

Definitions, proofs, and worked explanations develop the links between chapters. Previous/next navigation follows the same order as the path below. Later extensions may explore other preference objectives and decision processes in diffusion models. The conceptual starting point is Datawhale Easy RL, with independently developed derivations and additional material.

Reading path

The Reinforcement Learning Problem
What has to be built before a single gradient can be computed: why removing the teacher from supervised learning leaves a problem that is not yet well-posed, the interaction loop that replaces the dataset, the action space and the model that the loop leaves unspecified, and the two structural obstacles, credit assignment and exploration, that have no supervised counterpart.
The Markov Ladder
Three rungs, built one assumption at a time. The Markov property is what makes a state a state, and is exactly a sufficient statistic applied to time; it is also priced here, since conditioning on full history costs n^(T+1) parameters and stays unlearnable from finite data no matter how much memory you have, which is what makes the assumption enabling rather than merely convenient. Then the Markov process and reward process add transition and scoring but no agency, and the Markov decision process adds actions, forcing a second value function into existence.
Return, Value, and the Bellman Equation
One line does all the work: the return satisfies G_t = R_{t+1} + gamma*G_{t+1}. Everything in this chapter, the two value functions, the Bellman expectation equations, and the optimality equations, is that single recursion pushed through an expectation and then optimized over. Includes Bellman's principle of optimality, which is the actual reason a maximum may replace an average.
Planning with a Known Model
Turning the Bellman equations from identities into algorithms, when the model is known. One idea carries the whole chapter: both Bellman operators are gamma-contractions, so Banach's fixed-point theorem delivers existence, uniqueness, and geometric convergence at once. Covers policy evaluation, the policy improvement theorem, policy iteration, value iteration, and the exact point where a known model stops being available.
Policies as Probability Distributions
A policy can be a bare function from states to actions, or a full probability distribution over actions. This note works out why the second, harder-looking choice wins in practice, for two genuinely different reasons, and builds the categorical and diagonal Gaussian policies that make it concrete for discrete and continuous action spaces.
Policy Gradient
A rigorous derivation of the policy-gradient identity via the log-derivative (score-function) trick, from the trajectory-return objective through REINFORCE and the baseline/advantage variance-reduction argument, with the bridge to PPO.
Proximal Policy Optimization
One chain, each link forced by the last. Policy gradient is on-policy, so every expensive sample funds exactly one update and is then discarded. Importance sampling buys reuse, but its variance grows with the divergence between the policies and the surrogate it produces is only first-order accurate at the old parameters. Both failures say the same thing: stay near. KL divergence makes near precise, and clipping enforces it cheaply, though as this chapter's last section shows, it does not actually enforce anything.
Group Relative Policy Optimization
PPO carries a second network, a critic, to compute its advantages, and in a world where the reward lands once at the end of a long generation, that critic is both expensive and ill-posed. GRPO removes it: a group of samples drawn for the same prompt is its own baseline, the group mean standing in for the value the critic used to learn. What remains re-aims the KL divergence from the previous iterate to a fixed reference policy, turning a training diagnostic into a load-bearing term of the objective. This chapter builds that objective from PPO by subtraction, then reads it back onto language models, the setting where the whole apparatus is usually met first.
Direct Preference Optimization
DPO is a general conditional preference-learning paradigm: improve agreement with preferences without moving too far from a reference policy. Given preference pairs, must one first fit a reward model and then run reinforcement learning? No. Solve the KL-regularized objective for its optimal policy, substitute that form into a Bradley-Terry preference model, and the reward disappears. What remains is a binary loss that applies to any conditional probabilistic neural network.