What This Series Is

Nine chapters that build reinforcement learning from first principles. The series starts from the question of what makes reinforcement learning a genuinely different learning problem, climbs the Markov ladder from bare state transitions to decision processes, develops the Bellman machinery with full contraction proofs, derives the policy-gradient identity through REINFORCE and variance reduction, arrives at PPO, where the cost of sampling forces the whole apparatus of importance sampling, divergence, and clipping into existence, reaches GRPO, which removes PPO’s critic by sampling its baseline, and ends with DPO, which turns offline preference pairs into a policy-optimization loss from the same KL-regularized objective.

Every chapter aims to be rigorous for its stated scope: definitions are precise, theorems are proved rather than quoted, and each page stays independently readable while inline links carry the larger arc. English and Chinese versions are written together.

Reading Path

  1. The Reinforcement Learning Problem: why the supervised framing breaks, the interaction loop, action spaces, models, and the credit-assignment and exploration obstacles
  2. The Markov Ladder: the Markov property as a sufficient statistic for the future, then MP, MRP, and MDP one rung at a time
  3. Return, Value, and the Bellman Equation: the one-step return recursion, the two value functions, and every Bellman equation derived from that single lemma
  4. Planning with a Known Model: turning those identities into algorithms via contraction proofs: policy evaluation, policy improvement, policy iteration, value iteration
  5. Policies as Probability Distributions: the categorical and diagonal Gaussian policies, and the log-likelihood objective
  6. Policy Gradient: the full derivation, REINFORCE, reward-to-go, and baselines
  7. Proximal Policy Optimization: importance sampling buys sample reuse, its variance and the surrogate’s locality demand a short step, KL divergence measures the step, and clipping enforces it (or rather, does not)
  8. Group Relative Policy Optimization: replacing PPO’s learned critic with the mean of a sampled group, re-aiming the KL from the previous iterate to a fixed reference policy, and reading the whole apparatus back onto language-model RLHF
  9. Direct Preference Optimization: deriving the DPO loss from the same KL-regularized objective, and showing how reward modeling and the online reinforcement-learning loop disappear into preference pairs

Each chapter also carries previous/next navigation in this order, so any entry point works.

Where the Series Goes Next

From DPO, one route continues to other preference-optimization objectives and their assumptions. A second line applies these algorithms to diffusion models, where the denoising chain itself is the decision process. Those chapters are planned as the next batch.

Sources

The conceptual skeleton follows Datawhale Easy RL; definitions, proofs, and the connective argument are developed independently in these notes, with added material (contraction proofs, the policy improvement theorem, the reward-to-go causality argument, Gaussian reparameterization) beyond the source’s scope.