Noteartificial intelligence18 July 2026growing
DPO is a general conditional preference-learning paradigm: improve agreement with preferences without moving too far from a reference policy. Given preference pairs, must one first fit a reward model and then run reinforcement learning? No. Solve the KL-regularized objective for its optimal policy, substitute that form into a Bradley-Terry preference model, and the reward disappears. What remains is a binary loss that applies to any conditional probabilistic neural network.
Reinforcement LearningPreference LearningOptimization
Noteartificial intelligence18 July 2026growing
PPO carries a second network, a critic, to compute its advantages, and in a world where the reward lands once at the end of a long generation, that critic is both expensive and ill-posed. GRPO removes it: a group of samples drawn for the same prompt is its own baseline, the group mean standing in for the value the critic used to learn. What remains re-aims the KL divergence from the previous iterate to a fixed reference policy, turning a training diagnostic into a load-bearing term of the objective. This chapter builds that objective from PPO by subtraction, then reads it back onto language models, the setting where the whole apparatus is usually met first.
Reinforcement LearningPolicy GradientOptimization
Noteartificial intelligence15 July 2026stable
One chain, each link forced by the last. Policy gradient is on-policy, so every expensive sample funds exactly one update and is then discarded. Importance sampling buys reuse, but its variance grows with the divergence between the policies and the surrogate it produces is only first-order accurate at the old parameters. Both failures say the same thing: stay near. KL divergence makes near precise, and clipping enforces it cheaply, though as this chapter's last section shows, it does not actually enforce anything.
Reinforcement LearningPolicy GradientOptimization
BlogLinguistics05 July 2026
A position piece on why linguistics never left my study of AI, math, and CS: the same instinct for modeling structure through pattern shows up in language taxonomy, syntax trees, categorial grammar, and the old fight between linguists and statisticians in NLP.
LinguisticsAIComputer ScienceMathematics
Noteartificial intelligence03 July 2026stable
A policy can be a bare function from states to actions, or a full probability distribution over actions. This note works out why the second, harder-looking choice wins in practice, for two genuinely different reasons, and builds the categorical and diagonal Gaussian policies that make it concrete for discrete and continuous action spaces.
Reinforcement LearningPolicy Gradient
