Reinforcement Learning

What It Is

A machine learning paradigm where an agent learns by interacting with an environment — taking actions, receiving scalar reward signals, and updating its policy to maximize cumulative reward over time. No labeled data; only the reward signal guides learning.

Why It Matters

RL is how language models are trained to follow human preferences (RLHF), how AlphaGo learned to play Go, and how robots learn to walk. It’s the bridge between supervised learning (which requires labeled examples) and open-ended optimization (which requires only a reward function).

How It Works

The agent observes a state $s_t$, selects an action $a_t$ according to its policy ${\pi }_{\theta }(a\mid s)$, receives reward $r_t$, and transitions to state $s_{t+1}$. The goal is to find ${\pi }_{\theta }$ that maximizes expected discounted return $\mathbb{E}\left[{\sum }_{t=0}^{\infty }{\gamma }^t{r}_t\right]$. Policy gradient methods like PPO optimize ${\pi }_{\theta }$ directly by ascending the gradient of expected reward. Value-based methods like Q-learning learn a value function $V(s)$ or $Q(s,a)$ and derive a policy from it.

Key Sources

• proximal-policy-optimization

• grpo-deepseekmath-group-relative-policy-optimization

• learning-to-summarize-human-feedback

• deepseek-r1-reasoning-via-reinforcement-learning

• metis-hdpo-meta-cognitive-tool-use

Related Concepts

• policy-gradient
• ppo
• rlhf
• reward-model