Reinforcement Learning: Q-Learning, Policy Gradients, and Applications
RL learns behaviour from consequences, not examples. Read the two families, the sample-efficiency problem, and where the method actually applies.
Reinforcement learning is usually introduced with games, and the examples are memorable, but they obscure the method's actual shape. An RL agent is not given the right answer; it is given a measure of how well it is doing and must discover behaviour that improves that measure. That makes the central problem different from supervised learning: the data depends on the actions the agent chooses, so the agent has to decide what to try as well as what to learn.
Direct answer: The field divides broadly into value-based methods, which learn how good it is to take an action in a state, and policy-gradient methods, which learn a parameterised behaviour directly. Deep Q-learning established that control policies can be learned from raw sensory input; PPO made policy gradients practical by allowing multiple updates per batch of experience; and human-preference methods showed that the reward signal can come from comparisons rather than a hand-written function. The binding constraint across all of them is sample efficiency — how much interaction is needed to learn something useful — which is why most real-world applications are in simulation or in domains where mistakes are cheap.

The loop and the credit problem
An RL loop has a small number of parts: an agent observes a state, takes an action, receives a reward and a new state, and repeats. The objective is not the immediate reward but the discounted sum of future rewards, which introduces the central difficulty — credit assignment.
If a chess player wins on move forty, which of the preceding moves deserved the credit? The same problem appears in any sequential task where consequences arrive late. Value-based and policy-gradient methods attack it from different directions.
Value-based methods learn a value function: the expected return from a state, or from a state-action pair. The Q-learning line learns an action-value function directly, and the agent chooses actions by reference to it. Mnih et al. demonstrated the approach at scale with a convolutional neural network whose "input is raw pixels and whose output is a value function estimating future rewards". They applied the same architecture and learning algorithm to seven Atari 2600 games "with no adjustment", reporting that it "outperforms all previous approaches on six of the games and surpasses a human expert on three of them". The significance was not any single game; it was that one learning procedure handled seven different tasks without per-task engineering.
Policy-gradient methods skip the value function as the primary object and optimise the policy directly. The difficulty is that a single trajectory gives a noisy estimate, and standard methods make one gradient update per data sample, which is wasteful.
What PPO changed
Schulman et al.'s Proximal Policy Optimization addresses that waste directly. The methods "alternate between sampling data through interaction with the environment, and optimizing a surrogate objective function using stochastic gradient ascent", and the paper's key contribution is "a novel objective function that enables multiple epochs of minibatch updates" — where standard policy gradients would perform one update per sample.
The result, the authors report, is a method with "some of the benefits of trust region policy optimization (TRPO), but... much simpler to implement, more general, and have better sample complexity (empirically)". Their experiments covered "simulated robotic locomotion and Atari game playing", where PPO "outperforms other online policy gradient methods".
The engineering significance is that PPO made policy gradients usable in practice, not that it introduced a new theoretical idea. Simplicity and sample efficiency are what let a method spread, and the paper's emphasis on both is why it became a default.
When the reward cannot be written down
Many real tasks are easy to recognise and hard to specify.
Christiano et al. show that goals can be defined by "(non-expert) human preferences between pairs of trajectory segments", and that the approach can solve complex RL tasks "without access to the reward function", including Atari games and simulated robot locomotion. The decisive number is the cost of oversight: feedback on "less than one percent of our agent's interactions with the environment", enough to make the approach "practically applied to state-of-the-art RL systems". They also report training "complex novel behaviors with about an hour of human time".
That result reframes human involvement. In supervised learning, humans label the answer; in preference-based RL, they compare outcomes, which is easier to provide and cheaper to scale. The method does not remove the human — it relocates the human to a comparison task that fits their actual ability.
Bai et al. push the idea further in Constitutional AI: "the only human oversight is provided through a list of rules or principles". The method has a supervised phase of self-critique and revision and a reinforcement phase using "RL from AI Feedback" (RLAIF), producing an assistant that is "harmless but non-evasive" and "engages with harmful queries by explaining its objections to them". The governance consequence is that the principles become the artefact to review — a subject this publication covers in AI Governance.
Why sample efficiency dominates everything
In a simulated game, an agent can take millions of actions. In a real factory, a hospital, or a trading system, each action has a cost and some are irreversible.
That gap is why RL's most visible successes are in simulation, and why a model that learns well in simulation can fail when transferred.
Three properties determine whether RL is feasible in a given domain:
Can you simulate it? A sufficiently accurate simulator turns an impossible data requirement into a computable one — at the price of simulation bias, which is the risk that the agent learns to exploit the simulator rather than the task.
Are mistakes cheap? If a wrong action can harm a person or destroy capital, exploration has to be constrained, which removes much of the method's advantage.
Can you measure the outcome? A reward that is delayed, sparse, or contested makes learning slow and the resulting behaviour hard to evaluate. Preference learning helps when the outcome is recognisable but not specifiable.
A decision sequence
- State the reward in one sentence, and check whether you can actually write it as a function. If not, consider preference-based methods.
- Estimate the interaction budget. How many actions can the system take before the cost is unacceptable?
- Check whether a simulator exists, and how closely it matches the real environment.
- Choose the family deliberately — value-based for discrete action spaces and simpler settings, policy gradients for continuous control and stochastic policies.
- Start with the simplest baseline, including a non-learning heuristic. RL should beat it, not merely run.
- Evaluate on held-out episodes and on the real system separately. Simulation performance is a design signal, not a deployment result.
- Instrument the reward itself. An optimiser will find any loophole in the objective, and the first sign is usually a metric that improves while the task does not.
For the training mechanics behind these systems, see How AI Models Are Trained; for the governance of reward design, see AI Safety.
A worked example
The following is hypothetical and illustrative. Consider a hypothetical team weighing reinforcement learning for a warehouse routing task against a fixed heuristic. Three checks decide feasibility. First, can the task be simulated? Assume a simulator exists but is approximate, so the agent may learn to exploit model quirks - the simulation bias the sources warn about - and simulation results count only as a design signal. Second, are mistakes cheap? Suppose a wrong action can damage goods or delay orders, so exploration must be constrained and the sample-efficiency advantage of methods such as PPO, which Schulman et al. show tolerating multiple update epochs per batch, is spent cautiously rather than freely. Third, can the outcome be measured? If a clean reward is hard to write, assume the team follows Christiano et al. and compares trajectory pairs, using feedback on a small fraction of interactions instead of a hand-written function. The bounded outcome: the policy is trialled in simulation against the heuristic, any gain must survive a separate real-system test, and deployment stops if the reward metric rises while the task does not. No real facility is implied.
A counterargument
A fair objection is that the article treats sample efficiency as a fixed property of a domain when its own sources show the cost of interaction being engineered down. Christiano et al. report usable reward models from feedback on less than one percent of interactions and complex behaviours from about an hour of human time, and Bai et al. reduce oversight to a list of principles revised through self-critique and RLAIF.
On that evidence, the feasibility test is less about how many actions a domain permits and more about whether human feedback can be made cheap enough to fit the budget.
The framing also leans on results that are entirely simulated - Atari and robot locomotion - so extending them to factories and hospitals is an extrapolation the sources do not themselves make.
References & Further Reading
- Richard S. Sutton and Andrew G. Barto, 2018. Reinforcement Learning: An Introduction (2nd edition) — The standard textbook treatment of value functions, policy gradients, and temporal-difference learning.
- Volodymyr Mnih et al., 2015. Playing Atari with Deep Reinforcement Learning — The deep Q-learning result that established value-based deep RL.
- John Schulman et al., 2017. Proximal Policy Optimization Algorithms — The clipped-objective policy-gradient method behind most modern RL training.
- John Schulman et al., 2015. Trust Region Policy Optimization — The trust-region derivation that motivated PPO's clipping.
- Paul F. Christiano et al., 2017. Deep Reinforcement Learning from Human Preferences — The reward-model pipeline behind RLHF.