Reinforcement Learning for Robots

When nobody can write the controller, when the dynamics are too messy to model and the body too complex to hand-tune, the robot learns the behavior itself, from trial, error, and reward.

Everything in Chapters 3 through 6 shared a quiet assumption: that you, the engineer, know how the robot should behave, and your job is just to compute it. Kinematics gave you the hand's position from the joint angles. Control drove the joints to a commanded value. Planning found a path through configuration space. In every case the recipe was hand-designed and the math filled in the numbers. That works beautifully, when you have a model.

Now watch it break. Ask a quadruped to sprint across loose rubble, or a five-fingered hand to reorient a screwdriver by feel. Try to write the controller. What's the equation for "recover when your foot lands on a rock that rolls"? What's the closed-form law for the contact forces between four fingers and a tumbling object? There isn't one you can write down. The dynamics are contact-rich, discontinuous, and higher-dimensional than any human can reason about. The villain of this chapter is exactly that: you cannot hand-code the policy. And there's a second, sneakier villain riding along: the reward that tells you whether you're doing well is sparse and delayed. You only learn you fell over after a hundred good steps. Which of those steps was the mistake? That's the credit-assignment problem, and it's the thing that makes learning from consequences genuinely hard.

The hero this chapter is the value function: a learned estimate of "how good is this situation," which lets us spread a delayed reward backward in time onto the actions that earned it. Everything else is machinery around that one idea.

I'm assuming you've read Chapter 1 (the sense-plan-act loop is about to reappear wearing a new costume) and that you're comfortable with the idea of a parameterized function you can nudge with gradients, a neural net, basically. You don't need Chapters 3–6 in detail, but it helps to remember that they built the classical stack we're now going to reach past.

Here's the road, and every stop is a fix for the one before it. First, the honest question of when to even reach for RL. Then the MDP, the formal box we put the problem in. Then value functions and the Bellman equation, the credit-assignment hero. Then policy vs value methods and the exploration dilemma. Then policy gradients, which optimize behavior directly but drown in noise. Then actor-critic, which fixes the noise with a value function. Then PPO, the workhorse that keeps the whole thing from blowing up. And finally the two tricks, domain randomization and teacher-student distillation, that carry a policy trained in simulation onto a real robot walking on real ice.

Let's go.

Let's be honest up front, because RL is oversold and the honesty is the useful part.

Reinforcement learning is not magic and it is not free. It is sample-hungry: a locomotion policy might need the equivalent of years of robot experience to train, which is only feasible because we run it in simulation at thousands of times real speed. It is finicky about reward: you're not writing the controller anymore, but you are writing the reward function, and a badly-shaped reward produces a robot that games it in ways you didn't imagine (a walking reward that rewards forward velocity, with nothing penalizing thrashing, gives you a robot that lurches forward and faceplants, technically moving forward the whole way). And it is hard to debug: when a learned policy misbehaves you can't step through the logic, because there is no logic, just a matrix of weights.

So here's the honest framing. If you have a good model of the dynamics, classical control usually wins. A robot arm doing pick-and-place on a table has clean, well-understood dynamics; Chapter 4's control and Chapter 6's planning will beat RL on reliability, sample cost, and your sanity. Model-predictive control, solving a little optimization every timestep using a known dynamics model, is extraordinarily good when the model is trustworthy, and it's how Boston Dynamics made Atlas do backflips for years.

RL earns its keep precisely where the model runs out. Contact-rich manipulation, where the physics of fingers-on-object is a nightmare of making and breaking contacts. Legged locomotion over terrain you can't predict, where the "plan a footstep, track it" pipeline shatters the moment a foot slips. Whole-body athletic behavior where the interactions are too coupled to decompose. In those regimes nobody can write the controller, and learning it from trial and error is not a shortcut, it's the only door.

The mature view, and the one this chapter argues for: RL and classical control are not rivals, they're a spectrum, and the best real systems put learned policies exactly where the model is weakest and keep classical methods everywhere the model is strong.

Before we can learn behavior, we need a precise container for what "behavior" even means. That container is the Markov Decision Process, or MDP, the standard mathematical frame for sequential decision-making under uncertainty. It's five pieces, and once you see them you'll notice they're just Chapter 1's sense-plan-act loop written formally.

Here they are:

The word Markov buried in there means one specific thing: the future depends only on the current state, not on the whole history of how you got there. If your state captures everything relevant, position and velocity, say, then knowing it is enough to predict what comes next; the past is redundant. This is a modeling assumption (a "useful lie," in Chapter 1's phrase), and a lot of the art of setting up an RL problem is choosing a state that makes it true enough.

The loop runs like this, forever: the robot observes state $s_t$, picks an action $a_t$ according to its policy (its behavior, the thing we're trying to learn, written $\pi$), the environment transitions to $s_{t+1}$ and hands back reward $r_t$. Sense, plan, act, repeat: exactly Fig 1.1, now with a scorekeeper attached.

So what's the goal? Not "maximize the next reward," that's greedy and shortsighted. The goal is to maximize the return, the total reward accumulated over time, discounted so that sooner is worth more than later:

Reading the symbols:

Why discount at all? Two reasons. Practically, it keeps the sum finite for a task that never ends. Conceptually, it encodes a real preference: a reward now is worth more than the same reward later, because later is uncertain, you might fall over before you get there. A γ near 1 (say 0.99) is a patient, far-sighted robot that plans for the long haul; a γ near 0 is an impulsive robot that grabs whatever's immediately in front of it. For locomotion, 0.99 is typical: you want the robot thinking many steps ahead.

Worked example. Say a robot takes three steps and gets rewards $r_0 = 1$, $r_1 = 1$, $r_2 = 1$ (three good steps), with γ = 0.9. The return from the start is $G = 1 + 0.9\cdot1 + 0.9^2\cdot1 = 1 + 0.9 + 0.81 = 2.71$. Now suppose the same three good steps but then it falls, $r_3 = -10$. The return becomes $2.71 + 0.9^3\cdot(-10) = 2.71 - 7.29 = -4.58$. The fall, three steps in the future, drags the whole return negative, and it should, because those three "good" steps led somewhere bad. Hold that thought. Making the return reflect a future consequence is exactly the credit-assignment magic we're building toward.

Here's the central problem, stated plainly. The reward is sparse and delayed. You get +1 for reaching the goal and nothing along the way. So how does the robot learn that the step three moves before the goal was a good step? It got zero reward for it. Nothing in the immediate signal says "well done."

The fix is to stop asking "what reward did I just get?" and start asking "how good is it to be here?", to attach a number to every situation that summarizes all the reward you can expect from it onward. That number is the value, and it's the single most important object in RL.

There are two flavors, and you need both.

The state-value function, written V(s), is the expected return if you start in state $s$ and follow your policy from then on:

Reading it: $V(s)$ answers "starting here, and behaving the way I currently behave, how much total discounted reward should I expect?" A state one step from the goal has high value. A state teetering on the edge of a cliff has low value, even if the immediate reward there is zero, because of where it leads. That's the key move: value looks past the immediate reward to the whole future the state opens onto.

The action-value function, written Q(s, a), is almost the same but pins down the first move too:

Reading it: $Q(s, a)$ answers "if I do this specific action right now and then behave normally, how good is that?" The difference from V is subtle but decisive: Q evaluates each action, so if you know Q for every action available, you can just pick the best one. That "pick the highest-Q action" idea is the seed of a whole family of methods, coming up next.

Now, the beautiful part, the thing that makes value functions computable at all. Value is self-consistent in a recursive way, and that recursion is the Bellman equation. In words: the value of a state equals the reward you get now, plus the discounted value of wherever you land next.

Reading the symbols: $V(s)$ is the value of where you are, $r$ is the reward you collect on this step, $\gamma$ is the discount, and $V(s')$ is the value of the next state you transition into. (The full equation averages over all the next-states the dynamics might produce and all the actions your policy might take, but this stripped-down version carries the whole intuition.) The value of here is defined in terms of the value of there: value is a chain, each link pinned to the next.

Why does this matter? Because it tells you how the reward propagates backward. The state next to the goal gets its value directly from the goal's reward. The state next to that one gets its value from the first one, discounted. And so on, rippling outward, so that a distant reward eventually colors states many steps away, each a little dimmer, thanks to γ. This is the credit-assignment hero doing its work: a single reward at the goal, through repeated application of the Bellman relation, assigns credit to every state on the path that led there. The delayed reward flows backward in time until every good decision along the way is marked as good.

Let's do it by hand on a tiny gridworld. Picture a 1×4 corridor of cells (call them A, B, C, Goal) where the robot moves right one cell per step. The Goal cell gives reward +10; every other step gives 0. Discount γ = 0.9. What's the value of each cell under a policy that always moves right?

Start at the Goal side and work backward, which is exactly what Bellman lets us do. Cell C is one step from the Goal: $V(C) = 0 + 0.9 \cdot V(\text{Goal})$. Take $V(\text{Goal}) = 10$. So $V(C) = 0.9 \cdot 10 = 9$. Cell B: $V(B) = 0 + 0.9 \cdot V(C) = 0.9 \cdot 9 = 8.1$. Cell A: $V(A) = 0.9 \cdot V(B) = 0.9 \cdot 8.1 = 7.29$. Look at what happened: the +10 at the goal, which the robot doesn't see until the very end, has painted a gradient of value all the way back down the corridor (7.29, 8.1, 9, 10), each cell knowing it's a step closer to the payoff, even though every one of those cells hands out a reward of zero. That gradient is what lets the robot know which way to go before it ever reaches the reward. Sparse reward, solved, by value flowing backward.

Now we have value functions. There are, broadly, two philosophies for turning them into behavior, and the whole field splits along this line.

Value-based methods learn a value function (usually Q(s, a)) and then act by always picking the highest-value action. The policy is implicit: it's just "look up Q for every action here, do the best one." You never represent the behavior directly; you represent how good things are and let that dictate the behavior. The classic here is Q-learning (Watkins, 1989): repeatedly nudge your estimate of $Q(s, a)$ toward $r + \gamma \cdot \max Q(s', a')$, the Bellman equation again, now used as a learning rule. Do that enough and Q converges to the truth, and greedy action on the true Q is optimal. Q-learning powered the famous Atari-from-pixels result (DQN, Mnih et al., 2015).

Policy-based methods skip the value function as the end product and learn the behavior directly: a policy $\pi_\theta(a \mid s)$, a function with parameters $\theta$ (a neural net's weights) that takes a state and outputs an action (or a probability over actions). You then adjust $\theta$ to make good actions more likely. We'll spend the rest of the chapter here, because for robots with continuous actions ("set this joint to 1.37 radians," not "move left or right") policy methods are the natural fit, and they're what actually runs on legged robots.

But both philosophies hit the same wall, the exploration-exploitation dilemma, and it's worth meeting head-on because it's one of the deepest tensions in all of learning-from-experience.

Here's the bind. To get reward, you should exploit: take the action you currently believe is best. But your beliefs are based on limited experience, and the action you think is best might just be the best of the few you've tried. Maybe there's a far better action you've never sampled. To find it you have to explore: try something that looks worse, on the gamble that it's actually better. Explore too little and you lock into a mediocre habit forever, confidently. Explore too much and you never cash in on what you've learned. Every RL algorithm needs an answer to this.

The simplest answer, and a genuinely useful one, is ε-greedy (epsilon-greedy): most of the time (probability 1−ε) do the action you believe is best, but a small fraction of the time (probability ε) do something random. Set ε = 0.1 and the robot exploits 90% of the time and explores 10%. Often you start with ε high (explore a lot early, when you know nothing) and decay it over training (exploit more as your beliefs sharpen). It's crude (random exploration is inefficient in big spaces) but it captures the essential move: you must deliberately do things you don't think are optimal, or you'll never discover that they are.

Let's commit to the policy-based path, because it's where robots live. We have a policy $\pi_\theta(a \mid s)$, a neural net with weights $\theta$ that reads the state and outputs a distribution over actions. We want to find the $\theta$ that maximizes expected return. The question is: which direction should we nudge $\theta$?

The idea is almost embarrassingly intuitive. Run the policy, see what happened, then make the actions that led to high return more likely and the ones that led to low return less likely. If a rollout went well, whatever you did, do more of that. If it went badly, do less. Reinforce success, suppress failure. That's it, that's the whole instinct, and remarkably it can be made mathematically exact.

The exact version is the policy gradient theorem, and its simplest incarnation is the REINFORCE algorithm (Williams, 1992). The update direction for the parameters is:

Let's read every symbol, because this little expression is the beating heart of modern robot learning:

Put the pieces together and the mechanism is gorgeous. For each action you took, you compute "the direction that makes this action more likely" and then scale it by how well the episode went. If $G$ is big and positive, you take a big step toward making that action more likely, reinforcing it. If $G$ is negative, the sign flips and you step away, suppressing it. Actions from good episodes get pushed up, actions from bad episodes get pushed down, all automatically, just by multiplying the "make-it-likelier" direction by the return. The robot bootstraps a good policy out of nothing but "try stuff, keep what worked."

Worked example, kept tiny. A robot in some state can go left or right, currently 50/50. It runs ten episodes. The five where it went right averaged a return of +8; the five where it went left averaged −2. REINFORCE takes the "make-right-more-likely" direction and scales it by +8 (a strong push up), and the "make-left-more-likely" direction scaled by −2 (a push down, because of the negative sign). Net effect: after the update, $\pi_\theta(\text{right} \mid s)$ climbs above 50% and $\pi_\theta(\text{left} \mid s)$ drops. The policy has learned, from raw experience and nothing else, that right is better here. Multiply that across millions of states and you have a walking robot.

So why isn't this the end of the chapter? Because REINFORCE has a fatal flaw: enormous variance. The weight on each update is $G$, the whole-episode return, and whole-episode returns are wildly noisy. An episode might get a great return not because your action was good but because of a lucky bounce ten steps later; another might tank through no fault of the action you're crediting. You're using one very noisy number to judge every action in the episode, indiscriminately: the good actions in a bad episode get punished, the bad actions in a good episode get rewarded. The gradient estimate is correct on average, but any single estimate is so noisy that learning is glacially slow and unstable. You're trying to hear a whisper in a hurricane.

The variance problem has a clean diagnosis: we're judging each action by the entire noisy episode return when we should be judging it by how good that specific action was, right there. We already built the tool for "how good is a situation": the value function. Time to put it to work.

The move is called actor-critic, and it's exactly what it sounds like: two learned pieces working together.

Why does this cut variance? Because the critic provides a baseline, a sense of what return was expected from this state, so the actor can update on the difference between what actually happened and what was expected, rather than on the raw, absolute return. That difference is the star of this section: the advantage.

Reading it: $Q(s, a)$ is how good it is to take action $a$ in state $s$ (then behave normally); $V(s)$ is how good state $s$ is on average, across whatever actions the policy would take. Their difference, $A(s, a)$, answers the exact question we want: "how much better than average was this particular action?" If $A > 0$, the action beat expectations, so reinforce it. If $A < 0$, it underperformed the baseline, so suppress it. If $A \approx 0$, it was about what you'd expect, so leave it mostly alone.

Now re-read the policy-gradient update with $A$ in place of $G$:

Same shape as REINFORCE, one crucial swap: we weight the "make-this-action-likelier" direction by the advantage instead of the raw return. And this is a genuinely better signal. The raw return $G$ swings wildly with luck that has nothing to do with the action; the advantage subtracts off the state's baseline value, cancelling most of that common noise and leaving a much cleaner read on this action's contribution. Same unbiased direction as before, dramatically less variance. (Why is subtracting something legal? First principles: the "make-it-likelier" directions, averaged over every action the policy might pick, sum to exactly zero, because the probabilities have to keep summing to 1. So multiplying them by any number that depends only on the state, like $V(s)$, also averages to zero. Subtracting the baseline changes the noise, never the average direction.) The whisper gets louder because we turned down the hurricane.

Worked example. Your robot is in a state where the critic says the expected return is $V(s) = 5$. It tries an action and things work out to an estimated $Q(s, a) = 7$. The advantage is $A = 7 - 5 = +2$: this action beat the baseline by 2, so nudge it up, but only gently (it was a bit better than average, not a miracle). Contrast plain REINFORCE, which might have weighted that same action by a raw return of $+7$, a much bigger, blunter push that also swept in all the value that was coming anyway just from being in a good state. The advantage strips out the "anyway" and keeps only the credit the action actually earned. That's the whole trick.

One practical wrinkle worth naming, because you'll see it in every real system: we don't have true Q and V, only estimates, and how we estimate the advantage trades bias against variance. The standard tool is GAE, Generalized Advantage Estimation (Schulman et al., 2015), which blends short-horizon estimates (low variance, a bit biased) and long-horizon ones (low bias, higher variance) with a knob (usually written λ) to tune the mix. You don't need the derivation; you need the name and the idea: GAE is the practical recipe for computing the advantage that goes into the update, and it's essentially universal in modern policy-gradient robotics.

Actor–critic gives us a lower-variance gradient. But there's one more villain, and it's the one that historically made policy-gradient RL a nightmare to actually run: a single update that's too big can destroy the policy.

Here's the trap. The gradient tells you a direction to move $\theta$, and it's only trustworthy locally, near your current policy. But if you take a big step in that direction, you can overshoot into a region where the policy is much worse, and (this is the cruel part) because RL is on-policy, all your data was collected by the old policy. Once you've lurched to a bad new policy, the experience you have is no longer about the policy you're now running, so you can't easily recover; you just collected a batch of garbage and stepped off a cliff. Performance collapses, sometimes irreversibly. Old policy-gradient methods were maddeningly sensitive to step size for exactly this reason.

The fix is to limit how far the policy is allowed to move in a single update, to keep each step inside a "trust region" where the gradient's promise still holds. The method that made this practical, robust, and stupidly popular is PPO, Proximal Policy Optimization (Schulman et al., 2017), "proximal" as in stay close: never let one update carry the policy far from the one that gathered the data. PPO is, no exaggeration, the workhorse of robot RL, the default algorithm that trains the overwhelming majority of legged-locomotion and manipulation policies you've read about.

The mechanism is a small, clever tweak to the objective. Define the probability ratio, how much more (or less) likely the new policy is to take the action than the old one was:

A ratio of 1 means the policy hasn't changed on this action; 1.3 means the new policy is 30% more likely to take it; 0.7 means 30% less likely. The plain policy-gradient objective is basically $\text{ratio} \cdot A$: push the ratio up when the advantage is positive, down when it's negative. The danger is that $\text{ratio}$ can run away, the update making a good action wildly more likely in one step, far past where the gradient was trustworthy. Concretely: with ε = 0.2 the band is $[0.8, 1.2]$, so an action the old policy took 10% of the time can rise to at most 12% in this update; a 1.5× leap to 15% gets clipped, and the rest of the improvement has to wait for fresh data collected under the new policy.

PPO's answer is the clipped surrogate objective. In words: let the ratio help you, but only up to a limit; clip it so the policy can't move too far in one step. Formally you take the smaller of the raw objective and a version where the ratio is clipped to a band around 1 (typically $[1-\varepsilon,\ 1+\varepsilon]$ with ε ≈ 0.2):

Here's the intuition, which is the part that matters. Suppose the advantage is positive, a good action, you want to make it more likely. The objective rewards raising the ratio... but only until the ratio hits $1+\varepsilon$. Past that, the clipped term takes over and flattens out: pushing the ratio higher yields no more objective, so the gradient goes to zero and the update stops driving the policy further in that direction. You get the improvement, then the door closes. Now suppose the advantage is negative, a bad action, you want to make it less likely. Symmetrically, the objective lets you lower the ratio until it hits $1-\varepsilon$, then flattens. Either way, the objective is flat outside the trust region, so no single update can shove the policy more than a bounded amount from where it started. The $\min$ is what makes it safe in both directions: it always takes the more conservative (less optimistic) of the two terms, so PPO never rewards moving too far.

That's the whole idea. It's not a hard trust-region constraint like its ancestor TRPO; it's a cheap, soft, clip-based approximation that's good enough and vastly simpler to implement, which is exactly why it took over. Three properties make PPO the default for robots:

We now have a stable algorithm that eats parallel rollouts for breakfast. But rollouts on a real robot are precious and slow: you can't crash a real quadruped a million times, and even if you could, it'd take years and a warehouse of spare parts. So the entire modern approach is: train in simulation, where you can run thousands of robots at thousands of times real speed, then transfer the policy to the real robot. This is sim-to-real, and it's how essentially every impressive legged-RL result was made.

The enabling technology is massively-parallel physics simulation: think Isaac Gym-style GPU simulators that run thousands of robot instances simultaneously on a single graphics card, each experiencing slightly different situations, all pouring experience into one shared PPO update. A locomotion policy that would take years of real-world walking trains in hours this way. (We'll go deep on simulators in Chapter 11; here we just need that they exist and they're fast.)

But there's a villain waiting, and it's the recurring nemesis of this whole subfield: the reality gap. The simulator is a model of physics, and like every model it's wrong in a thousand small ways: the friction is a little off, the motors have unmodeled delay, the robot's true mass differs from the CAD file, the ground has give the sim doesn't capture. A policy that's perfect in sim can walk beautifully in the simulator and then fall flat on its face in reality, because it learned to exploit quirks of the fake physics that don't exist in the real world. This is exactly Chapter 1's Villain #2 (every measurement lies, every model is a little wrong) scaled up to an entire world.

The key trick that closes the gap is domain randomization (Tobin et al., 2017). The idea is almost paradoxical and completely brilliant: instead of trying to make the simulator exactly match reality, deliberately make it vary, randomizing the physics parameters on every episode, so the policy is forced to work across a huge range of physics, and the real world just becomes one more variation it already handles. Randomize the friction between feet and ground. Randomize each link's mass. Randomize motor strength, sensor noise, control latency, terrain roughness. Every episode, roll new dice. A policy that survives all of that can't have overfit to any one fake physics: it had to learn something robust enough to cover the whole range, and reality lands somewhere inside that range.

Think of it as the difference between studying for one specific exam versus studying so broadly that any exam is easy. Overfit to a single simulation and you're memorizing one exam; randomize aggressively and you're building genuine competence that transfers. The reality gap doesn't get closed so much as swallowed: reality becomes just another sample from the distribution the policy already masters. (Domain randomization gets its full treatment in Chapter 11; here it's the punchline of sim-to-real.)

Domain randomization makes a policy robust, but it exposes a second, subtler problem, and the solution to it is one of the most elegant ideas in modern robotics.

Here's the tension. In simulation you have access to the ground truth of everything: the exact friction under each foot, the precise height of the terrain around the robot, whether each foot is really in contact, the true mass of the payload. This is called privileged information: the simulator knows it perfectly, and it would make learning so much easier if the policy could use it. But the real robot can't measure most of it. It doesn't know the true friction of the ice it's stepping onto; it can't see the terrain height buried under its own body; it only has its onboard sensors: joint encoders, an IMU, maybe a camera. So a policy that leans on privileged information trains beautifully in sim and is useless in reality, because at deployment those inputs simply aren't there.

The fix is teacher-student learning (also called privileged learning), and it's a two-stage trick:

The student ends up with the teacher's robust behavior but running entirely on real-robot-available inputs. It's the deployable one. The teacher's job was never to be deployed; it was to make a hard learning problem easy, so that a good policy exists to imitate. You cheat during training precisely so you don't have to cheat at deployment.

This isn't a toy idea: it's how robust legged locomotion actually got solved. ETH Zürich's work on the ANYmal quadruped (Lee et al., 2020, Science Robotics) used exactly this teacher-student, privileged-learning recipe to walk over rough, unseen terrain, and the follow-up (Miki et al., 2022) fused proprioception with exteroception to cross moss, mud, snow, and rubble at speed. The results are striking for a reason that falls straight out of the method: robustness to disturbances you never explicitly trained for. A robot trained this way, across randomized physics, distilled to infer hidden terrain from its own sensor history, will carry an unknown payload it's never felt, absorb a kick it never saw coming, and stay upright on ice it was never told about, because from its perspective those are all just more variation of the kind it learned to handle. The competence generalizes because the training demanded generalization.

Worth situating the flagship platforms here. Boston Dynamics' Spot and Atlas are the most famous legged robots on Earth, and historically their jaw-dropping agility came from model-predictive control, the classical "know the dynamics, solve an optimization every timestep" approach from the top of this chapter, engineered to an extraordinary degree. That's the honest counterweight to the RL hype: hand-crafted model-based control, done superbly, produced backflipping robots years before RL locomotion matured. But the field is converging: Boston Dynamics and others are increasingly adopting learned and RL-based policies for the most athletic, contact-rich, hard-to-model behaviors, exactly where this chapter predicted the model would run out. The future isn't RL replacing control; it's each doing what it's best at, in one stack.

Step back and look at the arc. Chapters 3 through 6 built behavior by hand, with math, and that's still the right move whenever you have a good model. This chapter was about what to do when you don't: when the dynamics are too contact-rich, too discontinuous, too high-dimensional for any human to write the controller. The answer was to learn the policy from trial, error, and reward.

The chain of fixes, each answering the last one's flaw: we framed the problem as an MDP (states, actions, reward, dynamics, discount) whose goal is to maximize the discounted return $G = \sum \gamma^t r_t$. To handle sparse, delayed reward we built value functions ($V(s)$ and $Q(s, a)$) and the Bellman equation that lets a distant reward flow backward in time to credit the actions that earned it, our credit-assignment hero. We chose the policy-gradient path for continuous-control robots and hit its fatal variance flaw, fixed it with actor-critic and the advantage $A = Q - V$ (estimated in practice by GAE), then fixed policy-gradient's other flaw, catastrophic overshooting updates, with PPO's clipped objective, the stable, parallelizable workhorse that trains thousands of simulated robots at once. Finally we crossed from sim to reality: domain randomization swallows the reality gap by training across a distribution of physics, and teacher-student distillation lets a privileged-information teacher train a sensor-only student that inherits robust, deployable behavior, the recipe behind ANYmal's all-terrain walking and, increasingly, Spot and Atlas's athletic feats.

Be clear-eyed about the costs, because the honesty is the point: RL is sample-hungry, reward design is genuinely hard (a badly-shaped reward gives you a robot that games it), and a learned policy is hard to debug because there's no logic to step through, just weights. Reach for it when the model runs out, not before.

Where does this go next? Notice the quiet assumption we made all chapter: that the robot has a clean state to reason about, joint angles, terrain height, contact flags. But out in the world, that state has to come from raw pixels and point clouds, and turning an image into "there is a mug 40 cm ahead, slightly rotated" is its own enormous problem. That's Chapter 8, perception (vision and vision-language models), the sense beat of Chapter 1's loop, done with modern learning. Master perception and control-by-learning, and Chapter 9 fuses them into the vision-language-action models steering today's robots. Onward.