Interlude: Flow Matching & Diffusion Models, From Scratch

Chapter 8 waved at diffusion and flow matching. Chapter 9 leaned on them: π0 generates its action chunks with a flow head, and half the policies in Chapter 10 will too. This interlude stops waving and actually builds the machinery, from a single ordinary differential equation up to classifier-free guidance, following the MIT course notes by Holderrieth and Erives (2026).

Here is the one-sentence version of this whole chapter: modern generative models turn noise into data by following a velocity field, and training them is just regression on that field. Everything else is detail. But the details are exactly what you need when Chapter 9's π0 says "flow matching action expert" or Chapter 10's Diffusion Policy says "denoising", so let's do this properly.

The plan, from the ground up. First we agree on what "generate" even means, and the answer will be: sample from a probability distribution. Then we meet the machine that does the sampling, an ordinary differential equation (ODE) whose velocity field a neural network will learn, and its noisier sibling, the stochastic differential equation (SDE). Then the real question: what should the network learn? That leads to probability paths (a schedule for morphing noise into data), a beautiful dead end (the loss we want is uncomputable), and the escape hatch that makes the whole field work, the conditional flow matching loss. After that we re-derive everything through a second lens, score functions, which is where diffusion models and their denoising intuition live, and we pick up guidance, the trick that makes a model obey a prompt.

Then, because "a neural network supplies the velocity field" is a sentence hiding a lot of decisions, we open the box: the architectures that host the field (and the conditioning trick that robots reuse), the latent space trick that makes megapixel generation affordable at all, and how Stable Diffusion 3 and Meta's Movie Gen are assembled from exactly these parts. We take one detour into the discrete version of the whole story, because that is secretly where Chapter 9's FAST action tokens live. And we finish where a robotics guide should: taking a real flow-matching VLA apart, and looking at the one thing a robot's generative model does that an image generator never has to, which is tell you honestly that it knows what to do right now and has no idea what it will be doing in two seconds.

One promise before we start: nothing in this chapter is specific to images. The "objects" we generate are just vectors. For Stable Diffusion the vector is an image. For π0 the vector is a chunk of fifty robot actions. Same math, same training loop, different meaning attached to the numbers.

I am assuming you have read Chapter 8 (you know what a neural network and a training loss are, and you have seen the words diffusion and flow matching used in anger). Everything else we build here.

Start at the highest level: what does it mean for a model to "generate" something?

Think about asking for a picture of a dog. There is no single correct answer. There are billions of acceptable dog pictures, some more dog-like than others. Machine learning formalizes this spread of acceptable answers as a probability distribution over objects, written $p_{\text{data}}$. An object $z$ that looks very dog-like gets high probability; a smear of static gets essentially zero. "Generate a dog picture" then becomes a precise request: draw a sample $z \sim p_{\text{data}}$. That is the entire definition of generative modeling. A generative model is an algorithm that returns samples from $p_{\text{data}}$, or at least a good approximation of it.

Two details matter for everything downstream. First, we never actually know $p_{\text{data}}$. We only have a dataset: a finite pile of examples $z_1, \dots, z_N$ drawn from it. Every training recipe in this chapter consumes only those samples. Second, objects are vectors. An image is a vector of pixel values. A protein structure is a vector of atom coordinates. A robot action chunk is a vector of joint targets stacked over time, which is exactly how Chapters 9 and 10 use all of this. So from here on, $z \in \mathbb{R}^d$, and you can mentally substitute "50 future actions" wherever you like.

Often we also want to generate conditioned on something: an image given a caption, an action chunk given a camera view and an instruction. That is sampling from $p_{\text{data}}(z \mid y)$, where $y$ is the prompt. Hold that thought; it is the guidance section at the end. For now we build the unconditional machine, because conditioning turns out to be a small modification of it.

High-level idea first. Imagine space filled with little arrows, and imagine dropping a leaf into that space. The leaf drifts wherever the local arrow points, moment after moment, tracing out a smooth path. If someone arranged the arrows cleverly, every leaf you drop, no matter where, ends its journey sitting on your data distribution. That arrow arrangement is the entire model. Generation is: drop a leaf at random, let it drift, collect it at the end.

Now the same idea with its real names. The arrows are a vector field $u_t(x)$: a function that takes a location $x$ and a time $t$ between 0 and 1 and returns a velocity, which way to move and how fast. The leaf's journey is a trajectory $X_t$, and "drift wherever the arrow points" is an ordinary differential equation:

Reading the symbols: $\frac{\mathrm{d}}{\mathrm{d}t}X_t$ is the velocity of the point at time $t$, and the equation says that velocity must equal what the field $u_t$ prescribes at the point's current location. $X_0 = x_0$ just says where the leaf was dropped. Under mild conditions (which neural networks always satisfy) this equation has exactly one solution per starting point: drop the leaf in the same spot, get the same path, every time. The function that maps a starting point to its position at time $t$ is called the flow, which is where "flow model" gets its name.

How do you actually compute the path? You almost never solve the ODE with pen and paper. You simulate it with the Euler method, the most honest algorithm in numerical analysis: chop time into $n$ small steps of size $h = 1/n$ and repeatedly take a small step in the arrow's direction,

That is it. Look at the arrow where you are, walk a tiny bit that way, look again. Small steps track the true path closely; big steps cut corners and drift off. Fig 9.5.1 lets you feel this: set the step count to 4 and watch particles land sloppily, then set it to 96 and watch them snap onto the targets.

And where does learning enter? A flow model makes one substitution: the vector field is a neural network, $u_t^\theta(x)$, with parameters $\theta$. Generation then has a fixed recipe. Draw a random starting point $X_0 \sim p_{\text{init}}$, where $p_{\text{init}}$ is a distribution chosen to be trivially easy to sample, almost always a standard Gaussian $\mathcal{N}(0, I_d)$ (this randomness is what makes an otherwise deterministic ODE produce different samples each run). Then simulate the ODE from $t = 0$ to $t = 1$ with Euler steps, and return $X_1$. The training goal, which we have not yet earned, is to pick $\theta$ so that $X_1 \sim p_{\text{data}}$: noise in, data out.

(Side note: the network parameterizes the vector field, not the flow itself. To get a sample you must actually run the simulation, step by step. This is why sampling speed is counted in "number of function evaluations" and why Chapter 9 cared that flow-based action heads need only a handful of integration steps.)

Big picture: a flow model is a leaf drifting in a perfectly still pond. A diffusion model is the same leaf in a pond with wind. The drift is still there, pushing the leaf where the arrows point, but every instant also adds a random shove. Surprisingly, this randomness is not a defect. It gives the model different failure modes and, as we will see later, you can add it or remove it after training, for free.

To make "random shove" precise we need the fundamental noisy process, Brownian motion $W_t$. Think of it as a continuous random walk: it starts at zero, its increments over disjoint time intervals are independent, and each increment is Gaussian with variance proportional to elapsed time, $W_t - W_s \sim \mathcal{N}(0, (t - s) I_d)$. You can simulate it with one line: at each step of size $h$, add $\sqrt{h}$ times a fresh standard Gaussian. The $\sqrt{h}$ is the signature of diffusion; variances add linearly in time, so standard deviations grow like the square root.

A stochastic differential equation (SDE) is an ODE with a Brownian term stapled on. In the standard notation,

Reading the symbols: the first term is the same old drift, "walk where the arrow points". The second term says "then get shoved by noise", with the diffusion coefficient $\sigma_t$ setting the shove's strength at time $t$. Set $\sigma_t = 0$ and you recover the ODE exactly, so flows are a special case of diffusions. The simulation recipe, called Euler-Maruyama, is Euler plus the shove: $X_{t+h} = X_t + h\,u_t(X_t) + \sigma_t \sqrt{h}\,\epsilon_t$ with $\epsilon_t \sim \mathcal{N}(0, I_d)$ drawn fresh each step.

A worked miniature you can check by eye in Fig 9.5.2: take drift $u(x) = -\theta x$, which always points back toward zero, plus noise $\sigma$. This is the Ornstein-Uhlenbeck process. With $\sigma = 0$ every trajectory decays smoothly to zero, a pure flow. With $\sigma > 0$ trajectories jitter around zero forever and settle into a permanent cloud whose width is exactly $\sigma / \sqrt{2\theta}$. The figure measures the spread of its twelve trajectories and prints it next to that formula, so you can watch a theorem come true.

A diffusion model, then, is the generative recipe with the network in the drift seat: draw $X_0 \sim p_{\text{init}}$, simulate $\mathrm{d}X_t = u_t^\theta(X_t)\mathrm{d}t + \sigma_t \mathrm{d}W_t$ to $t = 1$, return $X_1$. The open question is unchanged, and it is the important one: what should $u_t^\theta$ be trained to equal? Neither the ODE nor the SDE picture answers that. The next two sections do.

Step back and look at what we are asking the model to do: start at noise ($t = 0$) and end at data ($t = 1$). We have said nothing about the middle. The insight that unlocks training is to choose the middle ourselves. We will write down, explicitly, a family of distributions $p_t$ that starts at $p_{\text{init}}$, ends at $p_{\text{data}}$, and morphs smoothly in between. That family is called a probability path, and once we have one, "train the model" becomes "make the model's trajectories march along this path", a concrete target instead of a vague wish.

The construction happens in two moves, and the first one looks weird until it doesn't. Move one: solve the problem for a single data point. Fix one example $z$ from the dataset and define the conditional probability path $p_t(\cdot \mid z)$: a morph that starts at noise and ends collapsed onto that single point $z$. The standard choice, used by essentially every large model, is the Gaussian probability path:

Reading the symbols: $\alpha_t$ and $\beta_t$ are the noise schedulers, two smooth increasing/decreasing dials with $\alpha_0 = 0, \beta_0 = 1$ (at $t = 0$ you have pure noise) and $\alpha_1 = 1, \beta_1 = 0$ (at $t = 1$ you have exactly $z$). Sampling from this path is one line: draw noise $\epsilon \sim \mathcal{N}(0, I_d)$ and output $x = \alpha_t z + \beta_t \epsilon$, a weighted blend of the data point and the noise. The simplest scheduler, $\alpha_t = t$ and $\beta_t = 1 - t$, is called the CondOT path, and it is the one π0 uses: $x = t z + (1-t)\epsilon$, literally a straight line from noise to data. Every figure in this chapter runs on it.

Move two: average over the dataset. The marginal probability path is what you get when the data point itself is random: first draw $z \sim p_{\text{data}}$, then sample from $p_t(\cdot \mid z)$. At $t = 0$ this is still pure noise; at $t = 1$ it is the full data distribution, because "pick a data point, then collapse onto it" is exactly sampling from the data. The marginal path is the thing we actually want our model to follow. Notice the asymmetry that will drive the whole next section: the conditional path is simple and fully known, while the marginal path is easy to sample from but impossible to write down (its density is an integral over the entire dataset).

Fig 9.5.3 shows both, side by side, computed from the same formula: the left cloud collapses onto one chosen point, the right cloud resolves into the whole dataset, and the $\alpha_t, \beta_t$ dials tick in the readout as you scrub $t$.

Now the payoff section. High level: we know where trajectories should be at every time (the probability path). Training should therefore be "make the network's velocity field push points along that path". That is a regression problem. The only obstacle is that the regression target seems uncomputable, and the escape from that obstacle is the single most important trick in this chapter.

First, the target. For the Gaussian conditional path there is an explicit velocity field that carries noise onto the single point $z$; you can derive it with two lines of calculus, and for the CondOT schedule it is wonderfully simple. If $x$ sits somewhere at time $t$, the velocity that keeps it on the straight noise-to-$z$ line is

Reading it: aim directly at $z$, with speed scaled so you arrive exactly at $t = 1$. This is the conditional vector field, and it is fully known, no learning required. Of course, on its own it is useless: it regenerates the data point you already had.

The magic is the marginalization trick (Theorem 9 in the notes): if you average the conditional fields of every data point, weighting each by the posterior probability that $x$ came from that point (a Bayes' rule weighting, "given that I am at $x$ at time $t$, which $z$ was I probably headed to?"), you get the marginal vector field, and its ODE provably transports $p_{\text{init}}$ into $p_{\text{data}}$ along the marginal path. (The proof runs through the continuity equation, the same conservation law that governs fluid flow: probability mass is conserved, so the rate of change of density equals the negative divergence of the probability flux. It is three lines of algebra once you have that language, and the notes spell it out.) The figures in this chapter can draw exact fields precisely because, for a small point dataset, this posterior average is directly computable.

So the ideal training loss writes itself: the flow matching loss. Sample a time and a point along the marginal path, and regress the network onto the marginal field,

And here is the dead end: for a real dataset, $u_t^{\text{target}}(x)$ is that posterior-weighted average over the entire dataset, recomputed for every training point. Uncomputable at scale. It looks like the whole plan collapses.

The escape (Theorem 12 in the notes, and the reason the field exists) is to regress on the conditional field instead, the one we can write in closed form. Define the conditional flow matching loss:

The theorem says: $\mathcal{L}_{\mathrm{FM}}$ and $\mathcal{L}_{\mathrm{CFM}}$ differ by a constant that does not depend on $\theta$, so their gradients are identical. Training on the cheap conditional loss is exactly training on the impossible marginal loss. The network, forced to output one velocity per location while being fed many conflicting conditional targets there, settles on their posterior-weighted average, which is precisely the marginal field. Nobody tells it to average; least squares does that automatically.

Written out for the CondOT path, the whole training loop is embarrassingly small. Sample a data point $z$, noise $\epsilon$, and a time $t$. Form the blend $x = t z + (1 - t)\epsilon$. Compute the loss $\| u_t^\theta(x) - (z - \epsilon) \|^2$ and take a gradient step. That is the entire algorithm behind Stable Diffusion 3, Meta's Movie Gen, and π0's action expert. The target velocity is just $z - \epsilon$, the straight-line direction from the noise you drew to the data you drew.

A worked number, one dimension, so nothing hides: say the data point is $z = 4$ and the noise draw is $\epsilon = -2$. Then the training pair at $t = 0.5$ is the input $x = 0.5 \cdot 4 + 0.5 \cdot (-2) = 1$, with regression target $z - \epsilon = 6$. And a sanity check shows why 6 is right: a point traveling at constant velocity 6 starting from $-2$ covers distance 6 in unit time and lands at $4 = z$, exactly on the data. One Euler step of size 1 gets there in a single jump because the path is straight.

Notice what is absent from that loop: simulation. Training never integrates the ODE, never runs the sampler, never generates anything. People call this simulation-free training, and it is why the method scales: each training step costs one forward pass, like ordinary supervised learning. Fig 9.5.4 makes all of this concrete by actually doing it in your browser: a 150-parameter model, real SGD on the real conditional flow matching loss, and you can watch the loss fall, the field organize itself, and samples start landing on the data.

Everything so far trained a velocity field and sampled with a deterministic ODE. But the papers that started this field (and Chapter 10's Diffusion Policy) speak a different dialect: noise prediction, denoising, score functions, stochastic samplers. High-level reassurance first: it is the same machinery in different clothes. This section is the translation guide, plus one genuinely new capability.

The central object of the diffusion dialect is the score function, $\nabla \log p_t(x)$: the gradient of the log-density of the probability path. Intuition: at every point, the score is an arrow pointing "uphill" toward higher probability. Noised-up samples sit downhill; the score tells you which way clean data lies. For the Gaussian conditional path the score has a tiny closed form, $\nabla \log p_t(x \mid z) = -(x - \alpha_t z)/\beta_t^2$, which points straight from $x$ back toward (the scaled) $z$. And the marginal score obeys the same posterior-averaging rule as the marginal velocity field, so everything we just learned transfers.

Train it the same way too: denoising score matching regresses a network $s_t^\theta(x)$ onto the conditional score, and the identical constant-difference theorem holds, so the cheap conditional loss has the same gradients as the impossible marginal one. Plug in the Gaussian formula and the target becomes $-\epsilon / \beta_t$: the network is literally being trained to point away from the noise that was added, which is why people say diffusion models "learn to denoise". The numerically stable variant drops the $1/\beta_t$ and predicts the noise itself, $\epsilon_t^\theta(x) \approx \epsilon$; that choice, made in the DDPM paper (Ho et al., 2020), is the "noise prediction network" you will see in every diffusion codebase, including robot ones.

Here is the punchline that unifies the dialects. For Gaussian paths, the marginal velocity field and the marginal score are two linear functions of the same underlying quantity (the posterior mean of $z$ given $x$), so each is recoverable from the other with a one-line formula (Proposition 1 in the notes). Learn a flow, get the score for free. Learn a denoiser, get the flow for free. Velocity-prediction, noise-prediction, score-prediction, denoiser-prediction: four parameterizations of one object.

And the new capability the score buys: the SDE extension trick (Theorem 17). Take the trained ODE and add any amount of noise, compensated by a score-following correction term,

and the marginal distributions $p_t$ are unchanged, for any noise level $\sigma_t \geq 0$ you pick after training. Reading the correction term: the injected noise constantly kicks samples downhill off the path, and the extra $\frac{\sigma_t^2}{2}\nabla\log p_t$ drift constantly herds them back uphill, in exact balance. (Set the path to be constant in time and this becomes Langevin dynamics, the classic sampler that MCMC and molecular dynamics run on. Diffusion sampling is Langevin dynamics sliding along a moving target.)

So "flow model versus diffusion model" is not a training decision at all; it is a sampling decision, one dial you can turn at inference time. In theory every $\sigma_t$ gives the same distribution; in practice the model is imperfect, and noise can help shake samples off errors (at the price of needing more, smaller steps). Fig 9.5.5 draws both samplers from the same exact field in 3D, space horizontal and time rising upward: the ODE paths are smooth ribbons, the SDE paths are jagged threads, and both funnels land on the same data.

Everything so far generates something from the data distribution. Almost always you want something specific: an image matching a caption, an action chunk for the current camera view and instruction. High level: conditioning is easy to bolt on, and then there is one strange, load-bearing trick, classifier-free guidance, that makes conditioning actually bite.

The easy part is vanilla guidance: hand the prompt $y$ to the network as an extra input, $u_t^\theta(x \mid y)$, and train exactly as before with (data, prompt) pairs sampled together. Nothing else changes; the conditional flow matching loss just carries $y$ along. For a robot policy, $y$ is the observation and the language command, which is precisely how the VLA heads of Chapter 9 consume it.

The strange part: models trained this way empirically under-obey the prompt. The fix comes from a Bayes' rule decomposition of the guided score: $\nabla \log p_t(x \mid y) = \nabla \log p_t(x) + \nabla \log p_t(y \mid x)$, an unconditional term plus a "does $x$ match the prompt" term. If the model under-weights the prompt, amplify that second term with a guidance scale $w > 1$. Early systems trained a separate noised-data classifier to supply $\nabla \log p_t(y \mid x)$ (classifier guidance); the modern move eliminates it with algebra. Substituting the decomposition back into itself gives classifier-free guidance (Ho & Salimans, 2022):

Reading the symbols: $u_t(x \mid y)$ is the prompted field, $u_t(x \mid \varnothing)$ is the unprompted field ($\varnothing$ means "no prompt"), and $w$ linearly extrapolates past the prompted field, away from the unprompted one. At $w = 1$ this is exactly vanilla conditioning. At $w > 1$ it exaggerates whatever the prompt contributes. And you do not need two networks: during training, randomly replace the prompt with $\varnothing$ some fraction of the time (10 percent is typical), so one network learns both fields. At sampling time, run it twice per step (once with the prompt, once without) and blend.

Be clear about what CFG is: a heuristic. For $w > 1$ the samples no longer follow the true conditional distribution; adherence to the prompt goes up while diversity goes down, and pushed hard it collapses variety entirely. Essentially every image you have seen from a modern generator used $w \geq 4$, cheerfully trading distributional correctness for "does what I asked". Fig 9.5.6 puts the tradeoff on instruments: crank $w$ and watch the measured adherence climb toward 100 percent while the measured diversity of the sample cloud shrinks.

Up to here the neural network has been a black box that we simply call $u_t^\theta(x \mid y)$. Time to open it, because two of the design choices inside turn out to be exactly the choices a robot policy makes.

Start by counting inputs. The network takes three things: the current noisy point $x$, the time $t$, and the conditioning $y$. It returns one thing: a velocity of the same shape as $x$. For the toy datasets in this chapter's figures, a plain multi-layer perceptron with all three concatenated is genuinely enough. For images, video, or robot actions, it is not, and the interesting question becomes how the time and the prompt get inside.

Embedding the time. You could feed the raw scalar $t$ into the network. In practice almost nobody does, because one number among thousands of activations is a whisper the network struggles to hear, and the velocity field's dependence on $t$ is sharp (remember the $1/(1-t)$ in the CondOT field). The standard fix is Fourier features: expand $t$ into a bank of sines and cosines at geometrically spaced frequencies,

Reading the symbols: $d$ is the embedding width, and the frequencies $w_i$ are spaced geometrically from $w_{\min}$ to $w_{\max}$, so low frequencies carry the coarse "where are we in the schedule" signal and high frequencies resolve fine differences. The $\sqrt{2/d}$ out front makes the embedding unit-norm, since $\sin^2 + \cos^2 = 1$, which is a free sanity check you can watch in Fig 9.5.7. If this smells like the positional encodings from Chapter 8's transformer section, that is because it is the same trick aimed at a different axis: there it told the model where in the sequence a token sat, here it tells the model where in the noise-to-data schedule it is.

Embedding the prompt. For a class label, learn one embedding vector per class and be done. For text, lean on a frozen pretrained encoder: CLIP (Chapter 8's contrastive model) gives one coarse vector for the whole caption, while a text transformer such as T5-XXL gives a sequence of embeddings the model can attend to piecewise. Large systems commonly concatenate several such encoders to get both the gist and the granularity.

The two architectures. Whatever supplies the field must map an input to an output of the same shape, which rules out the funnel-shaped classifier stacks you may be used to. Two designs dominate. The U-Net is a convolutional encoder that shrinks spatial size while growing channels, a processing block at the bottleneck, and a decoder that expands back up, with residual connections bridging matching levels; its output is image-shaped by construction. The diffusion transformer (DiT) is Chapter 8's ViT pointed at generation: chop the input into patches, embed them as tokens, run attention layers, then "depatchify" back to the original shape.

And now the choice that matters for robots. Inside a DiT block, how do $t$ and $y$ actually reach the computation? The lazy answer is to concatenate them to the input once at the front, which means the signal must survive being passed down through every layer, and it fades. The standard answer is adaptive normalization (AdaLN): push the time embedding through a small MLP to produce a per-channel scale and shift, $(\gamma, \beta) = g(\tilde{t})$, and use them to modulate the normalized activations inside every block:

Reading it: $\mathrm{Norm}$ is the usual layer normalization, $\odot$ is elementwise multiplication, and $\gamma, \beta$ are computed fresh from the timestep. Because they are injected at every block, the conditioning never has to survive a long journey. It is re-stated at every layer, so the deepest layer knows what time it is just as clearly as the first. Prompts typically enter alongside via cross-attention.

Hold onto AdaLN. When we get to π0.5 in a few sections, we will find it making exactly this choice under a slightly different name, and for exactly this reason. Fig 9.5.7 lets you watch the difference: flip to concatenation and the conditioning signal visibly decays through the stack while the readout counts it down, flip to adaptive normalization and it arrives at full strength everywhere.

Here is a problem you hit immediately at real resolutions. A $1024 \times 1024$ RGB image is a vector of about 3.1 million numbers. Our flow model's output has to be the same size as its input, so there is no funneling down to a small head like a classifier does. Simulating an ODE in three million dimensions, dozens of steps per sample, is not a thing you want to pay for.

The fix is the observation that real images do not fill that space. They cluster near a much lower-dimensional manifold inside it. So: compress first, generate in the small space, decompress at the end. Train an autoencoder, an encoder $\mu_\phi$ that maps an image to a small latent $z$ and a decoder $\mu_\theta$ that maps it back, with a reconstruction loss pulling $\mu_\theta(\mu_\phi(x))$ toward $x$. Downsample by a factor of 16 with 4 latent channels and those 3.1 million numbers become about 16,000, a compression of roughly 192 times. Now the ODE is affordable.

But a plain autoencoder has a subtle failure. Nothing constrains what the latent distribution looks like. You may have compressed the data and simultaneously turned it into a jagged, disconnected mess that is harder to model than what you started with. You solved the size problem and created a shape problem.

The variational autoencoder (VAE) fixes the shape by making the encoder probabilistic and regularizing it toward a distribution we like. The encoder outputs a Gaussian, $q_\phi(z \mid x) = \mathcal{N}(\mu_\phi(x), \sigma_\phi^2(x))$, and we add a penalty pulling that Gaussian toward the standard prior $\mathcal{N}(0, I)$, measured with the Kullback-Leibler divergence. The full objective is

Reading the symbols: the first term says latents must decode back into the original image, and the second says the cloud of latents must look like a standard Gaussian. $\beta$ sets the exchange rate between the two. They genuinely fight: the reconstruction term wants latents spread out and distinctive, the KL term wants them piled into a tidy Gaussian ball. Push $\beta$ too high and you get posterior collapse, where the encoder gives up and outputs the prior no matter the input, so reconstructions turn to mush; a common remedy is KL warm-up, starting $\beta$ at zero and easing it in. In modern image autoencoders $\beta$ is kept very small. For diagonal Gaussians the KL has a closed form worth seeing, because it makes the "be like a Gaussian" pressure concrete: it penalizes the mean for being away from zero and the variance for being away from one.

One mechanical wrinkle: the loss takes an expectation over $z \sim q_\phi(z \mid x)$, a distribution that depends on the parameters we are differentiating, so you cannot naively backpropagate through the sampling. The reparameterization trick sidesteps it by moving the randomness outside: instead of sampling $z$ directly, draw $\epsilon \sim \mathcal{N}(0, I)$ and set $z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon$. Now the only random quantity is $\epsilon$, whose distribution has nothing to do with $\phi$, and gradients flow straight through $\mu_\phi$ and $\sigma_\phi$.

Put it together and you have latent diffusion, the paradigm essentially every state-of-the-art image and video generator uses today: train the autoencoder first, then train the flow or diffusion model entirely inside its latent space, and at sampling time decode the result with the decoder's mean (not a random draw, to avoid injecting noise into the finished product). The intuition for why this works so well is that a good autoencoder strips out high-frequency detail that is perceptually irrelevant, letting the generative model spend its capacity on structure that matters. The catch, and it is a real one, is that your generator is now capped by your autoencoder: if the encoder throws something away, no amount of flow matching will bring it back.

Worth pausing to notice how little new machinery the frontier systems need. Two case studies from the notes, both of which you can now read end to end.

Stable Diffusion 3. It trains with the same conditional flow matching objective we derived, on a Gaussian probability path, in the latent space of a pretrained autoencoder, with classifier-free guidance trained by randomly dropping the label. Its own paper reports that the team extensively compared flow and diffusion variants and found flow matching best. The engineering on top: three different text embeddings (two CLIP variants plus the sequential output of Google's T5-XXL encoder), and a multi-modal DiT (MM-DiT) that extends the transformer to attend not just over image patches but over text tokens as well, so conditioning is granular rather than one squashed vector. Largest model: 8 billion parameters. Sampling: 50 Euler steps with a guidance weight between 2.0 and 5.0.

Meta's Movie Gen Video. Videos live in $\mathbb{R}^{T \times C \times H \times W}$, with a new time axis, and almost every design choice is the image recipe adapted to that extra dimension. Same conditional flow matching objective, same straight-line CondOT schedulers $\alpha_t = t$, $\beta_t = 1 - t$. The autoencoder becomes a temporal autoencoder (TAE) compressing by a factor of 8 in time and height and width, because memory pressure is far worse for video (which is exactly why generated clips are short); long videos are handled by chopping, encoding pieces separately, and stitching the latents. The backbone is a DiT that patchifies across both space and time, with self-attention among patches and cross-attention to language. Three text embeddings again, each for a reason: UL2 for granular text reasoning, ByT5 for character-level details when a prompt demands specific letters appear, and MetaCLIP for shared text-image space. Largest model: 30 billion parameters.

The lesson to carry: the math stops changing and the engineering takes over. Everything above is the conditional flow matching loss plus choices about what space to run in, what network hosts the field, and how the prompt gets in. When you meet the next 40-billion-parameter generator, those are the four questions to ask, and you now know all four.

Everything so far assumed the object being generated is a point in $\mathbb{R}^d$ that we can nudge with a velocity. Text is not. A token is a symbol drawn from a vocabulary, and there is no such thing as moving 0.3 of the way from "cat" to "dog". You cannot write an SDE on a discrete set, so the machinery seems to die at the border.

It does not, and the way it survives is worth seeing, both because language models built this way are a live research direction and because Chapter 9's discrete-token VLAs are secretly in this dialect. The replacement for a differential equation is a continuous-time Markov chain (CTMC): a process that sits in a state and occasionally jumps to another. Instead of a velocity field you specify a rate matrix $Q_t(y \mid x)$, the instantaneous rate of jumping from state $x$ to state $y$. It obeys two obvious rules: off-diagonal rates are non-negative (a negative rate of jumping somewhere is meaningless), and each diagonal entry is minus the sum of its row, which just says you either stay or you leave and the books must balance. Simulation is Euler's method again, in probability rather than position: over a small step $h$, jump to $y$ with probability $h\,Q_t(y \mid x)$, otherwise stay.

Now the problem that nearly kills it. The state space is $S = \mathcal{V}^d$, every possible sequence, so its size is exponential: vocabulary to the power of sequence length. A single column of that rate matrix could never be stored, let alone predicted. The fix is a factorized CTMC: forbid jumps that change more than one token at a time. The model then only has to output a rate for each (position, vocabulary word) pair, a $d \times V$ table that grows linearly in sequence length. A transformer emits exactly that shape without complaint, and you update every position in parallel each step.

From there the recipe is beat-for-beat the one we already know, which is the real payoff of having learned it structurally rather than as a list of formulas. Choose a conditional probability path: the factorized mixture path, where each token independently either shows its true value with probability $\kappa_t$ or shows noise, so as $\kappa_t$ climbs from 0 to 1 information fades in per token. (Note the honest difference from the Gaussian path: nothing moves here, because there is nowhere to move to. One distribution fades out and another fades in.) Marginalize over the dataset to get a marginal path. Prove a discrete marginalization trick saying the posterior-weighted average of conditional rate matrices follows the marginal path, using the Kolmogorov Forward Equation exactly where the continuous story used the continuity equation. And then the punchline lands:

Reading the symbols: $p_{1 \mid t}(z_j = v_i \mid x)$ is simply "given this partially-corrupted sequence, what is the probability that position $j$ was really the token $v_i$?", and the prefactor $\dot{\kappa}_t / (1 - \kappa_t)$ is an urgency term that grows as time runs out. The entire learned content of the rate matrix is that conditional probability, which is a classifier per token position. So the training loss is not a regression at all, it is plain cross-entropy:

Corrupt a sequence, ask the network to name the original tokens, take the cross-entropy. Simulation-free, exactly as before. Continuous flow matching collapses to regression; discrete flow matching collapses to classification. Same skeleton, different bones.

The most famous instance is the masked diffusion language model (MDLM): add a special $[\text{MASK}]$ token to the vocabulary, start from a fully masked sequence, and let the CTMC reveal tokens over time until $t = 1$ hands you a sentence. Generation as progressive unmasking, in any order, in parallel, rather than strictly left to right. Fig 9.5.9 runs the real path and shows the urgency prefactor exploding as $t \to 1$.

And now the callback that makes this section earn its place in a robotics guide. Chapter 9's FAST tokenizer and π0-FAST turn continuous robot actions into discrete tokens and generate them autoregressively. That is not a different universe from π0's flow head. It is this dialect: a discrete state space, a probability path that corrupts tokens, and a per-position classifier trained with cross-entropy. The fork in Chapter 9 between "discrete tokens" and "continuous flow" is, from up here, a choice of which state space to build the same generative machine on.

Now we cash everything in. Chapter 9 told you π0 has "a flow matching action expert." You can now read that sentence completely, so let us take it apart.

The substitution that makes it work. Every result in this chapter was about generating an object $z \in \mathbb{R}^d$, and we insisted the math did not care what the object meant. Here is the payoff: let $z$ be a chunk of 50 future actions. That is it. That is the whole adaptation. If the robot has 14 controllable joints, a chunk is a $50 \times 14$ array, which flattened is a point in $\mathbb{R}^{700}$, and every theorem above applies verbatim. The data distribution $p_{\text{data}}(z \mid y)$ is "the chunks a human demonstrator would produce, given what the robot currently sees and was told to do." The initial distribution is a Gaussian of the same shape. Generating a motor command is integrating an ODE from noise for a handful of steps. (Side note: real systems zero-pad the action dimension to a fixed size, commonly 32, so that one model can serve robots with different numbers of joints. Chapter 10's cross-embodiment story, in one implementation detail.)

The anatomy. π0 (Physical Intelligence, 2024) is two pieces bolted together. A PaliGemma vision-language backbone, about 3 billion parameters, reads the camera images and the instruction and produces the conditioning: this is our $y$, and it is where all the semantic understanding lives. Attached to it is a much smaller action expert, about 300 million parameters, whose entire job is to be $u_t^\theta(x \mid y)$, the velocity field over action chunks. The VLM knows what a pan is; the expert knows how to move.

The training loop is Algorithm 3 with different nouns. Sample a demonstration chunk $z$ and an observation. Sample noise $\epsilon$ of the same shape and a time $t \sim \mathrm{Unif}[0,1]$. Form the blend $x = t z + (1-t)\epsilon$. Regress the expert's output on $z - \epsilon$. That is the identical loss we derived, with no simulation in the loop, which is why a robot policy can be trained with ordinary supervised-learning economics.

Where the conditioning enters, revisited. Remember AdaLN from the architecture section. π0.5 makes precisely that choice, injecting the conditioning through adaptive RMSNorm at every layer rather than concatenating a timestep embedding at the input, so that (as Abhinav Jha puts it in his visualization writeup) the conditioning "is present at every layer." This is not a decorative detail. The velocity the expert must output depends violently on $t$, and a policy whose deep layers have only a faded rumor of what time it is will integrate badly.

Why so few steps. π0 generates a chunk in roughly ten integration steps. You now know exactly why that is allowed rather than lucky: the CondOT path trains the field on straight lines from noise to data, and Euler's error comes entirely from pretending the path is straight across a step. Train straight paths, and big steps stop lying. Count the cost honestly: network function evaluations = integration steps, and a 50 Hz control loop leaves 20 milliseconds per decision. Ten evaluations of a 300M expert fits. Fifty does not. This is also, concretely, why robot stacks usually leave classifier-free guidance off: CFG runs the network twice per step, doubling your NFE, and (per the earlier quiz) it distorts the very distribution whose calibrated spread you want.

And now the most interesting thing a robot's generative model does. Ask what the spread of the sampled chunks means, and you get an insight that no image generator can teach you.

Abhinav Jha ran PCA on π0.5's actual denoising trajectories on a Mobile ALOHA episode and measured how much the sampled actions disagreed with each other at each point in the chunk. The result: the immediate action, at horizon $h = 0$, converged tightly across noise seeds (dispersion 0.27), while the action fifty steps out, at $h = 49$, stayed wildly spread (dispersion 1.77). Call it temporal certainty decay: the model is confident about now and vague about later, and the vagueness grows smoothly with the horizon.

This is not a defect, and the mechanism falls straight out of what we built. The learned distribution is the demonstration distribution, so the model's spread is the demonstrators' spread. Given what the camera sees right now, every reasonable human does roughly the same next thing, so the data is nearly unimodal at $h = 0$ and the posterior over chunks agrees there. Fifty steps out, the demonstrators had genuinely diverged, some going left around the obstacle and some right, some already reaching for the faucet, because the far future depends on states nobody has observed yet. The data is multimodal there, so the model's samples are too. A well-calibrated policy should be uncertain about the far future. As Jha frames the model's implicit reasoning: it knows exactly what to do right now, but the correct action 49 steps from now depends on future states it has not seen.

Fig 9.5.11 reproduces the mechanism from scratch so you can watch it happen. The demonstrations agree at $h = 0$ and fan into three modes later, and the figure integrates the real ODE in $\mathbb{R}^{100}$ from twelve seeds and measures the spread at each horizon index. The spread grows about sixteen-fold from the first waypoint to the last, nobody drew that curve, and all three modes get hit, which is Chapter 10's "preserve the multimodality" requirement showing up as a number.

The practical consequence ties the whole guide together. This is why receding-horizon control is not a hack: you generate 50 actions, execute only the first handful where the model is actually confident, then throw the vague tail away and replan with fresh observations. Chapter 10's action chunking gave you the reason to predict a chunk (it kills compounding error), and this figure gives you the reason not to trust all of it. Both are true at once, and the tension between them is the design.

The whole chapter in six sentences. Generation means sampling from $p_{\text{data}}$, and the sampler is a differential equation: an ODE if you like determinism, an SDE if you add noise, with a neural network supplying the velocity field. You choose a probability path (almost always the Gaussian one, often the straight-line CondOT schedule) that morphs noise into data, and the marginalization trick turns "follow that path" into a computable regression: the conditional flow matching loss, whose gradients provably equal the loss you actually wanted. The score-function dialect (denoising, noise prediction, DDPM) is the same object in different coordinates, and it unlocks the SDE extension: one trained network, a family of samplers, noise level chosen at inference time. Guidance conditions the machine on a prompt, and classifier-free guidance is the heuristic that makes conditioning forceful, trading diversity for adherence. The network hosting the field is a U-Net or a DiT that takes Fourier-embedded time through adaptive normalization at every layer, it usually runs inside a VAE's latent space because raw pixel space is too big to integrate in, and if your data is discrete the whole skeleton survives with rate matrices and cross-entropy instead of velocities and regression.

The four questions. That is the durable takeaway. Any generative system you meet from here, image, video, protein, or robot, is pinned down by four answers: what space does it run in (pixels, a VAE latent, an action chunk, a token sequence), what probability path (almost always Gaussian, often straight-line CondOT), what network hosts the field (U-Net, DiT, an action expert bolted to a VLM), and how does the conditioning get in (concatenation, cross-attention, adaptive normalization). Stable Diffusion 3 answers: latent space, CondOT, MM-DiT, three text encoders plus AdaLN. π0 answers: action-chunk space, CondOT, a 300M expert on a PaliGemma backbone, adaptive RMSNorm at every layer. Same four slots. The papers get much shorter once you read them this way.

Now reread the rest of the guide with new eyes. Chapter 9's π0, "a flow matching action expert that integrates a learned velocity field from noise to a 50-step action chunk in about ten steps," is now a sentence where every word has a formula behind it, including why ten steps suffice. Chapter 9's π0-FAST, which discretizes actions into tokens and generates them autoregressively, is the CTMC dialect. Chapter 10's Diffusion Policy is the score dialect, ε-prediction and all. Chapter 11's "latent world models" run the same machinery inside an autoencoder. Four systems that read as four separate inventions are one machine with different answers in the four slots.

And the one lesson that is specific to robots, the thing the image world cannot teach you: the spread of your samples is not noise to be minimized, it is the model telling you what it does not know. Temporal certainty decay is that spread, honestly reported, and receding-horizon control is the engineering response to it. A generative policy is not just a fancy way to output a number. It is a way to output a number with its uncertainty attached, and on hardware, that attachment is the difference between a robot that replans and a robot that confidently drives its plan into a wall.

Sources and going deeper. This chapter is built from Holderrieth & Erives, An Introduction to Flow Matching and Diffusion Models (2026, arXiv 2506.02070), whose lectures and hands-on labs are at diffusion.csail.mit.edu and are worth your time, especially the lab where you build a model from scratch. The VLA framing and the temporal certainty decay measurement come from Abhinav Jha's Visualizing Flow Matching in Robotics (2026, abhinavjha.xyz), which PCA-projects π0.5's real denoising trajectories and is the empirical companion to this chapter's theory; he is appropriately careful that projecting from high dimensions to two can mislead, and that his visualizations show on-manifold samples rather than the global field. Federico Sarrocco's Flow Matching writeup (federicosarrocco.com/blog/flow-matching) is a good theory-to-PyTorch companion if you want to see each equation land as code.

Next: back to the main line. Chapter 10 asks where the demonstrations that feed all of this actually come from, and why collecting them is the real bottleneck.