Math of Neural Networks

In Chapter 1 we built the whole picture of how a network learns, and we stated the rules without proving them. We said the error at the output is the prediction minus the truth, that it propagates backward through the transposed weights, that the weight gradient is an outer product of an error signal and an activation. We used those facts. We never earned them.

This chapter earns them. Everything here is the calculus underneath Chapter 1, derived step by step so that by the end the four equations of backpropagation are not magic words you memorize but results you could rederive on a napkin. We are not going to re-explain what a neuron is or why gradient descent works; you have that. We are going deeper into the math of how.

The plan, in order. First we fix the notation, which matters more here than in any other subject because of the swarm of indices. Then we review the two pieces of calculus we lean on: gradients and Jacobians, and the chain rule that ties them together. Then we write forward propagation as clean math. Then we differentiate it the slow, honest way for a tiny network, watch a pattern appear, and watch that slow way fail to scale. Finally we fix the scaling problem by naming one reusable quantity and propagating it backward, which is backpropagation, and we derive its four equations and prove they match the slow way.

A quick word on what you need. From Chapter 1 you already have derivatives (a slope, a rate of change), partial derivatives (the slope with respect to one variable while the rest are frozen), the gradient (all the partials stacked into a vector), the dot product, and the basic shape of a matrix. We will build on those rather than restate them. The genuinely new piece of machinery is the Jacobian, and we will take that one slowly.

Most confusion in this subject is not confusion about ideas. It is misreading an index. So before any derivation, here is the bookkeeping, and it is worth fixing in your head now.

We use four typographic conventions. A lowercase italic letter like $x$, $w$, or $b$ is a single number, a scalar. A lowercase bold letter like $\mathbf{x}$, $\mathbf{w}$, or $\mathbf{b}$ is a vector, an ordered list of numbers, which we always treat as a column unless we say otherwise. An uppercase italic letter like $W$ is a matrix, a grid of numbers. And a superscript in parentheses, like $w^{(\ell)}$ or $\mathbf{a}^{(\ell)}$, is a layer label, not an exponent; $\mathbf{a}^{(\ell)}$ means "the activations of layer $\ell$," and the parentheses are there precisely so you never mistake it for raising something to the power $\ell$.

Subscripts pick out a component. The one to internalize is the weight index. We write

for the weight in layer $\ell$ on the connection going into the $j$-th neuron of layer $\ell$, from the $k$-th neuron of layer $\ell-1$. Read the order out loud: destination first ($j$), source second ($k$). This feels backward, because when you draw an arrow you naturally think source-then-destination. There is a concrete payoff for the inversion, and we will cash it in two sections from now: it is exactly what makes the layer's matrix multiplication work with no stray transposes.

Two more symbols carry the whole story. We write $z^{(\ell)}_j$ for the weighted input to neuron $j$ in layer $\ell$, the value before the activation function, and $a^{(\ell)}_j = \sigma(z^{(\ell)}_j)$ for the activation, the value after it. The cost is $C$. And one quantity we will define carefully later, the error of a neuron, is

You do not need to understand that last line yet. Just register that $\delta$ (delta) lives at the heart of backpropagation and is defined as a partial derivative of the cost with respect to a neuron's weighted input. Keep this notation nearby. When something looks impenetrable later, nine times out of ten it is a misread index, not a hard idea.

Start with what you know and stretch it by one step. A gradient is the object you get when a function takes in a vector and returns a single number. Take $f : \mathbb{R}^n \to \mathbb{R}$, read as "a function from $n$-dimensional vectors to single numbers" (the symbol $\mathbb{R}$ means the ordinary real numbers, and $\mathbb{R}^n$ means a list of $n$ of them). For example $f(\mathbf{x}) = x_1^2 + x_2^2 + x_3^2$. You can take a partial derivative with respect to each input, and the gradient simply stacks them into a column:

The $\nabla$ symbol (read "del" or "gradient of") just means "collect all the partials." For the example, $\nabla f = [2x_1, 2x_2, 2x_3]^\top$, where the small $\top$ ("transpose") is flipping the row I wrote on the page into the column it should be. We met the geometric meaning in Chapter 1: the gradient points in the direction of steepest ascent, and the negative gradient points downhill, which is why we walk against it. We will treat the gradient as a column throughout, because it makes the Jacobian conventions line up cleanly, which is the new idea we turn to now.

What if the function returns not one number but several? That is the common case in a network: a layer eats a vector and produces a vector. Take $\mathbf{f} : \mathbb{R}^n \to \mathbb{R}^m$, a vector in and a vector out, so $\mathbf{f}(\mathbf{x})$ has $m$ output components, each depending on all $n$ inputs. Stack the gradient of each output as a row and you get the Jacobian matrix:

There is one rule to hold onto, and it governs everything: rows are outputs, columns are inputs. Entry $(i, j)$ of the Jacobian is $\frac{\partial f_i}{\partial x_j}$, the sensitivity of output $i$ to input $j$. So row $i$ is the gradient of the $i$-th output, and the matrix is $m$ rows tall (one per output) by $n$ columns wide (one per input). That shape, output-by-input, is what makes the dimensions click together when we chain things in the next section.

The gradient is just a Jacobian in disguise. If $m = 1$, meaning the function returns a single number, the Jacobian collapses to a single $1 \times n$ row. Transpose it and you have the column-vector gradient from before. So gradients and Jacobians are the same animal; the gradient is the Jacobian of a function whose output happens to be one number.

One special shape shows up constantly in networks and is worth memorizing, because it makes the algebra later evaporate. Suppose $\mathbf{f}$ acts elementwise, meaning each output depends only on the input in the same position: $f_i(\mathbf{x}) = g(x_i)$ for some single-variable function $g$. Activation functions are exactly this; $\sigma$ hits each entry on its own. Then output $i$ does not depend on input $j$ at all when $i \neq j$, so every off-diagonal partial is zero, and the Jacobian is diagonal, carrying $g'(x_i)$ down the diagonal:

A diagonal matrix is one whose only nonzero entries sit on the top-left-to-bottom-right diagonal. Why care? Because a diagonal Jacobian, when it multiplies a vector inside a chain rule, behaves exactly like multiplying entry by entry. If $J = \mathrm{diag}(\mathbf{v})$ and $\mathbf{w}$ is any vector, then $J\mathbf{w} = \mathbf{v} \odot \mathbf{w}$, where $\odot$ is the Hadamard product, plain element-wise multiplication. That single fact is what turns the intimidating matrix expressions in backprop into the friendly $\odot \, \sigma'(\mathbf{z})$ you saw in Chapter 1.

A concrete worked example to make the rows-are-outputs rule stick. Let $\mathbf{f}(\mathbf{x}) = [x_1 x_2, \; x_1 + x_2]^\top$, with two inputs and two outputs. Differentiate each output by each input:

Row 1 is the gradient of $f_1 = x_1 x_2$ (its partials are $x_2$ and $x_1$), and row 2 is the gradient of $f_2 = x_1 + x_2$ (its partials are $1$ and $1$). Rows are outputs, columns are inputs. Always.

You know the basic chain rule from Chapter 1: if $y = f(u)$ and $u = g(x)$, then $\frac{dy}{dx} = \frac{dy}{du}\frac{du}{dx}$. Sensitivities multiply along a chain. Before we lift this to vectors, there is a slightly richer version that makes the vector case feel inevitable instead of surprising.

Suppose $y$ depends on $x$ through two intermediate variables, both of which are functions of $x$: $y = f\big(u_1(x), u_2(x)\big)$. Now $x$ has two separate routes to influence $y$, one through $u_1$ and one through $u_2$. The total derivative adds up both routes:

Read the structure, not just the symbols. Each term is one path from $x$ to $y$: travel from $x$ to $u_1$ (that is $\frac{du_1}{dx}$), then from $u_1$ to $y$ (that is $\frac{\partial y}{\partial u_1}$), and multiply the two sensitivities along that path. Do the same for the path through $u_2$. Then sum over all the paths. This "multiply along a path, sum over paths" idea is the entire content of the multivariable chain rule, and once it is in your head the matrix version below is just bookkeeping for doing many paths at once.

This matters for networks specifically because a single neuron's output usually fans out to every neuron in the next layer. So when we ask how wiggling one neuron changes the cost, there is not one path back to the cost, there are many, one through each downstream neuron, and we will be summing over exactly those paths. That sum is where the matrix transpose in backprop comes from. Keep the picture handy.

Now the generalization that runs the whole machine. Suppose $\mathbf{y} = \mathbf{f}(\mathbf{u})$ and $\mathbf{u} = \mathbf{g}(\mathbf{x})$, so a vector $\mathbf{x}$ produces a vector $\mathbf{u}$, which produces a vector $\mathbf{y}$. The chain rule says: multiply the Jacobians.

The only thing to check is that the shapes fit, and they do, by design. Say $\mathbf{x} \in \mathbb{R}^n$, $\mathbf{u} \in \mathbb{R}^k$, and $\mathbf{y} \in \mathbb{R}^m$. The first Jacobian $\frac{\partial \mathbf{y}}{\partial \mathbf{u}}$ has rows for $\mathbf{y}$ and columns for $\mathbf{u}$, so it is $m \times k$. The second, $\frac{\partial \mathbf{u}}{\partial \mathbf{x}}$, is $k \times n$. A matrix product is allowed exactly when the inner dimensions match, and they do (both are $k$), and the result is $m \times n$, which is precisely the shape $\frac{\partial \mathbf{y}}{\partial \mathbf{x}}$ should have, with rows for $\mathbf{y}$ and columns for $\mathbf{x}$. The output-by-input convention from the last section is what guarantees this lines up.

Write out a single entry of that product and the sum-over-paths idea from the previous section walks right back in:

Each intermediate component $u_p$ is one path from input $x_j$ to output $y_i$, the term $\frac{\partial u_p}{\partial x_j}$ is the first leg and $\frac{\partial y_i}{\partial u_p}$ is the second, and we sum over all $k$ paths. Matrix multiplication is quite literally an automated way of doing sum-over-paths for every input-output pair at once.

For a deeper stack you just keep multiplying. If $\mathbf{y} = \mathbf{f}(\mathbf{g}(\mathbf{h}(\mathbf{x})))$, then

That is, in one line, what backpropagation does. A network is a deep composition of functions, and computing how the cost depends on an early weight means multiplying a string of Jacobians together as you travel from the output back to that weight. Everything from here is figuring out what those particular Jacobians are and how to multiply them in an order that avoids redoing work.

We covered the forward pass conceptually in Chapter 1. Here we write it precisely, with the indexing from section 2, because the derivations later depend on getting these expressions exactly right.

A single neuron takes its inputs, forms a weighted sum, adds a bias, and applies the activation:

Here $\mathbf{w}^\top \mathbf{x}$ is the dot product (multiply matching entries, add them up), $b$ is the bias scalar, $z$ is the weighted input, and $a$ is the activation after the nonlinearity $\sigma$. Nothing new yet; this is Chapter 1 in symbols.

Now the indexing payoff promised earlier. Put every weight of layer $\ell$ into a matrix $W^{(\ell)}$ whose entry in row $j$, column $k$ is $w^{(\ell)}_{jk}$. Because we indexed destination-first, the $j$-th row of $W^{(\ell)}$ is exactly the list of weights belonging to neuron $j$. So the matrix-vector product $W^{(\ell)} \mathbf{a}^{(\ell-1)}$ produces, in its $j$-th entry, the dot product of neuron $j$'s weights with the previous layer's activations, which is precisely the weighted sum neuron $j$ wants. No transposes, no rearranging. That is the entire reason for the destination-first convention. The matrix $W^{(\ell)}$ has dimensions $n_\ell \times n_{\ell-1}$ (number of neurons in this layer, by number in the previous layer), and the bias $\mathbf{b}^{(\ell)} \in \mathbb{R}^{n_\ell}$ is a column with one entry per neuron. A whole layer is then:

where $\sigma$ applied to a vector means applied to each entry. To run the whole network you set $\mathbf{a}^{(0)} = \mathbf{x}$, the raw input, and apply this pair of equations for $\ell = 1, 2, \ldots, L$, and the final activation $\mathbf{a}^{(L)}$ is the prediction $\hat{\mathbf{y}}$.

A small worked example to see the matrix do its job. Say layer $\ell-1$ has 3 neurons and layer $\ell$ has 2, so $W^{(\ell)}$ is $2 \times 3$:

The product is

and each row is one neuron's weighted sum. Add the biases, apply $\sigma$, and you have $\mathbf{a}^{(\ell)}$.

Stacking all the layers, the entire network is one deeply nested function:

This is the object we are about to differentiate. It looks fearsome, but it is just our layer equation wrapped around itself $L$ times, and the Jacobian chain rule from the last section is built exactly for peeling apart compositions like this.

The cost is the single number we minimize, and we treat it, as in Chapter 1, as a function of all the weights and biases. Collect them into $\theta$. The network defines $\hat{\mathbf{y}} = f(\mathbf{x}; \theta)$, and we have training pairs $(\mathbf{x}^{(i)}, \mathbf{y}^{(i)})$ for $i = 1, \ldots, N$, where here $i$ indexes training examples, not layers. The default cost for this chapter is mean squared error:

Unpacking the symbols: the double bars $\| \cdot \|^2$ mean the squared length of a vector, which is just the sum of the squares of its entries, which is why the right-hand form expands into a sum over the output components $j$. The outer sum averages over all $N$ training examples, and the $\frac{1}{2}$ out front is pure convenience: when we differentiate a square we get a factor of $2$, and the $\frac{1}{2}$ is there to cancel it and keep the algebra tidy. It changes nothing about where the minimum is.

For the rest of the derivations we make one simplifying move: pretend there is a single training example, so $C = \frac{1}{2}\|\mathbf{y} - \hat{\mathbf{y}}\|^2$. The full-dataset cost is just the average of the single-example costs, and the gradient of an average is the average of the gradients, so the structure of every derivative we find is identical; we would just average at the end. Carrying the sum around would only clutter the page.

To differentiate the whole cost we first need the derivatives of the small operations a neuron performs, because the chain rule will glue these together. A layer does three things in sequence: it forms weighted sums (a matrix-vector product, or for one neuron a dot product, or for one weight a single multiplication), it adds the bias (a vector addition), and it applies the activation (an elementwise function). Let us find the Jacobian of each.

Begin with elementwise operations, since the activation is one and they have that pleasant diagonal structure. A binary elementwise function combines two vectors entry by entry: $f_i(\mathbf{u}, \mathbf{v}) = g(u_i, v_i)$. The Hadamard product $\mathbf{u} \odot \mathbf{v}$ (entry-wise multiplication) is the standard example. Because output $i$ depends only on $u_i$ and $v_i$ and nothing else, the Jacobian with respect to either input is diagonal. For the Hadamard product specifically, $f_i = u_i v_i$, so $\frac{\partial f_i}{\partial u_i} = v_i$, giving

In words, the derivative with respect to one operand is the diagonal matrix of the other operand. And recall the collapse from section 3: a diagonal Jacobian times a vector is just an element-wise product. So when a Hadamard product appears in a chain rule, the incoming gradient simply gets multiplied entry-by-entry by the other operand. This is exactly why the backprop equations are dotted with $\sigma'$ rather than carrying around full matrices.

Next, addition, which is the bias step. Take $\mathbf{f}(\mathbf{z}, \mathbf{b}) = \mathbf{z} + \mathbf{b}$, so $f_i = z_i + b_i$. Each output depends on its own $z_i$ and $b_i$ with derivative $1$ and on nothing else, so both Jacobians are the identity matrix $I$ (ones on the diagonal, zeros elsewhere):

This is the lazy case: addition passes gradients straight through, unchanged. Whenever you see a sum in the forward pass, the matching Jacobian is an identity, and identities do nothing in a product, so you can mentally skip them.

Finally, the lone sum that collapses a vector to a scalar, as in $z = \sum_k w_k x_k + b$ for one neuron. Treating "sum" as the function $s(\mathbf{u}) = \sum_i u_i$, its derivative with respect to each input is $1$, so its gradient is the all-ones vector $\mathbf{1} = [1, 1, \ldots, 1]^\top$. Differentiating a sum just adds up the upstream contributions equally. Pulling these together for a single neuron's $z = \mathbf{w}^\top \mathbf{x} + b$, the three derivatives we will reach for again and again are

Each reads naturally: the weighted input is sensitive to a weight in proportion to the input riding on it ($x_k$), sensitive to the bias with rate $1$, and sensitive to an input in proportion to that input's weight ($w_k$).

That leaves the activation. For a single neuron $a = \sigma(z)$, the derivative is just $\frac{da}{dz} = \sigma'(z)$. For a whole layer, $\sigma$ is applied elementwise, so by the diagonal rule the Jacobian is

The single most useful instance is the sigmoid, because its derivative is unusually clean. With $\sigma(z) = \frac{1}{1 + e^{-z}}$, differentiating gives

That last equality is worth savoring. The derivative of the sigmoid is expressible entirely in terms of the sigmoid's own value. So during the backward pass, if you already computed $\sigma(z)$ in the forward pass, you get its derivative almost for free, just multiply by $(1 - \sigma(z))$, with no exponentials to recompute. For ReLU the derivative is even simpler: $\sigma'(z) = 1$ when $z > 0$ and $0$ when $z < 0$ (it is undefined right at $0$, a single point we ignore in practice).

Now we differentiate the cost by hand, on the smallest network that still shows the structure: one neuron per layer, two layers. Watching this done explicitly is what makes the eventual backprop equations feel obvious rather than arbitrary. The forward pass is

We want $\frac{\partial C}{\partial w^{(2)}}$ and $\frac{\partial C}{\partial w^{(1)}}$.

Start with the last-layer weight, which is the short walk. The cost depends on $w^{(2)}$ only through $z^{(2)}$, then through $a^{(2)}$. So the chain rule runs three links deep:

Each link we already know how to compute. Differentiating $\tfrac{1}{2}(y - a^{(2)})^2$ with respect to $a^{(2)}$ gives $\frac{\partial C}{\partial a^{(2)}} = -(y - a^{(2)}) = a^{(2)} - y$. The activation link is $\frac{\partial a^{(2)}}{\partial z^{(2)}} = \sigma'(z^{(2)})$. And since $z^{(2)} = w^{(2)} a^{(1)} + b^{(2)}$, the last link is $\frac{\partial z^{(2)}}{\partial w^{(2)}} = a^{(1)}$. Multiply them:

Now the first-layer weight, the longer walk. The cost depends on $w^{(1)}$ through $z^{(1)}$, then $a^{(1)}$, then $z^{(2)}$, then $a^{(2)}$, so the chain is five links:

The two new links: $\frac{\partial z^{(2)}}{\partial a^{(1)}} = w^{(2)}$ (because $z^{(2)} = w^{(2)} a^{(1)} + b^{(2)}$), and $\frac{\partial a^{(1)}}{\partial z^{(1)}} = \sigma'(z^{(1)})$, and finally $\frac{\partial z^{(1)}}{\partial w^{(1)}} = x$. Multiply the whole string:

Stare at those two boxed results, because the patterns in them are the whole of backpropagation in miniature.

First, reuse. The first two factors of $\frac{\partial C}{\partial w^{(1)}}$, namely $(a^{(2)} - y)\,\sigma'(z^{(2)})$, are exactly the leading factors of $\frac{\partial C}{\partial w^{(2)}}$. We recomputed them. If instead we had saved that piece, we could have reached the first-layer gradient by just tacking on $w^{(2)}\,\sigma'(z^{(1)})\,x$. That saved, reused quantity is the seed of the error signal $\delta$.

Second, the **role of $\sigma'$**. Every layer the chain passes through contributes one factor of $\sigma'(z^{(\ell)})$. For the sigmoid, $\sigma'$ never exceeds $0.25$, so in a deep network you are multiplying many numbers each at most a quarter, and the product collapses toward zero. That is the vanishing gradient problem from Chapter 1, now visible as a literal product of small factors stacking up.

Third, a structural rhythm. Both formulas share one skeleton: an error at the output, $(a^{(2)} - y)$, propagated backward through the network, and then, at the very last step, multiplied by the input that fed the weight in question, which is $a^{(1)}$ for the second-layer weight and $x$ for the first. That skeleton, error-from-the-end times input-that-fed-the-weight, is precisely the form the four equations will take.

The bias derivatives confirm the rhythm. Same network, but now the final link changes, because $\frac{\partial z^{(2)}}{\partial b^{(2)}} = 1$ replaces the $a^{(1)}$, and $\frac{\partial z^{(1)}}{\partial b^{(1)}} = 1$ replaces the $x$:

So the bias gradient is the weight gradient with the trailing input factor stripped off. Said another way, the bias gradient is the propagated error at that neuron, which is the strongest hint yet that we should give that propagated error a name. We are about to.

We just differentiated a network with two parameters by hand. The obvious thought is to do the same for a real network: write a chain rule for every weight and grind through it. Let us see exactly why that is hopeless, because the failure points straight at the fix.

First, what we are even computing. For a weight matrix $W^{(\ell)}$, the gradient $\frac{\partial C}{\partial W^{(\ell)}}$ is itself a matrix the same shape as $W^{(\ell)}$, whose entry $(j, k)$ is $\frac{\partial C}{\partial w^{(\ell)}_{jk}}$. So we need one partial derivative per weight, arranged in a grid.

Now the cost of getting them the naive way. A network with $L$ layers and roughly $n$ neurons per layer has on the order of $L \cdot n^2$ weights. For each one, a chain rule walk back to the output passes through on the order of $L \cdot n$ intermediate quantities. Multiply: the total work is on the order of $L^2 \cdot n^3$ per training example. For anything beyond a toy, that is absurd, and worse, it is absurd while being wildly redundant, because, as we saw in the last section, the walks for different weights share almost all of their factors and we would be recomputing the same shared pieces over and over.

There is also an ugliness of representation lurking. The moment you try to write the Jacobian of a layer's activation vector with respect to its weight matrix, you are differentiating a vector with respect to a matrix, which is a three-dimensional array of numbers, a rank-3 tensor. The notation and bookkeeping get genuinely unpleasant fast.

Both problems have the same cure, and the last section already whispered it. Identify the one quantity that is reused across all of a layer's weight gradients, compute it once per layer, and propagate it backward instead of redoing full chain walks. That quantity is the error of a neuron. Naming it and propagating it turns the cost from $O(L^2 n^3)$ down to roughly the cost of a single forward pass, linear in the number of parameters. That move is backpropagation, and it is the rest of the chapter.

(For completeness: once we have all the gradients, we feed them to gradient descent, which we covered fully in Chapter 1. The one-line recap is that we update each parameter by stepping against its gradient, $\theta \leftarrow \theta - \eta \nabla_\theta C$, and in practice we average the gradient over a small mini-batch of examples per step rather than the whole dataset, repeating over many epochs. Nothing about that changes here; backprop is just the efficient way to get the $\nabla_\theta C$ that gradient descent consumes.)

Here is the quantity that fixes everything. For each neuron in each layer, define its error as the sensitivity of the cost to that neuron's weighted input:

In words, $\delta^{(\ell)}_j$ asks: if I wiggle the weighted input $z^{(\ell)}_j$ of neuron $j$ in layer $\ell$ by a hair, how much does the final cost move? It is a single number per neuron, and we gather a layer's worth into a column vector $\boldsymbol{\delta}^{(\ell)} \in \mathbb{R}^{n_\ell}$.

Why define the error at $z$, the weighted input, rather than at $a$, the activation, or at the weights directly? Because $z$ sits at the perfect hinge. Everything upstream of $z^{(\ell)}_j$, namely the weights $w^{(\ell)}_{jk}$, the bias $b^{(\ell)}_j$, and the incoming activations, feeds into $z^{(\ell)}_j$ through plain linear operations, so once we know how sensitive the cost is to $z^{(\ell)}_j$, the sensitivities to those upstream parameters are a trivial extra step. And everything downstream of $z^{(\ell)}_j$, the entire rest of the network up to the cost, is already bundled inside $\delta^{(\ell)}_j$ by definition. So $\delta$ cleanly separates the easy local part from the hard global part, and the global part is exactly what we will pass backward.

The strategy is now sharp. Compute $\boldsymbol{\delta}^{(L)}$ at the output layer. Then use it to compute $\boldsymbol{\delta}^{(L-1)}$, then $\boldsymbol{\delta}^{(L-2)}$, and so on backward to $\boldsymbol{\delta}^{(1)}$. Then, for every layer, read off the weight and bias gradients from $\boldsymbol{\delta}^{(\ell)}$ in one cheap step. Four equations make this concrete: one to start $\boldsymbol{\delta}$ at the output, one to pass it back a layer, and two to read off the parameter gradients. We derive them next.

These are the four equations Chapter 1 handed you as facts. Now we derive each one.

The cost depends on the output neuron's weighted input $z^{(L)}_j$ only through its activation $a^{(L)}_j = \sigma(z^{(L)}_j)$. So a two-link chain rule gives

Stacking this over all output neurons $j$, and using the diagonal-Jacobian collapse (an elementwise activation contributes a diagonal Jacobian, which acts as element-wise multiplication), the vector form is

Here $\nabla_{\mathbf{a}^{(L)}} C$ is the gradient of the cost with respect to the output activations, and $\odot$ is the element-wise product. For the mean squared error cost, $\nabla_{\mathbf{a}^{(L)}} C = \mathbf{a}^{(L)} - \mathbf{y}$, so $\boldsymbol{\delta}^{(L)} = (\mathbf{a}^{(L)} - \mathbf{y}) \odot \sigma'(\mathbf{z}^{(L)})$. Notice this matches the leading factors $(a^{(2)} - y)\,\sigma'(z^{(2)})$ from our hand derivation in section 9. Good sign.

This is the keystone, the equation that does the actual propagating. We want $\boldsymbol{\delta}^{(\ell)}$ assuming we already have $\boldsymbol{\delta}^{(\ell+1)}$ from the layer ahead.

Start from the definition $\delta^{(\ell)}_j = \frac{\partial C}{\partial z^{(\ell)}_j}$ and ask how $z^{(\ell)}_j$ reaches the cost. It does so only through the next layer, and specifically through every neuron of the next layer, because neuron $j$'s activation $a^{(\ell)}_j$ fans out to all of them. That fan-out is the sum-over-paths situation from section 4. So

where we recognized $\frac{\partial C}{\partial z^{(\ell+1)}_k}$ as exactly $\delta^{(\ell+1)}_k$, the error we already have. Now we just need $\frac{\partial z^{(\ell+1)}_k}{\partial z^{(\ell)}_j}$. Write out the next layer's weighted input:

Differentiate with respect to $z^{(\ell)}_j$. Only the $m = j$ term in the sum depends on it, and it brings down $w^{(\ell+1)}_{kj}$ times the activation's derivative:

Substitute back, and pull the $\sigma'(z^{(\ell)}_j)$ out of the sum since it does not depend on $k$:

Look closely at that remaining sum, $\sum_k w^{(\ell+1)}_{kj}\, \delta^{(\ell+1)}_k$. In the original matrix $W^{(\ell+1)}$, the row index $k$ is the destination and the column index $j$ is the source. Here we are summing over the destination $k$ and the free index is the source $j$, which means $j$ is now playing the role of a row. That is exactly the transpose $W^{(\ell+1)\top}$. So the sum is the $j$-th entry of $(W^{(\ell+1)})^\top \boldsymbol{\delta}^{(\ell+1)}$, and the whole thing becomes

This is the equation that propagates the error backward. Read it as two moves. The transposed weight matrix takes the error from the layer ahead and sends it back through the linear connections, handing each upstream neuron a share of the blame in proportion to how strongly it fed forward. Then the element-wise $\odot \, \sigma'(\mathbf{z}^{(\ell)})$ scales each neuron's share by how responsive its activation actually was at that point. And here is the vanishing gradient made precise: if a neuron sat on a flat tail of the sigmoid, $\sigma'(z^{(\ell)}_j)$ is nearly zero, so it throttles that neuron's error toward nothing, and every neuron further back gets starved of signal. The transpose is the same matrix doing the forward fan-out, now run in reverse.

Easy now that we have $\delta$. The bias $b^{(\ell)}_j$ enters the cost only through $z^{(\ell)}_j$, and $\frac{\partial z^{(\ell)}_j}{\partial b^{(\ell)}_j} = 1$ from section 8. So

or in vector form,

The bias gradient simply is the error. This formalizes the hint we noticed at the end of section 9.

The weight $w^{(\ell)}_{jk}$ enters the cost only through $z^{(\ell)}_j$, and since $z^{(\ell)}_j = \sum_m w^{(\ell)}_{jm} a^{(\ell-1)}_m + b^{(\ell)}_j$, only the $m = k$ term depends on it, giving $\frac{\partial z^{(\ell)}_j}{\partial w^{(\ell)}_{jk}} = a^{(\ell-1)}_k$. So

or, putting destination and source in their natural reading order,

This is the structural rhythm from section 9, now exact and general: a weight's gradient is the error of the neuron it feeds ($\delta^{(\ell)}_j$) times the activation it receives ($a^{(\ell-1)}_k$). Two numbers, multiplied.