Transformers

In 2017, eight researchers at Google published a paper called "Attention Is All You Need," and machine learning was never the same. The architecture they introduced — the Transformer — is the foundation of GPT, Claude, BERT, Gemini, LLaMA, and basically every modern foundation model. If you want to understand modern ML, this is the thing you have to understand.

So here's the plan. We're going to take this from first principles. We'll start high-level — asking why the transformer had to exist at all — then go deep, building the architecture up one component at a time, with diagrams to make the math visual. Along the way you'll meet every idea that lives inside the transformer: memory, gates, attention, parallelism. None of those ideas were invented in 2017. They were each invented to fix a specific problem with the model that came before, and the transformer is what you get when you keep all the good parts and drop the parts that slowed everything down.

Take it slow. By the end you'll be able to read any modern LLM paper and know exactly which piece it's poking at.

A quick note on what I'm assuming: you already know what a neural network is and roughly how a GPU works. Everything else, I'll explain as we go.

Let's get into it.

For a long time, the models we used for language were good at recognizing fixed patterns but fell apart the moment you handed them a sequence — and sequences are exactly what separate human language from a pile of features. These early models had no memory of order or context. Ring a bell? It should, because every model we're about to walk through is one more attempt to fix that single problem, and the transformer is where it finally gets fixed properly.

Here's the lineage we're going to trace, in order:

Feedforward → RNN → LSTM/GRU → Seq2Seq → Attention → Transformer.

Every arrow in that chain is a specific limitation that the next model was invented to solve. Hold that framing in your head the whole way through — it's the entire story, and it's what makes the transformer feel inevitable rather than magical.

These are the familiar networks — the kind you already know. We'll use them as our starting line.

A feedforward network (FFN) is the most basic neural network there is. Information flows in exactly one direction: input → hidden layers → output. No loops, no memory, no notion of time. Each layer is a matrix multiplication followed by a nonlinearity:

Let's read every symbol in that:

You feed in a fixed-size input vector $x$, the network runs it through a series of these transformations, and out comes a fixed-size output vector. Train it on enough examples and it can approximate basically any function from inputs to outputs — the universal approximation theorem says so.

For images, you'd flatten the pixels into one long vector. For tabular data, each column becomes an input dimension. For text, you'd... well, that's where the trouble started.

Why they were a breakthrough. In the 1980s, feedforward networks — trained with backpropagation, popularized by Rumelhart, Hinton, and Williams in 1986 — showed that neural networks could learn complicated functions automatically, straight from data, with nobody hand-coding the rules. They worked great on problems with fixed-size inputs and outputs: digit recognition, simple classification, regression.

How they handled sequences. Badly. And here's the root of it: a feedforward network has a fixed input size. To process "the cat sat," you'd have to pick a window size up front — say five words — and feed in five word embeddings glued together.

That choice immediately boxes you in three different ways:

First, fixed length. A five-word window can't handle a six-word sentence, let alone a paragraph. You're stuck truncating or padding everything to fit.

Second, no real sense of order. The network treats "word at position 1" and "word at position 2" as completely separate features. There's no shared understanding that they're the same kind of thing showing up in different spots. Trying to learn grammar this way is painfully inefficient — the network has to relearn what a verb is at every position separately.

Third, bag-of-words behavior. In a lot of setups, researchers used simpler representations like averaging the word vectors together. Do that and the network literally can't tell "dog bites man" from "man bites dog." Order just evaporates.

For anything sequential — language, audio, time series — feedforward networks were a dead end. You couldn't even decide what the right input format was supposed to be. This is what pushed the whole field toward architectures that had memory and some awareness of order.

As the name suggests, RNNs are networks that feed their own output back in as input at the next step. The network keeps a hidden state $h$ that carries across timesteps. At each step $t$:

Reading the symbols:

The hidden state is a "running summary" of everything the model has read up to now. And here's the most important part of the whole setup: the same weights are reused at every single timestep. There's only one $W_x$, one $W_h$, one $W_y$, no matter how long the sequence is. That weight-sharing is exactly what lets an RNN handle sequences of any length — you just keep applying the same transformation as new inputs roll in.

Let's walk an example. To process "the cat sat," you'd:

1. Feed in "the," get $h_1$.

2. Feed in "cat" along with $h_1$, get $h_2$.

3. Feed in "sat" along with $h_2$, get $h_3$.

By the end, $h_3$ is supposed to be a summary of the entire sequence.

This was a genuine leap. RNNs gave neural networks memory for the first time. In principle the hidden state could carry information from arbitrarily far back. They handled variable-length sequences naturally. And they produced an output at every timestep, so they could do part-of-speech tagging, language modeling (predict the next word), or translation. For roughly two decades, RNNs were the standard architecture for any sequence problem in deep learning.

Here's the unrolled view — the same cell repeated across time:

So why did RNNs fade out? One word: gradients. In theory the hidden state could carry information arbitrarily far. In practice it couldn't, and there were two compounding reasons plus a third structural one.

Vanishing gradients. To train an RNN you backpropagate the gradient through every timestep — this is called backpropagation through time. The gradient at step 1, coming from a loss at step 50, gets multiplied by the recurrent weight matrix 49 times on the way back. If those multiplications shrink the signal even a little — which is the default behavior with sigmoid or tanh nonlinearities — the gradient shrinks exponentially toward zero. By the time it reaches the early steps it's a rounding error. The early timesteps simply can't learn from mistakes made later on.

Exploding gradients. The mirror image. If the recurrent weights are a touch too large, the gradient grows exponentially through the backward pass and the loss blows up to NaN (Not a Number). It's less common than vanishing but harder to ignore when it hits. The standard fix became gradient clipping — capping the gradient's norm so it can't run away.

Compressed state. Even if the gradient flowed perfectly, the entire history has to be crammed into one hidden-state vector — a few hundred numbers. There just isn't enough room to remember everything, so new inputs end up overwriting old information.

In practice, RNNs reliably learned dependencies of about 5–10 tokens. Past that, they degraded. For language that's brutal. Understanding a sentence like "the dog that the cat that the rat bit chased ran" means linking words that sit far apart, and a plain RNN simply couldn't hold the thread that long. This is what sent everyone searching for an architecture with better memory.

An LSTM (Long Short-Term Memory network) was designed to fix that exact vanishing-gradient problem. The core difference from a plain RNN is that it gives the network a separate memory cell that information can flow through with almost no interference, controlled by learnable gates.

An LSTM cell keeps two pieces of state at each step, not one:

And it has three gates deciding what happens to that cell state at each step:

1. Forget gate $f_t$ — decides what to erase from the cell state. It's a sigmoid, so it outputs values between 0 (forget completely) and 1 (keep entirely).

2. Input gate $i_t$ — decides what new information to write into the cell state.

3. Output gate $o_t$ — decides what to read out of the cell state to form the hidden state.

Here are the equations:

Let's name every symbol:

The line that does all the work is the cell-state update:

Look at what it's doing. The old cell state $C_{t-1}$ is multiplied by the forget gate (which can sit near 1, leaving memory basically untouched) and then added to. There's no repeated matrix multiplication grinding the signal down. That additive path is why gradients can flow back across many timesteps without vanishing. This is the famous "memory highway": when the forget gate is open and the input gate is closed, information from far in the past slides through unchanged.

Here's the cell, gate by gate:

The GRU is a simpler take on the same idea. It merges the cell state and hidden state into one, and uses two gates instead of three:

1. Update gate $z_t$ — combines the jobs of the input and forget gates. It decides how much to update versus how much to preserve.

2. Reset gate $r_t$ — controls how much of the past to use when computing the new candidate state.

The symbols echo the LSTM: $z_t$ and $r_t$ are the gates, $W_z$, $W_r$, $W$ are their learnable matrices, $\tilde{h}_t$ is the candidate hidden state, and $\odot$ is again element-wise multiplication. Notice the final line is another convex blend — $(1 - z_t)$ of the old state plus $z_t$ of the new candidate — which keeps that same gradient-friendly additive flavor with fewer moving parts. Fewer gates means fewer parameters and faster training, often at basically the same quality.

LSTMs and GRUs were a huge step up. But even at their best, two problems stuck around — and a third one in translation setups turned out to be the spark for everything that followed.

Sequential training is slow. Computing $h_t$ requires $h_{t-1}$, which requires $h_{t-2}$, all the way back to the start. You cannot parallelize across time. On a GPU with thousands of cores, an LSTM uses roughly one core's worth of work per timestep. For a 1,000-token sequence, that's 1,000 operations that have to happen one after another. Modern GPUs scream through big dense matrix multiplications, but an LSTM can't feed them that kind of work. Training a large LSTM on a long sequence took days where a transformer does the same job in hours.

Long-range dependencies still degrade. LSTMs were far better than plain RNNs, but they still struggled past a few hundred tokens. The cell state has finite capacity, so over a long sequence the useful early information gets overwritten or muddied. Empirically, performance on long-range tasks just plateaued.

The information squeeze. This one was especially painful in encoder–decoder LSTM setups for translation. The encoder had to compress the entire source sentence into one fixed-size vector before the decoder could even start generating. For a 30-word sentence, maybe that vector could hold it all. For a 100-word paragraph, no chance. Everything had to pass through one narrow choke point, and detail got crushed.

That last problem is the one attention was invented to solve. Which brings us to Seq2Seq.

The Seq2Seq model, built by Ilya Sutskever and colleagues, used two LSTMs: an encoder that reads the input sentence and produces a single final hidden state, and a decoder LSTM that gets initialized with that state and generates the output one token at a time.

The flow looked like this:

It worked well for tasks like translation. But the architecture had one glaring weak point: every drop of information from the input had to flow through that single fixed-size vector $h_{enc}$. For short sentences, fine. For paragraphs, the model had forgotten the beginning of the input by the time it was generating the end of the output.

This is where attention enters the story — first as an add-on to Seq2Seq, introduced by Bahdanau and colleagues in 2014. The idea was simple and, in hindsight, enormous: don't force the decoder to lean on one summary vector. Instead, let it look back at every encoder hidden state and decide, at each generation step, which ones are relevant right now.

Here's the algorithm:

1. The decoder's current hidden state $s_{i-1}$ acts as a query.

2. Each encoder hidden state $h_j$ acts as a key/value.

3. Compute a compatibility score: how relevant is encoder state $h_j$ to the current decoder state $s_{i-1}$?

4. Softmax those scores into weights $\alpha_{ij}$ that sum to 1.

5. Compute a context vector $c_i = \sum_j \alpha_{ij} \, h_j$ — a weighted sum of the encoder states.

6. Use $c_i$ alongside the decoder hidden state to generate the next token.

Let's name those symbols, because they'll come back in a big way:

Take a sec to let that sink in, because here's the punchline: this is the exact same mechanism that becomes the centerpiece of the transformer. Modern self-attention is just this idea, generalized — instead of the decoder attending to the encoder, every token attends to every other token. Same query/key/value skeleton, same softmax-weighted sum.

One historical detail. In Bahdanau attention, the compatibility score was computed by a small feedforward network — different from the dot product the transformer would later use. But conceptually, this was the birth of attention.

Here's that original attention mechanism laid out:

What did attention buy us? It demolished the squeeze. Translation quality jumped, especially on long sentences. Suddenly the decoder could zero in on whichever input word mattered most at each output step — which is, not coincidentally, exactly how a human translator works.

And the lesson reached past translation. It showed that direct token-to-token interaction across the sequence, weighted by learned attention, was a more powerful idea than threading everything through a recurrent hidden state. The model was no longer limited by what it could squeeze into one vector — it could pull from anywhere it needed.

For a few years, the architecture of choice was "LSTM + attention." It worked — but the LSTM part was still slow and stubbornly sequential. Every step waited on the previous step. The attention was the good part; the recurrence was the part dragging everything down.

Think about what that meant. Even with attention bolted on, you still couldn't parallelize across timesteps. You still couldn't truly feed a GPU the dense work it loves. And you were now doing more total computation — both the recurrent step and the attention step — at every position.

So Vaswani and his coauthors asked the obvious question in 2017: what if we keep only the attention? What if we throw out the LSTMs entirely and let attention do all the work? If attention is the part actually solving the long-range problem, why are we still paying for recurrence at all?

The answer was "Attention Is All You Need," and the transformer was born. The very mechanism that started life as a helper for LSTMs became the entire architecture.

Let's start by looking at the transformer as a black box, then crack it open and study the system underneath.

At the highest level, a transformer is an architecture that takes a sequence in and produces a sequence out. The classic example is translation — an English sentence in, a French sentence out.

Why did this one architecture kick off the entire AI revolution? Because it fixed, all at once, everything the earlier models kept tripping over. The models we just walked through struggled to remember context over long paragraphs — when they tried, they either forgot too fast (small effective memory) or went unstable. They were hard to scale because they couldn't take advantage of GPUs, and all that compression crushed the detail out of long inputs. The transformer flips every one of those:

Internally, the transformer splits into two halves: an encoder that processes the input, and a decoder that generates the output. In the original paper, each half is a stack of 6 identical layers. Both halves are built from the same kit of components — they just use them a little differently.

A few things to lock in from the big picture. The encoder processes the entire input in parallel, producing a rich representation of every input token in context. The decoder generates the output one token at a time, and each step looks at (1) what the decoder has already produced and (2) the full encoder output, via cross-attention. That one-token-at-a-time pattern is autoregressive generation, and every modern LLM still works this way. (Quick definition: an autoregressive model predicts the next value in a sequence from the previous values in that same sequence.)

Also worth flagging early, since it reframes everything that follows: modern LLMs like GPT are technically decoder-only transformers — they keep the decoder stack and drop the encoder, because for pure text generation you don't need a separate "input" to encode. BERT is encoder-only for the opposite reason — it builds representations of text but never needs to generate. The original encoder–decoder design is most natural for input-to-output tasks like translation. We'll come back to all three.

Inside these blocks, five core mechanisms work together: attention (in four variants), feed-forward networks, layer normalization, positional encoding (added to the initial input embeddings), and residual connections. We'll cover every one.

Here's the whole thing, assembled — the full encoder–decoder transformer:

Before we get into the attention mechanisms themselves, there's some housekeeping to do. We need to talk about how raw text even becomes something a transformer can chew on. That's tokenization and embeddings.

Tokenization. Computers speak in numbers; humans speak in words. Tokenization is the bridge: we break raw text into smaller, manageable units called tokens. A token might be a whole word, a piece of a word ("token" + "ization"), or even a single character, depending on the scheme. Common approaches include word-level (split on spaces — simple but the vocabulary explodes and rare words break it), character-level (tiny vocabulary, but sequences get very long and meaning is thin per token), and the modern workhorse, subword tokenization like Byte-Pair Encoding (BPE) or WordPiece, which strikes a balance: frequent words stay whole, rare words split into reusable pieces, and you never hit a word you literally can't represent.

Token embeddings. Once text is split into tokens, each token is mapped to a unique token ID from a predefined vocabulary. But transformers don't compute on raw integers — they work with vectors. This is where embeddings come in. An embedding is a numerical representation of an object (like a word) that turns high-dimensional, sparse data into a dense, lower-dimensional vector living in a continuous space — the embedding space — where semantically similar items sit close together.

The machinery is an embedding matrix: a big table with $V$ rows and $d$ columns, where:

Each row of this matrix is a trainable vector for one token. For example: Transform → token ID 1231 → [0.1, 0.3, 0.4, 0.9, 0.8]. Once every input ID is swapped for its embedding, the entire input sentence becomes a 2D tensor of shape (number of tokens, $d$).

The takeaway: after tokenization and embedding, every token is a vector that carries semantic meaning, and similar concepts (bat, swung, hit, ball) land near each other in this high-dimensional space. These embeddings are what the transformer actually operates on.

Time for the main event. Let's build the intuition first.

Consider the sentence: "I like this girl." The word like is ambiguous on its own — is it the "similar to" like, or the "fond of" like? How do you, or a model, know which one we mean? The answer is context. As humans, we see that girl is sitting right there in the same sentence, so we connect like to fondness rather than similarity. Self-attention is the mechanism that lets each word do exactly this — gather context from the other words around it. Instead of treating each word in isolation, every word looks at every other word and decides how much each one matters for its own meaning.

Now the single most important idea in transformers: Q, K, V.

Self-attention gives each token three different vectors, all derived from its embedding:

1. Query (Q) — what this token is looking for.

2. Key (K) — what this token offers to others.

3. Value (V) — the actual content this token will contribute.

The classic analogy is a search engine. You type a search into the bar (your query). The engine matches your query against page titles and keywords (the keys). Where it finds good matches, it hands back the actual page content (the values).

In self-attention, every token does this at the same time. Each token acts as a query searching across all the other tokens (including itself), finds its best matches based on key similarity, and pulls in their values — weighted by how good each match was.

Q, K, and V are computed from the input embeddings via three learned weight matrices: $W_Q$, $W_K$, $W_V$. For an input embedding $x$:

Reading the symbols:

For a sequence of $n$ tokens you do this for the whole sequence at once with a single matrix multiplication, producing three matrices of shape $(n, d_k)$, $(n, d_k)$, and $(n, d_v)$ — where $d_k$ is the dimension of the query/key vectors and $d_v$ the dimension of the value vectors.

Once each token has its query and every token has its key, we need to measure how well each query matches each key. The transformer uses the dot product — a simple, fast operation that measures how aligned two vectors are. A large positive dot product means strong alignment (relevant); near zero means orthogonal (irrelevant); negative means anti-aligned.

For each query $Q_i$ and key $K_j$, compute $Q_i \cdot K_j$. Arrange all of these into a compatibility matrix of shape $(n, n)$, where entry $(i, j)$ answers "how much should token $i$ pay attention to token $j$?"

But there's a wrinkle. When you're working with high-dimensional vectors, those dot products between $Q$ and $K$ can get very large. Large values feed into the softmax and push it into a region where its gradients become tiny — and you'll recognize that immediately as our old enemy, the vanishing-gradient problem, showing up in a brand-new place. On top of that, raw dot products aren't probabilities; we want each query's attention to spread across the keys and sum to 1.

So we fix both issues in two steps.

Step 1 — scale. Divide the scores by $\sqrt{d_k}$:

Here $d_k$ is the dimensionality of the key vectors (a hyperparameter that sets the size of the subspace the embeddings get projected into), and $\sqrt{d_k}$ is the scaling factor. Dividing by it keeps the scores in a sensible range no matter how large the dimension gets, which keeps the softmax in its healthy, well-gradiented zone.

Step 2 — softmax. Apply softmax along each row, turning the scores into a probability distribution. Each row of the resulting attention-weight matrix sums to 1, telling us "of all the tokens, here's the fraction of attention this token should pay to each."

Finally, multiply those attention weights by the value matrix $V$. The values carry the actual content each token contributes, so the weighted sum produces a new representation for each token that blends in exactly the context it found relevant.

Put it all together and you get the whole mechanism in one line:

That's it — the entire self-attention mechanism in one equation. Read it right to left: take queries and keys, measure their alignment with a dot product, scale it down by $\sqrt{d_k}$, softmax to get probabilities, then use those probabilities to take a weighted blend of the values. For its time, this was a massive breakthrough — and it's still the beating heart of every model we'll discuss.

Here's the full mechanism, visualized:

Self-attention lets a model work out which words matter to each other. But there's more nuance in language than a single attention pattern can capture. Take the sentence "He swung the bat with incredible force." One relationship worth tracking is swung–bat; a totally different one is incredible–force. Multi-head attention lets us look at all of these relationships in parallel. Each head learns its own slightly different way of paying attention — one might focus on grammar, another on meaning, another on something like emphasis — and when you combine them, you get a far richer understanding of the sentence, much closer to how we read it.

Instead of one Q, K, V projection, you create $h$ different sets of them (the original paper used $h = 8$ heads). Each head gets its own learned $W_Q$, $W_K$, $W_V$, but each works on a smaller slice of the embedding space — typically $d/h$ dimensions per head. With $d = 512$ and 8 heads, each head gets 64 dimensions.

Each head independently runs the full scaled-dot-product attention we just built, producing its own output. Then the $h$ outputs are concatenated back together and passed through one final projection matrix $W_O$ that mixes the information from all heads:

The symbols: $h$ is the number of heads; $\text{head}_i$ is the output of the $i$-th head; $W_Q^{(i)}, W_K^{(i)}, W_V^{(i)}$ are that head's own projection matrices; $\text{Concat}$ glues the head outputs back into one vector; and $W_O$ is the output projection that blends them. The shape comes out the same as if you'd run a single attention over the full dimension — you've just done it in parallel, specialized slices.

Later research gave us a nice interpretation: different heads specialize. Some learn grammatical patterns (subject–verb agreement), some learn semantic links (which words refer to the same entity), some learn positional habits ("look at the previous token"). Nobody tells them what to learn — the model sorts it out through training.

Why 8 heads? Eight isn't magic — it's a hyperparameter chosen to balance two failure modes. Too few heads and each one has to learn too many relationships at once, losing its specialization. Too many heads and each head's slice of dimensions gets so small it can't represent anything meaningful, and you pay more in compute for the privilege. With $d = 512$ and $h = 8$, each head gets 64 dimensions — diverse enough to learn several patterns, large enough to stay useful. Modern large models often use far more heads (32, 64, even 128), with proportionally smaller per-head dimensions.

In the decoder, the first attention layer gets one small but critical modification. In regular self-attention, every token can "see" every other token in the sequence — which is totally fine in the encoder, since the whole input is known up front. In masked self-attention, the difference is, almost literally, just a mask: the model is forbidden from looking at future tokens when predicting the next word.

Mechanically, a look-ahead mask is applied to the scaled dot-product score matrix, setting every entry above the diagonal to negative infinity before the softmax. That guarantees each token can only attend to itself and the tokens before it, which preserves the autoregressive property — the model writes strictly left to right.

Why negative infinity, specifically? Because softmax turns $-\infty$ into $0$. You're effectively telling the softmax that those future positions don't exist, so they get zero attention weight.

And why bother with all this? Here's the intuition. If we didn't mask, the model would already be able to peek at the correct future tokens while predicting the next word — so there'd be no real learning, just copying. It would hit a perfect training loss by cheating and never learn to generate text on its own at inference, when those future tokens genuinely aren't available yet.

After the decoder applies masked self-attention to its own generated tokens, it still needs to actually look at the input. The encoder did all that work understanding "I like cats" — so how does the decoder get at it?

Cross-attention is the bridge. The mechanism is exactly the same scaled-dot-product attention as before, with one twist in where the vectors come from:

So when the decoder is about to generate the next French word, it forms a query that essentially asks "given what I've written so far, which English words are relevant right now?" — and pulls the matching values out of the encoder's representation. This is how the decoder lines up what it's generating with what the encoder understood. When the decoder produces "chats," cross-attention is what makes it look back at the encoder's representation of "cats" to know what to say.

One note: cross-attention only exists in encoder–decoder transformers (like the original). Decoder-only models like GPT don't have it — there's no separate encoder to attend to.

Attention has now gathered context for each token. But the model still needs to process that context — and that's the job of the feed-forward network inside every transformer layer.

If attention answers "what should I pay attention to?", the FFN answers "okay, now that I know what to focus on, what do I actually do with it?"

Each transformer layer has an FFN that processes each token independently — the same little network applied at every position. It's a simple two-layer MLP:

Symbol by symbol: $x$ is the token's representation coming out of attention; $W_1, b_1$ are the weights and bias of the first (expansion) layer; $\max(0, \cdot)$ is the ReLU nonlinearity; $W_2, b_2$ are the weights and bias of the second (contraction) layer. It runs in three steps:

1. Expansion — project from the embedding dimension $d_{model}$ up to a larger $d_{ff}$ (in the original paper, $d_{model} = 512 \to d_{ff} = 2048$, a 4× expansion). This gives the model room to detect complex features.

2. Nonlinear activation — apply ReLU (or GELU in more modern variants). This step is essential: without a nonlinearity, stacking layers would just collapse into one big linear transformation, and depth would buy you nothing.

3. Contraction — project back down from $d_{ff}$ to $d_{model}$, so the output matches the input shape and can flow into the next layer.

A practical fact that surprises people: the FFN holds most of the parameters in a transformer. With $d_{model} = 512$ and $d_{ff} = 2048$, each FFN layer has roughly $4 \times 512 \times 2048 \approx 4$ million parameters. Multi-head attention with the same $d_{model}$ has only about $4 \times 512^2 \approx 1$ million. In large LLMs, the FFN is where most of the model's knowledge actually lives.

That's exactly why an innovation like Mixture of Experts (MoE) targets the FFN specifically — it swaps the single dense FFN for many smaller "expert" FFNs and routes each token to just a few of them, letting you blow up the total parameter count without a matching blowup in compute. More on that later.

This one is essential for keeping training stable — and, you guessed it, for keeping our gradients from exploding or vanishing.

The idea: we normalize the activations so they always have mean 0 and standard deviation 1, then let the model learn how to rescale them through two trainable parameters:

The symbols:

The $\gamma$ and $\beta$ are the clever bit: they let the model "un-normalize" when that's actually useful, so normalization never costs it expressive power.

Layer norm vs. batch norm. Batch norm computes its statistics across a whole batch of examples. Layer norm computes them across the features of a single example. For transformers, layer norm wins for three reasons: it doesn't depend on batch size (transformers get trained with wildly varying batch sizes), it works fine with variable-length sequences, and it's compatible with autoregressive generation, where at inference you process one token at a time and simply don't have a batch to normalize over.

Modern variants like RMSNorm drop the mean-subtraction step entirely and just normalize by the root-mean-square, which saves a little compute. LLaMA and many recent LLMs use it — we'll dig into exactly why it works later.

Residual connections are the reason deep transformers work at all. Here's the problem they solve. If you stack many layers, the early layers have a hard time learning, because their gradient has to travel all the way back down through every layer above them — and we've seen what long backward paths do to a gradient. Layer norm helps, but it doesn't fully fix it.

The fix came from ResNet (2015): residual connections, also called skip connections. The idea is to add a direct path that bypasses each sublayer:

where $x$ is the input to the sublayer and $\text{Sublayer}(x)$ is whatever that sublayer computes (attention or the FFN). The magic is the $+ x$. Why does this help so much?

First, gradients flow freely. That $+ x$ creates a direct highway during backpropagation. Even if $\text{Sublayer}(x)$ produces a tiny gradient, $x$'s gradient passes straight through unimpeded. This is what stops gradients from vanishing across dozens of stacked layers.

Second, it's easy to learn the identity. Without residuals, every layer has to learn its full transformation from scratch. With residuals, a layer only needs to learn the delta — what to add to the input. If a layer doesn't need to do anything useful, it can just output zero and the residual passes the input through untouched. Learning "add nothing" is far easier than learning "be the identity function."

Residual connections wrap every sublayer in the transformer — both the multi-head attention and the feed-forward network. Without them, transformers with 6, 12, or 96 layers simply wouldn't train.

One modern variation worth knowing: pre-norm vs. post-norm. The original transformer applied the norm after the addition (post-norm). Modern models usually use pre-norm — applying LayerNorm to the input before the sublayer, then adding the unchanged residual. Pre-norm is more stable for very deep networks and is now the default in most modern LLMs.

There's one big problem we haven't addressed yet. Self-attention treats the input as a set of tokens, not a sequence. Without extra information, "I like cats" and "cats like I" would look identical to the model — the attention computation comes out the same regardless of the order the tokens arrive in.

But order is everything in language. Positional encodings fix this by injecting position information directly into the embeddings before they enter the first transformer layer. Each position in the sequence gets its own $d$-dimensional vector — the positional encoding for that position — and it's simply added to the token embedding sitting there:

where $\text{token\_embed}_i$ is the embedding of the token at position $i$, and $\text{pos\_encode}_i$ is the positional vector for that position. Add them and the token now "knows" where it sits.

The original transformer used a neat scheme built from sine and cosine waves at different frequencies:

The symbols: $pos$ is the position in the sequence (0, 1, 2, …); $i$ indexes the dimension of the encoding vector; $d$ is the embedding dimension; and $10000$ is a constant chosen to spread the frequencies across many scales. Even dimensions ($2i$) use sine, odd dimensions ($2i+1$) use cosine.

Each dimension of the encoding oscillates at a different frequency. Low dimensions wiggle fast (capturing fine, nearby position differences); high dimensions wiggle slowly (capturing long-range position). Together they produce a unique "fingerprint" for every position.

Why design it this way? Three reasons:

Every position gets a unique signature — no two positions share the same encoding, because the combination of frequencies never repeats over the range you care about.

The model can read off relative distances. This is the deep reason for using trig functions. Thanks to the sine/cosine angle-addition identities, the encoding for position $pos + k$ can be written as a fixed linear transformation (a rotation) of the encoding for position $pos$. So "look $k$ positions back" becomes a simple linear operation the model can learn — relative position awareness comes basically for free. The waves are periodic, but using many different frequencies (by varying $i$) guarantees each position still gets a unique overall combination.

It generalizes past the training length. Since sine and cosine are defined for every input, you can compute the encoding for any position, even one longer than anything seen during training.

Sinusoidal encodings are elegant, but most modern LLMs (LLaMA, GPT-4-era models) use Rotary Positional Embeddings (RoPE) instead. Rather than adding a positional vector to the embedding, RoPE rotates pairs of dimensions inside the query and key vectors by an angle that depends on position — so when you later compute the dot product $Q \cdot K$ for attention, the rotations naturally produce a term that depends on the difference between the two positions, not their absolute values. ALiBi (Attention with Linear Biases) is another modern alternative that biases attention scores directly by relative distance. We'll cover both in more depth in the modern-divergences section.

The core idea never changes: somehow inject position so the model knows the order. The specifics keep evolving.

Let's put every piece together. Here's exactly what happens when a sequence of tokens passes through a single transformer encoder layer:

1. Input: a sequence of token embeddings with positional encodings added, shape $(n, d)$.

2. Multi-head self-attention: project to Q, K, V across $h$ heads; compute scaled-dot-product attention per head; concatenate; project through $W_O$.

3. First Add & Norm: add the input back as a residual, then apply layer norm.

4. Feed-forward network: expand to $d_{ff}$, apply the nonlinearity, contract back to $d$.

5. Second Add & Norm: add the attention output back as a residual, then apply layer norm.

6. Output: same shape $(n, d)$, handed to the next layer.

That's one encoder layer. Stack 6 of them (or 12, or 96) and you have an encoder. A decoder layer adds one extra step in the middle: masked self-attention followed by Add & Norm, then cross-attention to the encoder output followed by Add & Norm, then the FFN and a final Add & Norm.

And here's the genuinely beautiful part: every layer has the same shape coming out as going in — $(n, d)$ to $(n, d)$. That means you can stack as many as you want — 6, 12, 96, 120 — and the data just flows straight through, each layer refining the representation a little more. That compositional simplicity is a huge part of why transformers scale so gracefully.

After the last decoder layer, you've got a sequence of $d$-dimensional vectors. To turn those into actual words, two more steps:

1. Linear projection to vocabulary size. A learned weight matrix maps each $d$-dimensional vector to a $V$-dimensional vector — one entry per vocabulary token. These raw scores are called logits.

2. Softmax over the vocabulary. Convert the logits into a probability distribution over the whole vocabulary. The token with the highest probability is the predicted next token.

At training time, you compute the cross-entropy loss between this predicted distribution and the true next token. At inference time, you sample from the distribution (or just take the argmax for greedy decoding) to pick the next token, then feed everything back through the decoder to predict the token after that, and so on — autoregression in action.

The original 2017 transformer is still the conceptual foundation. But modern LLMs have evolved several of its components — and here's the reassuring thing: every one of these is an optimization of the original recipe, not a replacement. Once you understand the architecture we just built, every modern paper clicks into place, because it's almost always improving one specific piece while leaving the overall shape intact.

Here's the short list before we go deep on each:

Let's go through them.

The original transformer had two stacks: an encoder for the input language and a decoder for the output language. But when the task is pure text generation — predict the next token given everything so far — you don't really need two streams. The "input" and the "output" are the same sequence, just shifted by one position.

That realization gave us decoder-only models: GPT-1 in 2018, then GPT-2, GPT-3, GPT-4, Claude, LLaMA, and basically every modern frontier LLM. The encoder is gone. The cross-attention block is gone. What's left is a tall stack of decoder layers, each with masked self-attention and a feed-forward network.

It almost seems too simple. How can a model that just predicts the next token translate languages, write code, answer questions, and reason? The answer: next-token prediction over a huge, diverse corpus turns out to be an extraordinarily general training signal. To predict the next token of a Python function, the model has to understand programming. To predict the next token of a Shakespearean sonnet, it has to understand iambic pentameter. To predict the next token of a math proof, it has to understand algebra. By being forced to model everything people write, the model implicitly absorbs the structure of language, knowledge, and reasoning.

Tasks like translation, summarization, and Q&A become special cases — you just frame them as text:

…and let the model predict what comes next. That's the unified interface decoder-only models gave us: one architecture handling every text task by reframing it as completion.

The masked self-attention is what makes this work. Each token can only attend to itself and earlier tokens, never future ones — which preserves the autoregressive property and lets the same model both train (predicting all positions in parallel) and generate (one token at a time at inference).

Encoder-only transformers stack self-attention layers to process input sequences bidirectionally — every token attends to every other token, both forward and backward. This makes them great for language understanding, representation learning, and structured prediction, rather than text generation.

(Quick definition: representation learning is the set of techniques that automatically discover compact, structured representations — embeddings — for things like feature detection or classification, replacing hand-engineered features.)

An encoder takes a sequence of tokens and produces a contextualized representation for each one. For "I like cats," the encoder outputs three vectors — one per token — where each vector has soaked up information from the whole sentence. The output vector for "cats" isn't just "the cats embedding"; it's "the cats embedding, having paid attention to 'I' and 'like.'" These representations aren't predictions — they're rich features that downstream tasks can build on. The encoder is fundamentally a representation learner, not a generator.

Encoder vs. decoder: the attention difference. The encoder uses bidirectional self-attention — every token attends to every other token, in both directions. That's fine, because the encoder isn't generating anything; it just needs to understand the input, and looking ahead doesn't help you cheat if you're not predicting the next token. The decoder uses causal (masked) self-attention — each token attends only to itself and prior tokens — which is required for autoregressive generation. This single difference leads to wildly different training objectives.

You can't train an encoder with next-token prediction, because it sees the whole sequence at once and the task would be trivial (it could just read the answer). So the objective that made encoders work is Masked Language Modeling (MLM), introduced in BERT (Bidirectional Encoder Representations from Transformers). The procedure:

1. Take a sentence: "I like cats and dogs."

2. Randomly mask out about 15% of the tokens: "I [MASK] cats and [MASK]."

3. Train the encoder to predict the masked tokens from the surrounding context.

Because the encoder is bidirectional, filling in a "[MASK]" in the middle of a sentence requires looking at the words both before and after it. That forces the model to build deep, two-sided understanding — context flows in from everywhere.

BERT's exact recipe was a touch more elaborate. Of the 15% of tokens selected for masking, 80% get replaced with [MASK], 10% get replaced with a random token, and 10% are left unchanged. This trick stops the model from learning that "[MASK] is the only signal that a prediction is needed" — which would hurt it on real downstream tasks where no [MASK] tokens appear.

BERT itself was a landmark: 12 encoder layers, 110M parameters in its base version, trained on Wikipedia and BookCorpus with MLM plus a next-sentence-prediction objective. From roughly 2018–2022, BERT and its descendants dominated NLP benchmarks.

Are encoders obsolete in the GPT era? Not at all — they've just specialized into the jobs where bidirectional understanding is the advantage:

So encoders didn't disappear; they migrated to the parts of the pipeline where building representations beats generating text.

A few important models still use the full encoder–decoder structure:

Encoder–decoder shines when the input and output are clearly distinct sequences with different roles — especially translation and summarization. Decoder-only models can handle these too, by treating input + output as one continuous sequence, but the explicit encoder–decoder split gives the model clearer built-in assumptions about which part is which.

Now the pieces modern models swap in. We'll take them one at a time, since each is a self-contained improvement to a part you already understand.

The original sinusoidal scheme worked, but it had two weaknesses. It encoded absolute positions when what really matters is relative distance ("the token 3 places back"), and it didn't extrapolate well past the maximum training length. Modern LLMs use two main alternatives.

RoPE (Rotary Positional Embedding). RoPE doesn't add a positional vector to the embedding. Instead it rotates pairs of dimensions in the query and key vectors by an angle that depends on position. Then, when you compute the attention dot product $Q \cdot K$, the rotations naturally produce a term that depends on the difference between the two positions, not their absolute values.

Mathematically, treat each pair of dimensions as a 2D vector and apply a rotation matrix:

where $\theta$ depends on both the position and the dimension. After rotation, the dot product between a rotated $Q$ at position $m$ and a rotated $K$ at position $n$ depends only on $(m - n)$ — the relative offset. You get relative-position information for free, without changing the attention formula at all.

Why it matters in practice: it extrapolates better to sequences longer than training (especially with NTK-aware scaling and YaRN, which stretch the rotation frequencies), it naturally captures the relative position the model actually cares about, and it's used by LLaMA, Mistral, Qwen, and most modern open LLMs.

ALiBi (Attention with Linear Biases). ALiBi takes a different route: don't touch the embeddings at all. Instead, add a position-dependent bias directly to the attention scores. Tokens that are far apart get a more negative bias, making them harder to attend to:

The symbols: $q_i$ and $k_j$ are the query and key vectors; $|i - j|$ is the distance between positions $i$ and $j$; and $m$ is a head-specific slope. Because different heads get different slopes, some heads attend mostly to nearby tokens while others can still reach far away. ALiBi is simpler than RoPE and extrapolates extraordinarily well — models trained at 2k context can sometimes handle 16k at inference. The tradeoff: it's a softer bias than RoPE and can be slightly weaker on tasks where exact distance matters.

Recall LayerNorm does two things: subtract the mean (re-centering), then divide by the standard deviation (re-scaling). Empirical research turned up something surprising: the re-centering step barely matters. The model trains nearly as well if you only do the re-scaling. So RMSNorm (Root Mean Square Normalization) drops the mean subtraction entirely:

The symbols: $x$ is the token's activation vector; $x_i$ is its $i$-th component; $d$ is the dimension; the denominator is the root-mean-square of the features; $\epsilon$ is a small stability constant; and $\gamma$ is a learnable scale. Notice what's missing compared to LayerNorm: there's no mean $\mu$ and no shift $\beta$. It just divides by the RMS of the features and applies a learnable scale.

The benefits are modest but real: roughly 10–15% faster than LayerNorm, slightly fewer parameters, and identical or marginally better quality. It's used in LLaMA, Mistral, PaLM, and many modern LLMs. It's one of those small wins that genuinely adds up when you're training billions of parameters over trillions of tokens.

The FFN in the original transformer was Linear → ReLU → Linear. Modern models almost universally use a gated variant called SwiGLU.

The idea comes from GLU (Gated Linear Units): instead of one linear projection followed by an activation, use two linear projections and multiply them together, with the activation applied to only one of them:

Here $\text{Swish}(x) = x \cdot \sigma(x)$ (also called SiLU) is a smooth nonlinearity, $\odot$ is element-wise multiplication, and $W_1, W_2$ are two separate learned projections. The output is then sent through a third linear layer to project back down:

with $W_3$ the down-projection. Because this uses three matrices instead of two, modern models shrink $d_{ff}$ to keep the parameter count fair — the original used $d_{ff} = 4 \times d_{model}$, while SwiGLU models typically use $d_{ff} \approx \tfrac{8}{3} \times d_{model}$.

Why does it work better? Intuitively, the multiplicative gate lets the network express more complex functions per parameter — one projection can dynamically modulate the other. Empirically, SwiGLU outperforms ReLU and GELU at scale across many benchmarks. The candor of the field is worth preserving here: Noam Shazeer's 2020 paper introducing it for transformers ended with the memorable line that they offer no explanation for why these architectures work and chalk it up, like all else, to divine benevolence. Funny as that is, gated activations have become standard in LLaMA, PaLM, Mistral, and most modern LLMs.

The biggest cost of full attention is its time complexity: $O(n^2)$ in the sequence length $n$, and it's the single biggest barrier to long-context LLMs. For $n = 1{,}000$, totally fine. For $n = 1{,}000{,}000$, you're looking at $10^{12}$ operations per layer per head — not fine. Sparse attention patterns trade a little flexibility for huge efficiency gains.

The simplest pattern is sliding-window attention: each token attends only to the $w$ tokens before it, not all of them. With a window of $w = 4096$ and a sequence of $n = 100{,}000$, you do work proportional to $n \times w = 4 \times 10^8$ operations instead of $n^2 = 10^{10}$ — about a 25× reduction.

But wait — if a token can only see 4,096 tokens back, how does a model ever use a 128k context? Here's the depth trick, and it's lovely. Stack layers. Layer 1 at a given position sees 4k tokens back. Layer 2's output at that position depends on Layer 1's outputs across its own 4k window — each of which already absorbed 4k more tokens back. So after $L$ layers, the effective receptive field is $L \times w$ tokens, even though every individual layer stays cheap. Information flows like ripples: each layer sees nearby context, but that nearby context already soaked up slightly more distant context from the layer below. With 32 layers and a 4096 window, the effective field is $32 \times 4096 = 131{,}072$ tokens.

This is exactly the strategy Mistral uses: Mistral 7B has a 4096-token sliding window across 32 layers, giving an effective context of ~128k tokens at modest compute.

What sliding window sacrifices: direct attention to faraway tokens. A token at position 50,000 can't directly attend to one at position 100 — the information has to flow up through the layers. For tasks that need exact long-range, pointer-like retrieval, that hurts.

A few richer patterns build on the basic window:

Longformer's pattern: local + global. Longformer added something simple but powerful — a few designated "global" tokens that attend to everything and that everything attends to (typically the [CLS] token or the start of each document). Most tokens use the cheap sliding window; the few special tokens get full attention to and from everyone. This is great when you have a fixed query at the front of the input (like extractive QA): the query tokens get global attention while the document body stays windowed, giving strong performance at sub-quadratic cost.

BigBird: local + global + random. BigBird (2020) is the most elaborate combination — local sliding window, a few global tokens, plus random connections (each token attends to a few random others). The random links are the clever part: they let information hop across the sequence efficiently, like a small-world graph where every node is a few hops from every other. Even though no single token sees everything, after a few layers the information has mixed globally. BigBird was proven to retain the theoretical expressivity of full attention (under mild conditions) while running at $O(n)$ compute — the cleanest theoretical justification for sparse attention.

And several more you'll run into:

The theoretical landscape is rich, but the production reality is mostly "sliding window plus full attention every few layers." The 2×–10× speedups are great; chasing every theoretical optimization for another 5% has historically not been worth the engineering pain.

One honest caveat: sliding window doesn't literally extend context for free. Watch for these failure modes — needle-in-a-haystack at long distance (if the answer is at position 1,000 and the question at position 50,000, windowed models can struggle even with enough effective receptive field, because the info has to survive propagation through many layers without being overwritten); multi-hop reasoning over long contexts (chains of references spanning the whole document degrade); and long-range copying (copying a specific phrase from far back gets harder). That's why pure sliding window is often paired with periodic full-attention layers, or with attention sinks (always attend to the first few tokens) to soften these effects.

The KV cache is the single most important inference-time optimization in LLMs. Here's the setup. When you generate text autoregressively, you produce one token at a time. To generate token $n+1$, you compute attention for position $n$ — and attention needs the $K$ and $V$ vectors of every previous position too.

The naive approach recomputes $K$ and $V$ for every position from scratch at each step. That's $O(n^2)$ work over the whole generation, and most of it is redundant: the $K$ and $V$ for position 4 don't change when you're generating position 100. Hugely wasteful.

The fix is to cache them. Once you compute the $K$ and $V$ vectors for a token, save them. When generating the next token, only compute $Q$, $K$, $V$ for the new position, then concatenate with the cached $K$, $V$ from before. Now attention does work proportional to $n$ per step instead of $n^2$. This optimization is so fundamental that every modern inference engine has it. The "KV cache" is just two big tensors of shape (num_layers, num_kv_heads, sequence_length, head_dim) — one for K, one for V — growing by one row per generated token.

How big does it get? The KV cache size is:

The leading $2$ counts both K and V. Plug in a 70B model with 80 layers, 8 KV heads, head_dim 128, in FP16 (2 bytes), at sequence length 100k:

That's for a single user's context. Now multiply by the number of concurrent users on a server and you see why KV cache management is the single biggest pressure point in production LLM serving. Innovations have piled up to deal with it: