What Attention Actually Computes

Imagine a Python dictionary d where you write d[k] and get back the one value stored under the exact key k. The lookup is brutally discrete: ask for a key that is even slightly off and you get nothing. Now soften it. Instead of demanding an exact match, score how well your request resembles every stored key, turn those scores into a probability distribution, and return a blend of all the values weighted by that distribution. You no longer retrieve one value; you retrieve a consensus of values, dominated by the entries whose keys most resemble your request. That soft, differentiable lookup is, almost exactly, what attention computes. The rest is bookkeeping about where the queries, keys, and values come from.

This essay unpacks the single line of algebra at the heart of modern sequence models and shows why it displaced the recurrent networks that preceded it. The nearest live network on this site is Plate V — the MNIST network; attention is the mechanism that lets a network of that kind look sideways at the rest of its input rather than only forward through its layers.

Queries, keys, and values

Suppose we have a sequence of $n$ tokens, each already turned into a vector by an embedding and earlier layers. Stack them as rows of a matrix $X \in \mathbb{R}^{n \times d}$ . From $X$ we manufacture three different views by three learned linear maps:

$Q = X W_Q, \qquad K = X W_K, \qquad V = X W_V.$

A query $q_i$ (row $i$ of $Q$ ) encodes what position $i$ is looking for. A key $k_j$ encodes what position $j$ offers as a handle for being found. A value $v_j$ encodes the content position $j$ will actually hand over if attended to.The query/key split is what lets a position advertise itself one way (“I am a verb”) while being retrieved on another (“looking for a subject”). Tying them would collapse this flexibility. These three roles are deliberately decoupled: the criterion by which a position is matched is separate from the content it contributes.

The score, and why $\sqrt{d_k}$

To measure how well query $i$ matches key $j$ , take their dot product $q_i \cdot k_j$ . Large when the two vectors point the same way, near zero when orthogonal. Collecting all pairs at once gives the score matrix $QK^\top \in \mathbb{R}^{n \times n}$ , whose $(i,j)$ entry is exactly $q_i \cdot k_j$ . These raw scores are the logits of the attention distribution.

Before normalizing, we divide by $\sqrt{d_k}$ , where $d_k$ is the dimension of the keys and queries. The reason is statistical. If the entries of $q_i$ and $k_j$ are independent with mean zero and variance one, then $q_i \cdot k_j = \sum_{m=1}^{d_k} q_{im} k_{jm}$ is a sum of $d_k$ independent terms, so its variance grows like $d_k$ and its typical magnitude like $\sqrt{d_k}$ .A sum of $d_k$ unit-variance independent terms has variance $d_k$ ; standard deviation is the square root. Dividing by $\sqrt{d_k}$ restores unit scale regardless of dimension. For large $d_k$ the dot products would otherwise be enormous, pushing the softmax into a regime where it saturates—one entry near 1, the rest near 0—and where its gradients nearly vanish. Dividing by $\sqrt{d_k}$ rescales the scores back to roughly unit variance, keeping the distribution responsive and the gradients alive. This is why the operation is called scaled dot-product attention.

Softmax into weights

We turn each row of scores into a probability distribution with the softmax function, applied along the key axis:

$a_{ij} = \frac{\exp\!\big(q_i \cdot k_j / \sqrt{d_k}\big)}{\sum_{\ell} \exp\!\big(q_i \cdot k_\ell / \sqrt{d_k}\big)}.$

Now $a_{ij} \ge 0$ and $\sum_j a_{ij} = 1$ : row $i$ is a genuine distribution over the $n$ positions, expressing how much position $i$ attends to each position $j$ . The exponential makes the competition winner-take-most without being winner-take-all—a higher score earns disproportionately more weight, but no position is ever hard-excluded, which is precisely what keeps the whole map differentiable.

The weighted sum of values

The output for position $i$ is the value vectors averaged under its attention distribution:

$\text{out}_i = \sum_j a_{ij}\, v_j.$

Each position emits a custom blend of the sequence’s contents, mixed in proportions it chose by matching its query against every key. Assembling all rows at once recovers the canonical formula:

$\text{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V.$

Read it left to right: $QK^\top$ scores every query against every key; $/\sqrt{d_k}$ tames the scale; softmax normalizes each row into weights; multiplying by $V$ takes the weighted sums. The whole thing is two matrix multiplications with a softmax between them—and because every step is differentiable, the maps $W_Q, W_K, W_V$ are learned by ordinary backpropagation.

Self-attention and the global receptive field

When $Q$ , $K$ , and $V$ are all derived from the same sequence $X$ , the operation is self-attention: every position attends to every other position, itself included. Note the structural consequence. After one self-attention layer, the representation at position $i$ can depend directly on position $j$ for any $j$ , no matter how far away. The receptive field is global in a single layer.

Contrast this with a recurrent network, which threads information through a chain of hidden states $h_1 \to h_2 \to \cdots \to h_n$ . For position 1 to influence position $n$ , its signal must survive $n-1$ sequential update steps—a path along which gradients tend to vanish or explode and along which nothing can be computed in parallel. Self-attention replaces that long serial path with a single $O(n^2)$ comparison that any position can make to any other in one shot.The cost is quadratic in sequence length: the $n \times n$ score matrix. Recurrence is linear in length but serial. Much later research trades one against the other.

Multi-head attention

A single attention distribution forces every position to summarize everything it cares about with one set of weights. But a token may need to track several relations at once—syntactic agreement, coreference, local phrasing. Multi-head attention runs $h$ attention operations in parallel, each with its own learned $W_Q^{(\cdot)}, W_K^{(\cdot)}, W_V^{(\cdot)}$ projecting into a lower-dimensional subspace of size $d_k = d/h$ . The $h$ outputs are concatenated and passed through a final linear map:

$\text{MultiHead}(X) = \big[\,\text{head}_1, \ldots, \text{head}_h\,\big]\, W_O.$

Each head is free to specialize in a different kind of relationship, and the model gets several distinct attention patterns for the price of one (the per-head dimension shrinks to keep the total compute roughly fixed).

Why it replaced recurrence

Two properties did the work. First, parallelism: the score matrix $QK^\top$ is one dense matrix multiply, so an entire sequence is processed simultaneously rather than one timestep at a time—a decisive fit for GPU hardware. Second, direct long-range dependencies: any two positions are one attention step apart, so the path length information must travel is constant in sequence length rather than linear. Stacking these layers, interleaved with feed-forward blocks and residual connections, gives the transformer, the architecture that now underlies essentially all large language models.Attention is permutation-invariant on its own—it sees a set, not a sequence. Order is reintroduced separately through positional encodings added to the embeddings. The idea did not arrive fully formed: it grew out of additive attention introduced to help machine-translation models align source and target words, then was distilled into the pure scaled-dot-product form below.

The line $\text{softmax}(QK^\top/\sqrt{d_k})\,V$ is, in the end, a soft dictionary that learns its own keys, learns what to return, and is smooth enough to train end to end. Everything that follows in a modern model is built from stacking that one idea.