A precise vocabulary.

A mechanism that computes a weighted combination of value vectors, where the weights measure relevance between a query and a set of keys. Scaled dot-product attention is $\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$, with the $\sqrt{d_k}$ factor stabilizing gradients. Self-attention lets every position in a sequence attend to every other, giving a global receptive field in a single layer; it is the core operation of the transformer.

An algorithm for computing the gradient of a scalar loss with respect to every parameter in a neural network by applying the chain rule from output to input. It performs one forward pass to cache intermediate activations, then a backward pass that propagates the derivative $\partial L/\partial z$ through each layer in reverse. It is reverse-mode automatic differentiation, computing all parameter gradients in time proportional to a single forward pass.

A layer that standardizes activations across a mini-batch to zero mean and unit variance, then rescales and shifts by learned parameters $\gamma,\beta$: $\hat{x}=\frac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}$, $y=\gamma\hat{x}+\beta$. It stabilizes and accelerates training by reducing internal covariate shift and smoothing the loss landscape. At inference it uses running estimates of the mean and variance instead of batch statistics.

A learned map from discrete or high-dimensional objects to dense vectors in $\mathbb{R}^d$, where geometric relationships encode semantic structure. Token embeddings turn vocabulary indices into trainable vectors that downstream layers consume. A well-trained embedding space places similar items near one another, so distances and directions become meaningful (e.g., analogies as vector offsets).

An iterative optimization method that minimizes a differentiable objective $L(\theta)$ by repeatedly stepping against the gradient: $\theta_{t+1}=\theta_t-\eta\,\nabla L(\theta_t)$, where $\eta$ is the learning rate. Stochastic gradient descent estimates $\nabla L$ from a mini-batch rather than the full dataset, trading variance for cheaper steps. Convergence to a global minimum is guaranteed for convex $L$ with suitable step size; for non-convex deep networks it converges to a local minimum or saddle region.

The scalar $\eta$ controlling how far each gradient-descent step moves the parameters, $\theta_{t+1}=\theta_t-\eta\,\nabla L$. Too large a value causes divergence or oscillation; too small wastes computation and may stall in flat regions. It is typically the most important hyperparameter and is often annealed over training via a schedule or set adaptively by optimizers like Adam.

The S-shaped function $\sigma(x)=\dfrac{1}{1+e^{-x}}$ mapping the real line to $(0,1)$, used to turn a single logit into a probability. Its derivative is the convenient $\sigma'(x)=\sigma(x)(1-\sigma(x))$. It is the two-class special case of softmax and saturates for large $|x|$, which can cause vanishing gradients.

The unnormalized real-valued score a model produces before a normalizing nonlinearity. For a probability $p$, the logit is the log-odds $\operatorname{logit}(p)=\ln\frac{p}{1-p}$, the inverse of the logistic sigmoid. In multiclass settings, logits are the raw inputs to softmax; only their differences affect the resulting probabilities.

A composition of parameterized affine maps interleaved with nonlinear activation functions, $f(x)=\sigma_L(W_L\cdots\sigma_1(W_1 x+b_1)\cdots+b_L)$. The universal approximation theorem states that a network with one sufficiently wide hidden layer can approximate any continuous function on a compact set to arbitrary accuracy. Depth makes many functions expressible with far fewer parameters than width alone, and parameters are fit by gradient descent via backpropagation.

The failure mode in which a model fits idiosyncratic noise in the training data and so generalizes poorly to unseen examples, exhibiting low training error but high test error. It worsens as model capacity grows relative to the amount and signal of the data. It is diagnosed by a widening gap between training and validation performance and is mitigated by regularization, more data, or early stopping.

Any technique that constrains a model to reduce variance and improve generalization, typically by penalizing complexity. Common forms add a norm penalty to the loss, such as $L_2$ (weight decay) $\lambda\|\theta\|_2^2$ or $L_1$ $\lambda\|\theta\|_1$, which induces sparsity. Dropout, data augmentation, and early stopping are also regularizers; from a Bayesian view, a penalty corresponds to a prior over parameters.

A method for taking gradients through a sampling step by expressing a random variable as a deterministic function of the parameters and an independent noise source. For a Gaussian, $z=\mu+\sigma\odot\epsilon$ with $\epsilon\sim\mathcal{N}(0,I)$ moves the stochasticity off the differentiation path, so $\nabla_{\mu,\sigma}$ flows through $z$. It is what makes the variational autoencoder trainable by ordinary backpropagation.

An additive shortcut that lets a layer learn a residual: the block computes $y=x+F(x)$ rather than $y=F(x)$. This gives gradients a direct path to earlier layers, mitigating vanishing gradients and enabling very deep networks to train. Residual connections are a defining feature of ResNets and appear around every sublayer of a transformer.

A function mapping a vector of real logits to a probability distribution: $\operatorname{softmax}(z)_i=\dfrac{e^{z_i}}{\sum_j e^{z_j}}$. The outputs are positive and sum to one, with relative magnitudes set by the exponentiated differences of inputs. It generalizes the logistic sigmoid to multiple classes and is the standard final layer for classification, typically paired with the cross-entropy loss.

The process of segmenting raw text into discrete units (tokens) that a model can index and embed. Subword schemes such as byte-pair encoding learn a vocabulary by greedily merging frequent adjacent symbol pairs, balancing vocabulary size against sequence length and gracefully handling rare or unseen words. The choice of tokenizer fixes the input alphabet of a language model and affects its effective context length.

A neural network architecture built from stacked blocks of multi-head self-attention and position-wise feedforward layers, each wrapped with residual connections and normalization. It dispenses with recurrence and convolution, processing all sequence positions in parallel and injecting order via positional encodings. Introduced in 2017, it is the backbone of modern large language models and many vision and multimodal systems.

The rule for inverting conditional probabilities: $P(A\mid B)=\dfrac{P(B\mid A)\,P(A)}{P(B)}$. It updates a prior belief $P(A)$ into a posterior $P(A\mid B)$ given evidence $B$, with the likelihood $P(B\mid A)$ doing the reweighting. It is the foundation of Bayesian inference, where the denominator $P(B)$ is the marginal likelihood normalizing the posterior.

The statement that the normalized sum of many independent, identically distributed random variables with finite variance converges in distribution to a Gaussian, regardless of the original distribution’s shape. Concretely, $\frac{1}{\sqrt{n}}\sum_{i=1}^n (X_i-\mu)/\sigma \Rightarrow \mathcal{N}(0,1)$. It explains the ubiquity of the normal distribution and underwrites much of statistical inference.

The probability of event $A$ given that $B$ has occurred, defined as $P(A\mid B)=\dfrac{P(A\cap B)}{P(B)}$ whenever $P(B)>0$. It restricts the sample space to outcomes consistent with $B$ and renormalizes. Independence is the special case $P(A\mid B)=P(A)$, and rearranging the definition gives Bayes’ theorem.

A measure of the joint linear variability of two random variables, $\operatorname{Cov}(X,Y)=\mathbb{E}[(X-\mathbb{E}X)(Y-\mathbb{E}Y)]$. It is positive when the variables tend to move together and zero when they are uncorrelated (which is implied by, but does not imply, independence). Normalizing by the standard deviations gives the Pearson correlation coefficient in $[-1,1]$, and stacking pairwise covariances forms the covariance matrix.

The probability-weighted average of a random variable, $\mathbb{E}[X]=\sum_x x\,p(x)$ in the discrete case or $\int x\,p(x)\,dx$ in the continuous case. It is linear, so $\mathbb{E}[aX+bY]=a\mathbb{E}[X]+b\mathbb{E}[Y]$ regardless of dependence. The law of large numbers guarantees that sample averages converge to the expectation as the sample size grows.

The property that a process’s future depends on its present state alone, not on its full history: $P(X_{t+1}\mid X_t,\dots,X_0)=P(X_{t+1}\mid X_t)$. A process with this property is a Markov chain, characterized by a transition matrix whose rows are probability distributions. Under mild conditions the chain converges to a stationary distribution, the basis of MCMC sampling.

A principle for estimating parameters $\theta$ by maximizing the probability the model assigns to observed data: $\hat{\theta}=\arg\max_\theta \prod_i p(x_i\mid\theta)$, equivalently maximizing the log-likelihood. For i.i.d. data it is consistent and asymptotically efficient under regularity conditions. Minimizing cross-entropy loss is exactly maximum likelihood, and adding a prior yields maximum a posteriori estimation.

The expected squared deviation of a random variable from its mean, $\operatorname{Var}(X)=\mathbb{E}[(X-\mathbb{E}[X])^2]=\mathbb{E}[X^2]-\mathbb{E}[X]^2$. It measures spread; its square root is the standard deviation, which shares the units of $X$. For independent variables, variances add, a fact central to the central limit theorem.

A measure of the expected coding cost of using distribution $q$ to encode samples drawn from the true distribution $p$: $H(p,q)=-\sum_x p(x)\log q(x)$. It decomposes as $H(p,q)=H(p)+D_{\mathrm{KL}}(p\,\|\,q)$, so minimizing cross-entropy over $q$ is equivalent to minimizing KL divergence. As a training loss with $p$ the one-hot label and $q$ the softmax output, it coincides with the negative log-likelihood.

The expected amount of information, in bits (base 2) or nats (base $e$), in a random variable: $H(X)=-\sum_x p(x)\log p(x)$. It quantifies uncertainty and equals the minimum average code length for the source under Shannon’s source coding theorem. Entropy is maximized by the uniform distribution and is zero for a deterministic outcome.

A measure of how one distribution $q$ differs from a reference $p$: $D_{\mathrm{KL}}(p\,\|\,q)=\sum_x p(x)\log\frac{p(x)}{q(x)}$. It is non-negative, zero only when $p=q$ almost everywhere, and asymmetric, so it is not a metric. It underlies variational inference, where minimizing KL to a posterior yields an evidence lower bound.

A function $f$ is convex if its domain is convex and $f(\lambda x+(1-\lambda)y)\le \lambda f(x)+(1-\lambda)f(y)$ for all $\lambda\in[0,1]$; geometrically, the line between any two points on the graph lies above it. For twice-differentiable $f$, convexity is equivalent to a positive semidefinite Hessian. Convexity guarantees every local minimum is global, making convex optimization tractable.

For a square matrix $A$, a scalar $\lambda$ is an eigenvalue with eigenvector $v\neq 0$ if $Av=\lambda v$, meaning $A$ acts by pure scaling along $v$. Eigenvalues are the roots of the characteristic polynomial $\det(A-\lambda I)=0$, and for symmetric matrices they are real with orthogonal eigenvectors. The spectrum governs stability, conditioning, and the behavior of iterated linear maps.

The gradient $\nabla f$ of a scalar field $f:\mathbb{R}^n\to\mathbb{R}$ is the vector of partial derivatives $(\partial f/\partial x_1,\dots,\partial f/\partial x_n)$. It points in the direction of steepest ascent, and its magnitude equals the maximal rate of change of $f$ at that point. The gradient is orthogonal to the level sets of $f$, and it vanishes at stationary points.

An integer greater than 1 whose only positive divisors are 1 and itself. By the fundamental theorem of arithmetic, every integer above 1 factors uniquely into primes, making them the multiplicative building blocks of the integers. Euclid proved there are infinitely many, and the prime number theorem states the count below $x$ is asymptotic to $x/\ln x$.

The function $\zeta(s)=\sum_{n=1}^\infty n^{-s}$ for $\Re(s)>1$, extended by analytic continuation to the rest of the complex plane except a pole at $s=1$. Euler’s product $\zeta(s)=\prod_p (1-p^{-s})^{-1}$ ties it directly to the primes. The Riemann hypothesis conjectures that all non-trivial zeros have real part $\tfrac12$, with deep consequences for the distribution of primes.