Library [16]
The works the rest of the atlas stands on. Each note says what the work established — not whether I liked it.
Introduced BERT, a bidirectional Transformer encoder pretrained with masked language modeling and next-sentence prediction, then fine-tuned for downstream tasks. Established the pretrain-then-fine-tune paradigm and set new state of the art across many NLP benchmarks.
Introduced the Transformer, an architecture based entirely on self-attention that dispenses with recurrence and convolution. Established multi-head scaled dot-product attention and positional encodings as the foundation for nearly all modern large language models.
Introduced Adam, an adaptive first-order optimizer combining momentum with per-parameter adaptive learning rates derived from running estimates of the first and second gradient moments, with bias correction. Became the default optimizer for training deep networks.
Introduced batch normalization, which normalizes layer inputs using minibatch statistics to stabilize and accelerate training and permit higher learning rates. Became a near-ubiquitous component of deep convolutional architectures.
Introduced residual connections (identity skip connections) that reformulate layers as learning residual functions, enabling stable training of networks hundreds of layers deep. Resolved the degradation problem and set new accuracy records on ImageNet.
Introduced dropout, a regularizer that randomly zeroes units during training to prevent co-adaptation, approximating an ensemble of subnetworks. Substantially reduced overfitting in large neural networks and became a standard technique.
Introduced generative adversarial networks, framing generative modeling as a minimax game between a generator and a discriminator. Established adversarial training as a powerful alternative to likelihood-based generative methods for producing realistic samples.
Introduced the variational autoencoder and the reparameterization trick, enabling gradient-based optimization of a variational lower bound (the ELBO) on the data likelihood. Unified deep learning with approximate Bayesian inference for latent-variable generative models.
Demonstrated that a deep convolutional network trained on GPUs could win the ImageNet challenge by a large margin, popularizing ReLU activations, dropout, and GPU training. Widely regarded as the work that ignited the modern deep learning era.
Popularized the backpropagation algorithm for training multilayer neural networks by efficiently computing gradients of an error function via the chain rule. Showed that hidden units could learn useful internal representations, making deep networks trainable in practice.
Founded information theory, defining entropy as a measure of information and establishing the source coding and noisy-channel coding theorems with the concept of channel capacity. Provides the quantitative foundation for compression, communication, and much of machine learning.
The definitive introductory text on reinforcement learning, developing Markov decision processes, dynamic programming, Monte Carlo and temporal-difference methods, function approximation, and policy gradients. The standard reference for the field.
The first comprehensive textbook on deep learning, spanning the mathematical prerequisites, feedforward and convolutional and recurrent networks, regularization, optimization, and generative models. A standard entry point to the field.
A comprehensive graduate-level reference on supervised and unsupervised learning, covering linear methods, regularization, trees, boosting, kernels, and the bias-variance tradeoff. A standard bridge between classical statistics and modern machine learning.
A widely used graduate textbook presenting machine learning from a probabilistic, largely Bayesian, perspective, covering linear models, kernels, graphical models, mixture models, and approximate inference. Known for its careful mathematical exposition.
A unified treatment connecting information theory, probabilistic inference, and learning, covering source and channel coding, Bayesian methods, Monte Carlo, and neural networks. Notable for its emphasis on the deep links between data compression and inference.